Now that Exchange 2007 SP1 Update Rollup 4 has shipped, it’s time to revisit recent events preceding it.
A few days before yesterday’s release, a pre-release version of Update Rollup 4 for Exchange Server 2007 SP1 made its way to Microsoft Update. Customers who had the Automatic Updates feature of Windows Server OS configured to automatically download and install updates got the pre-release version downloaded and applied automatically to those servers. Although it was detected and removed quickly from Microsoft Update, the update has left some customers affected by this issue quite annoyed— and understandably so.
Microsoft’s Scott Roberts posted the details on the Exchange team blog in INFO: Update Rollup 4 for Exchange Server 2007 Service Pack 1, including some of the issues faced by customers, and workarounds. Scott also responded to customers who left comments on the blog post, and frequently updated the post/comments.
Although this has proved to be a major annoyance for some customers, overall the number of customers affected was relatively quite low.
What’s of note is the upfront communication about this through the Exchange team blog. Rather than trying to sweep the issue under the carpet, it was actually talked about! Fessing up about such issues, apologizing where apologies are due, and ensuring adequate controls are in place so such things do not happen again is the right thing to do.
It’s also a sign of how Microsoft is increasingly being more open about such incidents.
Computerworld’s Gregg Keizer wrote about this in Microsoft issues wrong update for Exchange 2007. Surprisingly, other tech media outlets such as News.com and InfoWorld did not pick this up.
Keizer notes:
“For a brief period of time on 9/9, a pre-release version of Update Rollup 4 for Exchange Server 2007 Service Pack 1 was inadvertently made available to Microsoft Update, the Microsoft Update Catalog and WSUS servers for download,” an unidentified Microsoft employee said in a post to the official Exchange blog.
To set the record straight, the linked post is written by Scott Roberts, and clearly attributed to him with a link to his bio.
Auto-updating Servers and Server Apps?
Given the incident, it’s easy to respond with “We can’t trust Microsoft to automatically push patches that work!” — and you can’t be blamed for thinking that way. In fact, you shouldn’t trust any vendor to automatically push patches and updates to servers and server apps. In many organizations, patches for desktop/laptop OS and apps are also accorded similar treatment.
Although most software vendors test patches— some more extensively than others, there are a staggering number of variations in configurations, topologies, software and hardware deployed by customers. It is close to impossible to test a patch and account for these variations, and chances of a patch being tested for an environment exactly like yours are arguably quite slim.
It is a Patch Management best practice (and has been for as long as I can remember) to not auto-apply patches to servers and server applications without first testing these in a lab environment. A test and change control process— however rudimentary it may be, always helps in orderly deployment of patches, tracking of such updates, and forces you to think of a back-up plan.
It’s a good idea to always apply a patch or update on a test box or two, then roll it out to production servers— starting with low-impact/low-priority servers first to discover problems early on. This ensures that should things go wrong, the initial impact is low. As the patch or update is applied to more servers and you move to more critical/high-impact servers, you’ve gradually reduced the chances of things going wrong. (Of course, the exact method of rolling out and the order in which servers get a patch applied will vary in each organization and may depend on the type of patch being applied.)
Small businesses, some with no full-time IT staff, many with a single server, may not be able to justify the cost of a test environment or a consultant to test patches and updates.
One option is to use virtualization software such as Microsoft’s hypervisor-based Hyper-V (the standalone Hyper-V Server 2008, or the Virtualization/Hyper-V role of Windows Server 2008), the non-hypervisor-based Microsoft Virtual Server 2005 R2, or Microsoft VirtualPC 2007 for desktops— (all of them except Windows Server 2008 are free), to setup a virtual test environment.
If you are a consultant responsible for supporting many such small businesses, perhaps you can test patches on behalf of customers, and distribute the cost to a number of customers. You can generate additional revenue, and customers can get the assurance that the patches they deploy are tested by someone responsible for maintaining their servers— someone who knows their environment well. It can reduce the possibility of downtime, and is generally cheaper than actual downtime of critical services or applications.
Having patches and updates automatically applied to servers, without any testing, can and will land you in trouble at some point— regardless of the vendor.
{ 2 comments… read them below or add one }
Just a minor point, but Microsoft never fessed up to the exact times Windows Update had the bad Update version. “Brief Period of Time” is a weasel phrase.
Thanks for the update on the update – it helps to put things in the right perspective. You’re right on about not trusting any vendor to auto-apply patches to servers.