There’s a great write-up on building and maintaining a Windows tiered patching infrastructure over at Ars Technica today. It sets up like this:

Windows updates have historically been a constant annoyance for IT staff. Manual updates were a huge pain, and, while the advent of the Automatic Update feature improved the situation, it brought with it problems of its own. Specifically, Automatic Updates are simply too automatic. Automatic Updates grabs the latest updates, no matter what type, and applies them according to a schedule you set. The feature has no information and makes no judgments about service level agreements (SLAs), buggy updates, or anything else; it simply downloads and applies. While this may be acceptable for most home users, it is woefully inadequate in an enterprise.

Sound familiar? That’s not much different from some of the patching challenges we have in the control system space. Granted, this is just for Windows, but it’s a significant part of the problem for many shops. So here’s the goal:

By dividing systems into tiers and applying separate update policies to each tier, you can ensure that high-impact systems only receive a very limited set of extensively tested, approved updates, while low-impact systems (like lab systems) receive the latest updates for testing.

The article then goes in depth on building the tiered infrastructure, including how to set up the AD groups, group policies, WSUS groups, and WSUS approval policies. The example they use is a web service provider with a high uptime requirement SQL server clusters but it’s very easy to find the control system parallels.

If you haven’t implemented a patch management program for your control system yet, here’s my advice:

  • Go read this article to get an idea of how a tiered infrastructure can work, leverage your redundancy
  • Use a DMZ to limit direct traffic between the control system and the business network or Internet
  • Don’t forget non-Windows systems and embedded devices (automation isn’t always as easy here)
  • Inventory and patch application software, with a priority on network-facing apps like database servers, web servers, etc…

Have your own advice or war story for patching in control system networks? Leave a comment below.