1

TL;DR : How to make sure a list of package updates can be made in a reproducible way on many servers, even if newer security updates are available before the patching campaign is over?

In a company that has regulatory constraints, the security updates of servers need to be made in a reproducible way : a list of critical CVEs are identified, leading to a list of packages to update to a specific version. These updates need to be applied on many servers, from development to staging and production.

Applying these updates can take a long delay, and, even if newer security updates are available in between, it's important to make sure the same change is made on every server.

I suppose specifying a version with something like yum update package-version would not work because newer packages replace older ones on CentOS repositories (at least for security updates). So we might not always apply the exact same versions.

The ops team wants to use Landesk, and implemented a copy of CentOS repositories on the local network (used by every server instead of the normal CentOS repos), on which they plan to make some kind of "snapshots" of packages, in specific timestamped repositories (like 'repo_xxx_20191019'). In order to apply the security updates, they plan to update the repo config of each targeted server to add the timestamped repositories, and update packages this way.

I suppose it would work, but are there any caveats to care about? Or simpler ways to achieve the same result?

Mossroy
  • 121
  • 3

1 Answers1

1

Systematically and quickly.

Your patching procedures needs the ability to very rapidly apply updates for the most critical threats. Probably closer to hours than days for the worst case. Attackers won't wait for your paperwork, you can file the emergency change control after the fact.

Say you normally do monthly updates. Test systems on week 1, staging on week 2, production on week 4. Update package set each month is well defined in the local repos, and this works well.

On week 2, a critical update is released. CVSS score 10, systems are being compromised in the wild. Very bad. This must be patched as soon as possible and cannot wait.

Now you need to make a choice. Create a small ad-hoc package and install that everywhere, or add to the pending update and accelerate that into production.

How you stage the update repos could be done in a couple ways. What actually is important is that you set and measure metrics for how long your hosts are not patched. And have a process to move quickly if necessary.

John Mahowald
  • 32,050
  • 2
  • 19
  • 34
  • Thanks for your answer. Based on your example, let's say updating a package to 1.1 is planned and currently tested. On week 3 a security update 1.2 is released for this same package, with a very small CVSS score. It might be decided to keep 1.1 for this month, and postpone 1.2 for the next one. How to do that, as version 1.1 has been replaced by 1.2 in CentOS repos? – Mossroy Oct 20 '19 at 18:21
  • 1
    In that scenario, you have already made your local repo for the month and can use that. Updating direct from the mirrors with no version pinning is a different tactic. – John Mahowald Oct 20 '19 at 20:58