16

please forgive this rather straightforward question.

First off, I'm not a sysadmin, and my experience with Linux is somewhat limited.

About 3-4 months ago, I set up a CentOS server in work, for a variety of reasons. We are using it as a development server for web sites (which our clients have access to), subversion server, and we're hosting a wiki on there for internal communication too, so it has become quite an important tool for us. (Probably more important than we thought it would be when I set it up!)

It has come to my attention that Yum wants to update about 250 packages to the latest versions in the repo.

Since the server is working fine for us, should I take the risk of updating these packages? Do the security risks outweigh the risk of the server breaking when I update everything?

I should point out that while I have backups of everything, it would take time to set everything up the way it is just now, and I don't have very much free time at work at the moment!

If the advice is to update, are there any best practices that could be passed on to make the process as safe as possible?

Thanks in advance for any advice.

UPDATE - Thanks for your responses everyone. If I had enough rep to upvote everyone, I would. ;) I've decided to ghost the hard drive and update. Unfortunately, getting hold of a full or part time sysadmin isn't an option at the moment, so I'll just have to deal with the issue as well as I can!

John McCollum
  • 265
  • 1
  • 2
  • 7

8 Answers8

11

Quick and dirty (ie. Battlefield Administrator) solution:

  1. Take your system offline (I hope you can) and do a NortonGhost backup (or something similar) to a 2nd hard drive.

  2. Boot up the 2nd hard drive (to make sure your backup actually works) and do the yum update on THAT drive.

  3. If it all works... congratulations!

  4. If it screws something up... go ahead and put in your ORIGINAL drive and come up with a "Plan B".

UPDATE:

Just thought I'd mention that the real issue here is "Do I update my waaaay out of date system and risk messing it up?" or "Do I leave my perfectly good working system unpatched and risk having it hacked/compromised?"

The answer is... once you get your system patched via the steps above... try and stay on top of it by backing it up frequently AND patching it frequently.

Then you'll have the best of both worlds. ;-)

KPWINC
  • 11,394
  • 3
  • 37
  • 45
  • My pleasure... good luck with your backup/update. As a side note, I have personally done yum updates in CentOS when there were 200-300 updates and its been just fine. BUT... I've also done updates where it completely chunked and I had to do goofy voodoo/chicken rituals (and a lot of command line crap) just to get things working again. I wish you a speedy and successful update. ;-) – KPWINC Jul 07 '09 at 23:33
10

Yes, update.

RHEL (and therefore CentOS) are careful not to update versions to anything incompatible, instead they backport bugfixes and security fixes, so the actual changes to packages are minimal and reasonably unlikely to cause compatibility problems.

If any config files have changed, the packages will tell you about a .rpmorig or .rpmnew file that gets created. It depends on the configuration of the RPM itself. You can look for warnings about any of those being created and either put your old config back ("cp foo foo.bak; cp foo.rpmorig foo") or look at the .rpmnew files and incorporate any changes into your config.

The problem is less noticeable if you update regularly.

We have a lot of systems that get updated quarterly (every 3 months); and very rarely see any problems from package updates. (except on systems doing weird kernel things to access LUNs from a SAN)

freiheit
  • 14,544
  • 1
  • 47
  • 69
  • I like more KPWINC answer. Backup first. Example: httpd 2.2 got upgraded to 2.4 and suddenly config files don't work anymore. Panic and dev team idle for hours until the problem is diagnosed and fixed. – Jose Manuel Gomez Alvarez Dec 16 '18 at 23:35
  • Not to speak about kernel package upgrade, which could potentially break the booting of the machine https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/5/html/deployment_guide/s1-rpm-using – Jose Manuel Gomez Alvarez Dec 16 '18 at 23:51
  • @Jose Manuel Gomez Alvarez - backing up first is always nice, but if your system went from http 2.2 to 2.4, it didn’t match this question—CentOS doesn’t do that kind of thing, ever. – freiheit Dec 18 '18 at 02:30
6

While yes, it would take time to upgrade, And in the same manor, it would take time to restore if something went wrong, How much pain/suffering would it be if the data on that system was deleted through a exploit / hack?

For the most part upgrades from the CentOS base repositories are safe to install, The only time i've had update issues with CentOS is when i start / or needed to use a outside repository (DAG, RPMForge, Ect ect..)

The Best Setup for this kind of thing is to have a hot-swappable server ready, so you can test the updates on it before deploying them to the live server.

grufftech
  • 6,760
  • 4
  • 37
  • 37
4

If this system is so important, then the security updates become all the more important. Consider the implications if that system has to be taken down for a rebuild if (when?) an out of date package allows for a system compromise. Ideally, you would have a test server configured in a similar fashion that you could update first, and check to see if anything breaks.

When you do apply updates you need to make sure of a few things:

  1. The update time is publicized to everyone who uses the system
  2. You have a plan on how to both update and test each application
  3. You have a plan for how to undo the updates if (when?) the update breaks the app
  4. And current backups exist in case something goes really wrong

A good sysadmin would have experience in this kind of work, and should be doing all of those things anyway. If your organization has any, then this might be the time to dump administration of the system on them. Or if you are nervous of doing this yourself, then look into hiring somebody on contract to do this kind of routine maintenance. Either way, the updates need to happen, since you are opening yourself up to a much worse situation down the line.

Scott Pack
  • 14,907
  • 10
  • 53
  • 83
3

Sound like you need an actual system administrator to take a couple of hours to look over your system, update it and make sure everything runs again. Ideally, you would have this person come and do this for you a few times a month. A server is not an install-once-and-forget-about-it thing; it needs regular service.

Teddy
  • 5,204
  • 1
  • 23
  • 27
3

This is why today, I almost never run any production systems on real hardware. I run them in Virtual Machines. Then during a quick bit of downtime (5 minutes), I run either a snapshot from within ESX itself, or if I'm using a custom Xen/Solaris/OpenVZ setup, I do an LVM snapshot of the server image. I then boot the original back up, and now I have a copy I can do with as I please.

That said, start with updating the kernel, and apache, and then work backwards from there. You don't have to take the full package list that yum reports, but the top attack vectors should be the ones you patch the soonest.

Everytime I've ever had a Linux system hacked, it's because I left apache, openssh, or the kernel itself unpatched.

Chris K
  • 669
  • 4
  • 11
2

I would just update the security related packages.

fpmurphy
  • 841
  • 6
  • 13
2

I had that exact thing come up a year or so ago... I did the yum update on the CentOS box, running on Dell hardware and it installed a kernel that wouldn't boot. The box didn't have anything loaded on it yet (otherwise I would have been more cautious). Spent a lot of time messing around with it and it seems that there's some incompatibility between CentOS/Linux newer kernels and that Dell box. Be very cautious with your updates. I still recommend updating as it's the right thing to do, but be prepared to recover from a trashed system!

Brian Knoblauch
  • 2,196
  • 2
  • 32
  • 48