5

My boss and I were sitting in our server room today when we all of a sudden heard one of our servers go into hyper-speed, indicating that it was restarting. You can imagine the immediate "oh crap" expressions on our faces.

We dug into the logs, and it appears that there were some updates automatically installed that required a restart. AU, seeing that there was no one logged in, automatically did the restart. This is a production server, so we have automatic updates turned off (not even downloading then waiting.. It should wait for us to tell it before it does anything).

I ran both rsop.msc and gpedit.msc to check if there was a rogue group policy that forced the automatic updating. Nothing.

Our windowsupdate.log shows this:

2011-12-16  09:00:12:092     964    17f4    AU  Setting AU scheduled install time to 2011-12-16 20:00:00

(there were many more lines like that, and one pointing to a scheduled install just minutes before we heard the restart)

So, somewhere, AU is getting the bright idea that it should schedule automatic installs. Any ideas on why that might be happening?

A bit of pertinent information:

We recently (one month ago) installed a WSUS server, and two weeks ago pointed all of our servers at it. With WSUS came forefront client security, with a policy set up to do automatic definition updates every 6 hours. This could possibly be the problem, but it seems like a major flaw that by checking for definition updates it would automatically install other updates.

I also rolled out (I believe last Thursday) a new GPO for our workstations that forces automatic updating at 2:00 PM. This was applied to a select few workstations in the company and NONE OF THE SERVERS. I confirmed that that group policy wasn't applied through rsop.msc

As far as I can tell, this has only happened on or 2003 servers, but I can't make any promises that it isn't happening on the 2008 servers and I just haven't noticed.

Ideas?

jwegner
  • 493
  • 5
  • 10
  • Here's a quick fix to make sure this doesn't happen again (but not a diagnosis of the issue): use a GPO to disable the Automatic Updates service for all of the servers. – joeqwerty Dec 20 '11 at 22:20
  • 2
    Did you put a deadline on an approved update? – Shane Madden Dec 20 '11 at 23:26
  • @ShaneMadden Nope. We've been very intentional about not forcing servers to update. And as far as I'm aware, there's only a few updates that have required deadlines, and those should have been installed years ago. – jwegner Dec 21 '11 at 11:41
  • im with joeqwerty here, the only way to make sure this does not happen is to specifically make a GPO to disable to automatic updates. – Flash Dec 23 '11 at 13:06
  • @Flash I agree that that is a _solution_ to the problem, but I'd like to know the _cause_ of the problem. I'm not a fan of leaving unresolved issues lingering my network. Just seems like a bad practice. – jwegner Dec 23 '11 at 14:31
  • It's coming from somewhere, what's in the LOCAL policy? – SpacemanSpiff Dec 25 '11 at 14:35
  • @spacemanspiff as mentioned in the question, i checked gpedit.msc and there was nothing in there about updates. Unfortunately. – jwegner Dec 26 '11 at 15:20
  • 2
    The local policy just modified the registry. Entirely possible that something modified the appropriate keys manually which may not be reflected in a policy or RSoP report. – SpacemanSpiff Dec 27 '11 at 14:51
  • Can you show us the gpo that is applied to the servers to make them use the WSUS server? Otherwise, verify the only settings _applied_ in it are to associate the server (use the Settings tab in the GPMC to view the summarised resultant policy). If that has only the setting for the wsus server url in it, then explicitly roll-out your desired Windows Update settings to make sure that the servers are set right. – Ashley Dec 27 '11 at 21:59
  • What time were you sitting in the server room when it happened? around 2pm by any chance? – hookenz Dec 29 '11 at 00:08
  • @AshleySteel The GPO only has one item in it - setting the WSUS url. As I mentioned on one of the questions, I have already applied a specific GPO to force this not to happen, but I'm more interested in why it happened in the first place. – jwegner Dec 29 '11 at 18:04
  • @Matt Yes, it was around 2:00. Which is conveniently the same time that the unrelated GPO is set to run. However, I confirmed that this is not a result of that other GPO, by checking both RSoP and gpresult. – jwegner Dec 29 '11 at 18:06
  • Upvoting spaceman spiff. Have you checked the actual registry keys? – Dave P Dec 29 '11 at 22:03
  • @SpacemanSpiff That was a good check, but again no luck. The only value in the WindowsUpdate/AU folder is UseWUServer. – jwegner Dec 30 '11 at 16:18
  • I've had the same thing happen to myself a couple of weeks ago on a server, and I've so far failed to find an explanation myself. Will eagerly follow this thread, and update if I find something myself in other ways. Very interesting, but "scary". :/ – Mattias Ahnberg Jan 05 '12 at 00:36
  • I'm highly suspicious of that 2pm GPO seeing is that is about the time it happened. – hookenz Jan 05 '12 at 02:56

3 Answers3

1

On the server itself, from a command prompt, I'd recommend running gpresult (with a /v for Verbose output or a /z for the uber-detailed version) and see of you can locate a policy that is incorrectly applied (or alternative, not applied as it should be for some reason). Also, I'd recommend the >result.txt (or whatever you want to name it) with the /v and /z options - they can get rather wordy and exceed you command prompt screen buffer. Having the results in a text file also makes them searchable, which is nice...

Jes
  • 880
  • 2
  • 6
  • 10
  • Just ran gpresult /z (Thanks for the >result.txt tip.. that was way long!). That's a real handy flag - I never knew that. Unfortunately, nothing of interest came up. The only update related policies were WUServer, UseWUServer, and WUStatusServer. Unfortunately. – jwegner Dec 29 '11 at 18:11
0

Here is an interesting article, though I'm not sure if it applies here.

mgorven
  • 30,615
  • 7
  • 79
  • 122
user8128167
  • 101
  • 2
0

what are your automatic update settings? Check both group policy and local policy. With group policy (if aplicable) run a RSOP which will basically create a picture of what your GP settings look like including taking inheretance into consideration. My guess is you have auto install / reboot setup and MS released an out of band patch.

Secondly, be careful with what classifacation you auto approve. As you point out there's definition updates, but there's also security update, critical (or important) update, reccomended updates. If you have security or critical set to auto approve, and MS releases an out of band patch you're going to get it and it will reboot based on your WSUS scheduled install settings (in gorup policy).

Eric C. Singer
  • 2,329
  • 16
  • 17