5

Recently, we had some problems with one of our ESXi servers, caused by the NTP Server DRDoS Amplification Attack using ntpdc.

How do I configure the NTP server on ESXi to not be exposed to this DDoS attack?

Or, if I switch off the service, will that have any effect on my VMs?

HopelessN00b
  • 53,795
  • 33
  • 135
  • 209
fefe
  • 357
  • 1
  • 8
  • 18
  • 3
    Your ESXi server is directly accessible from the Internet, without any sort of firewall?! – Chris S Feb 06 '14 at 19:21
  • @fefe Can you please add some details on what problems you had on the ESXi host, and why you think that it was related to such an attack? People answering and commenting here (incl. me) might make false assumptions about what happened and give wrong or irrelevant answers as a result. So you better provide more details to get the best answers. – VFrontDe Feb 06 '14 at 20:01

5 Answers5

3

The question is not answerable - and you may not like the answers that can be given.

2 scenarios, you fail to mention any detail to filter out:

1 - you were used to amplify. In this case I wonder why the ESXi host was accessible from the internet. It should not be. It should not have a public IP. I do not run ESX but maintain a number of Hyper-V servers for customers and man, you would not find ANY of them on the internt. They live in an internal network, all access is done via VPN to that internal gateway. Yes, servers may have public IP addresses - virtual servers - but never the hosts. No need to. ALl traffic they ned from the internet goes through the firewall (via NAT), so they are protected. I consider this professional baseline in security.

2 - you were targetted. In this case - no way. That is like saying "what do I chane on my house so that people do not send me tons of packages I never ordered". by the time the traffic hits the ESX host it already overloads your bandwidth. BUt again, why is the host at all on the internet?

TomTom
  • 51,649
  • 7
  • 54
  • 136
  • You are right and make sense what you say. The sever was not configured by myself and being newbie I do not have to musch expreience on ESXi. How would I configure my external switch to not be exposed on internet? – fefe Feb 06 '14 at 11:43
  • Not at all. This is not something a switch does, but a router. Standard 101 setup - interneal network, internal addresses, edge router with NAT for outbound traffic and a VPN server for accessing the internal network. – TomTom Feb 06 '14 at 11:46
  • oh boy is getting complicated the server is not in here – fefe Feb 06 '14 at 11:47
  • @fefe It's also possible that you don't filter ntp traffic, (we certainly didn't care about ntp traffic coming into or going out of our network before this) and that's how it's reaching your ESXi server. If that's the case, you could block ntp traffic at the firewall/router for your network, but this might cause problems for legitimate ntp traffic you need. Sounds like you lot might benefit from having a network engineer on staff over there. :) – HopelessN00b Feb 06 '14 at 14:55
  • THe problem is that if you read his answer investigative, they have a esxi server with a switch in front. No router, no firewall.... otherwise it would not be a problem to go to a nice vpn based setup anyway ;( – TomTom Feb 06 '14 at 15:04
  • thanks guys for all of your feedback I try to make some conclusions and after all decision. Yeah is not the best scenario if esxi has a public IP. I will check our Router and switches to get deeper in this thing – fefe Feb 06 '14 at 15:34
2

The answer can be found in this blog post. All you need to do is disable the "monlist" command, which by the way was removed in ntpd 4.2.7 (our ESXi 5.1.0u2 servers are running 4.2.6p2).

  1. Access your server's console, either by enabling the local console or SSH.
  2. Edit /etc/ntp.conf by adding noquery to the first restrict line.
  3. Restart the NTP service with /etc/init.d/ntpd restart.

  4. Verify that the monlist command has been disabled:

    ntpdc -c monlist 1.2.3.4

Eric3
  • 302
  • 2
  • 10
  • The September 2014 version of ESXi 5.1.0u2 appears to fix this issue, as the `monlist` command now returns "***Server reports data not found" in all cases. – Eric3 Oct 08 '14 at 21:30
1

How do I configure the NTP server on ESXi to not be exposed to this DDoS attack?

At the moment, you can't, really. The ntpd in ESXi isn't really configurable (at least in a VMware-supported fashion), so your options are really on or off.

Presumably, VMware will release a patch or update sometime soon to address the issue, (I don't see one that says it addresses the issue right now), and you could apply that when it comes out, but that won't help right now.

You could manually update your ESXi host to a newer ntpd client that isn't vulnerable (ESXi is "just" a customized Linux distro, after all), but I wouldn't do that and risk being in an unsupported configuration with VMware.

Of course, as you suggest in your question, you can also prevent the attack by turning the service off (for the time being).

Or, if I switch off the service, will that have any effect on my VMs?

That entirely depends on how your guest operating systems are configured for ntp. If they're configured to synchronize time with the host system, they'll start to lose time sync (the more time they spend at idle CPU usage, the worse time drift you'll see).

If they're configured to get time from a different ntp source, they won't have problems... unless, of course, they're also running vulnerable ntp clients, in which case, they'll probably be getting the DRDoS attack instead of the host.

If you're currently configuring your virtual guest OSes to get time from their host servers, now might be a good time to consider a different approach, which would depend entirely on your use case. Running Windows domains makes this easy - the PDC emulator gets ntp time from a reliable (external) ntp source, all other Domain Controllers get their time from the PDC emulator, and all clients get their time from their local Domain Controller. Of course, your use case may well be different or more complicated.

HopelessN00b
  • 53,795
  • 33
  • 135
  • 209
  • 2
    SO, why would anyone put an ESXi server onto the internet in the first place then? No need to have that one with a public IP on a public network, or? – TomTom Feb 06 '14 at 11:31
  • @TomTom just because there's no good reason to do something doesn't mean that it's not happening. Seriously, at least half the questions on this site are an example of someone doing something stupid with no reason for doing it, and then having problems as a result. In these parts, saying `here's how you fix that problem you made` is generally preferable to saying `you're doing something stupid`, so... well, that. (But no, I can't think of a good reason to put an ESXi host on a public-facing IP address, either. Maybe I just lack imagination.) – HopelessN00b Feb 06 '14 at 11:35
  • Sorry @HopelessN00b, you made some false claims that I need to correct: 1. The NTP daemon is configurable on ESXi by editing /etc/ntp.conf (See e.g. http://kb.vmware.com/kb/1005092) 2. ESXi is NOT a customized Linux distro. It does use many of the GNU utilities and packages that are also used on Linux and other Unix OSs, but it does not use the Linux kernel. And you cannot just install some Linux package in ESXi. 3. Turning the NTP service off is not the only option. You can secure it through modifications of /etc/ntp.conf and/or by configuring the ESXi internal firewall. – VFrontDe Feb 06 '14 at 18:55
0

First of all, you should not have your ESXi retrieving its time from an outer source to avoid this.

I recommend you to create a NTP Server internally which will themself be pooled by the ntp.org servers and protected correctly to mitigate NTPD DDOS attack.

Dr I
  • 955
  • 17
  • 33
  • As mentioned by TomTom, this only applies if he's the **source** of the attack, not the target. (He didn't say which he was, of course.) – HopelessN00b Feb 06 '14 at 11:32
  • Yep, of course. But even if it is the target esxi should always rely for an internal time source ;-) Best practices advice x) then the internal serveur should be configurated to restrict sources and client on its config file + have an update upper than the 4.47 version of ntp. – Dr I Feb 06 '14 at 11:39
  • Those best practice don't really apply here - see my answer. For one thing, the ESXi ntpd is not really configurable in a vendor-supported manner (it's on, or it's off), and VMware has yet to release a patch addressing this relatively recent vulnerability... so you can't upgrade to a safe version, and stay in a vendor-supported configuration, which is an important consideration for a professional IT environment. – HopelessN00b Feb 06 '14 at 11:44
  • I didn't mention to replace the ESXi binary ;-) just adviced to change the NTP targeted server and THEN on the NTP server to operate maintenance and update ;-) But I may not have express my thoughts so well. – Dr I Feb 06 '14 at 14:43
  • 1
    There is nothing "wrong" with configuring your VMWare hosts to use an external time server for their reference clock (an internal server is generally a *better* solution, but certainly not a mandatory one). The problem is allowing *outside* systems to talk to the NTP daemon on the ESXi hosts. – voretaq7 Feb 07 '14 at 22:05
0

I'm really puzzled by all the long but irrelevant answers here ...

The answer is easy: Enable the ESXi bultin firewall!

By default it is enabled and blocks incoming NTP traffic. So why the heck have you disabled it in the first place??

VFrontDe
  • 1,508
  • 8
  • 13
  • 1
    And what makes you say the firewall is disabled? There's no evidence of that in the post. Not that blocking incoming ntp traffic helps you much if you're participating in a DDoS by sending ntp traffic out, either... which could well be the case (the asker never specified which side of the problem he was on). – HopelessN00b Feb 06 '14 at 19:04
  • What *inside* ESXi could cause it to be the *original* source of such an attack?? Once it was made part of the attack it is of course also *a* source, but it can only be made part of it by being accessed from the outside first. – VFrontDe Feb 06 '14 at 19:29
  • For some reason "malware" springs to mind immediately. What does it really matter? The question (which is admittedly poorly asked, and bound to generate confusion) is about how to defend against a given DDoS attack. "Enable the firewall" is one valid answer, (it's on by default, so I don't particularly like it as an answer, by itself) but so is everything else I've seen in this thread. For all we know, this guy's running an ntp server (as a time source) on his ESXi host, and that's why it accepts incoming traffic. Point is that "why" or "what" don't really matter to the question at bar. – HopelessN00b Feb 06 '14 at 19:40
  • In ESXi the NTP daemon runs by default, but it is supposed to be used in client mode only. This is why it is allowed to query other external servers to synchronize time, but the ESXi firewall will by default not allow it to be queried itself from the outside. There is not even an option available to open the NTP port for incoming traffic, so the only option to use it as a time source for external partners is to disable the firewall completely. I cannot think of any other scenario that would allow it to become part of an NTP based DDOS attack. – VFrontDe Feb 06 '14 at 19:52
  • Just to let you know firewall was not disabled – fefe Feb 06 '14 at 20:04
  • Okay, so was it maybe not the ESXi host itself, but a VM running on it that was involved in the DDoS attack? – VFrontDe Feb 06 '14 at 20:05
  • not on of the VM's IP was on target all was pointed to ESXi's public IP – fefe Feb 06 '14 at 20:07