4

I got a server from UbiquityServers about a week ago, I installed a simple Apache server and it just serves up images. The server is under very little load because it simply is an origin server behind Amazon's CloudFront but yesterday it suddenly became unresponsive to SSH to the point that I had power it off/on to SSH back in. I'm trying to find what caused this & I would appreciate any input from the community.

Here are some findings.

I noticed there was a spike in received multicast packets right around the time, here is a log:

sar -n DEV -f sa29 | less
08:30:01 PM      eth1     66.96     63.34     19.54     62.51      0.00      0.00      0.05
08:40:01 PM        lo      0.07      0.07      0.01      0.01      0.00      0.00      0.00
08:40:01 PM      eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
08:40:01 PM      eth1     65.05     70.51      5.63     84.70      0.00      0.00      0.02
08:50:01 PM        lo      0.04      0.04      0.00      0.00      0.00      0.00      0.00
08:50:01 PM      eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
08:50:01 PM      eth1     57.84     59.48      6.71     67.85      0.00      0.00      0.04
09:00:01 PM        lo      0.03      0.03      0.00      0.00      0.00      0.00      0.00
09:00:01 PM      eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
09:00:01 PM      eth1     48.55     47.35      4.30     53.78      0.00      0.00      0.03
09:10:01 PM        lo      0.01      0.01      0.00      0.00      0.00      0.00      0.00
09:10:01 PM      eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
09:10:01 PM      eth1     53.16     51.88      5.61     58.48      0.00      0.00      0.02
09:20:01 PM        lo      0.04      0.04      0.00      0.00      0.00      0.00      0.00
09:20:01 PM      eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
09:20:01 PM      eth1     61.80     63.91      7.75     73.46      0.00      0.00      0.05
09:30:01 PM        lo      0.03      0.03      0.00      0.00      0.00      0.00      0.00
09:30:01 PM      eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
09:30:01 PM      eth1     54.74     55.70      5.79     63.43      0.00      0.00      0.02
09:40:01 PM        lo      0.01      0.01      0.00      0.00      0.00      0.00      0.00
09:40:01 PM      eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
09:40:01 PM      eth1     27.83     28.57      3.17     32.59      0.00      0.00 1058754721.47
09:50:01 PM        lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00
09:50:01 PM      eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
09:50:01 PM      eth1      0.00      0.00      0.00      0.00      0.00      0.00 2142789576.69
10:00:01 PM        lo      0.05      0.05      0.01      0.01      0.00      0.00      0.00
10:00:01 PM      eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
10:00:01 PM      eth1      0.00      0.00      0.00      0.00      0.00      0.00 2152346090.50
10:10:01 PM        lo      0.01      0.01      0.00      0.00      0.00      0.00      0.00
10:10:01 PM      eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
10:10:01 PM      eth1      0.00      0.00      0.00      0.00      0.00      0.00 2142038999.87
10:20:01 PM        lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00
10:20:01 PM      eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
10:20:01 PM      eth1      0.00      0.00      0.00      0.00      0.00      0.00 2153457524.69
10:30:01 PM        lo      0.01      0.01      0.00      0.00      0.00      0.00      0.00
10:30:01 PM      eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
10:30:01 PM      eth1      0.00      0.00      0.00      0.00      0.00      0.00 2142646569.12
Average:           lo      0.03      0.03      0.00      0.00      0.00      0.00      0.00
Average:         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:         eth1     91.61     90.43     21.05     59.33      0.00      0.00 87333330.59

10:42:20 PM       LINUX RESTART

10:50:01 PM     IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
11:00:01 PM        lo      0.03      0.03      0.00      0.00      0.00      0.00      0.00
11:00:01 PM      eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:00:01 PM      eth1     31.57     28.14      2.54     30.25      0.00      0.00      0.05
11:10:01 PM        lo      0.11      0.11      0.01      0.01      0.00      0.00      0.00
11:10:01 PM      eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00

Server is using CentOS 6. I'm not quite sure as to what else I should be checking.

voretaq7
  • 79,879
  • 17
  • 130
  • 214
Emil
  • 191
  • 1
  • 5

1 Answers1

1

I just wanted to contribute to this, as I had the exact same problem, using the same hosting company as the OP. Our server would go unresponsive for a long time (sometimes hours) and that always coincided with a crazy amount of incoming multicast packets.

What I found out was that our server was not on a private VLAN and was being exposed to "public" multicast and broadcast traffic, specifically traffic probably directed to the previous owners of our IP address (web hosts recycle those). Our IP address was once used by a an online gaming community, so go figure.

Getting the guys at Ubiquity to place us on a private VLAN immediately solved this problem, and was $80 total (one-time fee). They should have probably warned me about that when I purchased my dedicated server about this vulnerability, but they did not.

I have nothing but great things to say for Ubiquity Hosting, and so I wanted to make sure the record was clear that it just boiled down to the fact that my IP was vulnerable to UDP traffic, and my box couldn't handle a billion+ erroneous UDP packets in such a short burst.

Hope this helps someone!

Jayse
  • 11
  • 1