3

I'm a bit out of me league here (we're a reasonably small firm, I'm a software dev stuck with doing sysadmin when needed), but I thought I'd ask the smart people at ServerFault about my problem before we called in our 3rd party IT support firm.

We're experiencing a massive traffic spike at the moment, similar to a spike we experienced in October, which went away by itself. If you'll see our ISP's internet usage monitor: Internode Traffic

You'll notice that in the last 2.5 days we've been maxing out our ADSL2 (~20mbps) connection. Ironically, it was Australia day (a public holiday) for one of those days.

We own a Fortinet Fortigate Internet appliance which does our logging and internet connectivity. Here is its snapshot of our usage: This one taken yesterday: fortigate traffic 1

This one today:

Fortigate traffic 2

You'll see that the connection was being absolutely maxed out until we arrived in the office yesterday morning, then it was pretty much maxed out (a lot higher than usual, as you can probably gather from the Internode monthly history image) until we left and then it started 100% usage again. Finally, at around 11ish Internode finally capped us (odd, given we'd been over our limit by heaps for the past 2 days).

We have a subscription to FAMS, Fortinet's online logging and reporting service. We also have our Fortigate export our logs to a syslog server. I've looked at FAMS and this is what the top service usage by destination log looks like: FAMS log

As you can see, there's only around 8 or 9 logged there, which is about normal for us, at least it's no where near the 167gb that we've been logged on Internode as using.

This puzzles me - clearly the Fortigate appliance has some sort of log of the traffic, as its utilization snapshot has it there, but in the detailed logs (syslogs didn't show much, but I don't know how to parse them in an efficient way, I've just been watching them stream in) there is nothing.

My question is, any ideas what sort of traffic this could be? I'm thinking perhaps the Fortigate doesn't bother logging certain types of traffic (ICMP ?) and we're being DOS'ed through that type of traffic. I should mention we do have publicly accessible URLs that are password secured, but our uploads are not included in our quota so I don't think that is it.

Any tips on where I should look? Or should I just call in the big guns (or perhaps just wait till it goes away like last time...)

EDIT: Here is another report from FAMS, this one goes by web requests I believe, unfortunately I can't get a report across all ports for this: FAMS report 2

Scott Pack
  • 14,907
  • 10
  • 53
  • 83
RodH257
  • 569
  • 6
  • 11
  • 23
  • That fortigate report states its only reporting on `:80` traffic so whatever it was it wasn't HTTP – Mark Henderson Jan 27 '11 at 23:56
  • Looking at the graph again, HTTP is the biggest consumer but it doesn't add up to anywhere near 167GB. Port 8000 shows quite a bit of traffic and FTP is at almost 1GB and 443 is almost at 1GB as well. – joeqwerty Jan 28 '11 at 00:15
  • In addition, a report showing the top conversations would probably yield better clues as to what's going on. – joeqwerty Jan 28 '11 at 00:23
  • the graph on the left has port 80 and port 8000 using the most, however in both of those the top IP address belongs to Internode, meaning that the bulk of that traffic is unmetered. Port 8000 is Internodes unmetered music streaming service. So once you take that out the report shows about 4gb of metered usage total across all ports, at least according to the FAMS report, which is why I suspect some services are being filtered from the results – RodH257 Jan 28 '11 at 00:26
  • So the graph shows all of internode's traffic and not just your traffic? I'm a little confused. – joeqwerty Jan 28 '11 at 00:27
  • the graph shows our traffic and its destination. The top destination IP's (see 150.101.195.89 in port 80 graph and 192.231.203.146) resolve to internode servers, meaning the traffic is unmetered. Point is that the graph I posted is missing ~170gbs of traffic from it. There's a mismatch from the preview traffic history from the fortigates web interface, to what is displayed in the detailed logs on FAMS. – RodH257 Jan 28 '11 at 00:32

3 Answers3

4

This link pointed me to the answer: http://forums.adobe.com/thread/391741

The problem was adobe updated and our fortigate router didn't like each other, causing an infinite loop. I would have thought that sort of thing should show up in the firewall logs, but I had a look at the 'requests' version rather than megabytes and one computer was trying to head to adobe for updates.

Looking at the thread, this was the problem:

  • Computer was trying to auto update adobe
  • Adobe starts the download of the update file
  • The Fortigate has http virus scanning, it caches the file and gets ready for virus scanning
  • Adobe thinks that the delay means that the download hasn't worked, so it scraps it and asks for it again
  • This goes on and on, the file never gets to the client PC, meaning that it is never actually logged in the full fortinet logs.

At least its something along those lines, for now I've turned off the fortigates virus scanning of HTTP requests. But I will look into just blocking adobe from everything, or modifying the scanner settings.

Thanks everyone for all your help - I appreciate it!

RodH257
  • 569
  • 6
  • 11
  • 23
  • thats ugly - some days I really think Old Steve (apple) had it right when he said no to adobe... but then again I fire up an adobe product daily ... ;- ) Also - thanks for updating the issue - really nice :-) – Glenn Kelley Feb 20 '11 at 00:57
1

Question:

On your webserver - are you seeing a ton of ip addresses in the logs over and over again... AND are they all pulling the same file or query ...

Generally a dos will have some stream to it that you can follow if you get deep into the logs

for a temporary (or perm solution - )check into a firewall off your network) This may help if it is a ddos

www.CloudFlare.com - we use this for a very highly political website about terrorism. The site has not seen a DDOS now for well over 4 months - and we used to battle it literally every single week.

Good news - its free -) and while it is meant as a firewall - it generally will also act as a free CDN service.

Glenn Kelley
  • 1,294
  • 6
  • 10
  • Hi Glenn, thanks for your answer - cloudfare looks like a great site, I'll keep it in mind for my web development activities, however the issue I'm having here isn't related to the web server we're hosting, I shut it down to see if it would stop and the data kept flowing. I've discovered what the issue is, see my answer on this question. – RodH257 Jan 28 '11 at 07:10
0

The traffic history shows the heavy traffic as inbound on the WAN interface and the graph shows the bulk of the traffic as being HTTP. Have you checked to see what the destination ip addresses are? Are they web servers, streaming media servers, etc? Could this be someone in your office downloading files from the internet, streaming music or videos, etc? I would be very surprised if a DOS attack were able to ramp up that much traffic. Are you able to see both the source and destination ip addresses in the graph? That would give you a better ides of what's going on.

joeqwerty
  • 109,901
  • 6
  • 81
  • 172
  • If it were someone in the office downloading files or streaming music, it should show up in the FAMS graph, but the FAMS graph only shows a normal days traffic, ~4gb metered and ~4gb unmetered, including some unmetered radio streaming. But all up there is around 170gb of data not showing up in the FAMS report. I can see source IP's (graph in OP shows destination), but that says a similar thing. – RodH257 Jan 28 '11 at 00:48