0

My web server has periodic spikes in bandwidth consumption:

Logs revealed these are coming from a single user:

(ip) - - [31/Aug/2011:20:55:31 -0400] "GET /train/16/32462 HTTP/1.1" 200 12909
(ip) - - [31/Aug/2011:20:55:32 -0400] "GET /css/global.1314709741.css HTTP/1.1" 200 9388
(ip) - - [31/Aug/2011:20:55:32 -0400] "GET /css/battle.1307899664.css HTTP/1.1" 200 1567
(ip) - - [31/Aug/2011:20:55:32 -0400] "GET /js/jquery.pulse.1313994229.js HTTP/1.1" 200 2534
(ip) - - [31/Aug/2011:20:55:32 -0400] "GET /js/jquery-1.6.2.min.1309653547.js HTTP/1.1" 200 91887
(ip) - - [31/Aug/2011:20:55:32 -0400] "GET /js/common.1313994229.js HTTP/1.1" 200 9128
(ip) - - [31/Aug/2011:20:55:32 -0400] "GET /js/battle.1313994229.js HTTP/1.1" 200 24155
(ip) - - [31/Aug/2011:20:55:32 -0400] "GET /js/prototype.1283487121.js HTTP/1.1" 200 140186
(ip) - - [31/Aug/2011:20:55:44 -0400] "GET /train/16/36685 HTTP/1.1" 200 13210
(ip) - - [31/Aug/2011:20:55:44 -0400] "GET /css/global.1314709741.css HTTP/1.1" 200 9388
(ip) - - [31/Aug/2011:20:55:44 -0400] "GET /css/battle.1307899664.css HTTP/1.1" 200 1567
(ip) - - [31/Aug/2011:20:55:44 -0400] "GET /js/jquery.pulse.1313994229.js HTTP/1.1" 200 2534
(ip) - - [31/Aug/2011:20:55:44 -0400] "GET /js/jquery-1.6.2.min.1309653547.js HTTP/1.1" 200 91887
(ip) - - [31/Aug/2011:20:55:44 -0400] "GET /js/common.1313994229.js HTTP/1.1" 200 9128
(ip) - - [31/Aug/2011:20:55:44 -0400] "GET /js/battle.1313994229.js HTTP/1.1" 200 24155
(ip) - - [31/Aug/2011:20:55:44 -0400] "GET /js/prototype.1283487121.js HTTP/1.1" 200 140186

As you can see, requests for static resources (css, js) are returning in full with status 200 OK, when they should instead be returning short 304 Not Modified responses (or not be requested at all).

What is causing this and is there something I can do on my end to fix it?

Ed Mazur
  • 133
  • 4
  • That chart looks like it has sustained periods of higher traffic. To sustain that throughput over that length of time with those small files, they'd need to be making hundreds of thousands of requests - two requests 12 seconds apart for different pages don't really fit this profile. It seems more likely to just be a more heavily trafficked time of day. Can you confirm that the majority of those spikes are all from this one client? – Shane Madden Sep 01 '11 at 02:25
  • @Shane I should have mentioned that this is a browser-based game, so sustained traffic from individual users is common. A little command-line fu tells me that of the 630 unique IPs that visited today, this particular IP accounted for ~933MB of today's bandwidth, which is around 35% of the total for the day. – Ed Mazur Sep 01 '11 at 02:45
  • Ah. Sounds like a misbehaving browser, then - check the user agent string with some more thorough logging to see if it's some piece of junk (that you could subsequently block), and maybe capture a request to see if they're properly sending an `If-Modified-Since` header? – Shane Madden Sep 01 '11 at 02:50
  • @Shane Currently my logs don't contain user agent or anything, but I will probably look into this and try examining what headers are being sent. I did contact the user and he said he's using IE9, but he's not too tech savvy so I wasn't able to debug it from his end. – Ed Mazur Sep 01 '11 at 02:55
  • I inspected the request headers. He's indeed using IE9 and the only unusual header is `X-Forwarded-For`. – Ed Mazur Sep 01 '11 at 04:55
  • But, is there an `If-Modified-Since`? That's necessary for a `304` to be sent for an unmodified asset. The `X-Forwarded-For` header would mean a proxy's involved - it may well be the cause. – Shane Madden Sep 01 '11 at 05:02

1 Answers1

1

What makes you think the requests shouldn't be returned with 200 OK? There's no information in the log provided that indicates that the client sent appropriate caching headers to allow your server to reply with anything else.

The problem could be at your end, by not sending good caching headers, or it could be a problem with the client(s) not sending the relevant caching headers back. Don't forget, also, that an IP address doesn't uniquely map to a single browser; you could have multiple users from the same NATted address all making requests to your website.

womble
  • 96,255
  • 29
  • 175
  • 230
  • I based that assumption off the fact that it's only this one user's traffic exhibiting this behavior. There are hundreds of other users and it's only happening with this one, so I assumed it was abnormal. If I grep through today's log, "jquery" appears 13,505 times: 7,583 returning `200 OK` (6,328 of these coming from the problem IP) and 5,811 returning `304 Not Modified`. For these reasons I think the server's cache settings are working correctly and suspect the problem is some client-side setting. If it's indeed the latter, what can I do to avoid wasting bandwidth? – Ed Mazur Sep 01 '11 at 02:53
  • There's nothing you can do to stop a client making requests. – womble Sep 01 '11 at 03:22