3

I have an odd problem-- on a high traffic website (millions of visitors a month), every day we get about 20 or so situations where one host begins incessantly requesting the same page, over and over-- multiple times per second, for any length of time from a few minutes to all day.

The attack is apparently not malicious, as I've backtraced the IP address and matched them to some of our registered users, whom I've interviewed. They say that when this happens, a javascript counter on our site 'keeps refreshing,' their computer becomes slow, but it is otherwise usable. It doesn't happen on every page load, but rather sporadically.

The log hits have the following characteristic:

  • They start 'normal'-- first page load actually accesses all the page's resources, as well as the .php
  • Then the host begins requesting JUST the php page, without the resources incessantly, usually one per second (but sometimes faster and sometimes a few seconds slower)
  • The remote browser is always Firefox 3.5.x
  • Subsequent hits have no referrer, even though the first page request had one

We are at wit's end with what to do with this. A simple DoS filter is not appropriate-- we have that and the threshold to trigger it is much higher than a single page request (without related images, css, etc.) a second.

The stack is LAMP, Redhat install, PHP 5.2, Apache 2.2.3, with an NGINX box operating as a software load balancer.

This is crushing our site-- please help! In the absence of good ideas, we are going to resort to writing a dummy filter that stores a key of IP+URI in memcached, and increments each page request. Once it crosses a certain threshold in a certain period of time, we'll 403 further requests. I do NOT think this is the appropriate place in the networking stack to handle this issue, however.

Thank you for anything you can contribute!

Dennis Williamson
  • 62,149
  • 16
  • 116
  • 151
  • 2
    Have your reported the bug to Mozilla? If it wasn't internal, I would say the attack sounds like click fraud, say a competitor exhausting the online ad budget of the advertiser found on that page (which may not be your company). – kmarsh Oct 06 '09 at 11:47

3 Answers3

5

From your own description I would suggest you start by removing that counter and see if that really does make a difference. It's so easily tested that I'm surprised you haven't already done so.

John Gardeniers
  • 27,458
  • 12
  • 55
  • 109
  • I know. However, we get over 5 million uniques a month, and just a few people a day trigger this issue, so it's unfortunately not that easy for me to pull the counter and wait to see if it happens again as I can't make the decision in a vaccuum. –  Oct 06 '09 at 10:17
  • 2
    Sorry, I don't understand. Why can't you remove that counter? The amount of traffic is irrelevant and if you are having a problem significant enough that you are asking for help I'd say it's far from making a decision in a vacuum. Rather than taking the obvious first step you want us to take wild guesses at what the problem could be, even though the evidence you provide points at only one thing. – John Gardeniers Oct 06 '09 at 10:26
  • oh don't misunderstand me, that's a logical thing to try, but if your 'failure rate' so to speak is that low (that's where the traffic comes in), and there's no reproducibility, then I can't go around disabling what others in my organization would consider important features. I asked the question here hoping someone else has had a similar issue-- I see a sprinkling of reports online about just this pattern of accesses that are not DoS's, and there is no uniformity in those folks having counters on their site. Nevertheless, thank you, and I do agree with your point. –  Oct 06 '09 at 10:42
  • 1
    If the question is worth asking it's worth investigating. If the counter is so important why not use a different one to do the same thing? The key here is to eliminate the one you are currently using to see if that changes things. – John Gardeniers Oct 06 '09 at 10:50
3

Looking for hits and sending a 403 is only really masking the issue. Sounds like a better way to fix the problem would be to fix the faulty javascript on the offending page.

JamesRyan
  • 8,166
  • 2
  • 25
  • 36
  • 1
    Try to reproduce the situation in your development or any other isolated environment, then fix the javascript bug. – Karsten Oct 06 '09 at 09:40
0

The problem with the memcached solution is that you're still getting the hits, but you're planning on avoiding any intensive work to serve it by checking memcached and making a determination if this is a bad request. That's work in itself, though sure, it may be saving your webserver or database server some cpu.

The other approach to using memcached for this is to calculate the response for that URI, and if it is unique to the IP then store the response keyed by the IP+URI in memcached, if not just key it by URI with any other unique request parameters that would vary the response. Then answer all requests with any cached response that is less than X seconds old. Now you're still recalculating every X seconds, but that's less than lots of ties a second. I believe a memcache aware proxy or webserver would be able to be configured to do this without writing anything extra, say MemProxy, or Nginx respectively.

Getting at the main cause of the bad behavior would be preferable. If it is JavaScript related, it could be JavaScript associated with a particular ad on your pages. You need to have a mechanism in development to reload the page with each possible ad. If you don't have that you can't end up catching the ads that are giving a couple users problems.

dlamblin
  • 939
  • 2
  • 10
  • 20