-4

I'm using the jquery cross-domain-request JS script to run a list of names from a database through a criminal record search engine. The JS script uses the "http://query.yahooapis.com/v1/public" vehicle to process the searches to get around the Same Origin Policy. The problem is, my script runs about 200 names over a 2 minute period a few times a day, and this has resulted in my server's IP address being blocked for periods of time and given an 999 error for overloading the server.

Can anyone think of a way around this? It isn't a huge amount of traffic. Like I said, a couple hundred searches maybe 3 or 4 times a day. I can't block my ip address because the script is being run on a webserver (not my personal pc).

I know this is a very random issue, but it is really causing me a lot of problems.

Justin Ervin
  • 13
  • 1
    If it's an ordinary web search, you can switch to the Bing API. It's one of the few providers which doesn't have a cap on requests. – Rob W Nov 01 '11 at 14:46
  • And why can you not write your own proxy on your server and process the data? – epascarello Nov 01 '11 at 14:48

1 Answers1

0

If you know which searches will be run ahead of time, you could schedule the searches to be doled out.

If the searches are live and dynamic you could perform them in smaller batches with a sleep between batches. Present the data returned by previous batches while you wait for the later batches to complete. See, for instance, the kayak.com interface for searching flights - early results populate while kayak searches for additional results. This approach would only really work well if small delyas were sufficient in overcoming yahoo's rate limiting.

Derek Prior
  • 3,497
  • 1
  • 25
  • 30
  • Thanks, that's been the only solution to come to my mind too. It looks like yahoo starts blocking if you submit 200 searches within a 10 minute period, so they'd have to be spread out over a pretty good period of time (in terms of how i have it running now). That may be the only way to go. Thanks. – Justin Ervin Nov 01 '11 at 16:01