0

I am creating a php script that requires the server to make several cURL requests per run. I'll be running this script through cron every 3 minutes. Im looking to maximize the amount of cURL requests I can make in a 24 hr period.

What I am wondering is if it would be better from a performance standpoint to get a dedicated server, or several small shared hosting accounts. With the problem being number of external connections and not system resources I'm wondering which is the best approach.

VolkerK
  • 95,432
  • 20
  • 163
  • 226
websiteguru
  • 493
  • 1
  • 11
  • 23
  • How many you want to run in every 3 minute cycle? Ever thought about NOT using CURL but something more efficient? – TomTom Mar 19 '10 at 08:09
  • You say you make several requests every 3 minutes, which implies a little amount of requests. You also want to maximize the amount of requests, what is this about? Are you trying to download the internet? – Sjoerd Mar 19 '10 at 08:09
  • @TomTom - I've never had any performance issues with cURL, can you suggest "something more efficient" for HTTP requests? If performance is key I'd be more concerned about PHP's performance than cURL and develop it as a native application. – Andy Shellam Mar 19 '10 at 08:16
  • Well, curl is not a bad utility, but you dont think large web crawlers use it? Why start / stop a proces 10.000 times to check 10.000 urls? That is the problem - if you really need performance, write a program that checks a LIST ot url's without a restart, in multiple threads, using async IO. Depends how many items you need. That is all I way. PHP i dont see relevant here- wrong tag. issue is CRON + script. Idoubt the user uses PHP on THAT part of his app. More on the front end ;) – TomTom Mar 19 '10 at 08:38

1 Answers1

1

Shared hosting accounts generally have a very low limit, making something like this difficult. I would not recommend this. It would be MUCH more effective to invest in a VPS hosting account (such as one with Linode or Slicehost). They generally don't have a connection limit (or at least not one that I've run into). They're also faster and much more cost-effective than straight-up shared hosting. A dedicated server would be overkill.

Also, since you're using PHP, feel free to take advantage of curl_multi_init, which will allow you to run many cURL requests at once.

Hope this helps!

mattbasta
  • 13,492
  • 9
  • 47
  • 68
  • AWESOME - I will be working on this tomorrow and will put this in place. I've never seen that function before, but it looks perfect – websiteguru Mar 19 '10 at 21:32