0

I have a php script that scrapes the web and inserts the scraped data into a database.
This php script runs for a very long time(about a couple of hours).
Sometimes, after the script runs for a long time, the php script just stops executing and shows no error.
The problem isn't caused due to the amount of execution time of the script because i set the php script to an unlimited amount of execution time. 

ini_set('max_execution_time', 0);

I also set the php script to show all errors.

ini_set('display_errors',1);
error_reporting(E_ALL);

But I get no error after the php script stops execution.
I also ran the script in several other computers and i still encounter the same problem, so the problem isn't due to server restrictions either.

I researched the issue and apparently it's a networking problem.
The php script stops communicating with the server and disconnects from it(probably because the php script sent a http request and didn't receive any response).

My question is this:

Is there any way I can check for network disconnections through the php script, and resume the script and try to reconnect if there was a network disconnection?

Eric Leschinski
  • 146,994
  • 96
  • 417
  • 335
yuval
  • 2,848
  • 4
  • 31
  • 51

1 Answers1

0

First "set_time_limit" is a better way to use that function and I would check your server's rules and regulations. If it is your private server, then great. If you are renting a VPS or other type of server, it is highly possible that they have restrictions to prevent misuse, like SPAMing and such. Make sure that you are within their rules and go from there.

morantis
  • 106
  • 8
  • well sometimes the script stops execution and sometimes it works and finishes the execution fully.If i would have restrictions from my server to prevent misuse, the script would never work properly.I also ran the script in several other computers and i still encounter the same problem.I probably just didn't receive a response from an http request, and the php script just stopped execution. – yuval Dec 10 '14 at 14:13
  • Not sure what it is that you are doing, but you can also be getting issues from the server of the page that you are trying to work with. It is possible that your structure is much like a Denial of Service attack. – morantis Dec 10 '14 at 14:16
  • It's unlikely that it's a Denial of Service attack, but even if it is i would still like some code that tries to restart the script, or detect the problem that is occurring. – yuval Dec 10 '14 at 14:23