0

I'm using the code below to download a zip file from another server which contains over 1,000 images. The source zip file is being downloaded and created / copied to my server, but the code seems to stop at that point. The echo output from the if else below doesn't even display.

I originally had the CURLOPT_TIMEOUT set to 240, but that was not long enough. I would get the curl timeout error. With the zip file being near 100MB I figured it would take about 15 minutes to complete. So I set CURLOPT_TIMEOUT set to 1200 since the source zip size will vary from day to day.

Any idea as to why it stops working? Is there a setting that needs to be changed?

$source_photos_file = 'source_photos_file.zip';
$curl = curl_init();
$destination_photos_file = fopen($_SERVER['DOCUMENT_ROOT'].'/files/photos.zip', 'w');

curl_setopt($curl, CURLOPT_URL, FTP_URL.$source_photos_file); #input
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curl, CURLOPT_CONNECTTIMEOUT, 30);
curl_setopt($curl, CURLOPT_TIMEOUT, 1200);
curl_setopt($curl, CURLOPT_FILE, $destination_photos_file); #output

if (curl_exec($curl) === false)
{
    echo '<p>Curl error: ' . curl_error($curl). ' - error # '.curl_errno($curl).'</p>';
}
else
{
    echo '<p>Done without any errors</p>';
}

curl_close($curl);
fclose($destination_photos_file);

$photos_unzip_result = shell_exec('unzip -P password '.$_SERVER['DOCUMENT_ROOT'].'/files/photos.zip -d '.$_SERVER['DOCUMENT_ROOT'].'/files/pics_test/');
spiderling
  • 139
  • 1
  • 2
  • 11
  • Have you looked into your web server log? – Olaf Dietsche Dec 16 '12 at 06:52
  • Is this being done in cron? If so, you should probably be aware that there are *much better* tools to use here than PHP. This entire thing can be condensed into a two line shell script. – Charles Dec 16 '12 at 06:53
  • You have a typo at `fclose`. – Olaf Dietsche Dec 16 '12 at 07:04
  • Olaf - no errors anywhere to be found. I've checked various logs and nothing. Charles - This will eventually be done in cron. If it is possible to get this to work using a shell script, I'm open to it to get it to work. I do however have a few other zip files being downloaded and extracted. Some are data files which are inserted into a database table. – spiderling Dec 16 '12 at 18:36

1 Answers1

2

Take a closer look what you actually do here. Next to basic trouble shooting like error logging with the highest level and following the error-log, take a look what you actually do:

curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);

You explicitly tell curl to return the download, that means it is copied into memory. Disable that option, especially as you already have specified an output file.

I can not say if this solves your issue, however I hope this is helpful. Do the proper error logging an provide concrete error information like notices, warnings and errors.

hakre
  • 193,403
  • 52
  • 435
  • 836
  • This didn't worked, unfortunately. I have checked all logs I can think of and turned on all error reporting and there are no errors being logged or displayed. As I mentioned above, the echo output from the if else statement doesn't even display after the destination zip file has been created. The if else statement and echoing $photos_unzip_result does display when working with other smaller zip files. The next size down from the 100MB is about 15MB. – spiderling Dec 16 '12 at 18:46
  • you can enable verbosity and directly log to STDERR/STDOUT (don't temp-store and rewind) for curl. Did you do that? Compare with http://stackoverflow.com/a/9707622/367456 – hakre Dec 16 '12 at 19:54
  • Thanks hakre. I finally got the STDERR to write to a file. Replaced the if else statement in my code with just curl_exec($curl). Any echos after this line are not displayed and no errors are being produced. The verbose in the file has no errors either. It shows the connection and the commands to change to the directory where the file I need is. Retrieves the file, completes the transfer and closes connection. It seems to obviously be an issue with the size of the zip I'm attempting to download and copy because I do not have this issue with smaller sized zip files. No errors is frustrating. – spiderling Dec 17 '12 at 01:20
  • Ended up using Charles' suggestion. I have a shell script getting the files with 'wget' and unzipping them with 'unzip'. I then have the shell script calling the PHP script to run the database insertion and update based on certain conditions. Thanks everyone. – spiderling Dec 18 '12 at 20:06
  • Yes, shell rocks. Often much faster and more flexible as coding PHP scripts alone - especially for data-processing. – hakre Dec 18 '12 at 20:08