Gateway Timeout 504 on multiple requests. Apache

Question

I have an XML file localy. It contains data from marketplace. It roughly looks like this:

<offer id="2113">
    <picture>https://anotherserver.com/image1.jpg</picture>
    <picture>https://anotherserver.com/image2.jpg</picture>
</offer>
<offer id="2117">
    <picture>https://anotherserver.com/image3.jpg</picture>
    <picture>https://anotherserver.com/image4.jpg</picture>
</offer>
...

What I want is to save those images in <picture> node localy.

There are about 9,000 offers and about 14,000 images.

When I iterate through them I see that images are being copied from that another server but at some point it gives 504 Gateway Timeout.

Thing is that sometimes error is given after 2,000 images sometimes way more or less.

I tried getting only one image 12,000 times from that server (i.e. only https://anotherserver.com/image3.jpg) but it still gave the same error.

As I've read, than another server is blocking my requests after some quantity.

I tried using PHP sleep(20) after every 100th image but it still gave me the same error (sleep(180) - same). When I tried local image but with full path it didn't gave any errors. Tried second server (non local) the same thing occured.

I use PHP copy() function to move image from that server. I've just used file_get_contents() for testing purposes but got the same error.

I have

set_time_limit(300000);
ini_set('default_socket_timeout', 300000);

as well but no luck.

Is there any way to do this without chunking requests?

Does this error occur on some one image? Would be great to catch this error or just keep track of the response delay to send another request after some time if this can be done?

Is there any constant time in seconds that I have to wait in order to get those requests rollin'?

And pls give me non-curl answers if possible.

UPDATE

Curl and exec(wget) didn't work as well. They both gone to same error.

Can remote server be tweaked so it doesn't block me? (If it does).

p.s. if I do: echo "<img src = 'https://anotherserver.com/image1.jpg'" /> in loop for all 12,000 images, they show up just fine.

Answer 1

Since you're accessing content on a server you have no control over, only the server administrators know the blocking rules in place.

But you have a few options, as follows:

• Run batches of 1000 or so, then sleep for a few hours.
• Split the request up between computers that are requesting the information.
• Maybe even something as simple as changing the requesting user agent info every 1000 or so images would be good enough to bypass the blocking mechanism.
• Or some combination of all of the above.

Answer 2

I would suggest you to try following 1. reuse previously opened connection using CURL

$imageURLs = array('https://anotherserver.com/image1.jpg', 'https://anotherserver.com/image2.jpg', ...);
$notDownloaded = array();
$ch = curl_init();
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_TIMEOUT, 10);

foreach ($imageURLs as $URL) {
    $filepath = parse_url($URL, PHP_URL_PATH);
    $fp = fopen(basename($filepath), "w");
    curl_setopt($ch, CURLOPT_FILE, $fp);
    curl_setopt($ch, CURLOPT_URL, $URL);
    curl_exec($ch);
    fclose($fp);
    if (curl_getinfo($ch, CURLINFO_RESPONSE_CODE) == 504) {
        $notDownloaded[] = $URL;
    }
}
curl_close($ch);
// check to see if $notDownloaded is empty

2. If images are accessible via both https and http try to use http instead. (this will at least speed up the downloading)
3. Check response headers when 504 is returned as well as when you load url your browser. Make sure there are no X-RateLimit-* headers. BTW what is the response headers actually?