Instagram-API ::: cURL error 18: transfer closed with 3789 bytes remaining to read

Question

I get this error in the middle of data mining from instagram (basically ~8000 images and comments were retrieved correctly and suddenly I receive the following error):

 cURL error 18: transfer closed with 3789 bytes remaining to read (see http://curl.haxx.se/libcurl/c/libcurl-errors.html)

The only part I have used curl in my code is:

function url_exists($url) {
    if (!$fp = curl_init($url)) return false;
    return true;
}

and the url is used here:

        $feed_img_url = $feed[$idx]->getImageVersions2()->candidates[0]->getUrl()."\n";
        if (url_exists($feed_img_url)==true) {
            $img = "results/".$feed_id_str."/".$feed_id_str.".jpeg";
            file_put_contents($img, file_get_contents($feed_img_url));
        }

It doesn't tell which line is producing the error but I guess this exception is coming from one of the above as I haven't used the url anywhere else. This part $feed[$idx]->getImageVersions2()->candidates[0]->getUrl()."\n"; is from Instagram PHP API as in https://github.com/mgp25/Instagram-API

Please suggest fixes to this problem.

Further information: This happens when retrieving data from https://www.instagram.com/gegengrader/ while it doesn't have many posts, posts have lots of likes and only 29 of the posts(images) were retrieved. That said, I am not confident if this is an API rate-limit problem or not. If it is, let me know how to get it fixed.

There's too much PHP involved for me to be totally sure here, but the curl error code 18 usually means that the connection was broken before all data was transferred. It is not an error caused by you, it is the server/network. — Daniel Stenberg, Mar 14 '17 at 07:13

score 1 · Accepted Answer · answered Mar 14 '17 at 02:56

So I realized when I am browsing this instagram account manually, not everything loads anyways and it takes a long time to load it. I used these and now at least I retrieve 70 of 130ish feeds:

function url_exists($url) {
    if (!$cURL = curl_init($url)) {
        return false;
    }

    curl_setopt($cURL, CURLOPT_HTTPHEADER, array('Expect:'));
    return true;
}

and

catch (Exception $e) {
    echo $e->getMessage();
    if (strpos($e->getMessage(), 'cURL error 18: transfer closed') !== false) {
        continue;
    }

}

Possibly not the best solution but it serves my needs. Please feel free to add your answers.

Instagram-API ::: cURL error 18: transfer closed with 3789 bytes remaining to read

1 Answers1