PHP fopen and file_get_contents limited download speed, why?

Question

I'm trying to retrieve a remote file (6MB text file) with PHP and I noticed that with fopen the speed is limited to 100KB/s and with file_get_contents is 15KB/s.

Howewer with wget from the server the speed is above 5MB/s.

What controls these speeds?

I checked the live speeds with nethogs.

Have you considered using cURL instead? – maaarghk Jun 25 '14 at 11:13 — maaarghk, Jun 25 '14 at 11:13
that's why it's a comment and not an answer :) – maaarghk Jun 25 '14 at 11:26 — maaarghk, Jun 25 '14 at 11:26
How exactly are you measuring these numbers? – deceze Jun 25 '14 at 11:37 — deceze, Jun 25 '14 at 11:37
I use nethogs and I watch the speed in real time. – Sandro Antonucci Jun 26 '14 at 19:30 — Sandro Antonucci, Jun 26 '14 at 19:30

score 3 · Answer 1 · answered Jun 25 '14 at 11:08

wget is great on it's own to mirror sites it can actually parse links from pages and download files.

file_get_contents doesn't send a "connection" HTTP header, so the remote web server considers by default that's it's a keep-alive connection and doesn't close the TCP stream until 15 seconds (It might not be a standard value - depends on the server conf).

A normal browser would consider the page is fully loaded if the HTTP payload length reaches the length specified in the response Content-Length HTTP header. File_get_contents doesn't do this and that's a shame.

SOLUTION

SO, if you want to know the solution, here it is:

$context = stream_context_create(array('http' => array('header'=>'Connection: close\r\n')));
file_get_contents("http://www.something.com/somepage.html",false,$context);

The thing is just to tell the remote web server to close the connection when the download is complete, as file_get_contents isn't intelligent enough to do it by itself using the response Content-Length HTTP header.

It doesn't change the speed. Still stuck at 109KB/s to be precise. — Sandro Antonucci, Jun 26 '14 at 19:38
Just wanted to add that if I open another page the speed is also 109KB, total 218KB/s...very weird, what decides this speed? — Sandro Antonucci, Jun 26 '14 at 19:53

PHP fopen and file_get_contents limited download speed, why?

1 Answers1