Download large files with php on appengine onto cloud storage

Question

My appengine project (running on App Engine Standard Environment), uses a media-converter service outside of appengine. I can easily start a new converting job and get notified whenever the job is done. The media-converter delivers a temporary url to retrieve the conversion-result (mp4 file).

Now i want, to start a background-job to download this converted file from the media-converter to my google-cloud storage.

Whatever i tried so far, i cannot download larger files that 32 mb.

These are my approaches so far:

First one by just copy with file_put_contents / file_get_contents as suggested on https://cloud.google.com/appengine/docs/standard/php/googlestorage/

$options = [
                'Content-Type' => 'video/mp4',
                'Content-Disposition' => 'inline',
];
$context = stream_context_create(['gs' => $options]);
file_put_contents($targetPath, file_get_contents($url), 0, $context);

Then i tried to work directly with streams:

$CHUNK_SIZE =  1024*1024;
$targetStream = fopen($targetPath, 'w');
$sourceStream = fopen($url, 'rb');
if ($sourceStream === false) {
  return false;
}
while (!feof($sourceStream)) {
  $buffer = fread($sourceStream, $CHUNK_SIZE);
  Logger::log("Chuck");
  fwrite($targetStream, $buffer);
}
fclose($sourceStream);
fclose($targetStream);

Then i was surprised that this actually worked (up to 32 mb)

  copy($url, $targetPath);

Im running out of ideas now. Any suggestions? I kinda need the cp function of gutil in php. https://cloud.google.com/storage/docs/quickstart-gsutil

I think this stackoverflow issue had a similar issue: Large file upload to google cloud storage using PHP

score 1 · Answer 1 · answered Aug 16 '18 at 08:15

There is a strict limit of 32MB of data for incoming requests.

Refer to Incoming Bandwidth for more details - https://cloud.google.com/appengine/quotas. This must be the reason why you are not able to go beyond the 32MB limit.

Possible Solution -

Can you modify the media-converter service?

If yes - Create an API in the media converter service and do the cloud storage upload at the media converter service itself by invoking the endpoint from your AppEngine application. Use the service account for cloud storage authentication (https://cloud.google.com/storage/docs/authentication#storage-authentication-php).

If no - You can do the same thing using Compute Engine. Create an API in the Compute Engine where you will be passing the URL of the file in response to the background job in AppEngine. Upload to cloud storage using the service account authentication.

Download large files with php on appengine onto cloud storage

1 Answers1