1

Broadly, I seek to answer the question, "Should I redevelop my app with the Amazon Web Service (AWS) platform?"

As described in another post, I have a google app engine (GAE) php55 service (which I am now contemplating migrating to AWS) that periodically checks a public web server and downloads a file directly to a Google Cloud Storage bucket. This file is typically small (<1MB), but occasionally over the 32MB individual response size limit on GAE. My simple app is based on the following:

<?php
$strSource = 'https://example.com/file.zip';

$strBucket = 'bucket-1234';
$strDirectory = '/path/to/file/'; // Google Cloud Storage directory
$strName = 'file.zip';
$strDestination = 'gs://' . $strBucket . '.appspot.com' . $strDirectory . $strName;

copy($strSource,$strDestination);
?>

My service needs to run about 100 times a day; often the files my service downloads are <<1MB and only about 1% of the time is the file over Google's current 32MB response size limit. I need the bucket to hold <1TB of data.

I am hesitant to implement Google's recommended solution (which is to run a Google Compute Engine virtual machine) because I would like to keep everything serverless.

If I redevelop this service as an AWS Lambda function (instead of GAE), will that immediately increase my app's individual file size limit up to 512MB?

(Note: The 512MB file size limit value assumes my Lambda function first downloads the file to /tmp, so long as I am diligent enough to clear the contents of this directory after transferring each file to an s3 bucket.)

Or is the Lambda function response size limit even higher and only based on what the Lambda function can download in the 15-minute execution time limit if I download directly to an s3 bucket?

My preliminary search suggests I should consider rebuilding in the AWS ecosystem, but I would like to avoid investing significant time learning a different ecosystem if a serverless solution for my needs on AWS is infeasible.

It also appears as though I would probably need to rewrite much of my current app in python so it is natively supported in AWS. Am I understanding these issues correctly? Are there any other quotas or limits that would restrict the size of a file I could transfer from a public web server into my s3 bucket using a Lambda function? Can I get around that 512MB limit by saving the file directly to the s3 bucket?

J. Doe
  • 23
  • 5
  • 1
    Use Google Cloud Run instead. With Cloud Functions, files are memory and have fixed limitations of 32 MB (64 MB for Go apps). Cloud Run is serverless containers that do not have the same limitations provided you are not serving files via HTTP. Cloud Run also has a runtime limitation of 15 minutes. – John Hanley Oct 28 '19 at 21:54
  • One issue is that you are trying to download files. Instead, download smaller parts of a file into memory (1 MB) and write that part to Cloud Storage. Repeat this process "streaming" the file data to storage. Then you are not limited by file size, just network bandwidth and runtime limits. – John Hanley Oct 28 '19 at 21:56
  • AWS Lambda now supports additional languages, such as PHP: [Scripting Languages for AWS Lambda: Running PHP, Ruby, and Go | AWS Compute Blog](https://aws.amazon.com/blogs/compute/scripting-languages-for-aws-lambda-running-php-ruby-and-go/) – John Rotenstein Oct 28 '19 at 23:08
  • Yes, the AWS Lambda function could download a file to the `/tmp/` storage, upload it to Amazon S3 and then delete the local file. Or, you could possibly do that transfer in memory without writing to local disk. – John Rotenstein Oct 28 '19 at 23:09
  • Thank you, @JohnHanley. You appear to have two different recommendations in your comments. In reverse order: (2) Modify my existing GAE code to save the zip file to memory in chunks less than the 32MB limit. Then save the data from memory to GCS bucket. This sounds the most simple. Can you direct me to an example? (1) Use Cloud Run instead ((why are there so many different products??? I can't keep up with them all -- sigh)). [This service appears quite new](https://cloud.google.com/run/docs/quickstarts/build-and-deploy). Do you have any sense of when this might come out of Beta? – J. Doe Oct 29 '19 at 00:27
  • Thank you, @JohnRotenstein. I may follow up depending on whether or not there is a way to minimally change my existing deployment in GAE based on what the other John is suggesting in his second comment. – J. Doe Oct 29 '19 at 00:31
  • I placed my comments from easy to harder. Developing with Containers and deploying to Cloud Run is very simple. Creating code that streams file data in Lambda is much harder but not difficult if you are an experienced developer. – John Hanley Oct 29 '19 at 00:35
  • See my answer on your other question here: https://stackoverflow.com/a/58600705/291180 – Travis Webb Oct 29 '19 at 02:26

0 Answers0