2

On a regular (non-flexible) instance of Google App Engine, you can use the Blobstore API and create a URL to allow a user to upload a file directly into your Blobstore. When it is uploaded, your app engine application is notified of the location of the file and can process it. An example of the python code is:

from google.appengine.ext import blobstore
upload_url = blobstore.create_upload_url('/upload_photo')

See the Blobstore docs.

Switching to Google App Engine Flexible Environment, usage of the Blobstore has been largely replaced by Cloud Storage. In such a case, is there an equivalent of create_upload_url?

My current implementation takes a standard file upload to a python Flask application. Then proceeds with something like:

from flask import request
from google.cloud import storage

uploaded_file = request.files.get('file')

gcs = storage.Client()
bucket = gcs.get_bucket(bucket_name)
blob = bucket.blob(blob_name)
blob.upload_from_string(
    uploaded_file.read(),
    content_type=uploaded_file.content_type
)

This seems like it is doubling the network load compared with create_upload_url because the file is coming into my app engine instance and then immediately being copied out. So the uploader will be made to wait extra time whilst this is happening. Presumably I will also incur extra App Engine charges for this. Is there a better way?

I have workers that later process the uploaded file, but I tend to download the file from Cloud Storage again in their code because I don't think you can assume that the worker will still have access to a file stored in the instance file system. Therefore I don't get any benefit of having the file uploaded to my instance rather than direct to it's storage location.

Jon G
  • 1,656
  • 3
  • 16
  • 42
  • I answered a similar question a couple of days ago https://stackoverflow.com/questions/42002013/replacing-blobstore-upload-handler-with-gcs/42002289#42002289 which may help you. You can use the same method of creating the upload handler. – Aaron Feb 07 '17 at 10:21
  • 1
    Thanks @Samson, but I don't think `google.appengine.ext.blobstore` package is available in the App Engine Flexible Environment. Maybe I am wrong? – Jon G Feb 07 '17 at 10:31
  • @JonG Did you found an answer for this? – Anees Hameed Feb 14 '18 at 21:24
  • Added my answer below @AneesHameed. – Jon G Feb 15 '18 at 09:45

2 Answers2

3

I have started using create_resumable_upload_session to create a signed URL that our client side application can upload a file to. Something like:

gcs = storage.Client()
bucket = gcs.get_bucket(BUCKET)
blob = bucket.blob(blob_name)
signed_url = blob.create_resumable_upload_session(content_type=content_type)

Then when the client has successfully uploaded a file to our storage, I subscribe to a Pub/Sub notification of the creation using this Cloud Pub/Sub Notifications for Cloud Storage.

Jon G
  • 1,656
  • 3
  • 16
  • 42
  • What do you do with the Pub/Sub notification when you get it? IIUC, that's going to arrive from a totally separate session, not the same as having the browser redirected to another action in your app. So I wouldn't know how, for example, to give the user a response based on a file they uploaded. (Just based on how I would guess it works. Documentation is lacking.) – philo Jun 03 '19 at 19:43
  • 1
    You are correct @philo, I don't receive the pub/sub message in the same request. Instead the process that handles the pub/sub message updates a database entry and then sends a message to the users app with the information. For me this is a Firebase Cloud Message, but could be something else: Web socket, the app polling an api, etc. – Jon G Jun 20 '19 at 09:18
0

Each blob created with the new Google Cloud Storage Client has a public_url property:

from flask import request
from google.cloud import storage

uploaded_file = request.files.get('file')

gcs = storage.Client()
bucket = gcs.get_bucket(bucket_name)
blob = bucket.blob('blob_name')
blob.upload_from_string(
    uploaded_file.read(),
    content_type=uploaded_file.content_type
)
url = blob.public_url

--

With the Blobstore, a GAE system handler in your instance takes care of the uploaded file you pass to the upload url created. I'm not sure if it's an issue handling it yourself in your code. If your current approach is problematic, you might want to consider doing the upload client side and not pass the file through App Engine at all. GCS has a REST API and the cloud storage client uses it underneath, so you can read and upload the file directly to GCS on the client side if it's more convenient. There's firebase.google.com/docs/storage/web/upload-files to ease you through the process

Jeffrey Godwyll
  • 3,787
  • 3
  • 26
  • 37
  • Thanks for your help. Your solution still requires the entire file to be sent to my app engine instance, and then forwarded on to Google Cloud Storage, incurring charges on the way in and out, right? With create_upload_url, the user uploads directly to the Blobstore and therefore I would only incur charges for that upload. Does that make sense? – Jon G Feb 08 '17 at 11:50
  • Hmm, with the Blobstore, a GAE system handler in your instance takes care of the uploaded file you pass to the upload url created. I'm not sure if it's an issue handling it yourself in your code. If your current approach is problematic, you might want to consider doing the upload client side. GCS has a REST API and the cloud storage client uses it underneath, so you can read and upload the file directly to GCS on the client side if it's more convenient. There's https://firebase.google.com/docs/storage/web/upload-files to ease you through the process – Jeffrey Godwyll Feb 08 '17 at 21:05