0

I'd like to find out what's the best architecture for a python app that gets triggered when a file is uploaded to Google Cloud Storage, does some processing and outputs a file to Google Drive?

I've tried using Cloud Functions but I'm getting a Function invocation was interrupted. Error: memory limit exceeded. in the logs.

I've also followed this tutorial Trigger Cloud Run with events from Eventarc so I know that one way is with EventArc and Cloud Audit logs.

2 questions:

  1. What other methods are there since I require higher memory limits?
  2. How do I get the bucket name and file name from cloud audit logs? through protoPayload.resourceName?
kenshima
  • 401
  • 4
  • 12
  • What type of processing are you using with your Cloud Function (CF)? what is the size of the data your are processing in CF ? have you tried to increase the memory of your CF: you can go up to 8GB – MBHA Phoenix Jul 26 '21 at 14:11
  • basically I'm doing some basic curve fitting for csv data and outputting a png image to google drive made with matplotlib. The size of the data would probably be max 12 mb. But thank you, I see now I can up the size of the cloud function but I'm gonna go with pub sub instead – kenshima Jul 27 '21 at 10:14

1 Answers1

2

You can use PubSub. You can create a PubSub notification and create a push subscription to the service that you want.

  • Http cloud function
  • App Engine
  • Cloud Run
  • Any HTTP service running somewhere (VM, Kubernetes, on prem,...)

EventArc is mainly a wrapper of this process and can call only Cloud Run service (for now)

guillaume blaquiere
  • 66,369
  • 2
  • 47
  • 76
  • thank you for a concise answer. I found some good resources to implement [GCS to Pub/Sub to Cloud Run](https://cloud.google.com/run/docs/tutorials/image-processing). – kenshima Jul 27 '21 at 10:16