0

I created an App Engine Service to transcode video files as well as images. Video files can be large and thus will take longer to process. Cloud Tasks seemed like a great fit but my front-end clients needs to monitor tasks during execution.

For example, if a user uploads a video file, I would like to keep the client informed on the progress of their upload. I can't really see anywhere in the docs that show how to request this information from an actively executing task (or an API I can send these updates to?). My current implementation uses web sockets to relay this information, however, this doesn't seem scalable if lots of clients start uploading videos. My thought is to store task state in a NoSQL db and return the DB task ID to the client when a video file is uploaded. Then, I would just poll the DB for updates.

Is this correct or is there a better approach?

tommyc38
  • 703
  • 1
  • 8
  • 21

1 Answers1

0

The approach come from different perspectives, but my suggestion its to move the less possible pieces of the board.

If you have developed a nodeJS application to perform your video operations, complete them with a notification system, or tool like Service Worker. https://www.youtube.com/watch?v=HlYFW2zaYQM

Meanwhile, you can use PubSub and make a the complete ring about notifications/events:

"Ingest events at any scale"

Data ingestion is the foundation for analytics and machine learning, whether you are building stream, batch, or unified pipelines. Cloud Pub/Sub provides a simple and reliable staging location for your event data on its journey towards processing, storage, and analysis.

Mario
  • 406
  • 2
  • 6
  • Thank you @Mario. The problem with pub/sub that I can’t reconcile is querying a specific job and relaying that job’s status back to the client that initiated it. This is why I thought it made more sense to persist the job in a db. Any direction here with pub/sub would be appreciated. – tommyc38 Aug 18 '19 at 18:05