0

I am building a python cloud video pipeline that will read video from a bucket, perform some computer vision analysis and return frames back to a bucket. As far as I can tell, there is not a Beam read method to pass GCS paths to opencv, similar to TextIO.read(). My options moving forward seem to download the file locally (they are large), use GCS fuse to mount on a local worker (possible?) or write a custom source method. Anyone have experience on what makes most sense?

My main confusion was this question here

Can google cloud dataflow (apache beam) use ffmpeg to process video or image data

How would ffmpeg have access to the path? Its not just a question of uploading the binary? There needs to be a Beam method to pass the item, correct?

Andrew Gaul
  • 2,296
  • 1
  • 12
  • 19
bw4sz
  • 2,237
  • 2
  • 29
  • 53

1 Answers1

0

I think that you will need to download the files first and then pass them through.

However instead of saving the files locally, is it possible to pass bytes through to opencv. Does it accept any sort of ByteStream or input stream?

You could have one ParDo which downloads the files using the GCS API, then passes it to a opencv through a stream, ByteChannel stdin pipe, etc.

If that is not available, you will need to save the files to disk locally. Then pass opencv the filename. This could be tricky because you may end up using too much disk space. So make sure to garbage collect the files properly and delete the files from local disk after opencv processes them.

I'm not sure but you may need to also select a certain VM machine type to ensure you have enough disk space, depending on the size of your files.

Alex Amato
  • 1,685
  • 10
  • 15
  • Just adding a comment here. I have been using google-cloud-storage python module to download the large (3 to 4 gb) video files. Talking to the package maintainer, using gsutil command line tool may be preferable and calling os.system() from within python. – bw4sz Aug 18 '17 at 21:07
  • when you said the "GCS API" do you mean using gsutil from command line? I'm trying to understand the best way to get local files, directly through python gcp modules does not seem to be handling the large files well. – bw4sz Aug 18 '17 at 21:51
  • It may help other users to know that according to the gsutil docs "Note: If you have previously installed the Google Cloud SDK or if you are using a Google Compute Engine instance, then you already have gsutil installed." – bw4sz Aug 29 '17 at 01:43