0

I am facing tough times deploying Dataflow pipelines. Thanks to GCP Documents.. Below is what I am trying to achieve.
I have 4 deep learning models (binary files each 1 gb). I want to get predictions from all the 4 models. So I stired all the 4 models in bucket. And in my pipeline I do.

download_blob(......, destination_file_name = 'model.bin')            
fasttext.load_model('model.bin')

It works fine but I have below concern.

Everytime a job is created it downloads these files which will consume lot of time. If I call 100 jobs, So the models will be downloaded 100 times. Is there any way I could avoid it?
Is there any way I could stage these files in some location so that even if I trigger job 100 times the model is downloaded just one time?

Chaitanya Patil
  • 183
  • 2
  • 16

1 Answers1

1

As mentioned at GCP Dataflow Computation Graph and Job Execution, you could put the model data in a custom container. Of course the container itself will still have to be staged on the workers.

You could also consider if a single pipeline (perhaps streaming if the input is not known ahead of time) would server your needs better than many successive runs.

robertwb
  • 4,891
  • 18
  • 21
  • Thank you! WIll take a look on it and share my findings.. I was also curious about loading loading the model files in staging location. Does that help in any way? https://stackoverflow.com/questions/30516965/staging-files-on-google-dataflow-worker – Chaitanya Patil Aug 16 '21 at 20:57
  • The staging location is still GCS. – robertwb Aug 16 '21 at 21:01