0

Dataflow currently supports custom containers as per the below page, https://cloud.google.com/dataflow/docs/guides/using-custom-containers

I want to know whether we can use our own VM image for starting the worker VM instances, currently the dataflow service account pulls the image for compute engine from "dataflow-service-producer-prod" project.

But we have restrictions to use only the images prepared within our org with the necessary patches that we need.

Stackoverflow has one more similar question asked reg this, but the answers doesn't address the question of using custom VM for worker instances, most answers are related to custom containers of apache beam,

Custom VM images for Google Cloud Dataflow workers

any help is much appreciated.

Sathish
  • 332
  • 4
  • 11

1 Answers1

1

I got the answer from google cloud community

https://www.googlecloudcommunity.com/gc/Data-Analytics/Using-Custom-VM-instance-images-for-worker-instances-in-dataflow/td-p/554882

Yes, Dataflow supports the use of custom containers. However, DF does not support the use of custom VM images for worker instances on Compute Engine

Similar clarification wrt to dataproc

In terms of Dataproc, you can indeed create custom images. The process involves using the generate_custom_image.py Python program, which creates a temporary Compute Engine VM instance with the specified Dataproc base image. The program then runs a customization script inside the VM instance to install custom packages and/or update configurations. After the customization script completes its task, the program shuts down the VM instance and creates a Dataproc custom image from the disk of the VM instance.

However, Dataflow does not currently support the use of custom VM images. While Dataflow does support custom container images, these are only for the Docker instances running inside the worker VM instances, and not for the worker VM instances themselves.

Sathish
  • 332
  • 4
  • 11