1

I'm running a kubeflow pipleine with different components for stages like data preprocessing, training etc.

i've understand that i can use specific image for each stage (component). (for example image that not or using a gpu for training.

my question is how the pipleline can set to scale out for different stages (components)?

or it just done automatically

thanks.

gogasca
  • 9,283
  • 6
  • 80
  • 125
tiktok
  • 157
  • 7

1 Answers1

1

The amount of resources allocated on each component (step), due to containerization, are capped by the resources on which it is running.
For GCP products (e.g. Dataflow, AutoML) that are used in the pipeline, the resources are automatically scaled based on that product scalability.
It is possible to create custom components, and for this Vertex AI allows for you to choose the amount of resources (e.g. machine type, CPU limit, GPU limit) for each step. For custom training there are more options available that you can also select in order to create more efficient flows.

Roberto Armenta Ahumada
  • 338
  • 1
  • 7