0

Is it possible to train a spark/pyspark ML lib model using VertexAI custom container model building? I couldn't find any reference in the vertex ai documents regarding spark model training. For distributed processing model building only options available are PyTorch or TensorFlow.

Ashwar Gupta
  • 1

1 Answers1

0

It is possible with custom containers if you leverage the Spark Kubernetes operator but this is not a well documented workflow and will require complex set up. GCP's preferred way to run Spark jobs is on Dataproc https://cloud.google.com/dataproc which supports PySpark, SparkR, Scala. You can still trigger a Dataproc Spark job from Vertex Pipelines and save the model for predictions in Vertex via MLeap.

mchrestkha
  • 66
  • 2