The approach used in this post is obsolete: you can find the solution by using VertexAI in the post linked below:
Automate batch predictions with VertexAI pipeline and Kuberflow component ***SOLVED***
###############################
I am new with cloud functions and AI Platform Pipeline.
I have my customised model stored in GCS that I run from my laptop with the script training.sh
where training.sh is
gcloud ai-platform jobs submit training model_training_$now \
--scale-tier basic \
--packages gs://my_project_bucket/my_package_model-0.1.2.tar.gz \
--module-name model.train_pipeline \
--job-dir=gs://my_project_bucket/trained_model \
--region europe-west1 \
--runtime-version=2.5 \
--python-version=3.7 \
-- \
--user_first_arg=first_arg_value --user_second_arg=second_arg_value
I am trying to automate the training every time a new file is uploaded in the input_data bucket. I am using cloud functions to do this. However it is not clear to me how to use Kuberflow to run the training.sh file.
I am using this tutorial
and its notebook
Here he defines a sequential pipeline by creating containers.
def sequential_pipeline(filename='gs://ml-pipeline-playground/shakespeare1.txt'):
"""A pipeline with two sequential steps."""
op1 = dsl.ContainerOp(
name='filechange',
image='library/bash:4.4.23',
command=['sh', '-c'],
arguments=['echo "%s" > /tmp/results.txt' % filename],
file_outputs={'newfile': '/tmp/results.txt'})
op2 = dsl.ContainerOp(
name='echo',
image='library/bash:4.4.23',
command=['sh', '-c'],
arguments=['echo "%s"' % op1.outputs['newfile']]
)
I cannot see how to define a similar function to run my training.sh. Do I need to containerise my model package my_package_model-0.1.2.tar.gz?
Is anyone familiar with this type of automation?
