How to use the same code for machine learning "data transformations" before prediction in both reshearch & production

Question

I wonder what is the best practice/s of taking for example a jupiyter notebook that contains the whole flow from eda to prediction and use the same code for "data transformations from the raw data" till using it for predictions, in case that external service also want its predictions but has a "raw data".

is there a ml-ops framework or design pattern for achiveing this goal?

how to use the same code for machine learning "data transformations" before prediction in both research & production

thanks.

I've tried to refactor the code

Answer 1

You have different ways to approach it at Google Cloud.

If you just need extra preparation/data preprocessing steps at inference time, you can use Vertex AI custom containers images for predictions, which will allow you to add extra functions ie. preparing/normalizing/scaling data before running the real prediction.

If you want to decouple the parts, like one API for data preprocessing and then later the inference itself, you can play with Cloud Functions or Cloud Run, where you may have an API for data preprocessing, one API for inference and you can encompass calls (ie. you may have one argument on the data preprocessing API to define if at the end it will call the API for inference or will simply return the data preprocessed).

If you have a data pipeline already established, ie usin Cloud Composer, which is a managed service based on Apache Airflow, you can add your preprocessing and inference code as steps on your DAG.

At last, but not least, you have Vertex AI Pipelines which gives you a managed way of using Kubeflow or TensorFlow Extended (TFX) -- but to be honest, for a simple pipeline just for preprocessing + inference, Pipelines may be "too much"... I would really go down the road of using custom inference containers or try using Cloud Run.

Hope that helps!