Is there a way to log stats/artifacts from AWS Glue Job using mlfow?

Question

Could you please let me know if any such feature available in the current version of mlflow?

Is there an API for Glue to access data from it? – Raphael K May 16 '19 at 02:02 — Raphael K, May 16 '19 at 02:02
Yeah, there are a couple of ways to access data from Glue. – Naga Budigam May 16 '19 at 08:52 — Naga Budigam, May 16 '19 at 08:52

score 2 · Accepted Answer · answered May 16 '19 at 18:52

I think the general answer here is that you can log arbitrary data and artifacts from your experiment to your MLflow tracking server using mlflow_log_artifact() or mlflow_set_tag(), depending on how you want to do it. If there's an API to get data from Glue and you can fetch it during your MLflow run, then you can log it. Write a csv, save a .png to disk and log that, or declare a variable and access it when you are setting the tag.

This applies for Glue or any other API that you are getting a response from. One of the key benefits of MLflow is that it is such a general framework, so you can track what matters to that particular experiment.

Hope this helps!

Is there a way to log stats/artifacts from AWS Glue Job using mlfow?

1 Answers1