MLflow: how to read metrics or params from an existing run?

Question

I try to read metrics in this way:

 data, info = mlflow.get_run(run_id)
 print(data[1].metrics)
 # example of output: {'loss': 0.01}

But it get only last value. It is possible to read manually all steps of a particular metric?

You are printing only the element at the index one. Please check it. — Srini, Mar 10 '20 at 11:18
No the element at position 0 is the string 'data' and at position 1 there is the dictionary. — Nicola Landro, Mar 10 '20 at 11:43

score 7 · Accepted Answer · answered Dec 02 '20 at 04:51

I ran into this same problem and was able to do get all of the values for the metric by using using mlflow.tracking.MlflowClient().get_metric_history. This will return every value you logged using mlflow.log_metric(key, value).

Quick example (untested)

import mlflow
trackingDir = 'file:///....'
registryDir = 'file:///...'
runID = 'my run id'
metricKey = 'loss'

client = mlflow.tracking.MlflowClient(
            tracking_uri=trackingDir,
            registry_uri=registryDir,
        )

metrics = client.get_metric_history(runID, metricKey)

From the docs

get_metric_history(run_id, key)[source] Return a list of metric objects corresponding to all values logged for a given metric.

Parameters run_id – Unique identifier for run

key – Metric name within the run

Returns A list of mlflow.entities.Metric entities if logged, else empty list

from mlflow.tracking import MlflowClient

def print_metric_info(history):
    for m in history:
        print("name: {}".format(m.key))
        print("value: {}".format(m.value))
        print("step: {}".format(m.step))
        print("timestamp: {}".format(m.timestamp))
        print("--")

# Create a run under the default experiment (whose id is "0"). Since this is low-level
# CRUD operation, the method will create a run. To end the run, you'll have
# to explicitly end it. 
client = MlflowClient() 
experiment_id = "0" 
run = client.create_run(experiment_id) 
print("run_id:{}".format(run.info.run_id))
print("--")

# Log couple of metrics, update their initial value, and fetch each
# logged metrics' history. 
for k, v in [("m1", 1.5), ("m2", 2.5)]:
    client.log_metric(run.info.run_id, k, v, step=0)
    client.log_metric(run.info.run_id, k, v + 1, step=1)
    print_metric_info(client.get_metric_history(run.info.run_id, k))
client.set_terminated(run.info.run_id)

score 3 · Answer 2 · answered Mar 18 '20 at 22:26

Yes, you can use MlffowClient APIs to get experiment and run info. It returns MLflow data structures as dictionaries and you iterate over it to extract what you need in your listcomp. Here's an example:


def print_experiment_details(experiment_id, run_id):
    """
    Method to print experiment run info and a specific run details
    :param experiment_id: MLflow experiment ID
    :param run_id: MLflow run ID within an experiment
    :return: none
    """
    print("Finished MLflow Run with run_id {} and experiment_id {}".format(run_id, experiment_id))

    # Use MlflowClient API to list experiments and run info
    client = MlflowClient()
    print("=" * 80)
    # Get a list of all experiments
    print("List of all Experiments")
    print("=" * 80)
    [print(pprint.pprint(dict(exp), indent=4))
     for exp in client.list_experiments()]
    print("=" * 80)
    print(f"List Run info for run_id={run_id}")
    print(pprint.pprint(dict(mlflow.get_run(run_id))))

This outputs:

Running local model registry=sqlite:///mlruns.db
Finished MLflow Run with run_id 3f3b827dd6814649a2f84ebae09b26c6 and experiment_id 0
================================================================================
List of all Experiments
================================================================================
{   'artifact_location': './mlruns/0',
    'experiment_id': '0',
    'lifecycle_stage': 'active',
    'name': 'ODSC_TUTORIALS',
    'tags': {   'mlflow.note.content': 'This is experiment for getting started '
                                       'with MLflow ...'}}
None
================================================================================
List Run info for run_id=3f3b827dd6814649a2f84ebae09b26c6
{'data': <RunData: metrics={'metric_1': 0.9236238251076615,
 'metric_2': 1.6732389715754346,
 'metric_3': 2.249979396736294}, params={'n_estimators': '3', 'random_state': '42'}, tags={'mlflow.log-model.history': '[{"run_id": "3f3b827dd6814649a2f84ebae09b26c6", '
                             '"artifact_path": "sklearn-model", '
                             '"utc_time_created": "2020-03-18 '
                             '22:25:33.083332", "flavors": {"python_function": '
                             '{"loader_module": "mlflow.sklearn", '
                             '"python_version": "3.7.5", "data": "model.pkl", '
                             '"env": "conda.yaml"}, "sklearn": '
                             '{"pickled_model": "model.pkl", '
                             '"sklearn_version": "0.22.2.post1", '
                             '"serialization_format": "cloudpickle"}}}]',
 'mlflow.note.content': 'This Run is for getting started with MLflow ...',
 'mlflow.runName': 'LOCAL_REGISTRY',
 'mlflow.source.git.commit': '0a3c6a3739deab77631318eca7fb9690b6dbad66',
 'mlflow.source.name': '/Users/julesdamji/gits/tutorials/mlflow/labs/00_get_started.py',
 'mlflow.source.type': 'LOCAL',
 'mlflow.user': 'julesdamji'}>,
 'info': <RunInfo: artifact_uri='./mlruns/0/3f3b827dd6814649a2f84ebae09b26c6/artifacts', end_time=1584570333841, experiment_id='0', lifecycle_stage='active', run_id='3f3b827dd6814649a2f84ebae09b26c6', run_uuid='3f3b827dd6814649a2f84ebae09b26c6', start_time=1584570332914, status='FINISHED', user_id='julesdamji'>}

You can get the full code here

Hope that helps.

score 2 · Answer 3 · answered Mar 14 '22 at 11:40

With MLflow client (MlflowClient) you can easily get all or selected params and metrics using get_run(id).data:

# create an instance of the MLflowClient,
# connected to the tracking_server_url
mlflow_client = mlflow.tracking.MlflowClient(
    tracking_uri=tracking_server_url)

# list all experiment at this Tracking server
# mlflow_client.list_experiments()

# extract params/metrics data for run `test_run_id` in a single dict 
run_data_dict = mlflow_client.get_run(test_run_id).data.to_dictionary()

# list all params and metrics for this run (test_run_id)
# pprint(run_data_dict)

print(run_data_dict['params']['algo'])
print(run_data_dict['metrics']['RMSE'])

score 1 · Answer 4 · answered Mar 10 '20 at 14:03

I solve it brutally: I read the raw file of specific [metric_name] with specific [run_id].

path = f'./mlruns/0/[run_id]/metrics/[metric_name]'
with open(path) as f:
    content = f.readlines()
metrics_for_step = [float(x.split(' ')[1]) for x in content]

MLflow: how to read metrics or params from an existing run?

4 Answers4