ClearML, how to query the best performing model for a specific project and metric

Question

I want to download the best performing model for a certain ClearlML project. I have the following content in my ClearML experiment platform:

According to: https://clear.ml/docs/latest/docs/clearml_sdk/model_sdk#querying-models I can get a list of models for a specific project:

model_list = Model.query_models(
    # Only models from `examples` project
    project_name='YOLOv5', 
    # Only models with input name
    model_name=None,
    # Only models with `demo` tag but without `TF` tag
    tags=['demo', '-TF'],
    # If `True`, only published models
    only_published=False,
    # If `True`, include archived models
    include_archived=True,
    # Maximum number of models returned
    max_results=5
)

print(model_list)

Which prints:

[<clearml.model.Model object at 0x7fefbaf22130>, <clearml.model.Model object at 0x7fefbaf22340>]

So I can run:

model_list[0].get_local_copy()

and get this specific model. But how do I download the best performing one for this project on a certain metric (in this case mAP_0.5:0.95 MAX)?

Mike B · Accepted Answer · 2022-09-07T12:23:52.007

I ended up doing the following:

try:
    import clearml
    from clearml import Dataset, Task, Model, OutputModel
    assert hasattr(clearml, '__version__')  # verify package import not local dir
except (ImportError, AssertionError):
    clearml = None

tasks = Task.get_tasks(project_name='YOLOv5', task_name='exp', task_filter={'status': ['completed']})

results = {}
best_task = None
for task in tasks:
    results[task.id] = task.get_last_scalar_metrics()['metrics']['mAP_0.5:0.95']['max']

best_model_task_id = max(results, key=results.get)
model_list = Task.get_task(best_model_task_id).get_models()
dest = model_list['output'][0].get_local_copy()
print('Saved model at:', dest)

ClearML, how to query the best performing model for a specific project and metric

1 Answers1