How to change and set up dvc config file so can manage the weights file?

Question

Im currently managing the weight file by dvc. This is the dvc file:

outs:
- md5: b2c80e73090cae013eef778308faf8fc
  size: 202055043
  path: checkpoint_1.pth.tar

So I would like to create a dvc config file so that when it's necessary to change the weight file then I change the dvc config file. My current config yaml file for the weights is like

...
robot_perception_detector:
    ros__parameters:
      weights: /home/ros2/foxy/src/robot_pkg/data/model_weights/checkpoint_1.pth.tar
 ....

Any help and advice how this config file should looks like so that works in both cases?

score 1 · Accepted Answer · edited Apr 27 '23 at 18:58

1

You are probably looking for Templating. The very first example templates an output file:

params.yaml:

models:
  us:
    threshold: 10
    filename: 'model-us.hdf5'

dvc.yaml:

stages:
  build-us:
    cmd: >-
      python train.py
      --thresh ${models.us.threshold}
      --out ${models.us.filename}
    outs:
      - ${models.us.filename}:
          cache: true

edited Apr 27 '23 at 18:58

TylerH

20,799
66
75
101

answered Dec 01 '22 at 17:06

Shcheklein

5,979
7
44
53

hmm, I'm not familiar with ROS, but if it can run Python and DVC it should be fine - DVC is not platform of framework specific. It was designed to be agnostic. In this sense to you think about it as a Makefile. – Shcheklein Dec 02 '22 at 17:50
@Bob9710 yes, if you experience some ROS + DVC specific problems I would describe them either here or in a separate question. If this one answers your specific question on templating path, I would keep it. Your call. – Shcheklein Dec 04 '22 at 02:38
So in my case the dvc.yaml file will be: stages: build-us: cmd: >- python train.py --thresh ${robot_perception_detector. ros__parameters.n_class} --out ${robot_perception_detector. ros__parameters. weights} outs: - ${robot_perception_detector. ros__parameters. weights}: cache: true – Bob9710 Dec 05 '22 at 04:59
You can either change it in the `params.yaml` file or using `-S` when you run `dvc exp run`. – Shcheklein Dec 05 '22 at 18:31
`build-us` - no need to use exactly the same stage name (it's just an example). Also `--thresh` was a specific param for that example. Besides that it looks about right to me. – Shcheklein Dec 05 '22 at 18:33

How to change and set up dvc config file so can manage the weights file?

1 Answers1