2

I am using PyTorch Lightning together with w&b and trying associate metrics with a finite set of configurations. In the LightningModule class I have defined the test_step as:

def test_step(self, batch, batch_idx):
  x, y_true, config_file =  batch
  y_pred = self.forward(x)
  accuracy = self.accuracy(y_pred, y_true)
  self.log("test/accuracy", accuracy)

Assuming (for simplicity) that the batch size is 1, this will log the accuracy for 1 sample and it will be displayed as a chart in the w&b dashboard.

I would like to associate this accuracy with some configuration of the experimental environment. This configuration might include BDP factor, bandwith delay, queue_size, location, etc. I don't want to plot the configurations I just want to be able to filter or group the accuracy by some configuration value.

The only solution I can come up with is to add these configurations as a querystring:

def test_step(self, batch, batch_idx):
  x, y_true, config_file =  batch
  # read values in config file
  # ...

  y_pred = self.forward(x)
  accuracy = self.accuracy(y_pred, y_true)
  self.log("test/BDP=2&delay=10ms&queue_size=10&topology=single/accuracy", accuracy)

Is there a better solution for this that integrates my desired functionality of being able to group and filter by values like BDP?

Kevin
  • 3,096
  • 2
  • 8
  • 37

1 Answers1

1

I work at W&B. You could log your config variables using wandb.config, like so:

wandb.config['my_variable'] = 123

And then you'll be able to filter your charts by whatever config you'd logged. Or am I missing something.

Possibly the save_hyperparameters call might even grab these config values automatically (from the WandbLogger docs here)

class LitModule(LightningModule):
    def __init__(self, *args, **kwarg):
        self.save_hyperparameters()
morganmcg
  • 460
  • 2
  • 5
  • Thanks, this seems to be the recommended way of doing this. However, I have multiple variables that are only valid for the test set. So I am training the model once and testing it on multiple test sets configured in some specific way. Is it possible to update the configuration between consecutive "test runs", or do I need to rerun the whole process? – Kevin Mar 04 '22 at 14:43
  • I posted a [similar question on the w&b community](https://community.wandb.ai/t/grouping-custom-metrics-by-configuration/1995/2) this week. – Kevin Mar 04 '22 at 14:44