is there a way to modify log directory in hydra?

Question

I am trying to modify log dir in hydra.
In default, output dir in hydra is like below.

outputs
└── 2022-11-12
    ├── 18-17-28
    │   └── train.log
    ├── 18-18-37
    │   └── train.log
    └── 18-19-01
        └── train.log

However, this folder structure is not suitable to find log by experiment name. tree structure I want is like below.

outputs
└── expname-2022-11-12-18-17-28-18-00-00
      └── train.log
└── expname2-2022-11-12-18-17-28-18-10-00
      └── train.log

which expname, expname2 is provided like this. any solutions?

python train.py expname=expname

plus. log file does not contain logs starting like this: [2022-11-12 17:51:53,936][HYDRA] which I want to log with callback
Thanks in advance

Answer 1

You can do this by overriding the hydra.run.dir setting Here is a working sample:

# config.yaml
hydra:
  run:
    dir: outputs/${expname}-${now:%Y-%m-%d-%H-%M-%S}

expname: ???

# script.py
import hydra

@hydra.main(config_name="config", config_path=".", version_base="1.2")
def main(cfg):
    ...

if __name__ == "__main__":
    main()

Running the script:

$ python script.py expname=my_experiment
$ tree
.
├── config.yaml
├── outputs
│   └── my_experiment-2022-11-16-17-33-28
│       └── script.log
└── script.py

Here are some references:

• info on hydra.run.dir
• info on the ${now:} resolver, which wraps strftime.