1

Suppose I have a server where many users run different experiments, possibly with different Trains Servers.

I know about the TRAINS_CONFIG_FILE environment variable, but I wonder if this can be made more flexible in one of the following ways:

  1. Specifying the Trains config file dynamically, i.e. during runtime of the training script?
  2. Storing a config file in each of the training repos and specifying its path relatively to the running script path (instead of relatively to ~/)?
DalyaG
  • 2,979
  • 2
  • 16
  • 19
Michael Litvin
  • 3,976
  • 1
  • 34
  • 40

1 Answers1

1

Disclaimer: I'm a member of Allegro Trains team

  1. Loading of the configuration is done at import time. This means that if you set the os environment before importing the package, you should be fine:
os.environ['TRAINS_CONFIG_FILE']='~/repo/trains.conf'
from trains import Task
  1. The configuration file is loaded based on the current working directory, this means that if you have os.environ['TRAINS_CONFIG_FILE']='trains.conf' the trains.conf file will be loaded from the running directory at the time the import happens (which usually is the folder where your script is executed from). This means you can have it as part of the repository, and always set the TRAINS_CONFIG_FILE to point to it.

A few notes:

  • What is the use case for different configuration files ?
  • Notice that when running with trains-agent , this method will override the configuration that the trains-agent passes to the code.
Martin.B
  • 599
  • 3
  • 9
  • Cool. Regarding (2), the docs say "After installing and configuring, you can access your configuration file at ~/trains.conf", so it first looks in the current working directory and then in the home dir? – Michael Litvin Jun 24 '20 at 15:04
  • The usecase for different configs is sharing a multi-GPU machine between projects that don't share the same Trains Server. – Michael Litvin Jun 24 '20 at 15:05
  • "so it first looks in the current working directory and then in the home dir?" What I mean is if you do not provide a full path (or ~ as home) it will look for a file relative to the current working directory. It will NOT first look for it there, only if you set your TRAINS_CONFIG_FILE it will. make sense ? – Martin.B Jun 24 '20 at 15:57
  • "The usecase for different configs is sharing a multi-GPU machine between projects that don't share the same Trains Server. " Is this like a multi-tenant use case ? where you want to have different "companies" share resources but not to mix credentials ? – Martin.B Jun 24 '20 at 15:59
  • I think I understand - default is `~/trains.config`, but if `TRAINS_CONFIG_FILE` is set then it will look relative to the current workdir. – Michael Litvin Jun 24 '20 at 19:17
  • Yes, it's a multi-tenant scenario similar to what you described. – Michael Litvin Jun 24 '20 at 19:19
  • Yes this exactly right :) (multi-tenant ... Nice J) – Martin.B Jun 24 '20 at 23:12