Managing configurations for different environments

Question

I've had a discussion with some people at work and we couldn't come to a conclusion.
We've faced a dilemma - how do you manage different configuration values for different environments?

We've come up with some options, but none of them seemed to satisfy us:
- Separate config files (i.e. config.test, config.prod etc.), and having a file pointing to the selected one (at ~/env for instance), or an environment variable pointing to it.
- Using a single DB to store all configurations (you query it with your environment and get the corresponding configuration values)
- Creating configuration files on deploy (Using CI/CD system like Atlassian Bamboo)

Which is the more widely used option? Is there a better way?
Should config file be kept in the git repository along with the rest of the code? Our systems are written in python (both 2.7 and 3)

score 15 · Answer 1 · answered May 30 '18 at 18:34

You can put all these configuration in single config file config.py.

class Base():
    DEBUG = False
    TESTING = False

class DevelopmentConfig(Base):
    DEBUG = True
    DEVELOPMENT = True
    DATABASE_URI = "mysql+mysqldb://root:root@localhost/demo"

class TestingConfig(Base):
    DEBUG = False
    TESTING = True
    DATABASE_URI = "mysql+mysqldb://root:root@test_server_host_name/demo_test"

class ProductionConfig(Base):
    DEBUG = False
    TESTING = False
    DATABASE_URI = "mysql+mysqldb://root:root@prod_host_name/demo_prod"

on the shell set environment variable like

APP_SETTINGS = config.DevelopmentConfig

In your main application app.py, load this environment variable (flask app as example)

from flask import Flask
import os

app = Flask(__name__)
app.config.from_object(os.environ['APP_SETTINGS'])

I hope this can help

score 1 · Answer 2 · answered Jun 13 '18 at 05:40

It's usually a bad idea to commit your configuration settings to source control, particularly when those settings include passwords or other secrets. I prefer using environment variables to pass those values to the program. The most flexible way I've found is to use the argparse module, and use the environment variables as the defaults. That way, you can override the environment variables on the command line. Be careful about putting passwords on the command line, though, because other users can probably see your command line arguments in the process list.

Here's an example that uses argparse and environment variables:

def parse_args(argv=None):
    parser = ArgumentParser(description='Watch the raw data folder for new runs.',
                            formatter_class=ArgumentDefaultsHelpFormatter)
    parser.add_argument(
        '--kive_server',
        default=os.environ.get('MICALL_KIVE_SERVER', 'http://localhost:8000'),
        help='server to send runs to')
    parser.add_argument(
        '--kive_user',
        default=os.environ.get('MICALL_KIVE_USER', 'kive'),
        help='user name for Kive server')
    parser.add_argument(
        '--kive_password',
        default=SUPPRESS,
        help='password for Kive server (default not shown)')

    args = parser.parse_args(argv)
    if not hasattr(args, 'kive_password'):
        args.kive_password = os.environ.get('MICALL_KIVE_PASSWORD', 'kive')
    return args

Setting those environment variables can be a bit confusing, particularly for system services. If you're using systemd, look at the service unit, and be careful to use EnvironmentFile instead of Environment for any secrets. Environment values can be viewed by any user with systemctl show.

I usually make the default values useful for a developer running on their workstation, so they can start development without changing any configuration.

Another option is to put the configuration settings in a settings.py file, and just be careful not to commit that file to source control. I have often committed a settings_template.py file that users can copy.

score 0 · Answer 3 · answered May 31 '18 at 17:28

One approach would be to write a "template" for each kind of configuration file, where the template contains mostly plain text, but with a few placeholders. Here is an example template configuration file, using the notation ${foo} to denote a placeholder.

serverName  = "${serverName}"
listenPort  = "${serverPort}"
logDir      = "/data/logs/${serverName}";
idleTimeout = "5 minutes";
workingDir  = "/tmp";

If you do that for all the configuration files used by your application, then you will probably find that performing a global-search-and-replace on the template configuration files with values for a relatively small number of placeholders will yield the ready-to-run configuration files for a particular deployment. If you are looking for an easy way to perform global search-and-replace on placeholders in template files and are comfortable with Java, then you might want to consider Apache Velocity. But I imagine it would be trivial to develop similar-ish functionality in Python.

score 0 · Accepted Answer · answered Jan 19 '19 at 08:01

We ended up using a method similar to this one. We had a base settings file, and environment specific files that simply imported everything from the base file base.py:

SAMPLE_CONFIG_VARIABLE = SAMPLE_CONFIG_VALUE

prod.py:

from base.py import *

So all we had to do when accessing values from the configuration is read an environment variable and creating the file corresponding to it.

Sairam Krish · Answer 5 · 2020-02-03T14:02:34.307

If we are using flask, then we could do environment specific config management like this :

-- project folder structure

config/
   default.py
   production.py
   development.py
instance/
  config.py
myapp/
  __init__.py

During app initialization.,

# app/__init__.py

app = Flask(__name__, instance_relative_config=True)

# Load the default configuration
app.config.from_object('config.default')

# Load the configuration from the instance folder
app.config.from_pyfile('config.py')

# Load the file specific to environment based on ENV environment variable
# Variables defined here will override those in the default configuration
app.config.from_object('config.'+os.getenv("ENV"))

While running the application:

# start.sh
# ENV should match the file name    
ENV=production python run.py

The above method is my preferred way. This method is based on best practices described here with few modifications like file name based on environment variable.

But there are other options

Managing configurations for different environments

5 Answers5

Linked