ConfigParser and String interpolation with env variable

Question

it's a little bit I'm out of python syntax and I have a problem in reading a .ini file with interpolated values.

this is my ini file:

[DEFAULT]
home=$HOME
test_home=$home

[test]
test_1=$test_home/foo.csv
test_2=$test_home/bar.csv

Those lines

from ConfigParser import SafeConfigParser

parser = SafeConfigParser()
parser.read('config.ini')

print parser.get('test', 'test_1')

does output

$test_home/foo.csv

while I'm expecting

/Users/nkint/foo.csv

EDIT:

I supposed that the $ syntax was implicitly included in the so called string interpolation (referring to the manual):

On top of the core functionality, SafeConfigParser supports interpolation. This means values can contain format strings which refer to other values in the same section, or values in a special DEFAULT section.

But I'm wrong. How to handle this case?

score 40 · Accepted Answer · answered Oct 27 '14 at 13:06

40

First of all according to the documentation you should use %(test_home)s to interpolate test_home. Moreover the key are case insensitive and you can't use both HOME and home keys. Finally you can use SafeConfigParser(os.environ) to take in account of you environment.

from ConfigParser import SafeConfigParser
import os


parser = SafeConfigParser(os.environ)
parser.read('config.ini')

Where config.ini is

[DEFAULT]
test_home=%(HOME)s

[test]
test_1=%(test_home)s/foo.csv
test_2=%(test_home)s/bar.csv

answered Oct 27 '14 at 13:06

Michele d'Amico

22,111
8
69
76

1

I can't modify the ini syntax because other software already use it and it is with the `$` syntax – nkint Oct 27 '14 at 13:42
You should preprocess the file before use SafeConfigParse(). But you can just replace `$` syntax by `%()s` syntax. the real problem will be `home=$(HOME)` that become recursive. You could assume that the capital words came from environ and replace by something like __ENV__KEY after change the `sys.environ` dict too. I can't do it now... but you can tray by yourself – Michele d'Amico Oct 27 '14 at 14:03
1

The one thing that got me about this implementation is if there are '%' marks in environment variables themselves, then the parser will throw an error. I had to write a function that would filter out those keys/values. Otherwise, I like this impl. – Chris Mar 26 '15 at 20:08
If you are working with other kinds of variables along with path variable, then use ConfigParser instead of SafeConfigParser else it will give an error %%' must be followed by '%%' or '(', found: %r" % (rest,)) – Deepak Sharma Mar 29 '17 at 13:40
home=${HOME} is the proper way to interpolate the environment variable – MrE Nov 12 '17 at 07:23
2

I like the simplicity of this answer, but was concerned how it casually imports all environment variables instead of just the relevant ones, so I passed on only the desired subset to `ConfigParser`: `{k: v for k, v in os.environ.items() if k in ('MY_ENV_1', 'MY_ENV_2')}`. – Taylor D. Edmiston Sep 09 '19 at 20:08
1

It should be noted that `SafeConfigParser` only exists in python 2. It is replaced by `ConfigParser` in python 3. – iron9 Dec 17 '20 at 12:40

Alex Markov · Answer 2 · 2020-06-19T07:53:27.430

12

You can write custom interpolation in case of Python 3:

import configparser
import os


class EnvInterpolation(configparser.BasicInterpolation):
    """Interpolation which expands environment variables in values."""

    def before_get(self, parser, section, option, value, defaults):
        value = super().before_get(parser, section, option, value, defaults)
        return os.path.expandvars(value)


cfg = """
[section1]
key = value
my_path = $PATH
"""

config = configparser.ConfigParser(interpolation=EnvInterpolation())
config.read_string(cfg)
print(config['section1']['my_path'])

edited Jun 19 '20 at 07:53

answered Mar 28 '18 at 08:32

Alex Markov

301
3
7

Hey @Alex, I want to fetch Environment variables and values from other sections also. How can I do this? Extended Interpolation doesn't look straight forward. – srand9 Aug 03 '18 at 11:17
2

please add missing call to `super().before_get(...)` – Andrei Pozolotin May 16 '19 at 16:44
@srand9, I have fixed the example, now you can use %-notation to reference values from other sections of the same config. See, [docs](https://docs.python.org/3.8/library/configparser.html#configparser.BasicInterpolation) for more details – Alex Markov Jun 19 '20 at 07:58
I've kind of jeopardized your answer to post my own answer addressing @srand9 concern (which is mine too). I hope you don't mind :) – Vser Jun 21 '21 at 13:37

gagnonlg · Answer 3 · 2016-05-24T13:58:53.897

3

If you want to expand some environment variables, you can do so using os.path.expandvars before parsing a StringIO stream:

import ConfigParser
import os
import StringIO

with open('config.ini', 'r') as cfg_file:
    cfg_txt = os.path.expandvars(cfg_file.read())

config = ConfigParser.ConfigParser()
config.readfp(StringIO.StringIO(cfg_txt))

edited May 24 '16 at 13:58

answered May 23 '16 at 19:00

gagnonlg

51
3

score 3 · Answer 4 · answered Nov 12 '17 at 07:19

3

the trick for proper variable substitution from environment is to use the ${} syntax for the environment variables:

[DEFAULT]
test_home=${HOME}

[test]
test_1=%(test_home)s/foo.csv
test_2=%(test_home)s/bar.csv

answered Nov 12 '17 at 07:19

MrE

19,584
12
87
105

In this case you are forced to use the basic ConfigParser interpolation, right? – Daniel Jan 23 '23 at 12:49

Mauro Baraldi · Answer 5 · 2019-07-26T17:10:59.063

ConfigParser.get values are strings, even if you set values as integer or True. But ConfigParser has getint, getfloat and getboolean.

settings.ini

[default]
home=/home/user/app
tmp=%(home)s/tmp
log=%(home)s/log
sleep=10
debug=True

config reader

>>> from ConfigParser import SafeConfigParser
>>> parser = SafeConfigParser()
>>> parser.read('/home/user/app/settings.ini')
>>> parser.get('defaut', 'home')
'/home/user/app'
>>> parser.get('defaut', 'tmp')
'/home/user/app/tmp'
>>> parser.getint('defaut', 'sleep')
10
>>> parser.getboolean('defaut', 'debug')
True

Edit

Indeed you could get name values as environ var if you initialize SafeConfigParser with os.environ. Thanks for the Michele's answer.

I can't modify the ini syntax because other software already use it and it is with the $ syntax — nkint, Oct 27 '14 at 14:47

score 2 · Answer 6 · answered Jun 21 '21 at 13:35

Based on @alex-markov answer (and code) and @srand9 comment, the following solution works with environment variables and cross-section references.

Note that the interpolation is now based on ExtendedInterpolation to allow cross-sections references and on before_read instead of before_get.

#!/usr/bin/env python3
import configparser
import os


class EnvInterpolation(configparser.ExtendedInterpolation):
    """Interpolation which expands environment variables in values."""

    def before_read(self, parser, section, option, value):
        value = super().before_read(parser, section, option, value)
        return os.path.expandvars(value)


cfg = """
[paths]
foo : ${HOME}
[section1]
key = value
my_path = ${paths:foo}/path
"""

config = configparser.ConfigParser(interpolation=EnvInterpolation())
config.read_string(cfg)
print(config['section1']['my_path'])

This works until you need to iterate over multiple items in a section, for k, v in config.items() it iterates and returns every environment variable set instead of the values in the ini — Brent, Oct 05 '22 at 13:01

score 1 · Answer 7 · edited Apr 19 '19 at 15:39

Quite late, but maybe it can help someone else looking for the same answers that I had recently. Also, one of the comments was how to fetch Environment variables and values from other sections. Here is how I deal with both converting environment variables and multi-section tags when reading in from an INI file.

INI FILE:

[PKG]
# <VARIABLE_NAME>=<VAR/PATH>
PKG_TAG = Q1_RC1

[DELIVERY_DIRS]
# <DIR_VARIABLE>=<PATH>
NEW_DELIVERY_DIR=${DEL_PATH}\ProjectName_${PKG:PKG_TAG}_DELIVERY

Python Class that uses the ExtendedInterpolation so that you can use the ${PKG:PKG_TAG} type formatting. I add the ability to convert the windows environment vars when I read in INI to a string using the builtin os.path.expandvars() function such as ${DEL_PATH} above.

import os
from configparser import ConfigParser, ExtendedInterpolation

class ConfigParser(object):

    def __init__(self):
        """
        initialize the file parser with
        ExtendedInterpolation to use ${Section:option} format
        [Section]
        option=variable
        """
        self.config_parser = ConfigParser(interpolation=ExtendedInterpolation())

    def read_ini_file(self, file='./config.ini'):
        """
        Parses in the passed in INI file and converts any Windows environ vars.

        :param file: INI file to parse
        :return: void
        """
        # Expands Windows environment variable paths
        with open(file, 'r') as cfg_file:
            cfg_txt = os.path.expandvars(cfg_file.read())

        # Parses the expanded config string
        self.config_parser.read_string(cfg_txt)

    def get_config_items_by_section(self, section):
        """
        Retrieves the configurations for a particular section

        :param section: INI file section
        :return: a list of name, value pairs for the options in the section
        """
        return self.config_parser.items(section)

    def get_config_val(self, section, option):
        """
        Get an option value for the named section.

        :param section: INI section
        :param option: option tag for desired value
        :return: Value of option tag
        """
        return self.config_parser.get(section, option)

    @staticmethod
    def get_date():
        """
        Sets up a date formatted string.

        :return: Date string
        """
        return datetime.now().strftime("%Y%b%d")

    def prepend_date_to_var(self, sect, option):
        """
        Function that allows the ability to prepend a
        date to a section variable.

        :param sect: INI section to look for variable
        :param option: INI search variable under INI section
        :return: Void - Date is prepended to variable string in INI
        """
        if self.config_parser.get(sect, option):
            var = self.config_parser.get(sect, option)
            var_with_date = var + '_' + self.get_date()
            self.config_parser.set(sect, option, var_with_date)

score 1 · Answer 8 · answered Aug 06 '22 at 06:29

Below is a simple solution that

Can use default value if no environment variable is provided
Overrides variables with environment variables (if found)
needs no custom interpolation implementation

Example: my_config.ini

[DEFAULT]
HOST=http://www.example.com
CONTEXT=${HOST}/auth/
token_url=${CONTEXT}/oauth2/token

ConfigParser:

import os
import configparser

config = configparser.ConfigParser(interpolation=configparser.ExtendedInterpolation())
ini_file = os.path.join(os.path.dirname(__file__), 'my_config.ini')
    
# replace variables with environment variables(if exists) before loading ini file
with open(ini_file, 'r') as cfg_file:
    cfg_env_txt = os.path.expandvars(cfg_file.read())

config.read_string(cfg_env_txt)

print(config['DEFAULT']['token_url'])

Output:

If no environtment variable $HOST or $CONTEXT is present this config will take the default value
user can override the default value by creating $HOST, $CONTEXT environment variable
works well with docker container

score 0 · Answer 9 · answered Jan 02 '19 at 12:29

It seems in the last version 3.5.0, ConfigParser was not reading the env variables, so I end up providing a custom Interpolation based on the BasicInterpolation one.

class EnvInterpolation(BasicInterpolation):
    """Interpolation as implemented in the classic ConfigParser,
    plus it checks if the variable is provided as an environment one in uppercase.
    """

    def _interpolate_some(self, parser, option, accum, rest, section, map,
                          depth):
        rawval = parser.get(section, option, raw=True, fallback=rest)
        if depth > MAX_INTERPOLATION_DEPTH:
            raise InterpolationDepthError(option, section, rawval)
        while rest:
            p = rest.find("%")
            if p < 0:
                accum.append(rest)
                return
            if p > 0:
                accum.append(rest[:p])
                rest = rest[p:]
            # p is no longer used
            c = rest[1:2]
            if c == "%":
                accum.append("%")
                rest = rest[2:]
            elif c == "(":
                m = self._KEYCRE.match(rest)
                if m is None:
                    raise InterpolationSyntaxError(option, section,
                                                   "bad interpolation variable reference %r" % rest)
                var = parser.optionxform(m.group(1))
                rest = rest[m.end():]
                try:
                    v = os.environ.get(var.upper())
                    if v is None:
                        v = map[var]
                except KeyError:
                    raise InterpolationMissingOptionError(option, section, rawval, var) from None
                if "%" in v:
                    self._interpolate_some(parser, option, accum, v,
                                           section, map, depth + 1)
                else:
                    accum.append(v)
            else:
                raise InterpolationSyntaxError(
                    option, section,
                    "'%%' must be followed by '%%' or '(', "
                    "found: %r" % (rest,))

The difference between the BasicInterpolation and the EnvInterpolation is in:

   v = os.environ.get(var.upper())
   if v is None:
       v = map[var]

where I'm trying to find the var in the enviornment before checking in the map.

ConfigParser and String interpolation with env variable

EDIT:

9 Answers9

Linked

Related