Adding and reading a config.ini file inside python package

Question

I am writing my first python package which I want to upload on PyPI. I structured my code based on this blog post.

I want to store user setting in a config.ini file. Read it once(every time the package is run) in separate python module in same package and save user setting in global variables of that module. Later import those in other modules.

To recreate the error I just edited few lines of code, in the template described in the blog post. (Please refer to it since it would take too much typing to recreate entire thing here in question.)

The only difference is that my stuff.py reads from config file like this:

from ConfigParser import SafeConfigParser

config = SafeConfigParser()
config.read('config.ini')

TEST_KEY = config.get('main', 'test_key')

Here are the contents of config.ini (placed in same dir as stuff.py):

[main]
test_key=value

And my bootstrap.py just imports and print the TEST_KEY

from .stuff import TEST_KEY

def main():
    print(TEST_KEY)

But on executing the package, the import fails give this error

Traceback (most recent call last):
    File "D:\Coding\bootstrap\bootstrap-runner.py", line 8, in <module>
    from bootstrap.bootstrap import main
    File "D:\Coding\bootstrap\bootstrap\bootstrap.py", line 11, in <module>
    from .stuff import TEST_KEY
    File "D:\Coding\bootstrap\bootstrap\stuff.py", line 14, in <module>
    TEST_KEY = config.get('main', 'test_key')
    File "C:\Python27\Lib\ConfigParser.py", line 607, in get
    raise NoSectionError(section)
ConfigParser.NoSectionError: No section: 'main'

Import keeps giving ConfigParser.NoSectionError, but if you build/run only stuff.py(I use sublime3), the module gives no errors and printing TEST_KEY gives value as output.

Also, this method of import does work when I just use 3 files(config, stuff, main) in a dir and just execute the main as a script. But there I had to import it like this

from stuff import TEST_KEY

I'm just using the explicit relative imports as described in that post but don't have enough understanding of those. I guess the error is due to project structure and import, since running stuff.py as standalone script raises no ConfigParser.NoSectionError.

Other method to read the config file once and then use data in other modules will be really helpful as well.

Answer 1

There are two aspects to this question. First is the weird behavior of ConfigParser. When ConfigParser is unable to locate the .ini file; it never gives, for some annoying reason, an IOError or an error which indicates that it is unable to read the file.

In my case it keeps giving ConfigParser.NoSectionError when the section is clearly present. When I caught the ConfigParser.NoSectionError error it gave an ImportError! But it never tells you that it is simply unable to read the file.

Second is how to safely read the data files that are included in your package. The only way I found to do this was to use the __file__ parameter. This is how you would safely read the config.ini in the above question, for Python27 and Python3:

import os

try:
    # >3.2
    from configparser import ConfigParser
except ImportError:
    # python27
    # Refer to the older SafeConfigParser as ConfigParser
    from ConfigParser import SafeConfigParser as ConfigParser

config = ConfigParser()

# get the path to config.ini
config_path = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'config.ini')

# check if the path is to a valid file
if not os.path.isfile(config_path):
    raise BadConfigError # not a standard python exception

config.read(config_path)

TEST_KEY = config.get('main', 'test_key') # value

This relies on the fact that config.ini is located inside our package bootstrap and is expected to be shipped with it.

The important bit is how you get the config_path:

config_path = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'config.ini')

__file__ refers to the location of current script that is being executed. In my question that means the location of stuff.py, that is inside the bootstrap folder, and so is config.ini.

The above line of code then means; get the absolute path to stuff.py; from that get path to the directory containing it; and join that with config.ini (since it is in same directory) to give absolute path to the config.ini. Then you can proceed to read it and raise an exception just in case.

This will work even when you release your package on pip and a user installs it from there.

As a bonus, and digressing from the question a bit, if you are releasing your package on pip with data files inside your package, you must tell setuptools to include them inside you package when you build sdist and bdists. So to include the config.ini in above package add the following lines to setup class call in setup.py:

include_package_data = True,
package_data = {
    # If any package contains *.ini files, include them
    '': ['*.ini'],
},

But it still may not work in some cases eg. building wheels etc. So you also do the same in your MANIFEST.IN file:

include LICENSE
include bootstrap/*.ini

Answer 2

abhimanyuPathania : The issue is with path of config.ini in stuff.py. Change config.read('config.ini') to config.read('./bootstrap/config.ini') in stuff.py. I tried the solution. It works for me.

Enjoying Pythoning...