Configparser Occasionally Returns Empty Result

Question

I am reading configurations from an ini file using Python configparser. 95% of the time, it works perfectly. Without any changes to the ini file (e.g., in the middle of a regression test), configparser will start returning empty results, and then of course have key errors as it searches for nested configs.

This suggests it is not a problem with the ini file, since it is not changing, and also because it happens occasionally. Once the problem happens, it continues to happen across all calls to config parser until i kill the program and re-run it.

I'm reading the config as such:

try:
    path = os.path.dirname(os.path.realpath(__file__))
    kml_ini = '/'.join([path, 'kml.ini'])
    config = configparser.ConfigParser()
    config.read(kml_ini)
    db_conn_returnable = config['database']['db_connection'].strip()
except Exception as e:
    print(e)        
    traceback.print_exc(file=sys.stdout)
    pprint.pprint(config.sections())
    pprint.pprint({section: dict(config.items(section)) for section in config.sections()})

I'm getting the error: File "/home/sahmed/anaconda3/envs/kml/lib/python3.6/configparser.py", line 959, in getitem KeyError: 'database'

My ini file looks as such:

[api]
server_debug=False
log_level=info
n_workers=10

[database]
db_connection=http://1.1.1.1:9191
user=admin
pass=whateverman

[cluster]
default_workers=3

My original thought was a thread issue since I've got 8 threads hitting this config file constantly (though it doesnt make sense...we're reading only) so I even put a fasteners block on the read, still no luck. What might cause such a situation?

totally unrelated, but you may want to use `os.path.join(path, "kml.ini")` instead. — bruno desthuilliers, Apr 16 '20 at 08:42
Note instead of kml_ini = '/'.join([path, 'kml.ini']) I also used a complete absolute path and still have the same failing. — CoderOfTheNight, Apr 16 '20 at 22:21
if you have the same issue with an hardcoded path then the cause is elsewhere indeed - but then you'll have to provide a proper [mcve] if you hope to get any useful answer. — bruno desthuilliers, Apr 17 '20 at 05:18

score 2 · Accepted Answer · answered Apr 16 '20 at 08:05

2

The only way I could replicate the problem with your provided code, was if I forcefully changed the working-directory during the code.

I put together this to test the problem:

from time import sleep
import configparser
import os

def readINI():
    try:
        path = os.path.dirname(os.path.realpath(__file__))
        ini = '/'.join([path, "test.ini"])
        config = configparser.ConfigParser()
        config.read(ini)
        string = config["things"]["abc"]
        print(string)
    except Exception as e:
        print("Error: {}".format(e))

count = 0
while True:
    count +=1
    readINI()
    sleep(1)
    if count % 3 == 0:
        os.chdir("..")
    elif count > 1 and count % 3 == 1:
        os.chdir("./TestFolder")

test.ini is very simple, it just contais two rows:

[things]
abc=123

I have both files in the same folder, and move back and forth between the "root" folder and TestFolder which contains the files. If you raise the exception instead of just printing it, then you will get the same error message.

Please verify that you're not accidentally changing the working directory during code-execution, and see if this indeed is the reason for the problems.

answered Apr 16 '20 at 08:05

Hampus Larsson

3,050
2
14
20

1

I confirm the observation. As a general rule, avoid using chdir in your code. – bruno desthuilliers Apr 16 '20 at 08:50
Thank you for the lead, let me check the theory by looking at the line below and seeing if the path changes. ini = '/'.join([path, "test.ini"]) – CoderOfTheNight Apr 16 '20 at 14:25
I tried to test this theory by printing the full path being read (thinking that perhaps the client program is changing directory causing this problem. However, the full path from which the ini file is read is consistently the same file throughout -- both while it worked and after it stopped working. So I dont think this is the cause in my case. – CoderOfTheNight Apr 16 '20 at 17:16
1

I'm marking this answer correct, because this is the only *direct* way for the ConfigParser to directly fail. Note in my case the cause was actually something completely different -- the Python logger was pulling too many file descriptors and exhausting the max limit -- so the overall Python process was not able to open the config file. – CoderOfTheNight Apr 22 '20 at 19:20

score 1 · Answer 2 · answered Apr 22 '20 at 19:23

I've resolved the issue. The Python logger was pulling too many file descriptors exhausting the max limit (there was a getLogger() inside a frequently called function, not globally) -- so the overall Python process was not able to open the config file.

The general answer here would be that ConfigParser is, of course, exposed to overall system issues where it may not be able to read a file it could read earlier. Not because of the file itself, but because of overall system limitations being reached.

Configparser Occasionally Returns Empty Result

2 Answers2