configparser loading config files from zip

Question

I am creating a program that loads and runs python scripts from a compressed file. Along with those python scripts, I have a config file that I previously used configparser to load info from in an uncompressed version of the program.

Is it possible to directly read config files in zip files directly with configparser? or do I have to unzip it into a temp folder and load it from there?

I have tried directly giving the path:

>>> sysconf = configparser.ConfigParser()
>>> sysconf.read_file("compressed.zip/config_data.conf")

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.4/configparser.py", line 691, in read_file
    self._read(f, source)
  File "/usr/local/lib/python3.4/configparser.py", line 1058, in _read
    raise MissingSectionHeaderError(fpname, lineno, line)
configparser.MissingSectionHeaderError: File contains no section headers.
file: '<???>', line: 1

Didn't work. no surprises there.

Then I tried using zipfile

 >>> zf = zipfile.ZipFile("compressed.zip")
 >>> data = zf.read("config_data.conf")
 >>> sysconf = configparser.ConfigParser()
 >>> sysconf.read_file(data)

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.4/configparser.py", line 691, in read_file
    self._read(f, source)
  File "/usr/local/lib/python3.4/configparser.py", line 1009, in _read
    if line.strip().startswith(prefix):
AttributeError: 'int' object has no attribute 'strip'

and found that it didn't work either.

so I've resorted to creating a temp folder, uncompressing to it, and reading the conf file there. I would really like to avoid this if possible as the conf files are the only limiting factor. I can (and am) loading the python modules from the zip file just fine at this point.

I can get the raw text of the file if there's a way to pass that directly to configparser, but searching the docs I came up empty handed.

Update: I tried using stringIO as a file object, and it seems to work somewhat. configparser doesn't reject it, but it doesn't like it either.

>>> zf = zipfile.ZipFile("compressed.zip")
>>> data = zf.read(config_data.conf)
>>> confdata = io.StringIO(str(data))
>>> sysconf = configparser.ConfigParser()
>>> sysconf.readfp(confdata)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.4/configparser.py", line 736, in readfp
    self.read_file(fp, source=filename)
  File "/usr/local/lib/python3.4/configparser.py", line 691, in read_file
    self._read(f, source)
  File "/usr/local/lib/python3.4/configparser.py", line 1058, in _read
    raise MissingSectionHeaderError(fpname, lineno, line)
configparser.MissingSectionHeaderError: File contains no section headers.
file: '<???>', line: 1
(continues to spit out the entire contents of the file)

If I use read_file instead, it doesn't error out, but doesn't load anything either.

>>> zf = zipfile.ZipFile("compressed.zip")
>>> data = zf.read(config_data.conf)
>>> confdata = io.StringIO(str(data))
>>> sysconf = configparser.ConfigParser()
>>> sysconf.read_file(confdata)
>>> sysconf.items("General") #(this is the main section in the file)
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/configparser.py", line 824, in items
    d.update(self._sections[section])
KeyError: 'General'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.4/configparser.py", line 827, in items
    raise NoSectionError(section)
configparser.NoSectionError: No section: 'General'

For your most recent edit, replace `str(data)` with `data.decode()`. — Robᵩ, Jul 13 '15 at 20:32

Robᵩ · Accepted Answer · 2015-07-13T20:29:13.167

can get the raw text of the file if there's a way to pass that directly to configparser

Try configparser.ConfigParser.read_string

When coupled with an appropriate ZIP file, this code works for me:

import zipfile
import configparser

zf = zipfile.ZipFile("compressed.zip")
zf_config = zf.open("config_data.conf", "rU")
zf_config_data = zf_config.read().decode('ascii')

config = configparser.ConfigParser()
config.read_string(zf_config_data)
assert config['today']['lunch']=='cheeseburger'

Upon reflection, the following might be more appropriate:

import zipfile
import configparser
import io

zf = zipfile.ZipFile("compressed.zip")
zf_config = zf.open("config_data.conf", "rU")
zf_config = io.TextIOWrapper(zf_config)

config = configparser.ConfigParser()
config.read_file(zf_config)
assert config['today']['lunch']=='cheeseburger'

jferard · Answer 2 · 2021-12-23T09:38:29.970

1

As written in comments, @matthewatabet answer won't work with Python 3.4 (and newer vesions). It's because ZipFile.open now returns a "bytes-like" object and not a "file-like" object anymore. You can use:

codecs.getreader("utf-8")(config_file)

To convert the config_file bytes-like object into a file-like object using the UTF-8 encoding. The code is now:

import zipfile, configparser, codecs

# Python >= 3.4
with zipfile.ZipFile("compressed.zip") as zf:
    config_file = zf.open("config_data.conf") # binary mode
    sysconfig = configparser.ConfigParser()
    sysconfig.read_file(codecs.getreader("utf-8")(config_file))

That seems more satisfactory than creating a string, but I don't know if it's more efficient...

EDIT Since Python 3.9, the zipfile module provides a zipfile.Path.open method that can handle text and binary modes. Default is text mode. The following code works fine:

# Python >= 3.9
with zipfile.ZipFile("compressed.zip") as zf:
    zip_path = zipfile.Path(zf)
    config_path = zip_path / "config_data.conf"
    config_file = config_path.open() # text mode
    sysconfig = configparser.ConfigParser()
    sysconfig.read_file(config_file)

edited Dec 23 '21 at 09:38

answered Jan 16 '19 at 21:37

jferard

7,835
2
22
35

Works with Python 3.8 but does not work on 3.9 and 3.10. `TypeError: can't concat str to bytes`. – Andrei Smeltsov Dec 22 '21 at 12:01
@AndrewSmeltzov Just tested a minimal example with 3.9 and it worked fine. Could you give the full stack trace? – jferard Dec 23 '21 at 08:07
My bad, yours works fine. Turns out my example is a little different. I use ZipFile.Path.open() instead of ZipFile.open(). Can't really spot the problem as the documentation states that one calls the other: https://docs.python.org/3.10/library/zipfile.html#zipfile.Path.open I use docker image python:3.9-alpine if that matters. The funny thing is it works on 3.8. Here's code and trace https://godbolt.org/z/xaTvdvn56 – Andrei Smeltsov Dec 23 '21 at 09:13
1

@AndrewSmeltzov Thanks for the feedback. From the doc: `ZipFile.open`: "Access a member of the archive as a binary file-like object." `zipfile.Path.open`: "Changed in version 3.9: Added support for text and binary modes for open." The `codecs` trick is now unnecessary since default mode is now text. I'll update my answer. – jferard Dec 23 '21 at 09:31
Right, how could I have missed it. No need for `codecs` manipulation in that case. Thanks! – Andrei Smeltsov Dec 23 '21 at 09:42

matthewatabet · Answer 3 · 2015-07-13T20:20:03.050

0

ZipFile not only supports read but also open, which returns a file-like object. So, you could do something like this:

zf = zipfile.ZipFile("compressed.zip")
config_file = zf.open("config_data.conf")
sysconfig = configparser.ConfigParser()
sysconfig.readfp(config_file)

edited Jul 13 '15 at 20:20

answered Jul 13 '15 at 20:14

matthewatabet

1,463
11
26

Did you test that? i couldn't get `configparser` to like the file-like object that `open` returned. – Robᵩ Jul 13 '15 at 20:14
Tested it a bit more, and it seems to work fine. Are you getting an error? – matthewatabet Jul 13 '15 at 20:22
Python 3.4.0 on Ubuntu 14.04, running your code, produces an error. `TypeError: startswith first arg must be bytes or a tuple of bytes, not str ` – Robᵩ Jul 13 '15 at 20:30
sorry, not running 3.4. – matthewatabet Jul 13 '15 at 20:44

configparser loading config files from zip

3 Answers3