1

In a Python package I have a data file which is nested 'above' a file which requires it. e.g.

package
  data
    data.csv
  utils
    util.py

In util.py I obtain the file as a stream using pkg_resources; this allows the code to be run from source and when installed as a package.

    data_stream = pkg_resources.resource_stream(
        __name__, os.path.join("..", "data", "data.csv")
    )

This works fine, and allows the data file to be retrieved as expected.

However, when I run tests which use the code in util.py, a Deprecationwarning is revealed, stating "DeprecationWarning: Use of .. or absolute path in a resource path is not allowed and will raise exceptions in a future release."

But I'm not sure what the correct way to approach this is?

ska_james
  • 59
  • 1
  • 6

2 Answers2

0

Maybe using the Path module would be a better way, it's also crossplatform.

import os
from pathlib import Path

data_stream = pkg_resources.resource_stream(
        __name__, os.path.join(Path(os.curdir())[0], "data", "data.csv")
    )
Harben
  • 1,802
  • 1
  • 11
  • 16
  • Thanks for the suggestion Harben, unfortunately this doesn't work as is for me, I did have a play around with the os.curdir variable, but that refers to the current dir, not the upper dir? – ska_james May 19 '20 at 18:45
0

It seems this issue did not only cause a DeprecationWarning, but led to an error when installing from the packaged .egg file (but not the .whl) - previously documented here: package data not installed from python .egg file

My workaround to both of these was to place a data_resources.py module at the same level as the data files, which defines the data structures from the on-disk resources. This avoids having to use paths with '..' in them.

E.g.

package
  data
    data_resources.py
    data.csv
  utils
    util.py

where data_resources.py may contain something like:

data_stream = pkg_resources.resource_stream(__name__, "data.csv")
data_df = pd.read_csv(data_stream)

and util.py can simply import:

from package.data.data_resources import data_df

I'm assuming this is a more correct approach to this situation, since it suppresses the DeprecationWarnings, and allows installation from .egg and .whl files.

ska_james
  • 59
  • 1
  • 6