22

I am in the process of packaging up a python package that I'll refer to as MyPackage.

The package structure is:

MyPackage/
    script.py
    data.json

The data.json file comprises cached data that is read in script.py.

I have figured out how to include data files (use of setuptools include_package_data=True and to also include path to data file in the MANIFEST.in file) but now when I pip install this package and import the installed MyPackage (currently testing install by pip from the GitHub repository) I get a FileNotFound exception (data.json) in the script that is to utilize MyPackage. However, I see that the data.json file is indeed installed in Lib/site-packages/MyPackage.

Am I doing something wrong here by trying to read in a json file in a package?

Note that in script.py I am attempting to read data.json as open('data.json', 'r')

Am I screwing up something regarding the path to the data file?

bad_coder
  • 11,289
  • 20
  • 44
  • 72
Mason Edmison
  • 594
  • 3
  • 16
  • 1
    Welcome to Stack Overflow. Could you share your code? It will be easier to help you if you do. –  Mar 14 '20 at 22:01
  • **Please provide the entire error message as well as a [mcve].** – AMC Mar 15 '20 at 02:14

2 Answers2

28

You're not screwing something up, accessing package resources is just a little tricky - largely because they can be packaged in formats where your .json might strictly speaking not exist as an actual file on the system where your package is installed (e.g. as zip-app). The right way to access your data file is not by specifying a path to it (like "MyPackage/data.json"), but by accessing it as a resource of your installed package (like "MyPackage.data.json"). The distinction might seem pedantic, but it can matter a lot.

Anyway, the access should be done using the builtin importlib.resources module:

import importlib.resources
import json

with importlib.resources.open_text("MyPackage", "data.json") as file:
    data = json.load(file)  
# you should be able to access 'data' like a dictionary here

If you happen to work on a python version lower than 3.7, you will have to install it as importlib_resources from pyPI.

Arne
  • 17,706
  • 5
  • 83
  • 99
  • Ah awesome! I just saw this after I posted my answer. I will try this, thank you! It's funny, I thought discretizing a piece of an existing script would be relatively easy but here I am nearly 2 hours later... ha – Mason Edmison Mar 14 '20 at 22:05
  • yeah, packaging code can turn into a mess really quickly =) – Arne Mar 14 '20 at 22:09
  • 1
    I'm wondering if something has changed in Python 3.9? An exact copy of your example code crashes for me with `AttributeError: 'WindowsPath' object has no attribute 'read'` The documentation suggests that "The context manager provides a pathlib.Path object" so I'm assuming it's working as intended, but that would invalidate your example. Am I missing something? – Mantas Kandratavičius Apr 01 '21 at 14:01
  • @MantasKandratavicius uhm.. no, you're right. It seems I uploaded buggy code and still managed to get a bunch of upvotes for it. `json.load` expects a file handle, not a path object, so you need to call `open()` there first. It's fixed now, thanks for pointing it out. – Arne Apr 01 '21 at 19:23
  • 1
    Instead of using the double with's you can also use `importlib.resources.open_text`, as in `with importlib.resources.open_text("MyPackage", "data.json") as data_file: data = json.load(data_file)` – boudewijn21 Aug 05 '21 at 10:03
  • @boudewijn21 I didn't know that function, thanks! – Arne Aug 05 '21 at 13:57
1

I resolved the issue by getting the 'relative path' to where the package is.

self.data = self.load_data(path=os.path.join(
                os.path.dirname(os.path.abspath(__file__)),
                'data.json'))

load_data just reads the data file

Any constructive criticism is still very much welcome. Not trying to write stupid code if I can't help it :)

Mason Edmison
  • 594
  • 3
  • 16
  • 1
    If this is a simple package that you have full control over, doing it like this will probably be good enough. One case where it might fail is if you install the package as a zipped archive. – Arne Mar 14 '20 at 22:15