4

Is pip capable of recursively searching through a file system path and installing a package and its dependencies? For example, given the following file structure (files not shown) pip install packageA -f C:\packages\ does not work.

C:\packages\
C:\packages\packageA\
C:\packages\packageA\1.0.0
C:\packages\packageA\1.0.1

C:\packages\packageB\
C:\packages\packageB\2.2.1
C:\packages\packageB\2.2.4

Additionally, can these packages be pure source with a setup.py file? Or do they need to be a binary like wheel or zip files. And finally, is there a way to resolve the dependencies? For example, if packageA requires a version of packageB, can pip get that version of packageB from my folders? Do I need html files indicating where and what things are?

I am aware I can point pip directly to local paths (pip install C:\packages\packageA\1.0.0), but I want this to function as if the packages were available in PyPI. For example, if a user types pip install packageB or pip install requirements.txt and that requirements file contains packages that exist locally but not in PyPI, it would just work. (I could set the local package storage path in a configuration file so the pip command doesn't need the -f argument)

Basically, I want to replicate PyPI functionality using a file system without a webserver (security won't let us run one). Any insight would be greatly appreciated.

supertommy
  • 351
  • 3
  • 11

2 Answers2

5

I figured this out. I used a package called pip2pi . This package has a command called dir2pi. I create a directory containing tar.gz files. Each tar.gz file was a Python package created with this layout. To identify versions, I added a hyphen and then used semantic versioning.

The directory looked like this:

C:\packages\package_a-1.0.8.tar.gz
C:\packages\package_a-1.1.0.tar.gz
C:\packages\package_b-2.0.0.tar.gz
C:\packages\package_c-1.0.5.tar.gz

Then I ran, dir2pi C:\packages -S. This created the necessary HTML files and folder layout. At the root of the folder, a folder called simple was created. The name simple appears to be a PEP 503 thing. Inside of simple, the folder structure looked like this:

C:\packages\simple\index.html

C:\packages\simple\package_a
C:\packages\simple\package_a\index.html
C:\packages\simple\package_a\package_a-1.0.8.tar.gz
C:\packages\simple\package_a\package_a-1.1.1.tar.gz

C:\packages\simple\package_b
C:\packages\simple\package_b\index.html
C:\packages\simple\package_b\package_b-2.0.0.tar.gz

C:\packages\simple\package_c
C:\packages\simple\package_c\index.html
C:\packages\simple\package_c\package_c-1.0.5.tar.gz

This is apparently what a compliant file/folder layout looks like for pip.

Each index.html file contains links to its neighboring files and folders. For example, C:\packages\simple\index.html looks like this:

<html><head><title>Simple Index</title><meta name='api-version' value='2' /></head><body>
<a href='package_a/'>package_a</a><br />
<a href='package_b/'>package_b</a><br />
<a href='package_c/'>package_c</a><br />
</body></html>

C:\packages\simple\package_a\index.html looks like this:

<a href='package_a-1.0.8.tar.gz'>package_a-1.0.8.tar.gz</a><br />
<a href='package_a-1.1.1.tar.gz'>package_a-1.1.1.tar.gz</a><br />

Now with the proper HTML files and folder structure, I told pip to look for packages in the C:\packages\simple folder. I did this by editing the pip.ini file (the location varies from system to system) and adding this line: 

[global]
extra-index-url = file:///c:/packages/simple

Alternatively, I could also pass this folder into pip as a parameter like so:

pip install --index-url=file:///c:/packages/simple package_a==1.1.1
supertommy
  • 351
  • 3
  • 11
  • This sounds better, but what if my packages/ folder contains many subfolders and subsub folders? Then pip2pi cannot give the correct simple/ folder. – ZHANG Juenjie Jan 16 '22 at 03:22
0

Is pip capable of recursively searching…?

No. Either one flat directory or a pre-generated html file that lists files in subdirectories.

Additionally, can these packages be pure source with a setup.py file?

Yes. pip can install source distributions, eggs and wheels.

And finally, is there a way to resolve the dependencies?

Yes, but they must be accessible. That is, a PyPI-like server (not necessary the PyPI), one flat directory or an html file.

PS. To prevent pip to look packages at a PyPI use option --no-index.

phd
  • 82,685
  • 13
  • 120
  • 165