0

Suppose I have the following PyPIs:

  • public PyPi (standard packages)
  • gitlab pypi (because internal team ABC wanted to use this)
  • artifactory PyPi (because contractor team DEF wanted to use this)

Now suppose package titled "ABC" exists on all of them, but are not the same thing (for instance, "apples," which are 3 entirely different packages on all pypis.). How do I do something in my requirements and setup.py to map the package name to the pypi to use?

Something like:

package_def==1.2.3 --index-url=artifactory
apples==1.08 --index-url=gitlab # NOT FROM PUBLIC OR FROM ARTIFACTORY
package_abc==1.2.3 --index-url=artifactory
package_efg==1.0.0 # public pypi

I don't even know how I'd configure the setup.py in this instance either.

I really don't want multiple requirements.txt with different index urls at the top. I also don't want --extra-index-url due to the vulnerabilities it could introduce when using a private pypi.

I tried googling around, messing around with the order of requirements.txt, breaking it up into different files, etc. No luck. Seems that the last --index-url is always used to install all packages.

Any ideas?

npengra317
  • 126
  • 2
  • 7
  • you can write a seperate script where it is mentioned what to install and from where, just a question why so much pypi artifactory in a single project ? – sahasrara62 Nov 22 '22 at 21:08
  • So there's no native way in requirements.txt to do this? That sucks :(. And there's so many pypis because it's just a large research-first-software-second corporation, many teams, some internal, some external. Everyone's got an opinion on where to put stuff and no one has decided to enforce a policy yet. – npengra317 Nov 22 '22 at 21:12
  • https://pip.pypa.io/en/stable/reference/requirements-file-format/#global-options "The following options have an effect on the *entire* `pip install` run, and must be specified on their individual lines." – phd Nov 22 '22 at 23:16
  • @npengra317 then I guess a proper policy should be there or project is decouple such a way that each team make there module seperately and use wheel or something else in main project like that – sahasrara62 Nov 23 '22 at 05:22
  • @sahasrara62 I totally agree that this is a bad situation, and that management not enforcing a policy earlier is annoying. It's not just because my requirements.txt is imperfect, but there's a whole bunch of other problems that come with it too. However, it's the situation I'm left with and I gotta solve it. If you (or anyone who wants free reputation) want to make an answer of "you can't" I'll accept it. – npengra317 Nov 23 '22 at 17:31
  • na i will pass to post an answer. it would be better if this problem is solved asap otherwise it is gona be pain the the project – sahasrara62 Nov 23 '22 at 18:33
  • Some reading: https://discuss.python.org/t/dependency-notation-including-the-index-url/5659 as well as [my answer to this related question](https://stackoverflow.com/a/67358322) -- My recommendation would be to build a proxy for all those index servers, which based on some simple rules would redirect to one server or the other. A very basic one is [_simpleindex_](https://pypi.org/project/simpleindex/), and some more complex solutions exist (bandersnatch, devpi). – sinoroc Nov 24 '22 at 16:01
  • Other similar Q&A: https://stackoverflow.com/a/64700106 – sinoroc Nov 24 '22 at 16:13

1 Answers1

0

The question gets back to the idea that a package dependency specification usually is a state of need that is independent of how that need should be satisfied.

So the dependency declaration “foo==1.0.0” (the thing declared as part of the package metadata) means “I need the package named foo with version 1.0.0" and that is in principle implementation independent. You can install that package with pip from PyPI, but you could also use a different tool and/or different source to satisfy that requirement (e.g. conda, installation-from-source, etc.).

This distinction is the reason why there's no good way to do this.

There are a few work arounds:

  • You can specify the full link to a wheel you want to pip install
  • You can use an alternative tool like Poetry, which does support this a little more cleanly.

For my particular usecase, I just listed the full link to the wheel I wanted to pip install, since upgrading to poetry is out of scope at the moment.

npengra317
  • 126
  • 2
  • 7
  • Yes, I thought of Poetry as well, I know it does something in that direction (probably some other comparable tools have similar features). But Poetry is a development tool. So as long as you are in development mode, Poetry will be able to fetch dependencies from one server or the other. But past the development phase, you will be confronted to the same issue again. And it is not a limitation of Poetry, it is a limitation of the design of the Python packaging ecosystem as a whole. – sinoroc Nov 24 '22 at 16:07