I have a repository, myBigRep, that contains a submodule, mySmallRep. This submodule is a python project that has a setup.py script that installs some commands. To set it, I run:
$ pip install --editable myBigRep/local/src/mySmallRep/
And that's it.
Now I want to clone mySmallRep somewhere else, set its submodules and install the Python commands. So I first freeze the Python packages:
$ cd myBigRep
$ pip freeze
-e git+git@bitbucket.org:me/mySmallRep.git@d702f237b4c5449dffe5f224a8c361bf983566fc#egg=mySmallRep&subdirectory=../../../../local/src/mySmallRep
$ pip freeze > requirements.txt
$ git add requirements.txt
$ git commit -m "added requirements."
$ git push origin master
Now I clone and set everything:
$ cd
$ git clone git@bitbucket.org:me/myBigRep.git myBigRepClone
$ cd myBigRepClone
$ git submodule init && git submodule update
$ git submodule foreach git pull origin master
$ pip install --requirement requirements.txt
Obtaining mySmallRep from git+git@bitbucket.org:me/mySmallRep.git@d702f237b4c5449dffe5f224a8c361bf983566fc#egg=mySmallRep&subdirectory=../../../../local/src/mySmallRep (from -r requ.txt (line 2))
Skipping because already up-to-date.
Error [Errno 2] No such file or directory: '/home/me/myBigRepClone/src/mySmallRep/../../../../local/src/mySmallRep' while executing command python setup.py egg_info
Exception:
Traceback (most recent call last):
File "/home/me/myBigRepClone/local/miniconda3/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/home/me/myBigRepClone/local/miniconda3/lib/python2.7/site-packages/pip/commands/install.py", line 335, in run
wb.build(autobuilding=True)
File "/home/me/myBigRepClone/local/miniconda3/lib/python2.7/site-packages/pip/wheel.py", line 749, in build
self.requirement_set.prepare_files(self.finder)
File "/home/me/myBigRepClone/local/miniconda3/lib/python2.7/site-packages/pip/req/req_set.py", line 380, in prepare_files
ignore_dependencies=self.ignore_dependencies))
File "/home/me/myBigRepClone/local/miniconda3/lib/python2.7/site-packages/pip/req/req_set.py", line 518, in _prepare_file
abstract_dist.prep_for_dist()
File "/home/me/myBigRepClone/local/miniconda3/lib/python2.7/site-packages/pip/req/req_set.py", line 129, in prep_for_dist
self.req_to_install.run_egg_info()
File "/home/me/myBigRepClone/local/miniconda3/lib/python2.7/site-packages/pip/req/req_install.py", line 439, in run_egg_info
command_desc='python setup.py egg_info')
File "/home/me/myBigRepClone/local/miniconda3/lib/python2.7/site-packages/pip/utils/__init__.py", line 667, in call_subprocess
cwd=cwd, env=env)
File "/home/me/myBigRepClone/local/miniconda3/lib/python2.7/subprocess.py", line 711, in __init__
errread, errwrite)
File "/home/me/myBigRepClone/local/miniconda3/lib/python2.7/subprocess.py", line 1343, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory: '/home/me/myBigRepClone/src/mySmallRep/../../../../local/src/mySmallRep'
It is obvious the last path is wrong. But if I execute the command manually it works:
$ pip install -e git+git@bitbucket.org:me/mySmallRep.git@d702f237b4c5449dffe5f224a8c361bf983566fc#egg=mySmallRep&subdirectory=../../../../local/src/mySmallRep
[1] 19227
Obtaining mySmallRep from git+git@bitbucket.org:me/mySmallRep.git@d702f237b4c5449dffe5f224a8c361bf983566fc#egg=mySmallRep
Skipping because already up-to-date.
Requirement already satisfied: click in ./local/miniconda3/lib/python2.7/site-packages (from mySmallRep)
Installing collected packages: mySmallRep
Running setup.py develop for mySmallRep
Successfully installed mySmallRep
And if I remove the submodule in the requirement field, it works also:
$ pip uninstall mySmallRep -y
Uninstalling mySmallRep-0.1:
Successfully uninstalled mySmallRep-0.1
$ cat requirements.txt
-e git+git@bitbucket.org:me/mySmallRep.git@d702f237b4c5449dffe5f224a8c361bf983566fc#egg=mySmallRep&subdirectory=../../../../local/src/mySmallRep
$ cat requirements.txt | sed 's/&subdirectory.*//g'
-e git+git@bitbucket.org:me/mySmallRep.git@d702f237b4c5449dffe5f224a8c361bf983566fc#egg=mySmallRep
$ sed 's/&subdirectory.*//g' -i requirements.txt
$ pip install --requirement requirements.txt
Obtaining mySmallRep from git+git@bitbucket.org:me/mySmallRep.git@d702f237b4c5449dffe5f224a8c361bf983566fc#egg=mySmallRep (from -r requirements.txt (line 1))
Skipping because already up-to-date.
Requirement already satisfied: click in ./local/miniconda3/lib/python2.7/site-packages (from mySmallRep->-r requirements.txt (line 1))
Installing collected packages: mySmallRep
Running setup.py develop for mySmallRep
Successfully installed mySmallRep
Why does it put the subdirectory in the first place?
Why processing the subdirectory from the requirements.txt file does not work while it does if installed manually?
In an automation context I need to flawlessly freeze packages and to install them from the requirements.txt file. A dirty workaround I found is to write the following in my script:
$ pip freeze | sed 's/&subdirectory.*//g' > requirements.txt
Is there a better way to solve this?
