3

I'm trying to create a singularity image based on docker://python:3.7-buster.

In the %post section, when I run /usr/bin/env python3 a different version of python is used than when I singularity exec the same thing.

Here is my test definition file:

$ cat test.def 
Bootstrap:docker
From:python:3.7-buster

%post
        /usr/bin/env python3 --version

And here is what happens when I try to build it:

$ sudo singularity build test.sif test.def 
INFO:    Starting build...
Getting image source signatures
Skipping fetch of repeat blob sha256:4a56a430b2bac33260d6449e162017e2b23076c6411a17b46db67f5b84dde2bd
Skipping fetch of repeat blob sha256:4b5cacb629f5c5323a32103e665756e5d50fe133b3db72d444f370566b713a6a
Skipping fetch of repeat blob sha256:14408c8d4f9a59a5da8f4cc40650be9a8d0991fa1ce1b2fb2767f289a9cc410d
Skipping fetch of repeat blob sha256:ea67eaa7dd42136287337f879ef20b4ee73baaa108d833d267ef99dd787cdcbf
Skipping fetch of repeat blob sha256:4d134ac3fe4b8dd8136d9e7acbb2708ead2154185b27c09ae62ca099070cfb27
Skipping fetch of repeat blob sha256:4c55f6f5d7f035e446f063331d9160bb00ed3da4632105ef5adedee3317c902f
Skipping fetch of repeat blob sha256:6ae475e50652d8ee1a2fdeb59ccce81d14c8c20e0fdfe94f22f1c69bd3e3befb
Skipping fetch of repeat blob sha256:6f41526442299286e270923d6cca3a516c3e1850f7e06c3facc0df7da8a5afbc
Skipping fetch of repeat blob sha256:6933d3d4604265f0c8f2a3806222749809c62b6e6a757d1f85720fa81622496d
Copying config sha256:5a5fb77dac35d62c5b062fc35b3b69e61ae68385fb4278ce6076532c3e50e316
 7.47 KiB / 7.47 KiB [======================================================] 0s
Writing manifest to image destination
Storing signatures
2019/09/16 11:07:07  info unpack layer: sha256:4a56a430b2bac33260d6449e162017e2b23076c6411a17b46db67f5b84dde2bd
2019/09/16 11:07:09  info unpack layer: sha256:4b5cacb629f5c5323a32103e665756e5d50fe133b3db72d444f370566b713a6a
2019/09/16 11:07:09  info unpack layer: sha256:14408c8d4f9a59a5da8f4cc40650be9a8d0991fa1ce1b2fb2767f289a9cc410d
2019/09/16 11:07:09  info unpack layer: sha256:ea67eaa7dd42136287337f879ef20b4ee73baaa108d833d267ef99dd787cdcbf
2019/09/16 11:07:11  info unpack layer: sha256:4d134ac3fe4b8dd8136d9e7acbb2708ead2154185b27c09ae62ca099070cfb27
2019/09/16 11:07:18  info unpack layer: sha256:4c55f6f5d7f035e446f063331d9160bb00ed3da4632105ef5adedee3317c902f
2019/09/16 11:07:18  info unpack layer: sha256:6ae475e50652d8ee1a2fdeb59ccce81d14c8c20e0fdfe94f22f1c69bd3e3befb
2019/09/16 11:07:19  info unpack layer: sha256:6f41526442299286e270923d6cca3a516c3e1850f7e06c3facc0df7da8a5afbc
2019/09/16 11:07:19  info unpack layer: sha256:6933d3d4604265f0c8f2a3806222749809c62b6e6a757d1f85720fa81622496d
INFO:    Running post scriptlet
+ /usr/bin/env python3 --version
Python 3.7.3
INFO:    Creating SIF file...
INFO:    Build complete: test.sif

The system version is used instead the one provided by the docker image, in contrast with what happens when I just exec the same command:

$ singularity exec docker://python:3.7-buster /usr/bin/env python3 --version
Python 3.7.4

What happens?

I tried to use $(which python3) instead of /usr/bin/env python3, and it is still the same version during the %post section (with singularity exec, it's the host system's version that is used).

My goal is actually to be able to install some personal python packages that I pull from a git repository, and that get installed using install scripts that run /usr/bin/env python3 -m pip install -e .

I noticed there was a problem because the system version of python 3 did not include pip.

HoldOffHunger
  • 18,769
  • 10
  • 104
  • 133
bli
  • 7,549
  • 7
  • 48
  • 94

2 Answers2

2

You're getting different versions of python because the environments are different.

If you add echo $PATH in %post you'll get: /bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin

If you run singularity exec docker://python:3.7-buster bash -c 'echo $PATH' you'll get: /usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

/usr/local/bin/python3 is the newer version that the docker container compiled for you and the PATH in %post prioritizes /usr/bin over /usr/local/bin. You can adjust the PATH variable at the beginning of the %post block to fix the issue.

As to the why it is using a different PATH there, I'm not sure. Might be worth raising an issue at the github repo.

tsnowlan
  • 3,472
  • 10
  • 15
0

I feel like there are multiple things going on here. Can you check that pip(3) is in there by trying (it could also be that you just have to use pip3 instead of pip):

$ which pip
$ which pip3

Unfortunately I'm not able to fully help you with the other part: why 3.7.3 over 3.7.4. Normally you could simply state python3.7 [arg] but that doesn't help here. Could it be that the 3.7.3 comes from elsewhere? As i don't know about the system using anything else from 2.7.

McBait
  • 57
  • 5
  • I run pip via `/usr/bin/env python3 -m pip` in order to have control on the python installation that will be affected. – bli Sep 16 '19 at 09:53
  • I see, missed that line. I'm fully able to reproduce your problem within a singularity container, but do not get the same issue using vanilla docker (where I can install packages defined in the dockerfile just fine using that specific image). Unfortunately, I'm also unable to diagnose something else. – McBait Sep 16 '19 at 11:32