Different behavior of Singularity image on different machines?

Question

I try to run my python3 based Singularity image on a remote machine, but I get the following error that I do not get with other machines:

singularity exec --nv --no-home --bind data/:/data/ image/ scripts/train.sh
+ singularity exec --nv --no-home --bind data/:/data/ image/ scripts/train.sh
WARNING: skipping mount of sysfs: no such file or directory
scripts/train.sh: line 8: warning: setlocale: LC_ALL: cannot change locale (en_US.utf8): No such file or directory
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = '/usr/local/bin/python'
  program name = '/usr/local/bin/python'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = '/usr/local/bin/python'
  sys.base_prefix = '/usr/local'
  sys.base_exec_prefix = '/usr/local'
  sys.executable = '/usr/local/bin/python'
  sys.prefix = '/usr/local'
  sys.exec_prefix = '/usr/local'
  sys.path = [
    '/usr/local/bin/python',
    '/usr/local/lib/python38.zip',
    '/usr/local/lib/python3.8',
    '/usr/local/lib/python3.8/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
LookupError: no codec search functions registered: can't find encoding

Current thread 0x00007f628ee51740 (most recent call first):
<no Python frame>

My definition file looks like this:

Bootstrap: docker
From: python:3

%post
    apt-get update  -y
    apt-get install -y git
    pip install torch torchvision
    git clone https://github.com/NVIDIA/apex
    cd apex
    pip install -v --no-cache-dir ./

%runscript
    echo "Running nnunet container..."

How can I fix this error? Why is the image not behaving the same way on my different machines?

Answer 1

This is often due to environmental variables being passed on, or not passed on, to the container without noticing. To ensure this is not an issue, you can use -e or --cleanenv. This will prevent any variables not prefixed with SINGULARITYENV_ from being loaded into the container.

That said, the warning WARNING: skipping mount of sysfs: no such file or directory is also concerning: singularity was unable to mount /sys into the image because it doesn't exist on the host server. That particular python error also seems to be specific to windows 10. Singularity doesn't currently support windows, even with the magic of WSL2.