3

Is there a way to track running singularity container like "docker ps" ? or singularity logs any start/stop info in somewhere.

I have tried the "singularity instance list" command but it doesn't work for the singularity container which doesn't start with "singularity instance start"

Singularity running with singularity exec

[vagrant@localhost ~]$ singularity exec hello-world_latest.sif sleep 600 &
[1] 31167
[vagrant@localhost ~]$ jobs
[1]+  Running                 singularity exec hello-world_latest.sif sleep 600 &

[vagrant@localhost ~]$ singularity instance list
INSTANCE NAME    PID    IP    IMAGE

Singularity running with singularity instance start

[vagrant@localhost ~]$ singularity instance start lolcow_latest.sif cow1
INFO:    instance started successfully
[vagrant@localhost ~]$ singularity instance list
INSTANCE NAME    PID      IP    IMAGE
cow1             31033          /home/vagrant/lolcow_latest.sif

Any thoughts? Thank you in advance!

ms1212
  • 33
  • 5

1 Answers1

2

As you found, instance list only lists singularity processes that were started as instances. Instance logs (both running and past) are stored in ~/.singularity/instances/logs/$HOSTNAME/$USER.

Active processes from non-instance commands aren't tracked anywhere. This is largely because Singularity does not use a centralized service model like Docker. Similarly, there's no "default" log location because stdout/stderr are handled that same as running any other command. As such, tracking down Singularity processes can be done using the standard shell tools.

Some examples:

# show your running Singularity processes (all users: aux)
ps ux | grep '[S]ingularity'

# Output:
#   tsnowlan 1350116  0.1  0.0 1237336 16804 pts/2   Sl   13:29   0:00 Singularity runtime parent
#   tsnowlan 1350273  0.7  0.0 1238488 16608 pts/2   Sl   13:29   0:00 Singularity runtime parent

# Check what the processes are doing
pstree -p 1350116

# Output:
#   starter-suid(1350116)─┬─foo.sh(1350131)─┬─foo.sh(1350153)───sleep(1350160)
#                         │                 ├─foo.sh(1350154)───sleep(1350162)
#                         │                 ├─foo.sh(1350155)───sleep(1350163)
#                         │                 ├─foo.sh(1350156)───sleep(1350161)
#                         │                 ├─foo.sh(1350157)───sleep(1350164)
#                         │                 ├─foo.sh(1350158)───sleep(1350166)
#                         │                 ├─foo.sh(1350159)───sleep(1350168)
#                         │                 ├─foo.sh(1350165)───sleep(1350172)
#                         │                 ├─foo.sh(1350167)───sleep(1350171)
#                         │                 ├─foo.sh(1350169)───sleep(1350173)
#                         │                 └─foo.sh(1350170)───sleep(1350174)
#                         ├─{starter-suid}(1350133)
#                         ├─{starter-suid}(1350134)
#                         ├─{starter-suid}(1350135)
#                         ├─{starter-suid}(1350136)
#                         ├─{starter-suid}(1350137)
#                         ├─{starter-suid}(1350145)
#                         └─{starter-suid}(1350146)
tsnowlan
  • 3,472
  • 10
  • 15
  • I'm seeing singularity instances running postgres "replicate" (I think?) and then go missing from `singularity instance list`. I _can_ find the process with `pgreg -fa Singularity`. Do you have any insight into this behavior? – jhchnc Jul 23 '22 at 20:30
  • (I'm using --pid-file if that is pertinent. – jhchnc Jul 23 '22 at 20:31
  • 1
    Sometimes when running singularity images as an instance / persistent service, the parent singularity process can exit without cleaning up all the child processes. I've noticed this is more likely to happen if the process (like postgres) forks internally. We use `supervisord` to manage services being run inside singularity and this _helps_, but things can still happen. `instance list` reads the info from in `$HOME/.singularity/instances/sing/$HOSTNAME/$USER/$INSTANCE_NAME/*.json` and you can check logs in `$HOME/.singularity/instances/logs/$HOSTNAME/$USER/$INSTANCE_NAME/*` – tsnowlan Jul 25 '22 at 08:43
  • 1
    Yeah that's exactly what I'm seeing with various services. Thank you very much for pointing me at supervisord. – jhchnc Jul 26 '22 at 04:10
  • 1
    Under the hood, it's all sunshine and green grasses ;) – jhchnc Jul 26 '22 at 04:10