0

We want to improve the reproducibility of the analyses at our institute. To this effect, we contemplate on implementing a system based on Singularity. The idea is that at the beginning of the analysis, the user can choose a machine configuration (later amendments must be possible) that sticks with them until the project is complete. Then, the image is archived with the analysis. Ideally, the user doesn't have to issue system admin commands (install packages etc.) in the process.

She just makes a request like "I need R with tidyverse and Python 3 and this and that in-house packages" and she gets a command that she can use to ssh into a singularity container that has those features. When she makes a new request, she gets the newest version of the programs but once the container has been deployed those versions don't change anymore.

It gets tricky when I think of the fact that multiple users will need different combinations of software. Do I need to provide an image for every combination of Software and software extension packages? If I only think of a scenario where users can choose of an arbitrary combination of {R, Julia, Python, r-tidyverse, r-data.table, r-whatever-genomic-analysis-package-on-bioconductor, python-...}

  • Is there a feature selection method in the veins of 

    singularity pull library://alpine:3.7 +r:3.2.1 +python3:3.7 +r-package:1.2.3

    such that the user can 

    ssh cluster01 -- singularity shell project-abc.simg

    and start/continue working?

  • If not, is there an alternative approach to supplying custom machine configurations to users using singularity?

I could find Singularity Compose, but this seems to just run multiple containers as services next to each other. So the images can stay separate. I have to merge them.

akraf
  • 2,965
  • 20
  • 44
  • ... just found https://singularity.lbl.gov/docs-scif-apps and https://sci-f.github.io/ ... will update if this turns out to be useful – akraf Nov 26 '19 at 16:16
  • SCIF apps just provide a means to register installed apps of a container in a structured way. A dedicated container image for each combination of apps is still nessecary – akraf Jan 06 '20 at 11:37

1 Answers1

0

Yes, with Singularity, a dedicated image must be provided for each possible combination of packages.

Selecting a set of applications per-user is possible by changing your server configuration to the package managers Nix or GUIX, a fork of nix. The concept here is that each application/library lives within its own directory, whose name is a hash of the app! Therefore, multiple application versions can coexist and each application can link to another version of the same library.

A user can select a set of those directories as a user profile. This is a folder of symlinks into binaries in the proper application folders. From the Nix manual:

scheme of Nix profile architecture as described above

So, each user can setup their environment as they like, down to bitwise reproducability.

After the analysis, the profile can be turned into an image. I know its possible with GUIX using guix pack (tar, Docker, Singularity).

For Nix, I'm not sure. There is a project on GitHub, datakurre/nix-build-pack-docker, but it's dormant since 2015. Maybe it's enough to copy the needed subset of /nix/store into a folder, pull a NixOS image, and bind /nix/store of that image to your own folder?

akraf
  • 2,965
  • 20
  • 44