How to use nix-shell properly and avoid "dumping very large path"?

Question

Based on my readings (notably the wiki and this blog post), I have come up with the following default.nix that I load with nix-shell:

with import <nixpkgs> {};

let emacs =
  emacsWithPackages (p : [ p.tuareg ]);
in

stdenv.mkDerivation rec {
    name = "env";

    src = ./.;

    # Customizable development requirements
    buildInputs = [
        pkgconfig
        ocaml
        ocamlPackages.merlin
        ocamlPackages.findlib
        ocamlPackages.lablgtk
        ocamlPackages.camlp5_transitional
        ncurses
        emacs
    ];

    # Customizable development shell setup
    shellHook = ''
        export PATH=`pwd`/bin:$PATH
    '';
}

But it always prints a warning:

warning: dumping very large path (> 256 MiB); this may run out of memory

and takes quite long to load (about 45 seconds the first time I call nix-shell after start-up, about 2 seconds on subsequent calls).

What is the meaning of this message? When I look for it on Google, I find a few GitHub issues but not expressed in a way that is easy to understand for the layman.

Can I speed up the load and remove this message? It seems to me that I'm doing something wrong.

Are there general recommendations on writing this kind of development environment that I might not be aware of?

score 12 · Accepted Answer · edited Jul 10 '18 at 06:50

12

Probably the src attribute (the current directory) is very big. nix-shell will copy it to the Nix store on each invocation, which is probably not what you want/need. The workaround is to write:

src = if lib.inNixShell then null else ./.;

(where lib comes from Nixpkgs).

This way, ./. will be copied when you invoke nix-build, but not when you run nix-shell.

edited Jul 10 '18 at 06:50

CMCDragonkai

6,222
12
56
98

answered Sep 14 '16 at 14:41

niksnut

1,129
1
8
5

That was quite helpful. However, shouldn't it read `"then null else './.';"` instead of `"then null else nix;"`? – knedlsepp Apr 12 '17 at 09:41
1

Wouldn't it cause the same warning to be issued when running `nix-build`? That's the issue that I have even without use of `lib.inNixShell`, `nix-shell` is happy, but `nix-build` gives warning (but still works). – LB2 Nov 21 '19 at 20:09

score 8 · Answer 2 · answered Aug 13 '19 at 14:57

Late to the party, but (since I can't comment on niksnut's correct answer) I wanted to mention a means of handling this if you want to add some subset of src to the Nix store, filtering out large/unneeded files.

This approach uses lib.cleanSource and friends from nixpkgs:

# shell.nix
{ pkgs ? import <nixpkgs> {} }:

with pkgs;

let
  cleanVendorFilter = name: type: 
    type == "directory" && baseNameOf (toString name) == "vendor";
  cleanVendor = src: lib.cleanSourceWith { filter = cleanVendorFilter; inherit src; };
  shellSrc = lib.cleanSource (cleanVendor ./.);
in mkShell {
  name = "my-shell";
  shellHook = ''
    printf 1>&2 '# Hello from store path %s!\n' ${shellSrc}
  '';
}

In the above snippet, shellSrc refers to an attribute set representing a store path that contains ./., but without the vendor subdirectory (cleanVendor) and without .git, .svn, files ending in ~, and other editor/VCS-related stuff (cleanSource).

Check out lib/sources.nix for more ways to filter paths.

This is the right way to do it, but this answer looks overly complicated because it deals with a `vendor` directory which the OP doesn't mention. It would be better to show `src = lib.cleanSource ./.;` first and then show how more complex things can be done. — Neil Mayhew, Jun 29 '20 at 14:26

How to use nix-shell properly and avoid "dumping very large path"?

2 Answers2