29

We use the standard practices of not including node_modules in version control. However, when moving through the CI/CD pipeline, we have to reinstall NPM dependencies in several places and it makes everything very slow.

Is there a way to somehow cache NPM dependencies with Docker? I searched Google "docker cache npm dependencies" and the first hit from 2014 yielded a dead link.

Nothing else of much value came up.

One solution is to include node_modules in version control, but I think that would be a huge mistake. I think caching the dependencies somehow would be the best option.

Here is the Dockerfile as is:

FROM node:6

COPY . .  # copy all files, but node_modules does not exist ( => gitignored)

RUN npm install --no-optional --only=production > /dev/null 2>&1
RUN npm install -g bower  > /dev/null 2>&1
RUN bower install --config.interactive=false --allow-root > /dev/null 2>&1

ENTRYPOINT ["/bin/bash", "/root/cdt/run.sh"]

Here is one possible solution, but I can't quite figure out how it works:

=> http://bitjudo.com/blog/2014/03/13/building-efficient-dockerfiles-node-dot-js/

Alexander Mills
  • 90,741
  • 139
  • 482
  • 817
  • 1
    How does your Dockerfile look like? – n2o May 04 '17 at 19:03
  • 1
    A possible solution is to use the [`-v`](https://docs.docker.com/engine/reference/commandline/run/#mount-volume--v---read-only) option in the Docker command line to map `~/.npm` to `~/.npm` in the container. This way on subsequent executions it uses an already populated cache. However, the improvement is not spectacular, it still needs ages to resolve the dependencies. – axiac May 04 '17 at 19:04
  • I just added my Dockerfile, thanks – Alexander Mills May 04 '17 at 19:13
  • 2
    I added an answer, I think this might be the best solution, although there may be some supplemental solutions on top of that. – Alexander Mills May 04 '17 at 22:47
  • Unless you go out of your way to change it, most docker files run with the root user. So you might run into permission issues in ~/.npm with that method. https://github.com/moby/moby/issues/3124 for more info, and work arounds – Eric Wooley Jan 17 '20 at 23:08

4 Answers4

24

This method works like magic:

https://blog.playmoweb.com/speed-up-your-builds-with-docker-cache-bfed14c051bf

Docker has a special way of caching things for you, and apparently it's best to use the inborn caching ability.

Cannot say I completely understand how it works, but it does work.

If you follow this pattern, it will work for you:

FROM mhart/alpine-node:5.6.0
WORKDIR /src

# Expose the port 3000
EXPOSE 3000

# Set the default command to run when a container starts
CMD ["node", "server.js"]

# Install app dependencies
COPY package.json /src
RUN npm install

# Copy your code in the docker image
COPY . /src
Alexander Mills
  • 90,741
  • 139
  • 482
  • 817
  • 9
    At the `COPY package.json /src` instruction, docker checks if the package.json file has changed. If it has changed it will rerun the `RUN` command below. it works but if you frequently change package.json it's a mess. Perhaps using shared volumes might install the modules faster – Maxim Jul 12 '17 at 08:11
  • 3
    I agree, if you change package.json frequently, it doesn't help anyone – Alexander Mills Jul 12 '17 at 19:13
6

Have you tried using yarn instead of npm which is way faster? Yarn does parallel package installations

https://yarnpkg.com/lang/en/compare/

Jamesed
  • 122
  • 5
  • 1
    yeah I was thinking about that, but then I would have to install Yarn first – Alexander Mills May 04 '17 at 19:47
  • Woudn't yarn add alot more benefits than using npm ? I think its worth it if you can use it with a base image. If not then im not sure. I am a newb when it comes to docker. – Jamesed May 04 '17 at 19:52
  • Yeah it might, but I don't think Yarn will come part as the base image, so I would have to do npm install -g yarn, then run yarn. I am also not that certain yarn is all that faster? Are there any stats on how much faster? – Alexander Mills May 04 '17 at 19:53
  • There is also npm-yarn-benchmark that can benchmark performance between both – Jamesed May 04 '17 at 19:57
  • thanks, I added an answer, this appears to be the best way to do it - the secret is that Docker has its own way of doing caching and it's different than you would expect. – Alexander Mills May 04 '17 at 22:32
  • 4
    Would like to mention that the new *npm 5* is faster or comparable in speed to *yarn*. Since yarn has to be installed in docker, it might be useful to just use node version 8+ instead, this comes with npm 5 pre-installed. – Maxim Jul 12 '17 at 08:12
  • Yarn should be safer because it is MITM proof, not sure if npm already worked on it, but last time I checked they didn't (about 2021) And yarn should come preinstalled in the latest node image – Overclocked Skid Jan 31 '23 at 04:52
2

Additional tricks for collection:
In square brackets is the relative time saved in my case

RUN mkdir node_modules
  • Skipping security audit on docker. [-30sec]
RUN npm ci --no-audit
  • Using other package managers like pnpm, even with additional install it's still faster.
RUN npm install -g pnpm
RUN pnpm install
AlexanderB
  • 879
  • 9
  • 12
-1

I have met this problem today, above solution doesn't work for me, my code had been already like that. I just find a simple way: update the command from 'npm install' to npm ci. In 'npm ci', 'ci' here means clean & install, I don't know how it works, but when I updated it, my installing in docker finished in 6 minutes which can not be finished before even spent 30 more minutes. Hope this can help.

kain
  • 214
  • 2
  • 4