Concurrent builds cause npm install to fail on TeamCity

Question

We are using TeamCity to build an Angular web application, which involves an npm install build step.

This intermittently fails with the error shown below (slightly edited to remove commercially sensitive details). We suspect that the failures are caused by two or more builds running concurrently, and both trying to access the same npm cache.

• There are three build servers, each with three build agents.
• All agents share the same service account (hence the same user profile and npm cache location). Since we are in a very large company, creating separate accounts would be non-trivial.

Is there anything we can do to ensure that concurrent npm installs do not clash with each other?

[11:04:27][Step 7/16] npm install (1m:23s)
[11:04:27][npm install] Executing npm via wrapping shell script
[11:04:27][npm install] Starting: cmd /c npm install
[11:04:27][npm install] in directory: D:\BuildAgent2\work\c97ce2d45f888f8a\My-Project
[11:05:51][npm install] npm ERR! Windows_NT 6.3.9600
[11:05:51][npm install] npm ERR! argv "C:\\Program Files\\nodejs\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\bin\\npm-cli.js" "install"
[11:05:51][npm install] npm ERR! node v6.9.1
[11:05:51][npm install] npm ERR! npm  v3.10.8
[11:05:51][npm install] npm ERR! path C:\Users\SVC_PITeamCityAgnt\AppData\Roaming\npm-cache\braces\0.1.5\package.tgz.2069119258
[11:05:51][npm install] npm ERR! code EPERM
[11:05:51][npm install] npm ERR! errno -4048
[11:05:51][npm install] npm ERR! syscall rename
[11:05:51][npm install] 
[11:05:51][npm install] npm ERR! Error: EPERM: operation not permitted, rename 'C:\Users\SVC_PITeamCityAgnt\AppData\Roaming\npm-cache\braces\0.1.5\package.tgz.2069119258' -> 'C:\Users\SVC_PITeamCityAgnt\AppData\Roaming\npm-cache\braces\0.1.5\package.tgz'
[11:05:51][npm install] npm ERR!     at Error (native)
[11:05:51][npm install] npm ERR!  { Error: EPERM: operation not permitted, rename 'C:\Users\SVC_PITeamCityAgnt\AppData\Roaming\npm-cache\braces\0.1.5\package.tgz.2069119258' -> 'C:\Users\SVC_PITeamCityAgnt\AppData\Roaming\npm-cache\braces\0.1.5\package.tgz'
[11:05:51][npm install] npm ERR!     at Error (native)
[11:05:51][npm install] npm ERR!   errno: -4048,
[11:05:51][npm install] npm ERR!   code: 'EPERM',
[11:05:51][npm install] npm ERR!   syscall: 'rename',
[11:05:51][npm install] npm ERR!   path: 'C:\\Users\\SVC_PITeamCityAgnt\\AppData\\Roaming\\npm-cache\\braces\\0.1.5\\package.tgz.2069119258',
[11:05:51][npm install] npm ERR!   dest: 'C:\\Users\\SVC_PITeamCityAgnt\\AppData\\Roaming\\npm-cache\\braces\\0.1.5\\package.tgz' }
[11:05:51][npm install] npm ERR! 
[11:05:51][npm install] npm ERR! Please try running this command again as root/Administrator.

Answer 1

We suspect that the failures are caused by two or more builds running concurrently, and both trying to access the same npm cache.

Assuming this is indeed the cause of your problem, there are a few ways you might mitigate this or outright prevent it.

1. Limit concurrent builds to 1. This would assume only one build configuration is used for the builds that are colliding. Yes, this will potentially slow down your pipeline slightly. If anything I would suggest doing this to see if the issue goes away, which would correlate concurrent npm installs with this build error. This setting is found in the General Settings section of a build configuration.

2. Use a Docker container. Depending on your familiarity with containerization, this may be non-trivial. Containers would isolate each build's file system, user, and so on, which would solve the presumed problem; unless you intentionally share volumes between containers, npm installs will be completely isolated from build to build. I say Docker specifically because I know they have support for Windows, which your agents seem to be.

3. Use on-demand cloud agents to cheaply run agents and service accounts 1:1. We use AWS EC2 instances as cloud build agents, and we only pay for the time we use with each agent. Cloud agent documentation can be found here.