How can i run npm install for different projects simultaneouly

Question

I am creating build which actually consists of 3 different projects. For 2 of them I need to do npm install and for third i need to do bower install before packaking and creating the build. So I was looking for optimization opportunity here and created this script:

install_dependency.sh

# $1 = project_1 $2=project_2 $3=project_3

npm install --global bower ng-cli &&              <!-- This runs 1st -->
echo "Global Installation complete" &&            <!-- This runs 2nd -->
node --version && npm --version &&                <!-- This runs 3rd -->
parallel --halt 2 ::: \                           <!-- From here till END runs in parallel -->
"cd $1; npm install" \
"cd $1; npm install --only=dev" \
"cd $2; npm install" \
"cd $2; npm install --only=dev" \                 <!-- "END" Till here it runs parallel-->
"cd $3; bower --allow-root install" && echo All is OK &&    <!-- It runs next -->
cd $1; npm run build_stage && echo build created && <!-- It runs next --> 
cd $2; npm run build  && echo build created    <!-- It runs next -->

But it seems I cant run npm install parallely as it is causing some conflicts and failing with some random error like this:

> node-sass@4.9.4 install /data/project/uiv2/node_modules/node-sass
> node scripts/install.js

npm WARN deprecated typings@1.5.0: Typings is deprecated in favor of NPM @types -- see README for more information
npm WARN deprecated browserslist@2.11.3: Browserslist 2 could fail on reading Browserslist >3.0 config used in other tools.
npm WARN deprecated gulp-util@3.0.7: gulp-util is deprecated - replace it, following the guidelines at https://medium.com/gulpjs/gulp-util-ca3b1f9f9ac5
npm WARN deprecated hoek@2.16.3: The major version is no longer supported. Please update to 4.x or newer
module.js:550
    throw err;
    ^

Error: Cannot find module 'inherits'
    at Function.Module._resolveFilename (module.js:548:15)

Please help me solve this if its possible. If it is not then what can be the best way to solve this.

are these projects depending on each other, like with `npm link`? If yes, then may be parallel install is not an option here. — muradm, Oct 30 '18 at 11:42
No they are independent. They have different package.json file and they just share same node version — undefined, Oct 30 '18 at 11:43
https://docs.npmjs.com/cli/cache should help may be. Default: `~/.npm` on Posix, or `%AppData%/npm-cache` on Windows. Then if you are running them simultaneously, they can try to modify same cache, causing random errors? If you do them one by one, do they install properly? — muradm, Oct 30 '18 at 11:46
Quick search points to a number of issues related: - https://github.com/npm/npm/issues/5948 - https://github.com/npm/npm/issues/2500 I don't know if they have solution, but intuitively I would look at `docker`, to run build within docker container. — muradm, Oct 30 '18 at 12:03

score 1 · Answer 1 · edited Jan 10 '22 at 17:34

Problem with parallel builds probably can be dockerized. For instance one command for building in $1:

docker run -it --rm -v $(pwd)/$1:/srv -w=/srv node:8.11.3 "npm install && npm install --only-dev && npm run build_stage && echo build created"

Then you don't have to even change directory. What this will do is:

docker run
interactively - -it
cleanup after process finished - --rm
mount project directory to container directory /srv - -v $(pwd)/$1:/srv
set working directory - -w=/srv
using image - node:8.11.3
and the command is - npm install --global bower ng-cli && npm install && npm install --only-dev && npm run build_stage && echo build created

Upon completion, project under $1 would be build as if it would be built on host machine.

Then you will have three such commands (as per your script), and you can run them in parallel. To illustrate script:

parallel --halt 2 ::: \ 
  "docker run -it --rm -v $(pwd)/$1:/srv -w=/srv \
    node:8.11.3 sh -c 'npm install --global bower ng-cli && npm install && npm install --only-dev && npm run build_stage && echo $1 build created'"
  "docker run -it --rm -v $(pwd)/$2:/srv -w=/srv \
    node:8.11.3 sh -c 'npm install --global bower ng-cli && npm install && npm install --only-dev && npm run build && echo $2 build created'"
  "docker run -it --rm -v $(pwd)/$3:/srv -w=/srv \
    node:8.11.3 sh -c 'npm install --global bower ng-cli && bower --allow-root install'"

Note that commands in this example are not accurate monkey typing for illustration purposes and not tested.

Your solution looks amazing and I think can solve my problem. I will upvote this for now because I am off work, I will try it in sometime and will accept your answer. Thanks a lot for your time. Very much appreciated. — undefined, Oct 30 '18 at 12:29

score 0 · Answer 2 · answered Jun 27 '23 at 15:28

You can use turbo: https://turbo.build/repo/docs/installing

Example:

package.json

{
  "name": "example",
  "private": true,
  "scripts": {   
-    "test": "yarn workspace project1 test && yarn workspace project2 test"
+    "test": "turbo test",
  },
  "workspaces": {
    "packages": [
      "projects/project1",
      "projects/project2"
    ]
  },
  "devDependencies": {
+     "turbo": "^1.10.6"
  }
}

How can i run npm install for different projects simultaneouly

2 Answers2