Efficient way to share node_modules between different Azure Pipeline jobs

Question

My project is a node project using yarn to manage npm dependencies.

In my CI azure pipeline, I have multiple jobs, all of which require running yarn to install npm dependencies. Because the number of dependencies is quite large, it takes almost 4 minutes to finish installing them.

I have tried using pipeline artifact to share the node_modules folder between them, but it took the same amount of time (or even slower) to publish and download the artifact between jobs.

I wonder if there's an efficient way to share large data between pipeline jobs as in this case?

score 2 · Answer 1 · answered Jul 24 '20 at 08:42

2

The easiest way is to run the tasks in one job. Or you can choose to run the jobs in same self-hosted agent, so that you don't need to install the duplicate dependencies in same environment.

If you have specific reason to use multiple jobs, you can also consider pipeline caching. Similar examples: link1,link2.

answered Jul 24 '20 at 08:42

LoLance

25,666
1
39
73

2

Thanks. The reason I had to split them into multiple jobs is that I want to parallel some tasks. Currently, they all run in one job serially and it takes too long. I am currently using pipeline caching but it doesn't help a lot, because it still takes a lot of time to download the cache content. We have a self-hosted agent pool. How do I request the jobs to run on a particular agent so that the jobs can run on the same agent and can still run in parallel? Is that even possible? – superkinhluan Jul 24 '20 at 14:22

score 1 · Answer 2 · answered Jul 23 '20 at 13:33

You didn't specify the type of agents you are using Self-Hosted or MS-Hosted. If you're not against using Self-Hosted agents I would recommend creating a agent with the necessary NPM dependencies already available. You can do this with a traditional VM or Docker. Other than this I believe the only alternatives for you would be to either run all tasks requiring these dependencies in the same job, or run your jobs in parallel to make up for the lost time installing the NPM dependencies. The parallel job option won't be available to you if you are using the free version of MS-Hosted agents only if you are paying for them.

Efficient way to share node_modules between different Azure Pipeline jobs

2 Answers2