0

I have a build that's currently using the old build flow plugin that I'm trying to convert to pipeline.

This build can be massively parallelized (many units of work can run on many different nodes) but we only want to extract the source code once at the beginning, preferably with the Pipeline script from SCM option. I'm at a loss to understand how I can share the source extract (which apparently is on the master) with all of the "downstream" nodes that will be used by the pipeline script.

For build flow we extracted to a well-known location on a shared file system and all of the downstream jobs invoked by the flow were passed (or could derive) that location. That always felt icky & I was hoping that pipeline would have solved this problem but I can't find anything to suggest that it has. What am I missing?

Vadim Kotov
  • 8,084
  • 8
  • 48
  • 62
carej
  • 596
  • 1
  • 6
  • 18
  • You can check http://stackoverflow.com/questions/36843215/how-can-i-use-the-jenkins-copy-artifacts-plugin-from-within-the-pipelines-jenki/36843927#36843927 – Daniel Hernández Oct 19 '16 at 18:56
  • @DanielHernández I'm not looking to push built artifacts around, I'm referring specifically to source code. – carej Oct 20 '16 at 20:09

2 Answers2

1

I believe the official recommendation for this is to make bundles of the source and then use "stash" and "unstash" to make them available to deeper steps of your pipeline script.

See https://www.cloudbees.com/blog/parallelism-and-distributed-builds-jenkins

Keep in mind that this doesn't do anything to help with line-endings. If you have builds that span OSs with different line endings you either need to make OS-specific stashes, or just checkout to a safe label in each downstream step.

Peter McNab
  • 1,031
  • 1
  • 10
  • 11
  • Using "stash" seems drastically inefficient for our use case. I'm talking about hundreds of units of work that can be parallelized against the same source code. Having to stash hundreds of slices of our source tree (i.e. one slice for each work unit) would, I have to believe, take much, much longer than simply extracting once to a shared, common location. – carej Oct 20 '16 at 20:13
1

After further research it seems like the External Workspace Manager Plugin does what I'm looking for.

carej
  • 596
  • 1
  • 6
  • 18