3

Hya,

i'm trying to configure correctly a project in Jenkins.

I want to use the parallelization of jobs to use different machine to compute some calculations.

My script has three parts : A - one script with a parameter. this parameter is a big file. the script is gonna calculate how many little calculations jobs I need to launch. B - a variable number of little and parallels jobs. This is the same jobs I launch on different little files. C - a final job which has to wait all the little jobs (part 2) have finished.

The problem is I want to monitor the progress.

My idea was to use the join plugin for the third part. But I have some problems to launch a variable number of downstreams jobs for part 2....(so that i can monitor them with pipeline downstream plugin)

have you any idea how I could do this?

for launching a variable number of the same job in the part 2, I was thinking of calling the jenkins api in my script A. Is there any way to notify a parent project when you call the api so that in the parent you'll see all the downstreams jobs called via the api?

Akhil Jain
  • 13,872
  • 15
  • 57
  • 93
user706355
  • 321
  • 1
  • 3
  • 4

1 Answers1

1

I apologize that this is not a direct answer to your question, but I'm not sure Jenkins is well set up for managing the variable distribution of jobs that you describe. However, there are other tools that might be useful.

  • I've used parallel both to do simple batch operations and to automate an analysis operation across a lot of files on several machines.
  • There are lots of links to tools in this ServerFault question about how to distribute the same command to many machine.

I can see how Jenkins would be useful if each of the jobs in B required the standard checkout, build, archive steps. But if the jobs in B are more independent, then I'd use a simpler tool. Using a tool like parallel, all your steps would become one job that can be monitored via the console output of parallel.

Community
  • 1
  • 1
Dave Bacher
  • 15,652
  • 3
  • 63
  • 86