1

I'm using the aws cli and job submission takes a long time , if done one by one. Is there a way to submit more than one job at a time in a single request?

Protostome
  • 113
  • 1
  • 6
  • 1
    Even if there is not a native way to do this (I'm not sure at the moment), it is trivial to write a script to do it for you. – EEAA Mar 03 '17 at 13:33
  • I'm not sure.. the request API accepts only one json object per request, so it seems: http://docs.aws.amazon.com/batch/latest/APIReference/API_SubmitJob.html – Protostome Mar 03 '17 at 15:37
  • No, what I'm saying is you write a shell script to run the command repeatedly on your behalf. – EEAA Mar 03 '17 at 15:54
  • Of course, that's what i'm doing. :) The problem is, I have 300k jobs and their submission can last several days... (as each submission request lasts ~0.5 second) – Protostome Mar 03 '17 at 16:02
  • 1
    Ahh, ok. That's a good start. I guess the next step would be to write a more complex script that can parallelize the job submissions across multiple processes. – EEAA Mar 03 '17 at 16:09
  • Yea... that's what i'll do if there's no other way. Of course, it would be much more elegant if there was a way to prepare one json request string for all the jobs and just submit them at once. That's what I'm trying to find out if possible :) – Protostome Mar 03 '17 at 16:21

1 Answers1

2

This is handled by "Array Jobs" which let you submit up to 10,000 jobs with a single request, See https://docs.aws.amazon.com/batch/latest/userguide/array_jobs.html

From the docs:

"AWS Batch array jobs are submitted just like regular jobs. However, you specify an array size (between 2 and 10,000) to define how many child jobs should run in the array. If you submit a job with an array size of 1000, a single job runs and spawns 1000 child jobs. The array job is a reference or pointer to manage all the child jobs. This allows you to submit large workloads with a single query."

Dane Macaulay
  • 136
  • 4