Task scheduler, keep a certain number of tasks running

Question

I have 400ish time consuming web extraction tasks that I want to run, but rather than just running them one after the other I want to run i.e. 6 at a time to speed the overall task time up. What I want to achieve ideally is to create the 400 tasks, select that I want 6 to be running concurrently at all time and then the scheduler software will make sure that 6 are always running, when one finishes the next one in the list is started. Is this possible?

If the above ideal solution is not available, I was thinking about creating 6 separate task lists, each with a 1/6 of the total list in. Then the tasks can be started all at the same time, and each separate task list will start the next in the list when the previous finishes.

Any ideas?

What platform? And you trying to write software, write a script, use an existing program, or are all options open? — David Schwartz, Apr 22 '12 at 22:29
I'd recommend he Use the Schwartz, but I would probably have my pun license revoked. — adaptr, Apr 22 '12 at 22:53

score 4 · Answer 1 · edited May 23 '17 at 12:41

4

On Linux, this can be achieved with xargs:

cat cmds | xargs -n 1 -P 6 -0 bash -c

where cmds is a list of commands to execute using bash

On Windows I suggest you try to solve this with PowerShell. There are multiple questions on this topic over at stack overflow

edited May 23 '17 at 12:41

Community

1

answered Apr 22 '12 at 23:12

dwurf

920
8
15

Thanks for the response. Sorry I should have specified that the platform is Windows Server 2003. I don't mind how it is achieved. – Stephen Apr 25 '12 at 21:46

Task scheduler, keep a certain number of tasks running

1 Answers1