5

I'd like to run several long-running processes on several inputs. E.g.:

solver_a problem_1
solver_b problem_1
...
solver_b problem_18
solver_c problem_18

I know how to run multiple arguments for the same command - that is the core use case. This is more like the opposite case: multiple commands for the same argument.

Of course you could always run multiple instances of parallel - but then are the instances on the same machines or under the same user aware of each other when scheduling resources?

Leo
  • 2,775
  • 27
  • 29

1 Answers1

7

I think you want GNU Parallel to run each of 18 problems through each of 3 solvers:

parallel echo solver_{1} problem_{2} ::: {a..c} ::: {1..18}

Sample Output

solver_a problem_1
solver_a problem_2
solver_a problem_3
solver_a problem_4
...
...
solver_c problem_16
solver_c problem_17
solver_c problem_18

Or, changing the other parameter faster:

parallel echo solver_{2} problem_{1} ::: {1..18} ::: {a..c}
Mark Setchell
  • 191,897
  • 31
  • 273
  • 432
  • I had never realised until now that those are just strings being passed to some kind of exec/eval statement. So in fact there is no difference between commands and arguments beyond what the shell understands. There is a sentence to that effect right at the beginning of the manual and it went over my head until now. Thanks! – Leo Apr 28 '20 at 15:14
  • I used this example to clarify the point: `parallel "{1} {2}" ::: 'printf "%02d "' 'printf "%03d "' ::: 1 2` – Leo Apr 28 '20 at 15:15
  • 1
    You and me both! I love **GNU Parallel** and use it all the time because CPUs are getting *"fatter"* (more cores) rather than *"taller"* (more GHz) all the time, so it makes sense to multi-process... but every time Ole Tange (the author of **GNU Parallel**) writes a new answer on Stack Overflow, I always learn something new about its abilities that I never knew :-) – Mark Setchell Apr 29 '20 at 08:15
  • 4
    When I run solvers on problems with multiple parameters I often use `--shuf`, because then I can plot results from the whole input range fairly fast. The plot can often show a trend that makes unnecessary to run the full set, but instead stop and focus on the range that looks promising. Whether that is relevant here or not, I do not know. – Ole Tange May 02 '20 at 08:29