1

I am trying to use GNU make to run some old code in parallel (say, 50 or 100 instances at a time); but because of a legacy resource issue (sigh), each parallel instance has to execute in a separate directory.

Hence I would like to use -j 50 but where:

  • job #0 runs in directory0
  • job #1 runs in directory1
  • job #2 runs in directory2, etc

I can see horrible ways of achieving this (I'm running under Windows 10), but no good way. :-(

Does GNU make offer any mechanism a given parallel job can use to determine the index of the job slot it is running in?

nickpelling
  • 119
  • 9
  • 1
    Maybe use **GNU Parallel** instead, which gives you that feature as `{#}`. – Mark Setchell Jan 23 '20 at 16:29
  • Maybe you could post some Makefile illustrating what you are trying to achieve. One thing I can think of is to use job's name as part of output directory to make it distinguishable, but of course it all depends on how exactly your Makefiles are built. – raspy Jan 24 '20 at 12:44
  • 1
    Sadly, none of the approaches suggested here solved the problem as described: GNU Parallel's clever tricks don't quite mesh with GNU make's clever tricks, and I genuinely want to run all the tasks as a variable make, all or some of which might need to be triggered. So, to get around the problem I've instead written a Windows batch script, which - though somewhat clunky - seems to be doing the right kind of thing when triggered by GNU make. – nickpelling Jan 31 '20 at 16:00

2 Answers2

1

You could wrap it in GNU Parallel like this:

parallel 'cd subdir{} && make' ::: {1..50}

If you want to see what it would do, without actually doing anything, add the --dry-run option:

parallel --dry-run 'cd subdir{} && make' ::: {1..10}

Sample Output

cd subdir1 && make
cd subdir2 && make
cd subdir3 && make
cd subdir4 && make
cd subdir5 && make
cd subdir6 && make
cd subdir7 && make
cd subdir8 && make
cd subdir9 && make
cd subdir10 && make

Or, if you want to specify the subdirectories as parameters:

parallel 'cd {} && make' ::: subdir*
Mark Setchell
  • 191,897
  • 31
  • 273
  • 432
0

You can probably fake it easily with a wrapper makefile.

Let's say you want to run a makefile (Make.slow say) thousand times, each time with a different parameter, and each time in a different folder.

jobs := $(shell seq 1 1000)

.PHONY: all
all: ${jobs} ; : $@ Success

.PHONY: ${jobs}
${jobs}:
    mkdir -p _$@
    ${MAKE} -C _$@ -f ../Make.slow jobno=$@

If you have 48 CPUs

$ make -j49 all

You will end up with 1,000 folders _1, _2 … _1000. In each, Make.old would have been run with jobno set to the number encoded in the folder name.

bobbogo
  • 14,989
  • 3
  • 48
  • 57