1

We have a proprietary compiler that can take a number of input files and process them at once:

compiler a.in
# produces a.out

compiler a.in b.in c.in
# produces a.out b.out c.out

The reason to do that is that is saves a lot of time for initialization. For thousands of files the batch version is orders of magnitude faster than compiling files individually. We also run a post-processor on the files.

Now, I have this in the (GNU) makefile, which is not taking advantage of the batch processing capabilities and updates files one by one. I want to update it to use batch compilation:

.INTERMEDIATE: $(TMP)
$(TMP):  $(TMPDIR)/%.tmp: $(SRCDIR)/%.in |$(TMPDIR) 
        compiler $< -o $@

$(RESULT): $(RESDIR)/%.out: $(TMPDIR)/%.tmp $(SRCDIR)/%.in
        post-process $< -o $@

How would I rewrite the first rule to recompile all files that have been modified with a single command, perhaps, using $?? The second rule needs to stay there and work the same.

Fyodor
  • 13
  • 3

1 Answers1

1

If you are able to require GNU make 4.3+, then your life is quite simple, you can take advantage of grouped targets, like this (note the &:):

a.out b.out c.out &: a.in b.in c.in
        compiler $^

If you can't require a recent version of GNU make, you're relegated to using "sentinel files", like this:

a.out b.out c.out : .sentinal ;

.sentinal: a.in b.in c.in
        compiler $^
        @touch $@

(be sure to include the trailing semicolon on the first rule...)

MadScientist
  • 92,819
  • 9
  • 109
  • 136
  • Wow, thanks, grouped targets are exactly what I needed. However, the second part won't really work: how do I make sure that when x.in changes and x.tmp is regenerated, only x.tmp gets post-processed? Anyway, I think the fact that they added the feature proves that it was not available before. – Fyodor Jun 15 '20 at 14:14
  • I'm not sure what you mean by "tmp"? There is no reference to "tmp" in my examples. I'm simply providing a way to implement the initial statement in your question, where a single invocation of the compiler with many `.in` files generates many `.out` files. My second example will definitely do that. If there are other requirements about some `tmp` files, that wasn't made clear in your question. – MadScientist Jun 15 '20 at 18:32
  • The second option definitely works, but it has issues. For example, deleting the .out files by hand won't cause them to be rebuilt: you have to delete the `.sentinel` file. That's why grouped targets were added; it makes things simpler to write and avoids some annoying things about the "traditional" solution. – MadScientist Jun 15 '20 at 18:34
  • I stated in the end of the original question that I still need the post-process rule to work. E.g., when I run `make result/foo.out`, and `src/foo.in` has been modified, it must be able to figure out that it needs to run both the compiler (batch) and post-processor (works on single files). In other words, I'm wondering if it's possible to have a complete and correct dependency graph while doing batch compilation, particularly, if output files of a batch compile can be other targets' prerequisites. And, it sounds like, prior to grouped targets, the answer was "no" – Fyodor Jun 16 '20 at 11:21
  • No, that's not my answer. I'm just not exactly clear on the full set of dependency requirements. If what you're saying is that the batch-build needs all the `tmp` files having been built, then you just have to tell make that. For example, you could change the `.sentinel` rule so it lists the `tmp` files as prerequisites, that way they're all built before the batch command runs. It's certainly _nicer_ to do it with grouped targets, but it's possible to hack it so it works more-or-less correctly without them. – MadScientist Jun 16 '20 at 18:03
  • No, I need the results of batch builds to be other targets' prerequisites. (in other words, that's just normal functioning of make: any file can be part of the dependency graph as a prerequisite or target). The most sophisticated thing I tried was something like: `$(RESULT): $(RESDIR)/%.out: $(SRCDIR)/%.in $(TMPDIR)/%.tmp | .sentinel` - this will run the batch compile and then rebuild individual files. However, it's still fragile in many ways. I was wondering if there was another solution. – Fyodor Jun 22 '20 at 10:49