Parallelize recursive Makefiles for building multiple Latex Documents in CICD Pipeline

Question

I want to speed up the build process on my Latex documents by parallelizing the building of the docs. I know that the latex compiler itself is not able to parallelize but i can at least execute the building of the separate docs in parallel. The whole thing is running on a Gitlab CICD Runner that is just executing the top level Makefile.

My Makefiles currently look like this:

Makefile 1 for top level and mid level folders:

# Makefile to go through all the subdirs and execute the makefile there.
SUBDIRS := $(dir $(wildcard */Makefile))

all: $(SUBDIRS)
$(SUBDIRS):
    $(MAKE) -C $@

.PHONY: all $(SUBDIRS)

Makefile 2 for lowest level folders where documents are placed

# Makefile to build all latex docs in current folder
DOC_SRCS := $(wildcard ./*.tex)
DOC_SRCS_NAME := $(basename $(DOC_SRCS))

.PHONY: all clean $(DOC_SRCS)

all: $(DOC_SRCS)
$(DOC_SRCS):
    mkdir -p build/img
    latexmk -outdir=build -shell-escape -pdf $@
    cp build/$(basename $@).pdf $(basename $@).pdf

clean :
    rm -rf build/*

The folder structure looks something like this (I shortened it to make it less cluttered):

project
│   Makefile 1    
│
└───folder
│   │   Makefile 1
│   │
│   └───subfolder
│       │   Makefile 2
│       │   file1.tex
│       │   ...
│   
└───folder
|   │   Makefile 1
│   │
│   └───subfolder
│       │   Makefile 2
│       │   file1.tex
│       │   ...
│   ...

My question now is, is it enough to add the -jN flag to the make calls or do i need to take special care of something? Does the flag only need to be at the Makefile 1 at the $(MAKE) -C $@ call?

I couldn't really get the flag for the jobs to work properly how i intended to.

score 1 · Accepted Answer · answered Jun 12 '23 at 14:13

Your lower makefile is wrong. In make the target of a recipe is the file you want to BUILD, not the source file you build FROM.

Like this:

# Makefile to build all latex docs in current folder
DOC_SRCS := $(wildcard *.tex)

DOC_PDFS := $(DOC_SRCS:.tex=.pdf)

.PHONY: all clean

all: $(DOC_PDFS)

%.pdf : %.tex
        mkdir -p build/img
        latexmk -outdir=build -shell-escape -pdf $<
        cp build/$@ $@

As for your question, it looks to me like this will work fine as-is for parallel builds without needing anything extra. You should add the -jN only to the invocation of make you do from your CI/CD system: do NOT add it into the recipe for the sub-makes.

Thanks for the answer and also the cleaned up Makefile. I updated the Makefile and changed my `make` call to `make -j$(nproc)` (which is 12 in my case) in the CI Pipeline file and it shaves off around 40% build time. — AcademicCoder, Jun 13 '23 at 07:20

Parallelize recursive Makefiles for building multiple Latex Documents in CICD Pipeline

1 Answers1