Using the job-server FDs to deduce the actual available job count
I've come up with a different approach which I prefer over parsing the command line as it was suggested in the accepted answer. The main issue I have with the latter is that it doesn't work in a recursively called makefile, as the jobs parameter is passed via the env there, and then it's representing the jobserver which tells you nothing of how many job tokens it's going to provide
So I have decided to embrace the jobserver thing and make a small Python script that consumes all tokens from the jobserver, counts them, and then puts them back.
This was a neat idea that appeared to be surprisingly easy to implement, and it worked very well for me.
There are a few notes on how it should be used:
- the script must be called with a
+ prefix on the recipe because that enables the "submake" functionality which duplicates the jobserver descriptors for the child process to be able to "spawn workers"
- to give the correct job count the rule containing this script in the recipe should be made a common order-only phony prerequisite to all your targets so that it will be run by
make prior to anything else and never in parallel to other things, cause then the count would be off
- The current script implementation will max out at 1024 tokens. Feel free to improve the code to avoid the limitation, can be done with a simple loop around the read and a little logic.
The Python script:
import argparse, os
def safe_int(s):
try:
return int(s)
except:
return -1
class JobserverArgs:
known_names = ["jobserver-fds","jobserver-auth"]
def __init__(self):
self.fds = "-1,-1"
@staticmethod
def from_argv():
ja = JobserverArgs()
parser = argparse.ArgumentParser()
for name in JobserverArgs.known_names:
parser.add_argument('--'+name, dest="fds")
parser.parse_known_args(namespace=ja)
return ja
def get_fds(self):
return tuple([safe_int(fd) for fd in (self.fds+",").split(",")][:2])
fd_in, fd_out = JobserverArgs.from_argv().get_fds()
if fd_in == -1 or fd_out == -1:
print(1)
else:
os.set_blocking(fd_in, False)
tokens = os.read(fd_in, 1024)
os.write(fd_out, tokens)
print(len(tokens)+1)
example makefile:
TARGETS := a b c d e
.PHONY: $(TARGETS)
$(TARGETS):
@for i in 1 2; do echo "$@$$i"; sleep 1; done
.PHONY: all
all: $(TARGETS)
@echo done all
$(TARGETS): | common
.PHONY: common
common:
+$(eval JOBS:=$$(shell python jobs.py $(MAKEFLAGS)))
@echo JOBS=$(JOBS)
running it:
projects/make-fun » make all -j100
JOBS=100
a1
b1
c1
d1
e1
a2
b2
c2
d2
e2
done all
