Extra build/missing object files with header-tracking Makefile

Question

I have written a (GNU make) Makefile designed to perform automatic dependency tracking in header includes. Everything works great except that upon typing make a second time, the entire code base rebuilds. Only typing make the third time and successive times gives the message that nothing is to be done.

SRCDIR := src
INCDIR := inc
ifeq ($(DEBUG),1)
    OBJDIR := debug_obj
    BINDIR := debug_bin
else
    OBJDIR := obj
    BINDIR := bin
endif

BINS := prog1 prog2 prog3 prog4
SRCS := $(wildcard $(SRCDIR)/*.cpp)
OBJS := $(patsubst $(SRCDIR)/%,$(OBJDIR)/%,$(SRCS:.cpp=.o))
DEPS := $(OBJS:.o=.d)

CC := g++
COMMON_FLAGS := -Wall -Wextra -Werror -std=c++11 -pedantic
ifeq ($(DEBUG),1)
    CXX_FLAGS := $(COMMON_FLAGS) -Og -g
else
    CXX_FLAGS := $(COMMON_FLAGS) -O3 -D NDEBUG
endif

all: $(addprefix $(BINDIR)/,$(BINS)) | $(BINDIR)

$(OBJDIR) $(BINDIR):
    @ mkdir -p $@;

$(BINDIR)/%: $(OBJDIR)/%.o | $(BINDIR)
    $(CC) $(CPP_FLAGS) $< -o $@;

$(OBJDIR)/%.o: $(SRCDIR)/%.cpp | $(OBJDIR)
    $(CC) $(CPP_FLAGS) -MMD -MP -c $< -o $@;

-include $(DEPS)

.PHONY: all clean

clean:
    - rm -f $(OBJS);
    - rm -f $(DEPS);
    - rm -f $(addprefix $(BINDIR)/,$(BINS));
    - rmdir $(OBJDIR) $(BINDIR) 2> /dev/null || true

Clearly some dependency had changed, so I tried running make -n -d | grep 'newer' following the first invocation of make, which shows this:

Prerequisite obj/prog1.o' is newer than targetbin/prog1'.
Prerequisite obj/prog2.o' is newer than targetbin/prog2'.
Prerequisite obj/prog3.o' is newer than targetbin/prog3'.
Prerequisite obj/prog4.o' is newer than targetbin/prog4'.

And ls -la obj/*

Showed the existence of the dependency (*.d) files but not the object (*.o) files. I assume that this is related to how g++ -MMD -MP works, but despite the apparent absence of object files, binaries are present after the first make.

The answer to this question suggests that both are generated at the same time, and man g++ does not dispute this as far as I can tell.

I've read a couple other questions and answers related to automatic dependency tracking, but I don't see this issue arising. Why is this happening? Can you suggest a fix?

Update

A more careful look at the first invocation of make shows this unexpected (to me) line at the end:

rm obj/prog1.o obj/prog2.o obj/prog3.o obj/prog4.o

That answers one question but raises another.

Update

I also found this in the debugging output.

Considering target file `prog1'.
 File `prog1' does not exist.
make: *** No rule to make target `prog1'.  Stop.
 No implicit rule found for `prog1'.
 Finished prerequisites of target file `prog1'.
Must remake target `prog1'.

For which I note that prog1 is missing the bin/ prefix. Nothing explains why the first run removes the object files, but the second run leaves them, however. That seems to be at the heart of the issue.

Unrelated to your question, though good practice: Why don't you use `CXX`, and `CXXFLAGS`? (Note that `CPPFLAGS` are preprocessor flags). — Micha Wiedenmann, May 05 '13 at 14:30

score 0 · Accepted Answer · answered May 05 '13 at 14:50

0

make was treating the object files as intermediates and deleting them accordingly. Adding:

.SECONDARY: $(OBJS)

solved the problem. I do not know why it was doing this the first invocation but not the second invocation. Comments are welcome.

answered May 05 '13 at 14:50

rg6

329
2
10

I'll leave this open awhile in case anybody has an answer to this. If one materializes I'll accept it. Otherwise, I'll accept mine. – rg6 May 05 '13 at 14:52

score 0 · Answer 2 · answered May 05 '13 at 15:23

The reason that the .o files are not present is that they're considered intermediate files so make deletes them. However, that shouldn't cause any problems in your build, because as long as make can envision the intermediate file it will realize it doesn't need to be rebuilt if its prerequisites are older than its parents (in this case, as long as prog1 is newer than prog1.cpp for example).

I was not able to reproduce your experience with the second build rebuilding everything. More details will be needed. The output you showed is not interesting because that's just saying that make does NOT need to rebuild the .o file (it's newer than the prerequisite). You need to find the lines in the output that explain why make does need to rebuild the .o file. If you provide that info we may be able to help.

Just a couple of comments on your makefile: first, I don't think it's a good idea to force the mkdir rule to always succeed. If the mkdir fails you WANT your build to fail. Probably you did this so it would not be a problem if the directory already exists, but that's not needed because the mkdir -p invocation will never fail just because the directory exists (but it will fail if the directory can't be created for other reasons such as permissions). Also you can combine those into a single rule with multiple targets:

$(BINDIR) $(OBJDIR):
        @mkdir -p $@

Next, you don't need the semicolons in your command lines and in fact, adding them will cause your builds to be slightly slower.

Finally, a small nit, but the correct order of options in the compile line is -c -o $@ $<; the source file is not (this is a common misconception) an argument to the -c option. The -c option, like -E, -s, etc. tells the compiler what output to create; in the case of -c it means compile into an object file. Those options do not take arguments. The filename is a separate argument.

Of course you're right about the mkdir handling, so I fixed my question. A small nit about your nit: of course -c says to compile without linking, but g++ does not require that its arguments be POSIX-LY correct, so in that sense the two forms are equally correct, and I find it conceptually clearer to say *to what* the -c operation applies immediately after -c than to put $@ $< both at the end like you did. If others share my thoughts, it may not be a misconception as much as a preference. I still await an answer about the single rebuild issue though. — rg6, May 05 '13 at 16:54
If you're willing to restrict your compiler options to GCC, then sure they're equivalent. However IMO that ordering is misleading and of course will not necessarily work correctly if you choose a different value for CC. As for the output, that output is perfectly normal. That's what make prints when it decides to remove some intermediate files. It's OK for make to do that since those files are not needed once the link is complete. What we need to see is the part of the debug output that shows why it decided to rebuild the `.o` file. — MadScientist, May 05 '13 at 18:20
I've included additional debugging output for one of the binaries that corresponds to the `make -n -d` command in the second invocation (in which it rebuilds, and in which it does __not__ delete the object files). — rg6, May 05 '13 at 19:01

Extra build/missing object files with header-tracking Makefile

Update

Update

2 Answers2