14

In relatively big projects which are using plain old make, even building the project when nothing has changed takes a few tens of seconds. Especially with many executions of make -C, which have the new process overhead.

The obvious solution to this problem is a build tool based on inotify-like feature of the OS. It would look out when a certain file is changed, and based on that list it would compile this file alone.

Is there such machinery out there? Bonus points for open source projects.

Elazar Leibovich
  • 32,750
  • 33
  • 122
  • 169

5 Answers5

15

You mean like Tup:

From the home page:

"Tup is a file-based build system - it inputs a list of file changes and a directed acyclic graph (DAG), then processes the DAG to execute the appropriate commands required to update dependent files. The DAG is stored in an SQLite database. By default, the list of file changes is generated by scanning the filesystem. Alternatively, the list can be provided up front by running the included file monitor daemon."

Happi
  • 166
  • 2
  • I wonder, what happens when you edit a file and save incomplete versions of it. Must be that every save triggers `Tup` to compile it, right? – Maxim Egorushkin Mar 02 '11 at 14:30
  • Tup only compiles when you ask, but it always watches the file system for changes. – Joseph Lisee May 28 '11 at 16:01
  • 2
    `tup monitor -a` – So far, looking good – The relevant files are compiled and the program is ready before I even got around to change to the terminal :-D – Frank Dec 31 '11 at 19:50
4

I am just wondering if it is stat()ing the files that takes so long. To check this here is a small systemtap script I wrote to measure the time it takes to stat() files:

# call-counts.stp

global calls, times

probe kernel.function(@1) {
    times[probefunc()] = gettimeofday_ns()
}

probe kernel.function(@1).return {
    now = gettimeofday_ns()
    delta = now - times[probefunc()]
    calls[probefunc()] <<< delta
}

And then use it like this:

$ stap -c "make -rC ~/src/prj -j8 -k" ~/tmp/count-calls.stp sys_newstat
make: Entering directory `/home/user/src/prj'
make: Nothing to be done for `all'.
make: Leaving directory `/home/user/src/prj'
calls["sys_newstat"] @count=8318 @min=684 @max=910667 @sum=26952500 @avg=3240

The project I ran it upon has 4593 source files and it takes ~27msec (26952500nsec above) for make to stat all the files along with the corresponding .d files. I am using non-recursive make though.

Maxim Egorushkin
  • 131,725
  • 17
  • 180
  • 271
  • You might be correct, and this might be a premature optimization. BTW was the cache hot? It is interesting to look at a project with 100,000, and see if your stat times scales. Note that nonrecursive make is many times not an option for really big projects. +1 for taking the time to profile. – Elazar Leibovich Mar 02 '11 at 13:29
  • The file system cache was hot (the workstation I compile on has 12Gb of RAM). And it is on a local ext3 filesystem. It takes orders of magnitude longer to `stat()` files when building on a NFS though. – Maxim Egorushkin Mar 02 '11 at 14:02
  • the author of `tup`, measures a few seconds for making a non-changed project with 100k files http://gittup.org/tup/make_vs_tup.html – Elazar Leibovich Mar 02 '11 at 14:06
  • Interesting. I wonder if the filesystem in the test was mounted with `noatime`. – Maxim Egorushkin Mar 02 '11 at 16:34
  • 2
    The author of `ninja` agrees that `inotify` doesn't worth the trouble. "inotify. I had originally intended to make Ninja be memory-resident and to use inotify to keep the build state hot at all times. But upon writing the code I found it was fast enough to run from scratch each time. Perhaps a slower computer would still benefit from inotify; the data structures are set up such that using inotify shouldn’t be hard." http://martine.github.com/ninja/manual.html – Elazar Leibovich Jul 21 '11 at 11:33
2

If you're using OSX, you can use fswatch

https://github.com/alandipert/fswatch

Here's how to use fswatch to for changes to a file and then run make if it detects any

fswatch -o anyFile | xargs -n1 -I{} make

You can run fswatch from inside a makefile like this:

watch: $(FILE)
  fswatch -o $^ | xargs -n1 -I{} make

(Of course, $(FILE) is defined inside the makefile.) make can now watch for changes in the file like this:

> make watch

You can watch another file like this:

> make watch anotherFile
d13
  • 9,817
  • 12
  • 36
  • 44
1

Install inotify-tools and write a few lines of bash to invoke make when certain directories are updated.

As a side note, recursive make scales badly and is error prone. Prefer non-recursive make.

Maxim Egorushkin
  • 131,725
  • 17
  • 180
  • 271
  • Huh? Invoking make touches each and every file on the project. I don't want to do that! The whole point to have make never touch all the files, even in order to get their modification time. – Elazar Leibovich Feb 26 '11 at 17:57
  • @Elazar Leibovich: make does not touch any files. It stat's source files to see if a target needs to be rebuild. – Maxim Egorushkin Feb 26 '11 at 19:02
  • Maxim, I don't know what about you, but I consider `stat`ing a file, touching it. And if you need to stat 100,000 files each and every time you do a single make, it can take a while (99.9% of those files aren't interesting for you, you already compiled them to libsomething.so, and that's all you need). – Elazar Leibovich Feb 27 '11 at 14:02
  • Oh, are you referring to file access time? This feature is totally flawed that for every read you do a write, use `-noatime` when mounting your filesystems. See http://thunk.org/tytso/blog/2009/03/01/ssds-journaling-and-noatimerelatime/ for more. – Maxim Egorushkin Feb 27 '11 at 14:50
  • @Maxim, that's a nice trick, but my approach guarantee reducing the overhead at 99.9% (access one changed file, and non of the other 100,000 file at all, the inotify-make will ignore them). – Elazar Leibovich Feb 27 '11 at 21:14
0

The change-dependency you describe is already part of Make, but Make is flexible enough that it can be used in an inefficient way. If the slowness really is caused by the recursion (make -C commands) -- which it probably is -- then you should reduce the recursion. (You could try putting in your own conditional logic to decide whether to execute make -C, but that would be a very inelegant solution.)

Roughly speaking, if your makefiles look like this

# main makefile

foo:
    make -C bar baz

and this

# makefile in bar/

baz: quartz
    do something

you can change them to this:

# main makefile

foo: bar/quartz
    cd bar && do something

There are many details to get right, but now if bar/quartz has not been changed, the foo rule will not run.

Beta
  • 96,650
  • 16
  • 149
  • 150
  • This is a compromise. Going with your approach gives me a huge bloated makefile which is faster. I want to have my cake, and eat it too. And this is possible with the inotify magic. Never touch any file which wasn't modify, not even for the modification date. – Elazar Leibovich Feb 26 '11 at 17:59