Boost's bjam is running tests before the build has finished

Question

I have a fairly simple Boost.Python extension that I am building with bjam. The problem is that the order that things happen in doesn't make sense to me, and I can't see how to fix it.

My project consists of a root directory, with a Jamroot, and a single project subdirectory with a Jamfile, C++ file, header file, and Python script.

In the root I have a Jamroot file that looks like this, largely scraped together from examples and the docs. It is separate from the project's Jamfile because I actually want to share this amongst several projects that will exist in other subdirectories.

import python ;

if ! [ python.configured ]
{
    ECHO "notice: no Python configured in user-config.jam" ;
    ECHO "notice: will use default configuration" ;
    using python ;
}

use-project boost
  : ./boost ;

project
  : requirements <library>/boost/python//boost_python ;

# A little "rule" (function) to clean up the syntax of declaring tests
# of these extension modules.
rule run-test ( test-name : sources + )
{
    import testing ;
    testing.make-test run-pyd : $(sources) : : $(test-name) ;
}

build-project hello_world ;
# build-project [[other projects]]... ;

Then I have a subdirectory containing my 'hello_world' project (name changed to protect the innocent), which contains a Jamfile:

PROJECT_NAME = hello_world ;

import python ;

python-extension interpolation_ext :
  $(PROJECT_NAME).cpp
:
  <define>FOO
;

# Put the extension and Boost.Python DLL in the current directory, so that running script by hand works.
install convenient_copy
  : $(PROJECT_NAME)_ext
  : <install-dependencies>on <install-type>SHARED_LIB <install-type>PYTHON_EXTENSION
    <location>.
  ;

# Declare test targets
run-test $(PROJECT_NAME) : $(PROJECT_NAME)_ext test_$(PROJECT_NAME)_ext.py ;

That 'convenient_copy' sure is convenient, but I haven't found much documentation about it, unfortunately.

Anyway, the idea is that while I'm in the "hello_world" project directory, I make code changes and type 'bjam' regularly. This has the effect of building the Python extension and then running the test_hello_world_ext.py file, which does an 'import hello_world_ext' to test that the extension has built correctly, and then a bunch of rather trivial unit-tests. If they all pass, then bjam reports success.

The problem seems to be that sometimes bjam runs the Python test before it has run the 'convenient_copy' rule, which means that it performs the test on the previous version of the extension, and then overwrites it with the new version. This means I'm frequently having to run bjam twice. In fact, the second time bjam knows that something is out-of-date because it actually does something. The third and subsequent time it does nothing until I make further source changes. It's like the classic double-make problem when a dependency isn't correct.

The main problem with this is that it's often failing a successful build (because the existing extension was bad), and other times it is showing a bad build as successful. It actually took me several weeks to notice this behaviour, around the same time I thought I was going insane, perhaps not coincidentally...

It also seems to do this more often on Linux than OS X, but I'm not completely sure. Feels that way though, and I divide my time between both environments fairly equally.

Also, am I the only person who finds bjam's 'jamfile' syntax utterly confusing? There's a lot going on under the hood that I simply don't understand, or can find adequate documentation for. I'd gladly use make or SCons instead but I wasn't able to get those working either due to broken examples here and there. What really confuses me is how bjam builds many, many other targets before getting on to my files, which would make writing a makefile quite tricky I think? As I am quite familiar with GNU Make and SCons, is it worth my time abandoning bjam to use one of those instead?

score 2 · Answer 1 · answered Oct 02 '12 at 10:35

The order of declaring targets in a jamfile doesn't determine the order of building the targets. Use dependencies to control the build order. It would be done like this here:

Change the run-test rule to accept requirements argument:

rule run-test ( test-name : sources + : requirements * )
{
    import testing ;
    testing.make-test run-pyd : $(sources) : $(requirements) : $(test-name) ;
}

Modify $(PROJECT_NAME) target declaration to add a dependency requirement on convenient_copy:

run-test $(PROJECT_NAME) : $(PROJECT_NAME)_ext test_$(PROJECT_NAME)_ext.py : <dependency>convenient_copy ;

Regarding the part on jamfile syntax etc.:

If you do anything with Boost.Build apart from really trivial things, you should definitely read its User Manual. My personal experience is that after reading it from start to the end, I choose Boost.Build over other build systems any day. YMMV

Boost's bjam is running tests before the build has finished

1 Answers1