4

We use TeamCity with subversion and MSBuild and we have a problem with the continuous build that is triggered by Subversion commits.

The continuous build is set-up to do incremental builds (the nightly build is full and clean).

The problem occurs if a developer changes and commits file for the second time after the build has started (commit triggered) but before the object that uses the file is built. Now the object file gets a timestamp that is after the timestamp of the second commit. This will cause all later incremental builds to skip the changes to the file.

For extra clarity here is the time line:

T1: Developer commits file.cpp (file.cpp has time T1)
T2: First incremental build starts on the build server
T3: Build server gets the files for the latest change (file.cpp at T1)
T4: Developer commits file.cpp for the second time (file.cpp has T4)
T5: Buildserver compiles file.cpp of T1 into file.obj (now file.obj has time T5)
T6: First Build finishes (result is good)
T7: Second incremental build starts on the build server
T8: Build server gets the files for the latest change (file.cpp at T4)

And now the problem:

T9: Build server doesn't compile file.cpp (of T4) into file.obj because file.obj is of T5 hence the compiler thinks it is newer than the source file.

The problem is easily fixed with a full build but those take a long time (30 minutes without unit tests).

Is incremental building possible in combination with Continuous Integration?

Edit: This problem only seems to happen when using server side checkout mode. With build agent side checkout mode changed files get the timestamp of the time of retrieval while with server side checkout they get the commit time as timestamp.

tshepang
  • 12,111
  • 21
  • 91
  • 136
Halt
  • 2,924
  • 4
  • 22
  • 26
  • You CI server could interrogate your VCS about code changes. – jfs Nov 18 '10 at 19:37
  • It does. The problem is the timing. If a file is changed after the CI server got the prev. version but before it is actually used in the build there seems to be a problem. – Halt Nov 19 '10 at 09:55
  • @Halt: I mean your VCS can tell what files are really changed, so you can touch them in order your build system to work correctly. Something like `svn diff -r PREV:HEAD --summarize | awk '{print $2}' | xargs touch` – jfs Nov 20 '10 at 09:29
  • @J.F. Sebastian: That is a good suggestion but for that to work TC must let the agent do the checkout and not the server otherwise you don't have a versioned checkout available on the build agent. As I found out my problem only happens when NOT using agent side checkout. (See my Edit in the question). – Halt Nov 21 '10 at 08:25
  • @Halt: 1. you could split your build on two tasks and pin the first one that updates timestamps to the CI server. Or 2. avoid relying on timestamps in your build process (use checksums e.g., `git svn` client gives them for free; `make` can be taught to use checksums too). – jfs Nov 21 '10 at 18:24
  • @J.F. Sebastian:We use MSBuild and I doubt we can teach it to use checksums. I'm not sure what you mean to accomplish with your 1st suggestion. You would need to have access to the location where TC server manages the cashed checkout files and mess with those. I'm not sure if this is what you mean. I'm afraid this may interfere with TeamCity itself. – Halt Nov 22 '10 at 21:11

1 Answers1

2

Yes, you definitely have a race condition. I suppose you could try to get clever by interrogating the change log, and touching any files listed in there - or better yet, if it's supported have Subversion not preserve file modification times, rather have the date stamp of the files be the date they were updated located.

One of the kludges I've seen around this is to only run incremental builds most of the time, but have some fraction of the builds run a clean (perhaps nightly). You might get caught in this race condition periodically, but you'll break out of it on a regular basis. Depending on how frequently this occurs, this kludge may be good enough.

EricMinick
  • 1,487
  • 11
  • 12
  • I thought about periodically cleaning but decided against it because after the problem happens and if it breaks the build (that's how we found out) it will break every build afterwards. And if it doesn't break the build you can't trust it to include your change so succeeding test will not mean a lot until the next clean. – Halt Nov 20 '10 at 08:10
  • 1
    "if it's supported have Subversion not preserve file modification times, rather have the date stamp of the files be the date they were updated" This is the default behaviour of subversion, unless you explicitly set the "use-commit-times" option in the config file ( http://svnbook.red-bean.com/nightly/en/svn.advanced.confarea.html#svn.advanced.confarea.opts.config ) – slowdog Nov 24 '10 at 23:51