17

A git tool that meets the specs below is needed. Does one already exist? If not, I will create a script and make it available on GitHub for others to use or contribute. Is there a completely different and better way to solve the need to build/test every commit to a branch in a git repository? Not just to the latest but each one back to a certain starting point.

Background: Our development environment uses a separate continuous integration server which is wonderful. However, it is still necessary to do full builds locally on each developer's PC to make sure the commit won't "break the build" when pushed to the CI server. Unfortunately, with auto unit tests, those build force the developer to wait 10 or 15 minutes for a build every time.

To solve this we have setup a "mirror" git repository on each developer PC. So we develop in the main repository but anytime a local full build is needed. We run a couple commands in the mirror repository to fetch, checkout the commit we want, and build. It's works extremely lovely so we can continue working in the main one with the build going in parallel.

There's only one main concern now. We want to make sure every single commit builds and passes tests. But we often get busy and neglect to build several fresh commits. Then if the build fails you have to do a bisect or manually figure build each interim commit to figure out which one broke.

Requirements for this tool:

  1. The tool will look at another repo, origin by default, fetch and compare all commits that are in branches to 2 lists of commits. One list must hold successfully built commits and the other lists commits that failed.

  2. It identifies any commits not yet in either list and begins to build them in a loop in the order that they were committed. It stops on the first one that fails.

  3. The tool appropriately adds each commit to either the successful or failed list after it as attempted to build each one.

  4. The tool will ignore any "legacy" commits which are prior to the oldest commit in the success list. This logic makes the starting point possible in the next point.

  5. Starting Point. The tool building a specific commit so that, if successful it gets added to the success list. If it is the earliest commit in the success list, it becomes the "starting point" so that none of the commits prior to that are examined for builds.

  6. Only linear tree support? Much like bisect, this tool works best on a commit tree which is, at least from it's starting point, linear without any merges. That is, it should be a tree which was built and updated entirely via rebase and fast forward commits.

  7. If it fails on one commit in a branch it will stop without building the rest that followed after that one. Instead if will just move on to another branch, if any.

  8. The tool must do these steps once by default but allow a parameter to loop with an option to set how many seconds between loops. Other tools like Hudson or CruiseControl could do more fancy scheduling options.

The tool must have good defaults but allow optional control.

  1. Which repo? origin by default.

  2. Which branches? all of them by default.

  3. What tool? by default an executable file to be provided by the user named "buildtest", "buildtest.sh" "buildtest.cmd", or buildtest.exe" in the root folder of the repository.

  4. Loop delay? run once by default with option to loop after a number of seconds between iterations.

Cœur
  • 37,241
  • 25
  • 195
  • 267
Wayne
  • 2,959
  • 3
  • 30
  • 48
  • https://github.com/spotify/git-test: `Run tests on each distinct tree in a revision list, skipping versions whose contents have already been tested.` – hlovdal Nov 30 '18 at 12:37

1 Answers1

15

I wrote a tool I called git test-sequence a while back to do exactly this.

I was just considering blogging about it because a friend was telling me it saved him a ton of work this week.

Dustin
  • 89,080
  • 21
  • 111
  • 133
  • Interesting, this tool very elegant. And it iteratively builds all the commits in a branch. How does it "remember" the next time which commits it already tried, success or fail? If builds only take a few seconds, brute force is fine. But my builds take about 10 to 15 minutes. So perhaps cloning your repo and adding a "memory" then offering you to update your branch will work? – Wayne Mar 28 '10 at 02:35
  • Do you or anyone have a suggestion as to where to "store" the memory? In my case, this is a mirror repo used only for builds so adding the two files to .ignore will be fine. But is there a more standard way to store that kind of memory like bisect in tools? Perhaps folder under .git? – Wayne Mar 28 '10 at 02:38
  • It's easiest just to give it a range. Let's say you have a pointer branch called `known_good` that you move up every time a build is successful. Then you'd run `git test_sequence known_good..master 'do_build.sh && git update-ref HEAD refs/heads/known_good` – Dustin Mar 28 '10 at 06:22
  • Oops, I think I mixed up the args to update-ref... Might want to test that before deploying it. :) – Dustin Mar 28 '10 at 07:32
  • Dustin, I thought of that also. Buy my workflow is that I do lots of commits before publish publicly. Then if a commit fails the build/test, I then perform a rebase interactive to fix that build and rebase the rest. So you see using a known_good branch will also result in non-fast forward updates that force a rebase which can product conflict and other mess. Instead, I want the build to to simply look at SHA-1 refs and see if it has build that on the tree successfully or not. If never built, it will build it sort it. The advantage of this is that it can't be foiled by rebase -i. – Wayne Mar 28 '10 at 17:07
  • Dustin, you think your approach will solve the rebase problem above? s there something I'm missing? – Wayne Mar 28 '10 at 17:15
  • What you're describing is exactly why this tool exists. I typically just track it as `origin/master` or similar, though. A rebase that rewrites `known_good` would obviously be harder than one that does not. I'd argue that if you invalidate `known_good` with a rebase, you should retest anyway. If you dont intend to, just rebase from `known_good` on. – Dustin Mar 28 '10 at 23:39
  • Yes. The plan is definitely to retest all commits from the rebase point. But the plan is also to definitely invalidate known_good from time to time. – Wayne Mar 30 '10 at 02:36
  • Hey, Dustin, the other problem is that this tool needs to run on multiple branches. We have feature branches under parallel development. Any commit to any of them needs to trigger a test. So that would create the requirement of many "known_good" branches, one for each real branch. Really, wouldn't it be easier to just use a single text file that lists SHA-1 refs which already were tested for ANY branch in the repo? That list can have SHA-1 & pass/fail on each line. That list can simply always grow and always be used to lookup any ref to see if it already passed or failed. – Wayne Mar 30 '10 at 02:45
  • Yes, this simply text list eliminates all the issues with branches, rebases, etc. And it can be maintained in sorted order for quick lookup via a binary search. I think there's shell tools for doing that kind of thing, right? – Wayne Mar 30 '10 at 02:48
  • The awk tool will handle this perfectly so that given a SHA-1 ref and the "tested" file it will return the text pass/file using awk '/sha1-ref/ {print $2}' tested. awk is fast enough that it's never necessary to sort the tested list file. Eventually, the test-sequence tool can have the ability to "prune" the tested list by removing all SHA1-refs that are no longer longer valid. – Wayne Mar 30 '10 at 03:02
  • I think anything you do that isn't a branch will just reinvent branches. `known_good_from_master` means all existing tools work, otherwise you have to do special stuff to keep things updated. – Dustin Mar 30 '10 at 16:49