1

When I use the git bisect command, I run only the failing tests at each bisection point for Java programs. However, I see that many tutorials related to git bisect propose running "make; make test". Is there any reason why I should run all the tests at each step? Thanks a lot in advance.

Ripon Saha
  • 43
  • 6

2 Answers2

1

If:

  • all tests pass at the commit marked as good
  • some tests fail at the commit marked as bad

Then yes, it's safe to only run the failing tests to speed up the bisection process. You can infer from the test results at the good and bad commits that the rest of the tests should pass.

You would probably re-run the full test suite after fixing the bug in question in any case, which covers you for the case where your bugfix introduces a regression.

bcmcfc
  • 25,966
  • 29
  • 109
  • 181
1

I would have to say that the conditions mentioned by @bcmcfc are necessary but not sufficient. For reference, his conditions are

  • all tests pass at the commit marked as good
  • some tests fail at the commit marked as bad

My problem is not knowing what has happened in between the good commit and the bad. For example, was there another bug discovered and fixed in the intervening commits? It's conceivable that that bug or its fix influenced this bug.

Another issue is the possible presence of "dirty" commits in the history. I don't know your usage patterns, but some people allow commits with failing tests to be present on feature branches. bisect can land on those commits, and if you only run the tests that you expect to fail you may not fully understand what's happening in that commit, and that may lead you astray in fixing the bug. It may even be that the bug was introduced and then fixed in that feature branch, then introduced again later on another feature branch in a slightly different way, which will really confuse your efforts to fix it.

This seems to me to be an example of the old adage, "In theory there's no difference between theory and practice, but in practice there is." I would run every test every time. And if they all pass where you expect, then you shouldn't feel like you've wasted your effort, you should glow with confidence knowing that you know what's going on.

Erick G. Hagstrom
  • 4,873
  • 1
  • 24
  • 38
  • 1
    Hi @Erick, thanks a lot for your insightful answer. It provides a very complete scenario to understand what may go wrong in debugging if we don't run all the test cases. So basically what I understood is that if we want to save time, we can run only the failing test cases. But that may or may not be helpful to understand the bug completely. On the other hand, running all the test cases will give us more confidence to fully understand the bug. – Ripon Saha Jan 13 '16 at 21:11