3

Basically I created a new test file in a particular package with some bare bones test structure - no actual tests...just an empty struct type that embeds suite.Suite, and a function that takes in a *testing.T object and calls suite.Run() on said struct. This immediately caused all our other tests to start failing indeterministically.

The nature of the failures were associated with database unique key integrity violations on inserts and deletes into a single Postgres DB. This is leading me to believe that the tests were being run concurrently without calling our setup methods to prepare the environment properly between tests.

Needless to say, the moment I move this test file to another package, everything magically works!

Has anyone else run into this problem before and can possibly provide some insights?

enter image description here

Jonathan Hall
  • 75,165
  • 16
  • 143
  • 189
Zhuiguang Liu
  • 510
  • 5
  • 8
  • I don't know testify at all. (And I don't know if that WAT duck is gonna get edited out, haha.) But wonder if it could be rogue concurrency in the code being tested, which the framework might have no control over: background goroutines unintentionally left running or whatever. – twotwotwo Oct 29 '14 at 23:13
  • 4
    The WAT duck should be a thing, like 50% of the questions answered here should have that. – OneOfOne Oct 29 '14 at 23:56
  • Could log on entering and leaving tests to see if they overlap, log when your test and setup functions run, etc. Seems we don't have a general answer (we share your WAT), so all that's left is digging into what's happening in your tests. – twotwotwo Oct 30 '14 at 06:59

2 Answers2

4

As it turns out, this is a problem rooted in how go test works, and has nothing to do with testify. Our tests were being ran on ./... This causes the underlining go test tool to run tests in each package in parallel, as justinas pointed out. After digging around more on StackOverflow (here and here) and reading through testify's active issue on this problem, it seems that the best immediate solution is to use the -p=1 flag to limit the number of packages to be run in parallel.

However, it is still unexplained why the tests consistently passed prior to adding these new packages. A hunch is perhaps the packages/test files were sorted and ran in such a manner that concurrency wasn't an issue prior to adding the new packages/files.

Community
  • 1
  • 1
Zhuiguang Liu
  • 510
  • 5
  • 8
3

What I've found from my use, is that "go test" runs a single package's test cases sequentially (unless t.Parallel() is called), but if you supply multiple packages (go test ./foo ./bar ./baz), each package's tests are run parallel to other packages. Definitely caused similar headaches with database testing for me.

justinas
  • 6,287
  • 3
  • 26
  • 36