2

I understand that in order to avoid cached results in Go tests you can use the -count=1 flag in the go test command, but why?

This is from the docs:

The idiomatic way to disable test caching explicitly is to use -count=1

The explanation for the count flag is:

-count n
    Run each test, benchmark, and fuzz seed n times (default 1).
    If -cpu is set, run n times for each GOMAXPROCS value.
    Examples are always run once. -count does not apply to
    fuzz tests matched by -fuzz.

It doesn't say anything about caching and the default value is 1, but skipping cached tests aren't ignored by default.

asaf92
  • 1,557
  • 1
  • 19
  • 30
  • What do you mean by "skipping cached tests aren't ignored by default"? – jub0bs Aug 21 '22 at 08:30
  • Because it‘s documented. (Makes forcing to run the tests simpler.) – Volker Aug 21 '22 at 08:44
  • Notice the "(default 1)" part. It says the default `-count` value is "1", so does that mean that when I don't use the `count` flag than the `count` flag is set to `1`? – asaf92 Aug 21 '22 at 08:44

1 Answers1

4

The simple answer is because this is how the go tool is written.

The reasoning is: Test outputs are cached to speed up tests. If the code doesn't change, the test output shouldn't change either. Of course this is not necessarily true, tests may read info from external sources or may use time and random related data which could change from run to run.

When you request multiple test runs using the -count flag, obviously the intention is to run the tests multiple times, there's no logic running them just once and show n-1 times the same result. So -count triggers omitting the cached results. -count=1 will simply cause running the tests once, omitting previous cached outputs.

icza
  • 389,944
  • 63
  • 907
  • 827
  • 1
    Isn't `count=1` by default? What I mean is that if you don't set this flag, the `count` value would be `1` internally, right? – asaf92 Aug 21 '22 at 09:01
  • @asaf92 The default is `1`, but there's a difference between explicitly providing the flag, or using the default value because the flag is not provided. – icza Aug 21 '22 at 10:53
  • @icza Can you clarify? It's illegal to use `-count` without providing it a value. – jub0bs Aug 21 '22 at 19:26
  • @jub0bs Since `count` is not a `bool` flag but an integer, _if_ you provide it, you have to provide its value too. The default value means if you don't provide it, the default value will be loaded into the variable associated with it. But there's a difference between providing it with the default value, or not providing it at all: in the first case cached outputs will not be used, and in the latter case they will be. – icza Aug 21 '22 at 19:50
  • @icza I still don't understand. I know that's the behaviour, but to me, the doc should say the default value is `0`, not `1`. – jub0bs Aug 22 '22 at 07:13
  • 2
    @jub0bs If the default would be `0`, that would mean to not run the tests. – icza Aug 22 '22 at 07:23
  • @icza I get it now. Thanks for your patience. – jub0bs Aug 22 '22 at 07:34
  • @icza This is extremely weird and unexpected. Why would anyone assume that there's a difference in behavior between explicitly using a default value to implicitly using a default value? I honestly never seen any program that does that. The fact that they would go through the trouble of implement this strange behavior instead of adding a simple bool flag called `--no-cache` or something is beyond me. – asaf92 Aug 28 '22 at 13:14