Is there a way to improve starlark's unittest error reporting?

Question

I'm writing analysis-time tests for my project as per https://docs.bazel.build/versions/master/skylark/testing.html and I'm wondering what the most useful way of reporting failures is.

Using the unittest.bzl module, I am using asserts.equals, asserts.true etc and find the error reporting in the logs somewhat lacking. For example, if an asserts.true fails the error message is Expected condition to be true, but was false, with no mention of which line, or what the condition it expected to be true was. In a file full of lots of tests, this isn't very useful! I'm aware one can add a message as an argument to these assertions, but having tailored messages for every assertion doesn't feel ideal either. Is there a way to get at the backtrace caused by the assertion failure at all? Or any other way of accessing the line number/details of assertion failure?

Laurenz · Answer 1 · 2021-02-06T22:16:39.030

I took the minimal example from your link and added the latest Skylib release to a new WORKSPACE. I then changed the expected value in _provider_contents_test_impl to make the test fail.

Below is the full output, note the DEBUG that contains a lot of useful info (file, line, expectation, actual value).

$ bazel test //mypkg:myrules_test                                             
DEBUG: /home/user/.cache/bazel/_bazel_user/863abec759a50d843603ddf033727331/external/bazel_skylib/lib/unittest.bzl:351:10: In test _provider_contents_test_impl from //mypkg:myrules_test.bzl: Expected "some valuexxxxx", but got "some value"
INFO: Analyzed target //mypkg:provider_contents_test (0 packages loaded, 0 targets configured).
INFO: Found 1 test target...
FAIL: //mypkg:provider_contents_test (see /home/user/.cache/bazel/_bazel_user/863abec759a50d843603ddf033727331/execroot/__main__/bazel-out/k8-fastbuild/testlogs/mypkg/provider_contents_test/test.log)
Target //mypkg:provider_contents_test up-to-date:
  bazel-bin/mypkg/provider_contents_test.sh
INFO: Elapsed time: 0.108s, Critical Path: 0.04s
INFO: 2 processes: 2 linux-sandbox.
INFO: Build completed, 1 test FAILED, 2 total actions
//mypkg:provider_contents_test                                           FAILED in 0.0s
  /home/user/.cache/bazel/_bazel_user/863abec759a50d843603ddf033727331/execroot/__main__/bazel-out/k8-fastbuild/testlogs/mypkg/provider_contents_test/test.log

INFO: Build completed, 1 test FAILED, 2 total actions

Tracebacks are also supported, see this bug report for example.

tion, actual value).

$ bazel test //mypkg:myrules_test                                             
DEBUG: /home/user/.cache/bazel/_bazel_user/863abec759a50d843603ddf033727331/external/bazel_skylib/lib/unittest.bzl:351:10: In test _provider_contents_test_impl from //mypkg:myrules_test.bzl: Expected "some valuexxxxx", but got "some value"
INFO: Analyzed target //mypkg:provider_contents_test (0 packages loaded, 0 targets configured).
INFO: Found 1 test target...
FAIL: //mypkg:provider_contents_test (see /home/user/.cache/bazel/_bazel_user/863abec759a50d843603ddf033727331/execroot/__main__/bazel-out/k8-fastbuild/testlogs/mypkg/provider_contents_test/test.log)
Target //mypkg:provider_contents_test up-to-date:
  bazel-bin/mypkg/provider_contents_test.sh
INFO: Elapsed time: 0.108s, Critical Path: 0.04s
INFO: 2 processes: 2 linux-sandbox.
INFO: Build completed, 1 test FAILED, 2 total actions
//mypkg:provider_contents_test                                           FAILED in 0.0s
  /home/user/.cache/bazel/_bazel_user/863abec759a50d843603ddf033727331/execroot/__main__/bazel-out/k8-fastbuild/testlogs/mypkg/provider_contents_test/test.log

INFO: Build completed, 1 test FAILED, 2 total actions

Tracebacks are also supported, see this bug report for example.

Note that asserts.equals will print the actual value on failure, but asserts.true will not - but you can easily work around this by using asserts.equals(env, True, test_val, msg) instead.

Is there a way to improve starlark's unittest error reporting?

1 Answers1