1

I'm building some tests around my pipeline and particularly I have two branches (one where errors are considered, another where successes), on the errors side I have an incrementing counter (ScioMetrics.counter("MetricName").inc()) and when building the tests for the other branch I want to assert that the errors counter is 0.

JobTest.
 ...
.counter(ScioMetrics.counter("MetricName"))(_ shouldBe 0)
 ...

I get a NoSuchelementException when the test runs successfully with a message explaining that the metric wasn't found as it might not have been accessed inside the pipeline, which is ok as I can assert that the exception is raised, but. Shouldn't there be a "nicer" way of testing that?

Thanks!

Carlos
  • 2,883
  • 2
  • 18
  • 19
  • Are the counter objects initialized inside of a transform step? If they are, it might help to move initialization outside of the pipeline. Assign the counter object to a variable and call `inc()` on the variable inside the transform. – Andrew Nguonly Mar 04 '18 at 02:06
  • I tried setting it to a `val` within the `main` method but before any processing code and then just calling `inc()` within the transform as you suggest, but same result. – Carlos Mar 05 '18 at 10:54

1 Answers1

0

Due to the Beam metrics API design, a counter that's not touched inside a transform will produce NoSuchElementExcception since the name is never registered.

Use ScioContext#initCounter to workaround this. It basically runs a dummy map transform to initialize each counter to 0. https://github.com/spotify/scio/blob/master/scio-core/src/main/scala/com/spotify/scio/ScioContext.scala#L716

Neville Li
  • 420
  • 3
  • 10