0

I would like to consistently apply a custom RicherIndicatorCoder for my case class RicherIndicator. Moreover if I fail to provide a new Coder for Tuples or KVs containing RicherIndicator then I would like to obtain a compile-time or runtime error rather than fall back on a suboptimal coder.

However Scio does not seem to honor the @DefaultCoder annotation:

@DefaultCoder(classOf[RicherIndicatorCoder]) // Ignored
case class RicherIndicator (
  policy: Policy,
  indicator: Indicator
)

Nor does Scio give priority to custom coders registered with the CoderRegistry, instead falling back on its own default coder:

val registry = sc.pipeline.getCoderRegistry
registry.registerCoderForClass(classOf[RicherIndicator], RicherIndicatorCoder.of) // Not used

Therefore I must use setCoder(RicherIndicatorCoder.of) wherever an SCollection of this type appears, and carefully comb through the pipeline in case there are composite types which include a RicherIndicator.

Is there a way to set my custom coder as the default, or to disable falling back on the default Magnolia or Kryo based coder?

Ralph Gonzalez
  • 69
  • 7

1 Answers1

0

Java annotation does not work in Scala. You can wrap your Beam Coder as an implicit Scio Coder like this:

implicit val richIndicateCoder = Coder.beam(RicherIndicatorCoder)

It should be picked up as long as the implicit is in scope.

Neville Li
  • 420
  • 3
  • 10
  • I was under the impression that Scala supports Java annotation, so was thinking this was an issue with Scio having its own concept of coder defaults and prioritization? – Ralph Gonzalez Jun 13 '19 at 19:22
  • Sorry I was thinking annotation processor. In this case yeah it's most likely Scio's coder resolution takes precedence, and/or the annotation doesn't work well on Scala classes. – Neville Li Jun 13 '19 at 23:24