1

I'm new on Scala and I'm trying to filter a KStream[String, JsonNode] based on the second component fields.

As a example, the working Java code is this:

import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.Consumed;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.Produced;
...
import com.fasterxml.jackson.databind.JsonNode;
...
...
final KStream<String, JsonNode> source = streamsBuilder.stream(inputTopic,
                    Consumed.with(Serdes.String(), jsonSerde));


// filter and producer preprocessed
source.filter((k, v) -> v.get("total_cost").asDouble() > 0 && v.get("num_items").asInt() > 0)
    .to(outputTopic, Produced.with(Serdes.String(), jsonSerde));

I have tried this:

import org.apache.kafka.streams.kstream.{Produced,Consumed,KStream};
import org.apache.kafka.streams.StreamsBuilder;
...
import com.fasterxml.jackson.databind.JsonNode;
...
var source:KStream[String, JsonNode] = streamsBuilder.stream(inputTopic, Consumed.`with`(Serdes.String(), jsonSerde));

source.filter({
  case (k:String ,v:JsonNode) => 
    (v.get("total_cost").asDouble() > 0 && v.get("num_items").asInt() > 0)
  })
  .to(outputTopic, Produced.`with`(Serdes.String(), jsonSerde));

In the above try i got:

Description Resource Path Location Type missing parameter type for expanded function The argument types of an anonymous function must be fully known. (SLS 8.5) Expected type was: org.apache.kafka.streams.kstream.Predicate[? >: String, ? >: com.fasterxml.jackson.databind.JsonNode]

I also tried this:

source.filter((_._2.get("total_cost").asDouble() > 0 && _._2.get("num_items").asInt() > 0))
  .to(outputTopic, Produced.`with`(Serdes.String(), jsonSerde));

How could i filter this object in Scala? Thanks in advance.

bottaio
  • 4,963
  • 3
  • 19
  • 43
Ángel Igualada
  • 891
  • 9
  • 13
  • Kafka Streams offers a first class Scala API. Check out the docs on how to use it: https://kafka.apache.org/24/documentation/streams/developer-guide/dsl-api.html#scala-dsl – Matthias J. Sax Mar 28 '20 at 19:40

1 Answers1

3

org.apache.kafka.streams.kstream.Predicate comes from JavaAPI, in Scala 2.11 (I assume you use it) you have to implement interface explicitly, thus:

source.filter(new Predicate[String, JsonNode]() {
  override def test(k: String, v: JsonNode): Boolean = {
    v.get("total_cost").asDouble() > 0 && v.get("num_items").asInt() > 0
  }
}).to(outputTopic, Produced.`with`(Serdes.String(), jsonSerde));

should work.

More on SAMs (Single Abstract Methods) can be found here

Note that you don't have to use Java API - there is first-class Scala API.

bottaio
  • 4,963
  • 3
  • 19
  • 43