0

We have a kTable where we map the values and materialize to KeyValueStore using the following code:

    @Bean
    public KTable<String, CancelEvent> kTable(StreamsBuilder kStreamBuilder,
                                          ValueMapper<CancelEvent, CancelEvent> mapper,
                                          SpecificAvroSerde<CancelEvent> serde) {

      return kStreamBuilder
            .table(properties.getProperty("topics.cancel"), Consumed.with(Serdes.String(), serde))
            .mapValues(mapper, Materialized.as("cancel-event-store"));

    }

And mapper code:

    @Override
    public CancelEvent apply(CancelEvent value) {

      if (value.getData().getCancelled()) {
        return null;
      }

      return value;

    }

We use Avro for serialization and deserialization. When we try to query the state store with a key, it always returns null.

Code for querying the state store:

    ReadOnlyKeyValueStore<String, CancelEvent> store = streamsFactory.getKafkaStreams()
            .store("cancel-event-store", QueryableStoreTypes.keyValueStore());

    CancelEvent event = store.get(queryEvent.getMeta().getId().toString());

event is always null. While debugging, I iterated through the keyValuestore, and realized something is appened to the key (magic byte may be!). Please refer the image below

enter image description here

So, the question is how to query with key?

Update 1

    private Map<String, Object> kStreamsConfigsProperties() {
    final Map<String, Object> config = new HashMap<>();

    config.put(APPLICATION_ID_CONFIG, "app.name" + new Date().getTime());

    config.put(BOOTSTRAP_SERVERS_CONFIG, properties.getProperty("spring.kafka.bootstrap-servers"));
    config.put(AbstractKafkaAvroSerDeConfig.SCHEMA_REGISTRY_URL_CONFIG, properties.getProperty("spring.kafka.schema-registry"));

    config.put(DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass().getName());
    config.put(DEFAULT_VALUE_SERDE_CLASS_CONFIG, SpecificAvroSerde.class.getName());
    config.put(DEFAULT_TIMESTAMP_EXTRACTOR_CLASS_CONFIG, WallclockTimestampExtractor.class.getName());

    return config;
}
Thiru
  • 2,541
  • 4
  • 25
  • 39
  • Did you check the data in KTable if value contains null or data? – Nishu Tayal Nov 08 '18 at 14:18
  • Yes, it has values and its not null – Thiru Nov 08 '18 at 14:29
  • What `Serde` are specified in you config? Because `Materialized` does not specify serdes explicitly, the config ones will be used. Side remark: why not use `KTable#filter()`? – Matthias J. Sax Nov 08 '18 at 22:35
  • Thanks @MatthiasJ.Sac, I have updated the question with config – Thiru Nov 09 '18 at 06:28
  • And how does `KTable#filter()` will solve the issue? I'm making the value as `null`, because it has to be considered `tombstone` message on certain criterial. Pleas correct me if I'm doing something wrong – Thiru Nov 09 '18 at 06:32
  • Your code seems to be correct. Using `#filter()` is just easier to read (and less code) that implementing filter logic within `mapValues()`. – Matthias J. Sax Nov 10 '18 at 01:37
  • The config looks ok. What key do you refer to with `and realized something is appened to the key`? The key that you pass into `store.get()`? What type is `queryEvent` that you use there? Maybe `queryEvent.getMeta().getId().toString()` does something unexpected. What type does `getId()` return? – Matthias J. Sax Nov 10 '18 at 01:44
  • The issue is because of confluent's `kafka-avro-console-producer`. I was generating the event with console producer. Apparently it added something in front with the key. Thanks for the support. – Thiru Nov 12 '18 at 08:19

0 Answers0