I am trying to leftJoin events from 2 streams. Initially, I joined 2 KStreams and everything was working fine. However, when I try to convert the second stream to a KTable, I get an error. Here is the code with the 2nd stream transformed to a KTable:
@Bean
public KafkaStreams kafkaStreams() throws IOException {
final Properties props = configureKafkaStreamsProperties();
ObjectMapper mapper = new ObjectMapper();
mapper.disable(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES);
final StreamsBuilder builder = new StreamsBuilder();
// 1st Structured stream
KStream<String, String> firstStream = builder.stream("topic-1", Consumed.with(Serdes.String(), Serdes.String()));
KStream<String, String> firstStreamTransformed = firstStream.map((k, v) -> {
try {
InputModelOne model = mapper.readValue(v, InputModelOne.class);
return new KeyValue<>(model.getId(), v);
} catch (Exception e) {
logger.error(e.toString());
return new KeyValue<>(k, v);
}
});
// Second stream
KStream<String, String> secondStream = builder.stream("topic-2",
Consumed.with(Serdes.String(), Serdes.String()));
KStream<String, String> secondStreamTransformed = secondStream.map((k, v) -> {
try {
InputModelTwo model = mapper.readValue(v, InputModelTwo.class);
return new KeyValue<>(model.getId(), v);
} catch (Exception e) {
logger.error(e.toString());
return new KeyValue<>(k, v);
}
});
// Build KTable from second topic
KTable<String, String> secondTable = secondStreamTransformed.toTable(Materialized.as("topic-2-table"));
// Valuejoiner
ValueJoiner<String, String, String> joiner = (one, two) -> {
try {
InputModelOne modelOne = mapper.readValue(one, InputModelOne.class);
InputModelTwo modelTwo = new InputModelTwo();
// Create output object with properties
OutputModel out = new OutputModel(modelOne.getId());
out.setOneTimestamp(modelOne.getTimestamp());
out.setTwoTimestamp(modelTwo.getTimestamp());
return mapper.writeValueAsString(out);
} catch (JsonProcessingException e) {
// TODO Auto-generated catch block
e.printStackTrace();
return null;
}
};
KStream<String, String> joined = firstStreamTransformed leftJoin(secondTable,
joiner);
joined.to("joined-topics", Produced.with(Serdes.String(), Serdes.String()));
Here is the error:
org.apache.kafka.streams.errors.ProcessorStateException: Error opening store joined-topics at location /var/folders/lx/dz_x9j5d7lz4mfymgzkcn7wr0000gn/T/kafka-streams/streams-pipe/2_0/rocksdb/joined-topics
at org.apache.kafka.streams.state.internals.RocksDBTimestampedStore.openRocksDB(RocksDBTimestampedStore.java:87) ~[kafka-streams-2.8.0.jar:na]
at org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:186) ~[kafka-streams-2.8.0.jar:na]
at org.apache.kafka.streams.state.internals.RocksDBStore.init(RocksDBStore.java:254) ~[kafka-streams-2.8.0.jar:na]
at org.apache.kafka.streams.state.internals.WrappedStateStore.init(WrappedStateStore.java:55) ~[kafka-streams-2.8.0.jar:na]
at org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStore.init(ChangeLoggingKeyValueBytesStore.java:55) ~[kafka-streams-2.8.0.jar:na]
at org.apache.kafka.streams.state.internals.WrappedStateStore.init(WrappedStateStore.java:55) ~[kafka-streams-2.8.0.jar:na]
at org.apache.kafka.streams.state.internals.CachingKeyValueStore.init(CachingKeyValueStore.java:75) ~[kafka-streams-2.8.0.jar:na]
at org.apache.kafka.streams.state.internals.WrappedStateStore.init(WrappedStateStore.java:55) ~[kafka-streams-2.8.0.jar:na]
at org.apache.kafka.streams.state.internals.MeteredKeyValueStore.lambda$init$1(MeteredKeyValueStore.java:122) ~[kafka-streams-2.8.0.jar:na]
at org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl.maybeMeasureLatency(StreamsMetricsImpl.java:884) ~[kafka-streams-2.8.0.jar:na]
at org.apache.kafka.streams.state.internals.MeteredKeyValueStore.init(MeteredKeyValueStore.java:122) ~[kafka-streams-2.8.0.jar:na]
at org.apache.kafka.streams.processor.internals.ProcessorStateManager.registerStateStores(ProcessorStateManager.java:201) ~[kafka-streams-2.8.0.jar:na]
at org.apache.kafka.streams.processor.internals.StateManagerUtil.registerStateStores(StateManagerUtil.java:103) ~[kafka-streams-2.8.0.jar:na]
at org.apache.kafka.streams.processor.internals.StreamTask.initializeIfNeeded(StreamTask.java:216) ~[kafka-streams-2.8.0.jar:na]
at org.apache.kafka.streams.processor.internals.TaskManager.tryToCompleteRestoration(TaskManager.java:433) ~[kafka-streams-2.8.0.jar:na]
at org.apache.kafka.streams.processor.internals.StreamThread.initializeAndRestorePhase(StreamThread.java:849) ~[kafka-streams-2.8.0.jar:na]
at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:731) ~[kafka-streams-2.8.0.jar:na]
at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:583) ~[kafka-streams-2.8.0.jar:na]
at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:556) ~[kafka-streams-2.8.0.jar:na]
Caused by: org.rocksdb.RocksDBException: Column family not found: keyValueWithTimestamp
at org.rocksdb.RocksDB.open(Native Method) ~[rocksdbjni-6.29.4.1.jar:na]
at org.rocksdb.RocksDB.open(RocksDB.java:306) ~[rocksdbjni-6.29.4.1.jar:na]
at org.apache.kafka.streams.state.internals.RocksDBTimestampedStore.openRocksDB(RocksDBTimestampedStore.java:75) ~[kafka-streams-2.8.0.jar:na]
... 18 common frames omitted
I am using Docker for Kafka and Zookeeper and the Kafka is running locally. Any help or suggestions will be greatly appreciated. I do hope I can keep on using my Mac, instead of switching to a lower quality display. Cheers guys !
