Questions tagged [kafka-join]
21 questions
4
votes
1 answer
Kafka Stream work with JoinWindow for data replay
I have 2 streams of data and I want to be able to join them for a window of 1 month let's say. When I have a live data everything is fun and super easy with KStream and join. I did something like this;
KStream stream1 =
…
Am1rr3zA
- 7,115
- 18
- 83
- 125
2
votes
1 answer
Kafka Streams API: I am joining two KStreams of empmodel
final KStream empModelStream = getMapOperator(empoutStream);
final KStream empModelinput = getMapOperator(inputStream);
// empModelinput.print();
// empModelStream.print();
empModelStream.join(empModelinput,…
Abhinav
- 41
- 5
2
votes
2 answers
Kafka stream join with a specific key as input
I have 3 different topics with 3 Avro files in schema registry, I want to stream these topics and join them together and write them into one topic. the problem is the key I want to join is different with the key I write the data into each…
Am1rr3zA
- 7,115
- 18
- 83
- 125
1
vote
1 answer
Kafka streams join duplicates
Please don't mark this question as a duplicate of kafka-streams join produce duplicates. I think my scenario is different. I'm also already using kafka EOS via processing.guarantee=exactly_once
I have an input topic transactions_topic with json…
slowratatoskr
- 11
- 1
1
vote
1 answer
How to use Kafka Streams to only process messages sent less than 3 times per key (UserId) over a given time window
I want to limit the number of message sent to the User over a given time period. I have topic of message to be sent and I can get a count of message per key (UserId) over a time window, but if I try and join that count KTable to a KStream of the…
user13517253
- 13
- 4
1
vote
0 answers
Kafka streams join: How to wait a duration of time before emitting records?
We currently have 2 Kafka stream topics that have records coming in continuously. We're looking into joining the 2 streams based on a key after waiting for a window of 5 minutes but with my current code, I see records being emitted immediately…
user10751899
- 147
- 1
- 9
1
vote
1 answer
Kafka- join KStream and KTable on composite key
I have two topics- plan and supplier
plan topic has a composite key (avro) based on two columns, planCode + memberAge.
supplier topic is keyed on column supplierId and it contains the column planCode but not memberAge.
KStream…
Ishani Vij
- 138
- 1
- 12
1
vote
1 answer
Can't we join two tables and fetch data in Kafka?
I have joined two tables and fetched data using Postgres source connector. But every time it gave the same issue i.e.
I have run the same query in Postgres and it runs without any issue. Is fetching data by joining tables not possible in Kafka?
Sajita
- 41
- 5
1
vote
2 answers
how to handle one to many relationship using kafka streams join operations
can you please help me how to achieve this using Kafka streams?
Scenario: Grouping all the invoices for an order data. In real-time streaming, there might be a delay in receiving invoices. so we want to wait 20 mins to group all the invoices before…
Devika Elangovan
- 11
- 1
- 6
1
vote
1 answer
Kafka - Joining Data from two different Streams when Data comes at different times
I have a scenario where we have two different streams and we get data on them at two different times and i need to join them on the basis of the timestamp that is there in the value. I will try to explain through below example.
inputStream1 ->
key…
Shashank
- 11
- 3
1
vote
0 answers
Embedded Kafka: KTable+KTable leftJoin produces duplicate records
I come seeking knowledge of the arcane.
First, I have two pairs of topics, with one topic in each pair feeding into the other topic. Two KTables are being formed by the latter topics, which are used in a KTable+KTable leftJoin. Problem is, the…
Freestyle076
- 1,548
- 19
- 36
0
votes
0 answers
KStream join with KTable record drops if key not exist in KTable
I have a use case in which I don't want to drop the record from the KStream if KTable key doesn't exist & I want to wait for the KTable key (record) to arrive & then sending fully populated data to a new kafka topic.
I have found a reference in…
Sharry India
- 341
- 1
- 9
0
votes
1 answer
Tombstone records not processed in Kstream-Ktable join
I have this kstream topology:
KStream eventStream = streamsBuilder.stream("event",
Consumed.with(Serdes.String(), eventSerde);
KTable resultKTable = streamsBuilder.table("result",
…
cppcoder
- 22,227
- 6
- 56
- 81
0
votes
1 answer
How to implement KStream-Ktable leftJoin, how to get and set the field by using Envelope object and implement the join for KStream-Ktable?
If I have one topic schema (that is Kstream):
{
"type": "record",
"name": "Value",
"namespace": "test1",
"fields": [
{
"name": "id",
"type": {
"type": "long",
…
Ragini Gupta
- 147
- 1
- 6
0
votes
1 answer
Relative amount of events in kafka streams using join
I'm writing a kafka streams app in which I'm producing statistics for web pages.
I have a stream of information about web pages which includes the page type (news, gaming, blog, etc.) and the page language (en, fr, ru, etc.) in a struct.
I've…
meirgold
- 83
- 6