1

Problem Statment:

Topic 1: "key = empId, value = empname, deptName, ..."

Topic 2: "key = deptName, value = deptName"

I need the data from Topic 1 where the deptName(value attribute in topic 1) is equal to key of Topic 2.

Steps:

  1. Create a stream from Topic 1, group it by deptName, and do aggregatation. It will return Ktable (key =deptName, value = "empId1,empId2,empId3 ..")
  2. Create a stream from Topic 2 (key ="deptName" value = "deptName")
  3. Perform a left join operation on Ktable (Step 1) and KSteam (Step2). (KStream-Ktable)
  4. And join returns desired result.

Everything works as expected in single partition, however, after switching to multiple partitions, join doesn't return any data.

Step 1:

KGroupedStream<String, Object> groupedStream = adStream.groupBy((key, value) -> value.getOrganizationId().toString());
        groupedStream
                .aggregate(() -> (new String()),
                        (aggKey, newValue, aggValue) -> addCurrentValue(aggValue,
                                String.valueOf(newValue.getOriginId())),
                        Materialized.as("aggregated-stream-store").with(strSerde, strSerde))
                .toStream().to(Constant.AD_AGGREGATED_DATA, Produced.with(strSerde, strSerde));

Step 2:

KStream<String, String> swgOrgStream = builder.stream(Constant.SWG_ORG_TOPIC,Consumed.with(strSerde, strSerde));

Step 3:

 KStream<String, String> filteredOrgStream = swgOrgStream.leftJoin(aggregatedTable,
                (leftValue, rightValue) -> rightValue);
Vijay
  • 11
  • 1
  • I see that in step 1 you are grouping using ` value.getOrganizationId().` is that correct? – Tuyen Luong Apr 26 '20 at 07:38
  • Why are you reading the second topics as `KStream`? Seems it would be better to read it as a `KTable` and do table-table join instead? -- In fact, you would even read the first topic as a `KTable`, too (instead of a stream and an aggregation) and do FK-table-table join? – Matthias J. Sax Apr 26 '20 at 19:24
  • I tried FK table join , but the approach didn't give the expected result. – Vijay May 08 '20 at 20:09
  • @TuyenLuong, yes thats correct. – Vijay May 22 '20 at 18:28
  • @MatthiasJ.Sax, Thanks for your opinion. I can try that out. However, do you see any issue in implemetation? – Vijay May 22 '20 at 18:29

0 Answers0