1

I have two topics- plan and supplier

plan topic has a composite key (avro) based on two columns, planCode + memberAge.
supplier topic is keyed on column supplierId and it contains the column planCode but not memberAge.

KStream<String, GenericRecord> supplier = builder.stream(supplier_topic);
KTable<GenericRecord, GenericRecord> planTable = builder.table(plan_topic);

SUPPLIER (KStream) : supplierId -> ....,planCode,....
PLAN (KTable/ GlobalKTable) : {planCode, memberAge} -> value

I want to do a left join of supplier with plan on plan_code. How can I do this?

Ishani Vij
  • 138
  • 1
  • 12

1 Answers1

0

You cannot do this via the streams dsl, as the dsl requires keys to be equal. Actually this is not only an api dsl restriction, but an actual physical requirement that forces keys to coexist locally on the same physical streams client.

To do what you describe I believe is possible only if the input topics of your subtopology have 1 partition and you can do 2 transformer punctuators that are fead the toStream() of each ktable. Each transformer punctuator will manually query the other table to find the matching keys and forward them downstream. It is like a custom inner join.

Practically, you have to make sure the joined tables are part of the same subtopology (you can print your topology to verify this) and the keys you are joining on though different must be on the same partition so that they are executed by the same streams task.

To do that you have to alter the custom partitioner of the producer of your topic that has the composite key, to hash only the first/ common part of the key.Or as mentioned above both input topics to have 1 partition.

Vassilis
  • 914
  • 8
  • 23