Spring Batch : One Reader, composite processor (two classes with different entities) and two kafkaItemWriter

Question

ItemReader is reading data from DB2 and gave java object ClaimDto. Now the ClaimProcessor takes in the object of ClaimDto and return CompositeClaimRecord object which comprises of claimRecord1 and claimRecord2 which to be sent to two different Kafka topics. How to write claimRecord1 and claimRecord2 to topic1 and topic2 respectively.

Exactly but I am not able to figure out how to add to delegates of two different entities in writer class and get the each entity from the CompositeClaimRecord to write to correspinding topic. — Sonia, Dec 14 '20 at 09:10
1, writer, 2 topics, get the field values, put on topic? Why is that so hard? — M. Deinum, Dec 14 '20 at 09:12
Can you share some example in spring-batch application. I am using KafkaItemWriter class and trying to add two delegates whose key is same but value is different . — Sonia, Dec 14 '20 at 09:16
Don't use the `KafkaItemWriter`. Or use 2 in a composite itemwriter you write yourself. — M. Deinum, Dec 14 '20 at 09:17
I don't think that will make any difference as my challenge is to write two delegates into arraylist and then in write method with parameter List extends CompositeClaimRecord > claimRecs, writing to both topics using delegates. I am getting class casting error as it seems to think that the incoming record is of type CompositeClaimRecord for both teh delagtes but it is claimRecord1 and claimRecord2 for both the topics resp. — Sonia, Dec 14 '20 at 09:32
Why would you need a list. Just inject the 2 kafka writer, writer recrod1 to writer1 and record2 to writer2. Just create a dedicated writer for this. — M. Deinum, Dec 14 '20 at 09:55

score 2 · Accepted Answer · answered Dec 14 '20 at 10:11

Just write a custom ItemWriter that does exactly that.

public class YourItemWriter implements ItemWriter<CompositeClaimRecord>` {

  private final ItemWriter<Record1> writer1;
  private final ItemWriter<Record2> writer2;

  public YourItemWriter(ItemWriter<Record1> writer1, ItemWriter<Record2> writer2>) {
    this.writer1=writer1;
    this.writer2=writer2;
}

  public void write(List<CompositeClaimRecord> items) throws Exception {

    for (CompositeClaimRecord record : items) {
       writer1.write(Collections.singletonList(record.claimRecord1));
       writer2.write(Collections.singletonList(record.claimRecord2));

    }
  }
}

Or instead of writing 1 record at a time convert the single list into 2 lists and pass that along. But error handling might be a bit of a challenge that way. \

public class YourItemWriter implements ItemWriter<CompositeClaimRecord>` {

  private final ItemWriter<Record1> writer1;
  private final ItemWriter<Record2> writer2;

  public YourItemWriter(ItemWriter<Record1> writer1, ItemWriter<Record2> writer2>) {
    this.writer1=writer1;
    this.writer2=writer2;
}

  public void write(List<CompositeClaimRecord> items) throws Exception {

    List<ClaimRecord1> record1List = items.stream().map(it -> it.claimRecord1).collect(Collectors.toList());
    List<ClaimRecord2> record2List = items.stream().map(it -> it.claimRecord2).collect(Collectors.toList());

    writer1.write(record1List);
    writer2.write(record2List);


  }
}

Would you call the YourItemWriter directly in the main step of job or you will add the delegates to the writer and then use them in write method of YourItemWriter ? — Sonia, Dec 14 '20 at 12:10
I'm sorry, I provided the whole implementation of the writer, what isn't clear about that? — M. Deinum, Dec 14 '20 at 12:46
I am able to write to two topic following your code. How to write them in a single transaction ? — Sonia, Jan 21 '21 at 13:51

Mahmoud Ben Hassine · Answer 2 · 2020-12-16T08:37:59.823

0

You can use a ClassifierCompositeItemWriter with two KafkaItemWriters as delegates (one for each topic).

The Classifier would classify items according to their type (claimRecord1 or claimRecord2) and route them to the corresponding kafka item writer (topic1 or topic2).

edited Dec 16 '20 at 08:37

answered Dec 14 '20 at 09:25

Mahmoud Ben Hassine

28,519
3
32
50

Can you share an example? I can see articles only with same entity types not with different. – Sonia Dec 14 '20 at 11:28
Looks like I misunderstood your question because it's unclear. In the title you said `composite processor (two classes with different entities)`, but from the description you seem to be encapsulating things of different types in `CompositeClaimRecord`. A processor that takes `ClaimDto` and returns `CompositeClaimRecord` is a regular processor and not a composite one. A `CompositeItemProcessor` accepts a list of delegates processors and pass items to that chain of processors one at a time. – Mahmoud Ben Hassine Dec 16 '20 at 09:40
1

This answer does not work with your approach based on encapsulation, the answer by @M. Deinum is the way to go. I upvoted it. – Mahmoud Ben Hassine Dec 16 '20 at 09:40

Spring Batch : One Reader, composite processor (two classes with different entities) and two kafkaItemWriter

2 Answers2

Linked