0

I'm using Spring Batch with Mongo database.

I need to get the documents based on status (status=PENDING), write in a Kafka queue and update documents field with new status(status=FILLED).

So I used a MongoItemReader, CompositeItemWriter (KafkaItemWriter, MongoItemWriter) to write in to Kafka queue and update it. But when I run the job I can see that some documents are skipped, and the number of skipped documents is equal to the chunk size.

For example, my collection has 15 documents, chunk size equals to 5 and the MongoItemReader reads the lines 1,2,3,4,5 then lines 11,12,13,14,15 (skipping the lines 6,7,8,9,10).

My ItemProcessor that performs modification to the POJOs Entities. Since MongoItemReader made a flush between each reading, the Entities are updated. But it seems that the cursor paging is also incremented (as can be seen in the log: row ID 6, 7, 8, 9 and 10 have been skipped). I have seen it has a work around in Relational Database by using "SqlPagingQueryProviderFactoryBean" but it not suites for NoSQL DB. I had tried everything to find the solution but nothing helped me.

So, how can I manage this issue ?

AlexZeDim
  • 3,520
  • 2
  • 28
  • 64
Naveen
  • 1
  • Welcome to StackOverflow community, please read [this guide](https://stackoverflow.com/help/how-to-ask) before asking and also [try to format](https://stackoverflow.com/editing-help) your question is more appropriate way. – AlexZeDim Aug 03 '20 at 18:09
  • 1
    maybe you can check this https://stackoverflow.com/a/45890047 for an answer – solujan Aug 03 '20 at 19:45
  • Does this answer your question? [Spring MongoItemReader not reading all records on single execution](https://stackoverflow.com/questions/45605056/spring-mongoitemreader-not-reading-all-records-on-single-execution) – Mahmoud Ben Hassine Aug 14 '20 at 08:33
  • Yes Mahmoud, I have tried the same way and succeeded. I would like to see is any inbuilt method in MongoItemReader, unfortunately not. Thank you all for the response – Naveen Aug 18 '20 at 18:40

0 Answers0