Spring batch item reader reading from multiple databases

Question

I have an Item object as below:

class Item{
   private String idProperty;
   private String prop1Db1;
   private String prop2Db1;
   private String prop1Db2;
   private String prop2Db2;

   // getters and setters here
  }

Now, for a given idProperty, two of the properties comes from one database, while other two properties comes from another database (as clear from the property names). The tables in both the databases have idProperty as the common property.

One approach I could take is read the first three properties from the first database. Pass the incomplete Item to a Processor and fire a query in the Processor to get other two properties to complete the Item before passing it to Writer.

However, for a huge number of records (millions), it will be a huge performance hit just with the number of queries I will have to fire in the Processor.

Is there any way to do it in the Reader itself ? A Reader which reads from two database simultaneously to create an Item before sending it to Processor and Writer ahead ?

possible duplicate of http://stackoverflow.com/questions/21304364/spring-batch-job-read-from-multiple-database — Karthik Prasad, Sep 05 '14 at 05:54
@KarthikPrasad: No, its not a duplicate. The question you link reads same kinds of record from two databases, say a Book object. While in my case the properties of a single Book object is divided across two databases. — Vicky, Sep 05 '14 at 12:41

score 0 · Accepted Answer · answered Sep 05 '14 at 06:43

Take this answer as a 'workaround' because I think more elegant solutions are around the corner...
You should change your reader/processor/writer let them return a List<Item> instead of a single Item object; in this way in your processor you can fire a single query using the SQL IN() fetching Item.idProperty from processing List<>.
According to List<>'s size you can reduce the amount of fired queries.

Spring batch item reader reading from multiple databases

1 Answers1