2

there is one case to handle more than 3M records which in the flat file, and we want to read 3k records one time and then process it to writer, is there any better way to process that? beside that, we have some other conditions, since our flat file format like below:

userid1, transaction1 userid1, transaction2 userid1, transaction3 userid2, transaction1 userid2, transaction3

one user may have many transactions, the item reader will read the transaction for one user to different threads, we want to put all the transaction for this user in the same thread, so we want to dynamically change the chunksize, so is there any better way for that?

Paul Samsotha
  • 205,037
  • 37
  • 486
  • 720
ShawnYJG
  • 23
  • 3

1 Answers1

0

To answer at your question:
yes, you can change commit-size using scope step and properties substitution with #{} syntax or declaring a custom completion-policy.

But I think you don't need to change commit-size but implements a custom reader that build up a domain object with all transactions for a single user.

class UserTransaction {
  String transactionId;
}
class UserWithTransaction {
  String user;
  List<UserTransaction> transactions;
}

class UserWithTransactionReader implements ItemReader<UserWithTransaction> {...}

There are a lot of custom reader implementation around the net (and here in SO) that build a complex object from a CSV file (like here or here)

Community
  • 1
  • 1
Luca Basso Ricci
  • 17,829
  • 2
  • 47
  • 69
  • How to mapping the Object UserWithTransactions? – ShawnYJG Sep 10 '14 at 07:32
  • In the AggregatePeekableReader, where do we define the method "hasSameInvoiceNumberAndVendorNumber" for T? – ShawnYJG Sep 10 '14 at 07:34
  • I gave you an idea about how to achieve your goal; I can't write code for you because I don't known you domain data,file format,business concern and more. Check http://stackoverflow.com/questions/7836689/spring-batch-multi-line-record-item-writer-with-variable-number-of-lines-per-rec and SB samples and do some tries. – Luca Basso Ricci Sep 10 '14 at 07:41
  • Thank you Ricci, I have thought out the way to resolve the cases. – ShawnYJG Sep 10 '14 at 09:29