SpringBatch Reading and Writing in chunks

Question

I have a program that contains some for-loops. The idea of the program is to log into a website using multiple accounts and retrieve a list (each login brings a different list). So the way I have it setup is with an enhanced for loop:

loginsList.put( "firstUsername", "firstPassword" );
loginsList.put( "secondUsername", "secondPassword" );
loginsList.put( "thirdUsername", "thirdPassword" );
loginsList.put( "fourthUsername", "fourthPassword" );
loginsList.put( "fifthUsername", "fifthPassword" );

for ( Entry<String, String> nextLogin : logins.entrySet() ) {
    String nextUser = nextLogin.getKey();
    String nextPass = nextLogin.getValue();

    Response authenticateUserResponse = Jsoup.connect( WEBSITE_I_NEED_TO_LOGIN_TO )
            .data( "username", nextUser )
            .data( "password", nextPass )
            .execute();

Basically here is what i want the flow to be:

read()--> obtain list----> send list to write() method to write it to the database--> loop back around and get the next login-->read()--> obtain list-->send it to the write()....etc..

however the issue I'm having is that my loop runs in the read method and does not go to the write method until all the lists have been traversed in all the accounts. Essentially the write is only being called once at the end, so what I have right now is something like this(this is the flawed design):

read()--->obtain list-->next account--->obtain list---next account--->obtain list--->write()

How can I organize the chunk processing in Spring to write after I read a chunk only?

olexity · Answer 1 · 2016-07-18T21:15:50.873

0

for ( Entry<String, String> nextLogin : logins.entrySet() ) {
    String nextUser = nextLogin.getKey();
    String nextPass = nextLogin.getValue();
     //do something
       ......
     //call write function 
writeValues(x, y, z); 
}

Is this all you want? Otherwise it seems like a traditional SpringBatch: Read > Process > Proceeed case. You will have your reader = gets a record Procesor > saves a record

Spring batch moves you to next record if the was no error.

    <step id="processUpdates">
        <tasklet task-executor="batchThreadPoolTaskExecutor" throttle-limit="${batch.cviscoreupdate.threadcount}">
            <chunk reader="Reader" processor="ItemProcessor" writer="ItemWriter" commit-interval="${batch.commit.interval}" skip-limit="${batch.skip.limit}" >
                <skippable-exception-classes>
                    <include class="batch.support.SkipRecordException" />
                </skippable-exception-classes>
            </chunk>
        </tasklet>
        <next on="FAILED" to="errorExit"/>          
        <next on="*" to="moveFilesFromWorkToDone" />
        <listeners>
            <listener ref="UpdateSkipListener"/>
        </listeners>
    </step>

<bean id="CVIScoreUpdateItemProcessor" class="com.batch.MyUpdateItemProcessor" scope="step" init-method="init" />

edited Jul 18 '16 at 21:15

answered Jul 18 '16 at 20:33

olexity

105
4

You need a step like this – olexity Jul 18 '16 at 20:54
so you're saying if it saves the record then the writer is called up automatically? – Salman Salman Jul 18 '16 at 21:01
well yes and no, something like you want to do, I would save to DB within the processor. Writer traditionally writes to that record. But ones again, Processor and Writer are classes that are written by you. See edit – olexity Jul 18 '16 at 21:14
right...after I save to the db the writer runs and Im only calling that once – Salman Salman Jul 18 '16 at 21:28
Yes, so put it into processor, processor runs for every record in your record set. – olexity Jul 19 '16 at 14:24

SpringBatch Reading and Writing in chunks

1 Answers1