10

So I am creating a Spring Batch job for reading a CSV file and for certain rows which contain incomplete data; it checks, outputs to the log that the row is incomplete, and skips. It works great except at the end of the job I want it to log how many rows it found that were incomplete. Just something simple like "X incomplete rows were found".

I've Googled and searched around for a solution but not found anything really.

Any help is appreciated and any more info needed just ask.

Luca Basso Ricci
  • 17,829
  • 2
  • 47
  • 69
dogfight
  • 193
  • 1
  • 1
  • 7
  • 1
    Not much we can tell you about how to change your script if we can't see it. – RGuggisberg Sep 11 '13 at 14:13
  • What do you want to see? I did say if any more info needed just ask. – dogfight Sep 11 '13 at 14:15
  • It's the usual Spring Batch stuff... an ItemProcessor is where it checks for incomplete data. It is Spring Batch, not a batch script. I have a feeling you may have misunderstood the question? http://docs.spring.io/spring-batch/ – dogfight Sep 11 '13 at 14:20
  • At the end of the step you can use the [StepExecution](http://docs.spring.io/spring-batch/apidocs/index.html) to retrieve the different skip counts (read, write, processing). So you basically could write a [StepExecutionListener](http://docs.spring.io/spring-batch/apidocs/index.html) which records this. – M. Deinum Sep 11 '13 at 14:21
  • Cheers I'll do some reading on StepExecutionListener :) Do you have any advice on how/where to store the count from the ItemProcessor so that I can access it from the StepExecutionListener? – dogfight Sep 11 '13 at 14:29
  • Spring Batch already does that for you... it keeps track of reads, writes, processing... So if you do it correctly, Spring Batch will provide you with those numbers... – M. Deinum Sep 11 '13 at 14:34

3 Answers3

14

Spring Batch itself keeps track of how many records it reads, writes, processes and how many it skips (for each of those numbers). That information is stored in the StepExecution. The StepExecution can be accessed from a StepExecutionListener. In this case an implementation of the afterStep method will suffice.

public class SkippedItemStepExecutionListener extends StepExecutionListenerSupport {

    @Override
    public ExitStatus afterStep(StepExecution stepExecution) {
        int skipped = stepExecution.getSkipCount(); // Total for read+write+process
        // Log it to somewhere.        
        return null;
    }
}

How to add it to your job/step is explained in the reference guide

Links

  1. StepExecution javadoc
  2. StepExecutionListener javadoc
  3. Listener Configuration Reference
M. Deinum
  • 115,695
  • 22
  • 220
  • 224
  • Thanks for the reply. I have set up a StepExecutionListener and it can log the number of skipped rows, etc. The thing is that it is just for certain conditions I want it to add to the count, not any skipped row. Just when a certain condition that I check for in the process() method where if it is 'one of those rows' it increments a count and then write it out at the end using the StepExecutionListener. – dogfight Sep 11 '13 at 14:48
  • 1
    If that is the only thing you do in the process method you can use the `getProcessSkipCount()`, whereas the `getSkipCount()` method returns thet total for all skipped records. If you really need something more fine grained you probably are going to need a `SkipListener` instead of add a property yourself to the `ExecutionContext` to keep the state. – M. Deinum Sep 11 '13 at 14:58
  • Managed to get this solved, check my answer. Thanks for the help :) – dogfight Sep 12 '13 at 08:38
9

Manage to solve this, here's how I did it:

In the ItemProcessor I added an attribute and a method for getting access to the ExecutionContext from within the process method,

private ExecutionContext executionContext;

@BeforeStep
public void beforeStep(StepExecution stepExecution)
{
    this.executionContext = stepExecution.getExecutionContext();
}

...and then in the process() method when I find one of the rows I want to log, I can do this,

this.executionContext.putInt( "i_ThoseRows", this.executionContext.getInt( "i_ThoseRows", 0 ) + 1 );

Finally I add another method to the ItemProcessor to print the result at the end of the step,

@AfterStep
public void afterStep(StepExecution stepExecution)
{
    System.out.println( "Number of 'Those rows': " + this.executionContext.getInt( "i_ThoseRows", 0 ) );
}

Hope it helps someone

dogfight
  • 193
  • 1
  • 1
  • 7
0

To complement @dogfight answer:

from spring batch docs:

The annotations are analysed by the XML parser for the elements, so all you need to do is use the XML namespace to register the listeners with a step

So to call the listener callback annotated functions beforeStep() and afterStep() you need to register you ItemProcessor as listener in the step:

<listeners>
    <listener ref="MyItemProcessor">
</listeners>

Otherwise you will have a NullPointerException when use the executionContext.

guilhermerama
  • 750
  • 9
  • 21