Spring Batch configure Multiple Readers/Processors/Writers for single JOB based on JobParameters

Question

I have a spring batch application configured as per below link.

https://spring.io/guides/gs/batch-processing/

Now I have to configure step in such a way that it has to take FlatFileItemReader if user input is file and JdbcPagingItemReader if the input is SQL. I would pass the inputs(file/sql) as JobParameters.

From my understanding, everything in the above example by spring are singleton beans which are loaded into ApplicationContext when the application is started. As step can be configured only once with one Reader. How would I configure it to take different Reader's based on user input

I don't prefer creating multiple jobs if only the reader is changing.

I thought of using Factory/Strategy patterns but this are only achievable if Step is not a bean. Here all of them are beans which are loaded during application startup.

Regardless of patterns, a solution to use different Readers in step based on JobParameters would be helpful.

score 4 · Answer 1 · answered Nov 08 '17 at 18:54

Your assumtion about the Beans is correct.

The fastest solution would be to just create two jobs, if you only have two flavors. But let's ignore reality and talk about theory:

You could create two steps, one for each flavor, and use a JobExecutionDecider to either do one or the other step (see docs).

Or you create your own ItemReader and let it create a delegate reader dynamically. If you use ItemStreamSupport or AbstractItemCountingItemStreamItemReader as base class, you get an open(ExecutionContext executionContext) method.

Sample code:

public class TicTacReader extends ItemStreamSupport implements ItemReader<TicTac>{

  protected ItemReader<TypedFieldSet<RecordType>> delegate;

  public TicTacReader() {
    super();
    setName(TicTacReader.class.getName());
  }

  @Override
  public void open(ExecutionContext executionContext) throws ItemStreamException {
    super.open(executionContext);
    // TODO: use the appropriate reader
    this.delegate = new VoodooReader();
  }

      @Override
  public TicTac read() throws Exception, UnexpectedInputException, ParseException, NonTransientResourceException {
    TypedFieldSet<RecordType> line = this.delegate.read();
    // TODO ...
 }
}

I don't feel right about creating multiple steps either as this would keep increasing steps when there are more readers/processor's & writers in future. I implemented the ItemStreamReader as per your solution for reader. This resolved the issue with Multiple readers. Is there a solution for processors and writer as well by avoid step & job creation???? — karthik, Nov 09 '17 at 00:43
I think there is a compositeItemProcessor and something in writer should be available to do this. I will update once I perform poc on processor and writer. Thanks for for head start & solution @dube — karthik, Nov 09 '17 at 01:08
Random hints from my latest batch project (about 10 files with ~15 record types): Create an configuration class for each step, then it doesn't hurt that much if you have similar (sub-)steps and flows. Naming them beans with some suffix also helped not to accidentally link the sub-step in the global job configuration. — dube, Nov 10 '17 at 10:17

score 2 · Answer 2 · answered Nov 09 '17 at 11:19

You can implement this by using @StepScope. Put your bean name i.e FlatFileItemReader / or SQL in job parameters then use below code

sample implementation

java config way

    @StepScope
        public Step step(@Value("#{jobParameters['beanId']}") final String beanId) {
        return stepBuilderFactory.get("step")
                .<POJO, POJO> chunk(10000)
                .reader(getReader(beanId))
                .processor(Processor)
                .writer(Writer)             
                .build();
     }


 getReader(String beanId)
     {
        return applicationContext.getBean(beanId);
     }

XML

<batch:chunk reader="#{jobParameters['beanId']}" writer="writer" processor="processor" commit-interval="100" />

score 0 · Answer 3 · answered May 13 '19 at 09:59

Remember that Spring configs (hence beans) can be conditioned to Spring profiles, environment variables, application context beans presence, properties, etc.

@Profile
@ConditionalOnBean @ConditionalOnMissingBean
@ConditionalOnClass @ConditionalOnMissingClass
@ConditionalOnExpression (SpringEL)
@ConditionalOnJava (JVM version)
@ConditionalOnJndi
@ConditionalOnProperty
@ConditionalOnResource
@ConditionalOnWebApplication @ConditionalOnNotWebApplication
@ConditionalOnEnabledEndpoint (see Actuator)

So you should be able to filter your factories (through @Configuration classes and @Bean methods as well), depending on the execution context.

Spring Batch configure Multiple Readers/Processors/Writers for single JOB based on JobParameters

3 Answers3