Spring boot repository does not save to the DB if called from scheduled job

Question

I have a spring boot application in which I need to schedule a job to read files from a specific directory and store the data into the DB.

I used Spring batch for handling the files part as the number of files is very large.

The application has a component named PraserStarer which has a method named startParsing. This method is annotated with @scheduled annotation.

@scheduled(fixedDelay = 60 * 1000)
public startParsing(){
    // start spring batch job
}

I have a repository interface NewsRepositry injected into the writer of the spring batch first step.

The application has a simple controller to manually call the startParsing method. When calling the startParsing method from the controller, everything works fine. The spring batch job starts normally, read the files, write the data into the DB, and archive the files.

When the method startParsing is invoked from the scheduling framework, the spring batch job starts normally, and read the files but nothing is stored the DB.

I suspect the problem here is that there are two different contexts, one for the scheduling part and another for the rest of the application.

For some reason, there is no transaction manager in the scheduling context which causes nothing to go to the DB.

1- Is my suspicion correct?

2- If yes, how can I force the transaction manager to be loaded to the other context?

EDIT

The code for the parser starter class is below

@Component
public class ParserStarter {

    @Autowired
    JobLauncher jobLauncher;

    @Value("${app.data_directory}")
    private String dataDir;

    @Autowired
    private ParserJobListener jobListener;

    @Autowired
    private JobBuilderFactory jobBuilderFactory;

    public Resource[] getResources() throws IOException {
        // return array of file resource to be processed
    }

//  @Scheduled(fixedDelay = 60 * 1000)
    public void startParsing() throws Exception {
        String jobName = System.currentTimeMillis() + " New Parser Job";

        JobParameters jobParameters = new JobParametersBuilder().addString("source", jobName).toJobParameters();

        jobLauncher.run(getParsingJob(), jobParameters);
    }

    @Bean(name="getParsingJob")
    private Job getParsingJob() throws IOException {

        jobListener.setResources(getResources());

        Step processingStep = jobListener.processingStep();

        Step archivingStep = jobListener.archivingStep();

        Job job = jobBuilderFactory.get("Store News").incrementer(new RunIdIncrementer())
                .listener(jobListener).start(processingStep).next(archivingStep).build();

        return job;
    }
}

The code for the job listener is below

@Component
public class ParserJobListener extends JobExecutionListenerSupport {

    @Autowired
    private StepBuilderFactory stepBuilderFactory;

    private Resource[] resources;

    @Value("${app.archive_directory}")
    private String archiveDirectory;

    @Autowired
    private Writer writer;

    public MultiResourceItemReader<DataRecord> multiResourceItemReader() {
        MultiResourceItemReader<DataRecord> resourceItemReader = new MultiResourceItemReader<DataRecord>();
        resourceItemReader.setResources(resources);
        resourceItemReader.setDelegate(new Reader());
        return resourceItemReader;
    }

    public Step archivingStep() {
        FileArchivingTask archivingTask = new FileArchivingTask(resources, archiveDirectory);
        return stepBuilderFactory.get("Archiving step").tasklet(archivingTask).build();
    }

    public Step processingStep() {
        return stepBuilderFactory.get("Process news file").<DataRecord, DataRecord>chunk(1000)
                .reader(multiResourceItemReader()).writer(writer).build();
    }

    @Override
    public void afterJob(JobExecution jobExecution) {
        if (jobExecution.getStatus() == BatchStatus.COMPLETED) {
            System.out.println("Job finished")
        }
    }

    public void setResources(Resource[] resources) {
        this.resources = resources;
    }

}

What is remaining is the writer and it is below

@Component
public class Writer implements ItemWriter<DataRecord>{

    @Autowired
    private DataRepository dataRepo;

    @Override
    public void write(List<? extends DataRecord> items) throws Exception {
        dataRepo.saveAll(items);
    }

}

Edit 2

I have changed the writer's write method to save and flush each item indiviually as following

@Transactional
    public void write(List<? extends GdeltRecord> items) throws Exception {
        for (GdeltRecord gdeltRecord : items) {
            dataRepo.saveAndFlush(gdeltRecord);
        }
//      dataRepo.saveAll(items);
 }

This time the application throws a TransactionRequiredException: no transaction is in progress exception.

Here is the full stack trace of the exception

Caused by: javax.persistence.TransactionRequiredException: no transaction is in progress
    at org.hibernate.internal.SessionImpl.checkTransactionNeeded(SessionImpl.java:3552) ~[hibernate-core-5.3.7.Final.jar:5.3.7.Final]
    at org.hibernate.internal.SessionImpl.doFlush(SessionImpl.java:1444) ~[hibernate-core-5.3.7.Final.jar:5.3.7.Final]
    at org.hibernate.internal.SessionImpl.flush(SessionImpl.java:1440) ~[hibernate-core-5.3.7.Final.jar:5.3.7.Final]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_191]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_191]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_191]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_191]
    at org.springframework.orm.jpa.ExtendedEntityManagerCreator$ExtendedEntityManagerInvocationHandler.invoke(ExtendedEntityManagerCreator.java:350) ~[spring-orm-5.1.4.RELEASE.jar:5.1.4.RELEASE]
    at com.sun.proxy.$Proxy87.flush(Unknown Source) ~[na:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_191]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_191]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_191]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_191]
    at org.springframework.orm.jpa.SharedEntityManagerCreator$SharedEntityManagerInvocationHandler.invoke(SharedEntityManagerCreator.java:308) ~[spring-orm-5.1.4.RELEASE.jar:5.1.4.RELEASE]
    at com.sun.proxy.$Proxy87.flush(Unknown Source) ~[na:na]
    at org.springframework.data.jpa.repository.support.SimpleJpaRepository.flush(SimpleJpaRepository.java:533) ~[spring-data-jpa-2.1.4.RELEASE.jar:2.1.4.RELEASE]
    at org.springframework.data.jpa.repository.support.SimpleJpaRepository.saveAndFlush(SimpleJpaRepository.java:504) ~[spring-data-jpa-2.1.4.RELEASE.jar:2.1.4.RELEASE]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_191]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_191]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_191]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_191]
    at org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments.invoke(RepositoryComposition.java:359) ~[spring-data-commons-2.1.4.RELEASE.jar:2.1.4.RELEASE]
    at org.springframework.data.repository.core.support.RepositoryComposition.invoke(RepositoryComposition.java:200) ~[spring-data-commons-2.1.4.RELEASE.jar:2.1.4.RELEASE]
    at org.springframework.data.repository.core.support.RepositoryFactorySupport$ImplementationMethodExecutionInterceptor.invoke(RepositoryFactorySupport.java:644) ~[spring-data-commons-2.1.4.RELEASE.jar:2.1.4.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) [spring-aop-5.1.4.RELEASE.jar:5.1.4.RELEASE]
    at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.doInvoke(RepositoryFactorySupport.java:608) ~[spring-data-commons-2.1.4.RELEASE.jar:2.1.4.RELEASE]
    at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.lambda$invoke$3(RepositoryFactorySupport.java:595) ~[spring-data-commons-2.1.4.RELEASE.jar:2.1.4.RELEASE]
    at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.invoke(RepositoryFactorySupport.java:595) ~[spring-data-commons-2.1.4.RELEASE.jar:2.1.4.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) [spring-aop-5.1.4.RELEASE.jar:5.1.4.RELEASE]
    at org.springframework.data.projection.DefaultMethodInvokingMethodInterceptor.invoke(DefaultMethodInvokingMethodInterceptor.java:59) ~[spring-data-commons-2.1.4.RELEASE.jar:2.1.4.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) [spring-aop-5.1.4.RELEASE.jar:5.1.4.RELEASE]
    at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294) ~[spring-tx-5.1.4.RELEASE.jar:5.1.4.RELEASE]
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98) ~[spring-tx-5.1.4.RELEASE.jar:5.1.4.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) [spring-aop-5.1.4.RELEASE.jar:5.1.4.RELEASE]
    at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:139) ~[spring-tx-5.1.4.RELEASE.jar:5.1.4.RELEASE]
    ... 66 common frames omitted

Your configuration is incorrect, I would take out step definitions from the listener and put them next to the job definition in a separate class (a typical job configuration can be found in the [getting started guide](https://spring.io/guides/gs/batch-processing/)). In regards to transactions, can you share the configuration of your transaction manager? — Mahmoud Ben Hassine, Jan 31 '19 at 11:15
I do not think the problem is with the steps. I tried what you suggested and the situation is the same. Regarding the transaction manager configuration, I did not do any configuration. It is configured using spring auto configuration. — Fanooos, Jan 31 '19 at 12:02
I also tried to configure the data source, transaction manager, and entity manager manually but I got two transaction manager beans exception. One of the two beans is provided by spring batch and I think this maybe the problem. — Fanooos, Jan 31 '19 at 12:11
Possible duplicate of https://stackoverflow.com/questions/22509529/jpaitemwriter-no-transaction-is-in-progress — Mahmoud Ben Hassine, Jan 31 '19 at 12:31
@MahmoudBenHassine +1 <3. I did not try it yet but I believe you are correct. — Fanooos, Jan 31 '19 at 12:37
@MahmoudBenHassine I tried the solution in the question you mentioned and it worked. thnx — Fanooos, Feb 03 '19 at 07:59

score 5 · Accepted Answer · answered Feb 03 '19 at 07:56

I tried the approach described in this question (JpaItemWriter: no transaction is in progress) and it worked for me.

I defined a JpaTransactionManager bean and used it with step configuration.

    @Bean
    @Primary
    public JpaTransactionManager jpaTransactionManager() {
        final JpaTransactionManager transactionManager = new JpaTransactionManager();
        transactionManager.setDataSource(dataSource);
        return transactionManager;
    }

and in the step configuration

    @Autowired
    JpaTransactionManager trxm;

    public Step processingStep(Resource[] resources) throws IOException {
        return stepBuilderFactory.get("Process CSV File")
                .transactionManager(trxm)
                .<DataRecord, DataRecord>chunk(1000)
                .reader(multiResourceItemReader()).writer(writer).build();
    }

well I tried this exact thing and my code JPA save and saveAll methods just "hang" and do nothing and this doesn't work for me , yet this all worked fine on spring 1.x.x (now I was trying Spring 2.2.1) — ennth, May 21 '21 at 02:46

score 1 · Answer 2 · answered Jan 31 '19 at 09:22

1

It's difficult to analyze what exactly is happening without seeing the full code. However, based on the documentation, spring uses chunk oriented processing. This is what it says:

Once the number of items read equals the commit interval, the entire chunk is written out via the ItemWriter, and then the transaction is committed.

This might be why you are not seeing any db writes immediately.

Regarding the transaction manager, you can define it like this (explained here):

@Bean
public Job sampleJob(JobRepository jobRepository, Step sampleStep) {
    return this.jobBuilderFactory.get("sampleJob")
                            .repository(jobRepository)
                .start(sampleStep)
                .build();
}

/**
 * Note the TransactionManager is typically autowired in and not needed to be explicitly
 * configured
 */
@Bean
public Step sampleStep(PlatformTransactionManager transactionManager) {
        return this.stepBuilderFactory.get("sampleStep")
                                .transactionManager(transactionManager)
                                .<String, String>chunk(10)
                                .reader(itemReader())
                                .writer(itemWriter())
                                .build();
}

answered Jan 31 '19 at 09:22

Darshan Mehta

30,102
11
68
102

Reagrding the number of items and commit interval, unfortunately this is not the case. When start the job from the controller, the records appears in the DB immadiatly but when start it from the scheduled method, the job start and end and nothing stored to the DB – Fanooos Jan 31 '19 at 09:42
So, it works when you call `startParsing` from controller and doesn't work when you call it from scheduler? – Darshan Mehta Jan 31 '19 at 10:53
It works when I call `startParsing` from the controller and it does not work when the `startParsing` method triggered by the scheduling framework. – Fanooos Jan 31 '19 at 11:08
Just to give it a try, can you write another method in controller, annotate it with `@Schedule` and call `startParser` from there? – Darshan Mehta Jan 31 '19 at 11:40
I already did that. I annotated the same method with scheduled and the situation still the same. I will add a new edit to the question right now. Kindly check it. – Fanooos Jan 31 '19 at 11:49

score 0 · Answer 3 · edited Dec 31 '20 at 10:37

0

We have to explicitly mention JpaTransactionManager instead of default spring batch transactions.

@Configuration
@EnableBatchProcessing
public class MyJob extends DefaultBatchConfigurer {
       
       @Autowired
       private DataSource dataSource;

       @Bean
       @Primary
       public JpaTransactionManager jpaTransactionManager() {
            final JpaTransactionManager tm = new JpaTransactionManager();
            tm.setDataSource(dataSource);
            return tm;
       }
}

edited Dec 31 '20 at 10:37

Eric Aya

69,473
35
181
253

answered Dec 31 '20 at 08:13

Bharathiraja

714
1
12
20

Do we use this with the STEP and a seperate Transaction manager for JOB REPOSITORY? what about the Entity Manager? I see people passing that to the JpaTransactionManager constructor – ennth Apr 15 '21 at 03:50

Spring boot repository does not save to the DB if called from scheduled job

EDIT

Edit 2

3 Answers3

Linked