Improve the performance of Spring boot JpaRepostiory on Save

Question

Am working on fetching the data from an third party API and then I want to insert those record in to Oracle database.This will occurs on hourly basis. Sometimes the API returns more than 50,000 records for an hour. So that time , it takes so much of time to insert the record(Absolutely record size too large). But I want to know is there any other way to increase the performance of the insert query.

Following is the code I used for inserting the records:

List<ClassName> retrievedList=util.getResultFromApi(); //The function getResultFromApi may returns more than 50,000 records.

userInfoRepository.save(retrievedList);

UserInfoRepository is my JpaRepository. Can anyone please help me with the suggestions.

Are you sure you have to improve `JpaRepository` and not the `database` itself? — Pijotrek, Aug 16 '18 at 08:44
What about: https://stackoverflow.com/questions/5649345/how-to-persist-a-lot-of-entities-jpa — Pijotrek, Aug 16 '18 at 08:45
Yes ,I want to improve JpaRepository, is there any options like Chunk @Pijotrek — kaviya .P, Aug 16 '18 at 08:48
Nope, the query will be generated dynamically, also using raw sql is not allowed in our project — kaviya .P, Aug 16 '18 at 08:50

score 0 · Answer 1 · answered Aug 16 '18 at 08:55

0

You can make it faster, by calling flush() and clear() after a certain number of inserts.

int i = 0;
for(ClassName className: classNames) {
    dao.save(className);
    if(++i % 20 == 0) {
        dao.flushAndClear();
    }

}

answered Aug 16 '18 at 08:55

Alien

15,141
6
37
57

score 0 · Answer 2 · answered Aug 16 '18 at 13:16

0

Perhaps a relational database is not the best way to do it. If you are going more write hard to your database. May be Mongo could be a better choice.

answered Aug 16 '18 at 13:16

CRISTIAN ROMERO MATESANZ

1,502
3
14
26

Improve the performance of Spring boot JpaRepostiory on Save

2 Answers2