How spring data clean persited entities in transactional method?

Question

I need to receive and save huge amount of data using spring data over hibernate. Our server allocated not enough RAM for persisting all entities at the same time. We will definitely get OutOfMemory error.

So we need to save data by batches it's obvious. Also we need to use @Transactional to be sure that all data persisted or non was persisted in case of even single error.

So, the question: does spring data during @Transactional method keep storing entities in RAM or entities which were flushed are accessible to garbage collector?

So, what is the best approach to process huge mount of data with spring data? Maybe spring data isn't right approach to solve problems like that.

Several hundreds thousand entities of variable size (10 kb - 2mb) — Artyom Karnov, Jan 11 '19 at 13:15
Like you said you have got to do batching. @Transactional hasn't got any direct relationship with batching. There are many ways you could insert batches. But usually it's a bad practice to keep the transaction open for the whole record set. What DBMS do you use? — Laksitha Ranasingha, Jan 11 '19 at 13:22
DB2. So using 2 phase commit - seems quite dangerous and difficult to implement. — Artyom Karnov, Jan 11 '19 at 13:23
Do you really want all or nothing when inserting hundreds of thousands of entities? I mean using one umbrella transaction? — Laksitha Ranasingha, Jan 11 '19 at 13:46
The requirement - all or nothing. I try to find best solution. — Artyom Karnov, Jan 11 '19 at 13:47
I don't think there's any reliable and robust way that you could achieve this task. I can point you to how batch inserts can successfully make this work, obviously it's not all or nothing. — Laksitha Ranasingha, Jan 11 '19 at 13:50
I just need to know best, time-proved method of solving this problem — Artyom Karnov, Jan 11 '19 at 14:02
You have to think about 1. the batching and batch size 2. if batching should be concurrent? (multiple batches concurrently) 3. How to configure DB2 batching with hibernate, 4. How you could use DB2 locking with hibernate. A good post at https://vladmihalcea.com/the-best-way-to-do-batch-processing-with-jpa-and-hibernate/ — Laksitha Ranasingha, Jan 11 '19 at 14:07
I think you won't come around dumping the data initially to (hard) drive, since you can predict outofmemory, but I wonder how the client can allocate all of it ;) some questions: "huge data" means "many rows" (in one dataset) or/and "many columns" or/and "many datasets" (many requests/"second")? How is data represented/structured? (xml/json/.../binary stream...ideally sql insert statements:) What do you want to do with the data? (INSERT only or also update, delete?)..and to the core of your question: I think "spring-data" is "no big help" in such case, but it also doesn't make things worse. — xerx593, Jan 11 '19 at 23:42

Ken Chan · Accepted Answer · 2019-01-12T03:35:45.407

Does spring data during @Transactional method keep storing entities in RAM or entities which were flushed are accessible to garbage collector?

The entities will keep storing in RAM (i.e in entityManager) until the transaction commit/rollback or the entityManager is cleared. That means the entities are only eligible for GC if the transaction commit/rollback or entityManager.clear() is called.

So, what is the best approach to process huge mount of data with spring data?

The general strategy to prevent OOM is to load and process the data batch by batch . At the end of each batch , you should flush and clear the entityManager such that the entityManager can release its managed entities for CG. The general code flow should be something like this:

@Component
public class BatchProcessor {

    //Spring will ensure this entityManager is the same as the one that start transaction due to  @Transactional
    @PersistenceContext
    private EntityManager em;

    @Autowired
    private FooRepository fooRepository;

    @Transactional
    public void startProcess(){

        processBatch(1,100);
        processBatch(101,200);
        processBatch(201,300);
        //blablabla

    }

    private void processBatch(int fromFooId , int toFooId){
        List<Foo> foos =  fooRepository.findFooIdBetween(fromFooId, toFooId);
        for(Foo foo :foos){
            //process a foo
        }

        /*****************************
        The reason to flush is send the update SQL to DB . 
        Otherwise ,the update will lost if we clear the entity manager 
        afterward.
        ******************************/
        em.flush();
        em.clear();
    }
}

Note that this practise is only for preventing OOM but not for achieving high performance. So if performance is not your concern , you can safely use this strategy.

Ken, thank you for your explanation! I have question 1) I understand correctly that whole transaction will be rollbacked if during even only one call of processBatch() error occurs? 2)What would you suggest as best performance solution? — Artyom Karnov, Jan 11 '19 at 15:10
@ArtyomKarnov Yes. It is all or nothing . Either all records are processed or nothing are processed. For the best performance , I would suggest using JDBC directly .It should be more easier than fine tuning hibernate . But of course when talk about performance seriously , you would better actually measure it.... — Ken Chan, Jan 11 '19 at 15:15

How spring data clean persited entities in transactional method?

1 Answers1

Linked