0

Please somebody help me with a memory issue while processing records with LOBs! So I have the following situation: I have two tables, and one of them is a table with some data. One of the columns of the table has "one-to-many" mapping to the second table. The second table keeps millions of photo images that are stored in LOBs. I use the "lazy loading" method to access the images. I need to iterate the first table's records to get photos and other information and send them to some SOAP-gate. Every time new data falls to the tables so I need the process to be restarted periodically. So I did and it works well for some amount of time but then suddenly stops with an exception like this:

июн 08, 2021 3:23:44 PM org.hibernate.engine.jdbc.connections.internal.DriverManagerConnectionProviderImpl$PoolState stop
INFO: HHH10001008: Cleaning up connection pool [jdbc:postgresql://x.x.x.x:yyyy/SomeDbInstance]
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at java.awt.image.DataBufferByte.<init>(DataBufferByte.java:92)

That is my app consumed out all the memory and stopped execution with this error.

So my question: is it possible to clean up processed entries to release the memory they start to consume after they got loaded?

Mohammad Mirzaeyan
  • 845
  • 3
  • 11
  • 30
Rome Novikow
  • 11
  • 1
  • How large are the images? How many are you loading into memory at a time? What is your approach for paginating the data you're working with (e.g. you're loading 10 records at a time and paging through data, you're not trying to load everything into memory in one go?) What heap size are you running with? – Kevin Hooke Jun 08 '21 at 15:31
  • Thank you for your comment! These images are from 1M to 5M. I developed a service which gets new data from one remote DB and post it to SOAP gate. To do initial step I select the data starting at an hour before now(). So I get about thousand of records which need to be processed. But it crashes before 10% is done... – Rome Novikow Jun 08 '21 at 16:01
  • What size is your heap? Are you loading one at a time for processing? When you say you get 1000 records for processing at a time, is this a list of ids for the records to be processed, or are you loading 1000 5MB images into memory in one go? – Kevin Hooke Jun 08 '21 at 16:45
  • Note, this is sounding more like a design question rather than help with a specific programming issue, so unless you can edit the question to ask about an issue with a specific api you have a problem with this is probably in danger of not being on topic and will be closed, please see https://stackoverflow.com/help/on-topic – Kevin Hooke Jun 08 '21 at 16:49
  • Probably this is a design question. O'k, I can stop to use HIbernate mechanism and do a separate request for related photos every time I get a row from the first table. Or I can restart the loop every 10 entries for example. But these cases are connected with some performance degradation. And it would be more strait and logical to release useless memory on the fly. Or what use of lazy loading mode when the memory is going to run out but later? – Rome Novikow Jun 08 '21 at 17:32
  • Lazy Loading doesn't prevent you from filling your memory and running out of heap. It only defers the memory usage until the point when you need it. You still need to manage removing stuff from memory. In Java terms, letting references go out of scope or setting refs to null to let the GC do its work. – Kevin Hooke Jun 08 '21 at 19:29
  • Rephrased this as an answer with some other suggestions, hopefully this is useful – Kevin Hooke Jun 08 '21 at 19:39

2 Answers2

1

Thank you, Kevin for your suggestions! I started Java Visual VM, used profiler, heap dump and monitor to analyze the executing process of my service. So, I found many heavy instances of LOBs hanging in the heap. And, yes, I have to invent method to sweep them out. And I seemed to get it!

for (PhotoContext pc:tr.getPhotoContext()) {

    . . .

    photo.setPhotoType(photoTypeMapping(pc.getInd()));
    GregorianCalendar gk = new GregorianCalendar();
    gk.setTime(pc.getDatetime());
    photo.setDatetime(DatatypeFactory.newInstance().newXMLGregorianCalendar(gk));
    photo.setPhoto(Arrays.copyOf(pc.getPhoto(), pc.getPhoto().length));
    photos.add(photo);

    sourceDao.getSession().evict(pc);
    pc.setPhoto(null);
}

I had to detach the photo entry and than set it's photo field (byte[]) to null. Maybe that is not particularly elegant but it works.

Rome Novikow
  • 11
  • 1
0

Lazy loading allow you to defer loading something into memory until you need it. It doesn't prevent you from loading too many things into memory and running out of heap. You need to think about your design for how you load the objects as needed, process them and then release the references so that the GC can free space in the heap.

Other things to consider:

  • use a profiler so you can understand how your app is using and consuming heap at runtime
  • use the JVM gc flags to watch gc performance at runtime - are you releasing references so the GC can perform collection? How long is the collection taking? Are you creating new instances faster than the GC is able to free space in the heap?
  • is your heap sized appropriately to work with the size of data you're attempting to process?
Kevin Hooke
  • 2,583
  • 2
  • 19
  • 33