0

I am converting list of strings to list of BigDecimal values in below scala code. But when queryResults list is very large (> 10 million), I'm getting below error.

We have some limitations on memory usage and cannot increase memory. Is there any way to handle this error, without increasing memory size?

This is the error I am getting

   java.lang.OutOfMemoryError: Java heap space
     at scala.Option.toList(Option.scala:301) 
     at scala.Option$.option2Iterable(Option.scala:17)

Code:

    val queryResults = List("11", "test", "444", "22", "33", "44", "1111", "2222", "3333", "4444")

    //getting error at below line
    val z = queryResults.flatMap(idPath => Try(idPath.toLong).toOption).map(BigDecimal.valueOf)
James Z
  • 12,209
  • 10
  • 24
  • 44
Ankitha Vaddeti
  • 1
  • 3
  • 1
    I really doubt you can do anything but maybe trying to "paginate" the queryResults (load only a few items, process them, load the next page, etc.) – spi Mar 29 '21 at 07:52
  • Well, you could try to split `queryResults` and work on smaller batches. Without more information at least I can't tell you a definite answer but I'd recommend you use some memory profiling (the JVM provides a sampler which should already help) to see where memory is held (I assume there's a lot of objects created in that call to `flatMap()` and the lambdas which only can be collected once the call is done. – Thomas Mar 29 '21 at 07:53
  • 1
    Term "in some crores" will be totally confusing for most people living outside India. – James Z Mar 29 '21 at 09:33
  • Don't fetch all query results into a list. Use iterator. – Dima Mar 29 '21 at 09:50
  • Actually we are using this query results to download into a file. due to this I cant use pagination here – Ankitha Vaddeti Mar 29 '21 at 11:31
  • 2
    So you are downloading query results and writing them to a file? Sounds like a streaming solution or even just an `Iterator` would fit perfectly here. – Jasper-M Mar 29 '21 at 12:12
  • I am downloading query results, performing some actions on the results and after that downloading them into file – Ankitha Vaddeti Mar 29 '21 at 12:53

0 Answers0