0

I'm generating a HashMap where I can estimate the requiered size, for factorials: 

  import scala.collection.mutable.HashMap

  val mm = new HashMap [Int, BigInt] 
  mm.put (0, 1)
  def fak (i: Int) : BigInt = mm.getOrElseUpdate (i, i * fak (i-1))

I frequently request the factorial (fak) of primes in ascending order, and like to reach pretty high values (> 10 Mio factorials).

Calling it with about 70000 results in an OutOfMemory-Error: Java Heap Space. I started the program with 

scala -J-Xmx4G TestFak 70000

With 60000 as parameter it works. I guess, it builds 70000 MutableMaps which get frequently thrown away and garbage collected. Since I know the requiered size in advance, is it possible to generate a mutableMap of the right size from start?

The error is thrown in the mm.getOrElseUpdate - Line.

Version: Scala version 2.11.6 (OpenJDK 64-Bit Server VM, Java 1.8.0_66-internal)

trincot
  • 317,000
  • 35
  • 244
  • 286
user unknown
  • 35,537
  • 11
  • 75
  • 121
  • It does not build 70000 maps, there is just one map with 70000 entries. One map entry without any data takes about 36 bytes, 36*70K is about 2.5 gig right there. Plus the actual size of BigInts you are storing (they arent small either). You are just hitting your memory limit. – Dima Mar 06 '16 at 00:12
  • @Dima: Hm, according to a fast estimation, 36*70K should be about 2.5M, not 2.5G, shouldn't it? – user unknown Mar 06 '16 at 00:16
  • Yeah, you are right about that :) YourKit? – Dima Mar 06 '16 at 00:40
  • You only need to store the prime ones, right? Why not throw away all the non-prime ones between p(n-1) and p(n), once you hit p(n)? – Rex Kerr Mar 06 '16 at 04:47
  • @RexKerr: In fact I need more than just the prime ones. About 50% of all numbers and in the end up to 100 million, so any trick which let's me get 2 times or 10 times what I have will not be sufficient. Meanwhile I tried to store just the last number but even this approach is to slow. It would run, optimistically, in 2 days instead of 1 minute. :) Since it is a quizzle, I don't want to ask for the underlying problem, which seems to need some more mathematical approach. But either I solve this by myself or it will just remain unsolved. – user unknown Mar 06 '16 at 05:37

1 Answers1

4

The factorial of 70000 is huge! The BigInt required to store that is going to be fairly large in and of itself! Just to give you an idea, the BigInt is probably backed by an Array[Int] in Java. That means the total size required to store 1!, 2!, ..., 70000! is going to be sum_(1 to n) of 4 * log_(2^32) n! for n = 70000, which is on the order of four gigabytes.

badcook
  • 3,699
  • 14
  • 25