2

Just out of curiosity, I tried this code in Frege:

println (mydrop 30000000 [1..30000001])

It goes without saying that a sequence of 30 million entries is kind of silly and I would have been ok with an OOME. I wanted to see whether lazy evaluation makes a difference here. The result was though that all my 8 cores were exhausted at 100% and stayed there until I hard-killed the process.

Have I hit a systematic upper bound?

I should have mentioned that I used the mydrop from the real-world Haskell exercise:

mydrop n xs = if n <= 0 || null xs
              then xs
              else mydrop (n-1) (tail xs)
Dierk
  • 1,308
  • 7
  • 13

2 Answers2

2

The issue has now being tracked down, but it turns out that it can't be avoided in all cases, due to lack of proper tail calls in the JVM. See here for an explanation: https://github.com/Frege/frege/issues/65

Interestingly, the following "equivalent" groovy program (my first one!) exhibits an interesting behaviour:

println (new IntRange(0,30_000_000).drop(29).take(3))

this takes substantial longer than

println (new IntRange(0,30_000_000).drop(29_999_990).take(3))

The Frege program always needs n+m steps, where n is the number of dropped items and m the number of items taken, so the first one is almost immediate, but the second one takes 2 to 3s. The grrovy program OTOH seems to actually realize the list that remains after drop, before continuing with take. Thus the first version is slower.

Ingo
  • 36,037
  • 5
  • 53
  • 100
1

There is no upper limit I am aware of. A short test in the REPL (try.frege-lang.org) shows that I can go up to

drop 16_000_000 [1..16_000_000]

which finishes after a few seconds only. Since this program is O(n) I'd estimate a maximum execution time of maybe 30 seconds for 30 million, but with 32_000_000 I get "Service unavailable" after some seconds, which usually hints at exhaustion of some limit of the free web service.

Also, the memory usage of the program above should be constant, irrespective of the number. If it didn't I would consider it a bug.

--- EDIT ---

Tried it on a 2 core 2.9GHz office PC: works like a charm and takes 5.7s. 64 million take 10.5s

Ingo
  • 36,037
  • 5
  • 53
  • 100
  • with 16_000_000 I get `runtime 42.802 wallclock seconds` but it works - still consuming all 8 cores at 100% on my MacBook Pro. I don't have an mental model that could explain this behavior. – Dierk Sep 30 '13 at 11:39
  • @Dierk Which frege version are you using? Pleaser un java -jar fregec.jar -version – Ingo Sep 30 '13 at 11:52
  • '3.21.190' 'g714a7cc', with Java7 I get OOME for 16_000_000, with Java 8 just as slow as mentioned above. I run with -Xss1m. – Dierk Sep 30 '13 at 11:58
  • @Dierk with `mydrop` I need 50s for 8 million, this is due to the function being very general (see the `null` discussion also): you could say `mydrop 4 "Dierk"` and it would be "k". There are 2 indirect calls (null and tail) and the costs sum up acordingly. – Ingo Sep 30 '13 at 12:29
  • @Dierk - The long run time is explainable (try adding a type signature like `mydrop :: Int -> [a] -> [a]` should be *much* faster) but the OOME is disturbing me .... will look into it further later. – Ingo Sep 30 '13 at 12:34
  • Accidentally, I used that exact signature already ;-) Sorry for not stating that in the question. – Dierk Sep 30 '13 at 15:54
  • I should have known by the time you said it's the same with `drop` that there must be something differently wrong. Would you mind posting the whole code verbatim, so we can hunt it down? BTW, it is too funny that some (IMHO unrelated) error led me to notice an issue with lazy list arguments I need to investigate further. – Ingo Sep 30 '13 at 17:28
  • https://github.com/Dierk/Real_World_Frege/blob/master/realworld/chapter2/Evaluation.fr and pretty much all the code that I refer to in recent questions is in that repo. Thanks so much for helping me with my questions! – Dierk Oct 01 '13 at 12:35
  • 1
    @Dierk, it seems to be the case that under circumstances to be investigated further the java code holds onto a reference to a lazy argument, thus preventing proper garbage collection. I am not sure yet whether this happens only in the IO monad or generally. Anyway, I could prevent it by compiling the stuff with option `-inline`, just add this to your gradle.build, clean up and recompile and it'll work with even greater numbers. – Ingo Oct 01 '13 at 17:50
  • Awesome! [30000001] runtime 4.115 wallclock seconds - I'm impressed! – Dierk Oct 01 '13 at 18:55
  • @Dierk - and this on only a single core, I guess :) You know what the 8 cores were doing then? Collecting garbage. – Ingo Oct 01 '13 at 19:34
  • It uses 5 of 8 cores (but none of them at 100%) when calculating. How much of that is GC, I did not inspect, but I could if that helps. I always thought there was one GC thread only. – Dierk Oct 01 '13 at 20:04