1

I have tested the following python script on 2 Windows machines, and onlinegdb's python compiler.

On the Windows machines, this code when run as it is, just exits midway, non-deterministically, with no error message or warning. I tested with python 3.9.6.
It works as expected and does not crash when the function nCr is a different function outside isPrime. You can comment and uncomment the appropriate text.
On onlinegdb's python compiler, it works as expected in both cases.

import sys
sys.setrecursionlimit(10**6)

# def nCr(n, r):
#     if r == 0:
#         return 1
#     else:
#         return nCr(n, r-1) * (n-r+1) / r

def isPrime(N):

    def nCr(n, r):
        if r == 0:
            return 1
        else:
            return nCr(n, r-1) * (n-r+1) / r
    
    if nCr(N, N//2) % N == 0:
        return True
    else:
        return False

for i in range(4050, 4060 + 1):
    print(f"{i}: {isPrime(i)}")
else:
    print("Done")

Any clues on what may be causing this? Is it possible to get this to run correctly on Windows? Or should I just avoid inner functions entirely?

Note: I know that the prime checker's logic is incorrect.
Note: You can try a larger range by changing the 4th last line if you are not able to reproduce the crash.

Edit 1:
We found that:

  1. If the recursive depth is sufficiently large, it will most likely cause a crash on all platforms. This number, although large, would still be small enough such that only a small portion of the machines memory is being used.
  2. Moving the function to module level does not prevent the crash.
  3. Increasing system recursionlimit does not affect the crash, if it is more than the depth at which the crash occurs.

So, the question now is:
Is there a way to estimate the depth at which the crash will occur? Also, the depth at which the crash occurs is very small, and if we use our own stack instead of calling the function recursively, then we can keep going till the machine is out of memory. So, should we just avoid using recursive function calls in python?

Venu Nathan
  • 81
  • 6
  • Do you know how deep your recursion is actually getting? I suspect you're running into some sort of memory problem because the stack can't grow as large as you want it to be. It shouldn't be that hard to eliminate the recursion by calculating nCr(n, r) by first calculating nCr(n, 0), nCr(n, 1), etc in a loop. – Frank Yellin Nov 29 '21 at 07:07
  • Are you even sure this algorithm is correct? It's telling me that 45 and 49 are prime – Frank Yellin Nov 29 '21 at 07:08
  • @FrankYellin The recursion can have a depth of N//2. It works when I find nCr by iteration. So, are the stacks of inner functions managed differently when compared to usual functions? – Venu Nathan Nov 29 '21 at 07:24
  • I tried the code on a phone with Pydroid. It failed multiple times for the same N for me with a "Segmentation fault". If "nCr" isn't an inner function it fails in the same way but for a much higher N. So I think it is a memory problem which can be mitigated but not solved by avoiding inner functions. – Michael Butscher Nov 29 '21 at 07:25
  • I don't think so. – Frank Yellin Nov 29 '21 at 07:27
  • @MichaelButscher Thanks for pointing it out. After you mentioned, I checked and found that for me, it fails around N = 5130 when nCr is not an inner function. I think I'll stick to finding it by iteration instead. – Venu Nathan Nov 29 '21 at 07:37
  • But even at a depth of 5000, on my machine the program only uses around 100 mb of ram (and I have lots of unused space). And the function nCr does not even need that much space on the stack. So, other that the recursion limit, could there be some threshold on the space used on stack as well? Can I change that threshold? – Venu Nathan Nov 29 '21 at 07:48
  • Crash on Linux and Python 3.10 and also 3.7 too at recursion levels about 14 and 18 thousand respectively. The program can be simplified to the simplest recursion nCr(n) = nCr(n-1), stopping the recursion at 0. It does not matter if nCr is at module level or an inner function. – VPfB Nov 29 '21 at 08:14
  • From `sys.setrecursionlimit(limit)` docs: _"The highest possible limit is platform-dependent. A user may need to set the limit higher when they have a program that requires deep recursion and a platform that supports a higher limit. This should be done with care, because a too-high limit can lead to a crash."_ So the question is what is the platform dependent limit. – VPfB Nov 29 '21 at 08:30
  • @VPfB Thanks. Yes, the question is about finding the platform dependent limit, and why it is so small. I would say even 18,000 is very small for the simplest recursion. The funny thing is if, instead of recursively calling the function, we process it with our own stack, then we can keep going till the machine is out of memory. I can go to a depth of 4,000,000 and have the same memory usage as recursively calling the function till a depth of 5000, and it also completes without crashes. I think we should avoid recursion in general when using python. – Venu Nathan Nov 29 '21 at 09:47
  • And there is also a system limit: `resource.setrlimit(resource.RLIMIT_STACK, )`. At least on Linux. Don't know about Windows. If I increase also that one, it works for 1_000_000 (1M) levels deep recursion and 10_000_000 is killed by the system's OOM (out of memory killer). The limit is then the memory only, but the max count of stack frames is not clear, because the size of one stack frame is not exaclty known. – VPfB Nov 29 '21 at 10:18

0 Answers0