Python - memory efficiency - range

Question

I am seeking ways to make the below minimal code for MaxPower() function more memory efficient & examples using my code.

Purpose of MaxPower(): for integers i and N_remainder, to return the maximum value of m such that i^m divides N_remainder.

MaxPower() is only called in the factorise() function if i divides N_remainder

Currently the linked code below has the following results for the number to factorise (wrote in standard form):

1x10^8 - works fine. Two arrays of results produced and are correct
5x10^8 - hangs Linux & OS is totally unresponsive. Computer needs hard restart.
1x10^9 - gives memory error in terminal.

Python version used is 2.74 on Linux Mint 17.

I am currently learning python.

def MaxPower(i,N_remainder):
    m=1
    MxP=1
    for n in range (2, N_remainder + 1):        
        b = i**n    
        a = N_remainder % b
        if a==0:    
            m = m+1                 
        else: 
            MxP = m     
            break
    return MxP

Revised code since originally posted:

def MaxPower(i,N_remainder):
    m=1
    MxP=1
    for n in xrange (2, N_remainder + 1):       
        b = pow(i,n)        
        a = N_remainder % b
        if a==0:    
            m = m+1                 
        else: 
            MxP = m     
            break
    return MxP

I am aware of the following (which I haven't tried since new & out of my depth currently);

Could do more work in compiled C code - converting to "list comprehension"
Use threading & make sure each thread is deleted after use.

Do you know which part of the code in particular is using all the memory? That’s a lot of code to go through when probably only a small part of it actually has the problem. — Daniel H, Oct 05 '17 at 16:39
I'm not sure, but I suspect it may be `maxpower()` function. One example is for 3rd example (1*10^9) - this factorises by hand as 2^9 * 5^9 - could it be something to do with storing large powers of numbers in memory? — unseen_rider, Oct 05 '17 at 16:45
Updated question with minimal code for `MaxPower()` function — unseen_rider, Oct 05 '17 at 16:54
Just so that you know: It looks like the size of some of those numbers could also be very large. `sys.getsizeof()` reports that `10**8` itself takes 16 bytes and `2**(10**8)` takes 13333346 bytes. — quamrana, Oct 06 '17 at 09:18
@Holloway it is supposed to return the maximum value of m such that i^m divides N_remainder. — unseen_rider, Oct 06 '17 at 11:13
@quamrana ok, anything you advise I change in my code, apart from `xrange`, when dealing with very large numbers such as `2**(10**8)`? — unseen_rider, Oct 06 '17 at 11:17
What if `i` doesn't divide `N_remainder`? I don't see how you can ever return 0. — trent, Oct 06 '17 at 21:01
@trentcl this should not happen since `maxpower()` is only called in the `factorise()` function if `i` divides `N_remainder`. See rest of the code linked in question. — unseen_rider, Oct 06 '17 at 23:13

Daniel H · Accepted Answer · 2017-10-05T18:07:14.107

3

The Python 2 range function creates a list with one entry for each number in the given range. This entire list needs to be stored in memory, so for large values of N_remainder it can get pretty big.

Instead, there is the xrange function, which does almost the same thing. This uses constant memory, only storing the parameters and calculating each value only when it’s needed and not storing old values once they’re used. If you replace the call to range with a call to xrange, your function should use almost-constant memory.

Note that if N_remainder + 1 is too big for a python int, the xrange function won’t work, but the documentation provides an alternative.

Not a memory issue, but you don’t need the explicit calls to long; Python 2 automatically converts to the long type when necessary.

edited Oct 05 '17 at 18:07

answered Oct 05 '17 at 17:25

Daniel H

7,223
2
26
41

`xrange` has been around a lot longer than Python 3. It's been around longer than Python 2, even; it's listed in the [Python 1.4 docs](https://docs.python.org/release/1.4/lib/node26.html#SECTION00330000000000000000). – user2357112 Oct 05 '17 at 17:37
@user2357112 Oh, I thought it was a backport. I’m correcting it now. – Daniel H Oct 05 '17 at 18:06

Python - memory efficiency - range

1 Answers1