2

I am trying to create a large Boolean array (for a prime number sieve). I used first Python lists, but at limit = 10^9 this created a MemoryError.

boolarray = [True] * limit

Then I learned about Numpy and read that it is better with space organisation, so I tried

boolarray = np.full(limit, True, dtype = bool)

The limit only marginally increased to 10^10, which is not sufficient, since I need 10^12. I find this surprising, you just need a bit for Boolean, don't you? Any idea, how to overcome this problem? Thanks in advance.

Mr. T
  • 11,960
  • 10
  • 32
  • 54
  • [Docs](https://docs.scipy.org/doc/numpy-1.13.0/user/basics.types.html): ```bool_ Boolean (True or False) stored as a byte```. Now i let others / you do some research on this design-decision. It's highly linked to CPU-architecture and intended use-cases. – sascha Nov 20 '17 at 11:59
  • Thanks, Sascha. I've seen this - that's why my surprise about Python/Numpy using more than a bit, when I specifically declare it with `dtype = bool`. Though as a Python and Numpy newbie, I don't know, in which direction to search. – Mr. T Nov 20 '17 at 12:08
  • I'm afraid you have to use some other lib (probably based on c-structs), if memory is your main-concern. – sascha Nov 20 '17 at 12:09
  • Do you realise that 10^12 bits are more than 116 Gigabyte? – MB-F Nov 20 '17 at 12:23
  • Good point, I haven't calculated this. So I shouldn't try to find a way to make this approach work in Python, but rather try to find a different strategy? – Mr. T Nov 20 '17 at 12:28
  • 1
    Well, I may have made a mistake so you will want to verify that :) (I calculated this number as `10**12 / (8 * 1024**3)`). Unless you have a crazy amount of ram you should probably look for a different approach. – MB-F Nov 20 '17 at 12:33
  • 1
    @Piinthesky You may find [this answer](https://stackoverflow.com/a/26849599/3005167) interesting. It discusses ways to store primes efficiently. However, I'm not sure if this helps in generating them in the first place... – MB-F Nov 20 '17 at 15:32
  • 1
    Excellent. This is also something I was thinking about. Just in case somebody else is not so much interested in programming and just wants to have a practical solution - there are C libraries like http://primesieve.org/ for that. – Mr. T Nov 20 '17 at 15:51
  • 1
    There are also [Python bindings](https://github.com/hickford/primesieve-python) for the primesieve library. – MB-F Nov 20 '17 at 16:16
  • Out of curiosity I have given this Python binding a try. Though it is factor 40 faster than my pure Python program, it works only up to `10^9`. I assume, because it has to convert the C list back into a Python list. – Mr. T Nov 20 '17 at 17:21

1 Answers1

2

Let's put aside the fact that 10^12 bits will probably not easily fit into memory. If you care more about memory usage than performance you can pack the bits into a byte array. This comes at the cost of additional computations when reading/writing bits (which is the reason numpy stores booleans as bytes). 

import numpy as np


def getbit(bitarray, index):
    i, j = index // 8, index % 8
    x = bitarray[i]
    return x & (1 << j) != 0


def setbit(bitarray, index, value):
    value = bool(value)
    i, j = index // 8, index % 8
    x = bitarray[i]
    bitarray[i] ^= (np.uint(-value) ^ x) & (1 << j)


n = 10**5 // 8
bitarray = np.zeros(n, dtype=np.uint8)  # initialize all bits to 0

print(getbit(bitarray, 19))  # False

setbit(bitarray, 19, True)
print(getbit(bitarray, 19))  # True

setbit(bitarray, 19, False)
print(getbit(bitarray, 19))  # False
MB-F
  • 22,770
  • 4
  • 61
  • 116
  • Thanks for the explanation and the examples. Speaking from experience with my first self-written summation functions, before I discovered the Python function `sum()`: I assume these operations mean A LOT more computation time, since they don't give parameters directly to Python to handle the data efficiently within their libraries written in C. If this is true, then I'm afraid that means back to the drawing board. – Mr. T Nov 20 '17 at 14:21
  • @Piinthesky Jep, if you call these functions a lot - which I assume you do - you will notice the interpreter overhead. Anyway, this does not solve the problem of squeezing 10^12 bits into the ram, but I thought it may be worth answering the question anyway because it can be useful for others too. – MB-F Nov 20 '17 at 15:23
  • I marked it as a good answer, because it shows that I am on the wrong track. This is valuable information to me (and maybe others as well). – Mr. T Nov 20 '17 at 15:42