0

I am using the scipy simpsons rule in a vectorized way for performance. (I am calculting integrals on a rolling window on a timeseries.)

For some reason I am getting a memory error when using simps on an 500MB array of floats on a 64bit machine which has 4GB of memory available.

Here's some example code:

import numpy as np
from scipy.integrate import simps
import psutil

print(psutil.virtual_memory().percent)  #gives 40%
# in the followinging, the number of rows can be increased up until 190'000 without issues.
# The memory usage during execution of the whole script grows linearly with the size of the input array
# I don't observe any exponential behaviour in Memory-usage

a = np.random.rand(195000, 150, 2)  
print(psutil.virtual_memory().percent)  #gives 51%
print(a.nbytes/1024/1024, "MB")  #gives 446MB
print(a[:,:,0].nbytes/1024/1024, "MB")  #gives 223 MB
I = simps(a[:,:,0], a[:,:,1])  #MemoryError
print(psutil.virtual_memory().percent)

I can see the virtual memory of my machine going up until 2GB are used and then I get a MemoryError even though not all of my free memory is used.

So I am confused: Why am I getting a MemoryError even though there is still about 2GB of unused Memory available? Can this be worked around?

I am using Winddows Server 2012 R2.

Edit: To illustrate the more or less linear scaling of memory usage vs. input size, I performed the following little experiment:

def test(rows):        
    a = np.random.rand(rows, 150, 2)
    I = simps(a[:,:,0], a[:,:,1])
    print(I.shape)

for numRows in [r for r in range(10000,190000,10000)]:
    test(numRows)

Which gives a rising memory consumption in the windows ressource monitor as well as in other tools like mprof:

Memory Usage during experiment

Mischa Obrecht
  • 2,737
  • 6
  • 21
  • 31
  • What's the significance of 53365? – Nick is tired Dec 27 '17 at 17:12
  • Nothing really, the problem occures in a bigger script with that exact number of entries, but a more complex application of simps. I clarified the example. – Mischa Obrecht Dec 27 '17 at 17:27
  • Refresh our memory about `simps`. How many temporary arrays of size like `a` will it have to make? What will be the size of the returned `I`? You might also test making several copies of `a` to see how memory usage changes. – hpaulj Dec 27 '17 at 19:52
  • I am not too sure about how simps is implemented. In principle it is a numerical quadrature, and memory and cpu usage should scale linearly with the size of the input data. I've modified my post with an experiment about memory usage to illustrate that point. The returned I will be a 1d-array with the same number of rows as the input – Mischa Obrecht Dec 27 '17 at 22:30

0 Answers0