0

I'm trying to create a large file with numpy memmap 

big_file = np.memmap(fnamemm, dtype=np.float32, mode='w+', shape=(np.prod(dims[1:]), len_im), order='F')

The system is a Windows 10-64bits operating in a 64bits python 

In [2]: sys.maxsize

Out[2]: 9223372036854775807

With enough virtual memory (maximum of 120000Megas)

However every time I try to create a file which resulting size should exceed 2Gigas I get a runtime error

In [29]: big_file = np.memmap(fnamemm, dtype=np.int16, mode='w+', shape=(np.prod(dims[1:]), len_im), order=order)
C:\Users\nuria\AppData\Local\Continuum\anaconda3\envs\caiman\lib\site-packages\numpy\core\memmap.py:247: RuntimeWarning: overflow encountered in long_scalars
  bytes = long(offset + size*_dbytes)
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<ipython-input-29-66578da2d3f6> in <module>()
----> 1 big_file = np.memmap(fnamemm, dtype=np.int16, mode='w+', shape=(np.prod(dims[1:]), len_im), order=order)

~\AppData\Local\Continuum\anaconda3\envs\caiman\lib\site-packages\numpy\core\memmap.py in __new__(subtype, filename, dtype, mode, offset, shape, order)
    248
    249         if mode == 'w+' or (mode == 'r+' and flen < bytes):
--> 250             fid.seek(bytes - 1, 0)
    251             fid.write(b'\0')
    252             fid.flush()

OSError: [Errno 22] Invalid argument

This error does not happen when the files sizes are under 2Gigas...

I have replicated the same problem with another windows 7 also 64bits

Have I forgotten something? Why is memmap acting as I have a 32bits system?

EDIT: The error is not exactly a runtime error. the variable "bytes" gets an runtime warning when trying to get the length of the file, resulting I guess in a bad argument that raises the Errno 22

senis000
  • 241
  • 2
  • 6

2 Answers2

1

I had a similar error and it turned out that it was because one of the shape=(A,B) argument was with int32 instead of int64. Try the following:

len_im64 = np.array(len_im,dtype='int64')
big_file = np.memmap(fnamemm, dtype=np.float32, mode='w+', shape=(np.prod(dims[1:]).astype('int64'), len_im), order='F')

It fixed it for me.

RemiDav
  • 463
  • 3
  • 16
0

Even though the system is 64 bit, problem may be because the application is built with 32 bit target. Check your shell execution mode (32 bit or 64 bit).

For such applications you have make them large address aware. Then the 32 applications can get access to 4GB memory on 64 bit machines.

How to do that? Here is someone's article.

https://github.com/pyinstaller/pyinstaller/issues/1288

Note: If your application is already built with 64 bit target.. ignore this and put in comment, Will delete this answer.

Pavan Chandaka
  • 11,671
  • 5
  • 26
  • 34
  • The application is python-anaconda which is supposed to be also 64bit. I'm using numpy, which when you import numpy.distutils.system_info as sysinfo and check the sysinfo.platform_bits gives 64bites as a result – senis000 Jul 30 '18 at 23:38
  • 1
    Ca you do simple check about your shell..# For 32 bit it will return 32 and for 64 bit it will return 64 import struct print(struct.calcsize("P") * 8) or https://www.w3resource.com/python-exercises/python-basic-exercise-42.php – Pavan Chandaka Jul 30 '18 at 23:54
  • Edited answer with "Check your shell execution mode (32 bit or 64 bit)." – Pavan Chandaka Jul 31 '18 at 00:13
  • print(struct.calcsize("P") * 8) == 64 – senis000 Jul 31 '18 at 18:56