0

I have a large binary file (~4GB) written in 4byte reals. I am trying to read this file using numpy fromfile as follows.

data = np.fromfile(filename, dtype=np.single)

Upon inspecting data, I see that all elements are zeros. However when I read the file in Matlab I can see that the file contains correct data and not zeros. I tested a smaller file (~2.5GB) and numpy could read that fine.

I finally tried using np.memmap to read the large file (~4GB), as

data = np.memmap(filename, dtype=np.single, mode='r')

and upon inspecting data, I can see that it correctly reads the data.

My question is why is np.fromfile giving me all zeros in the array. Is there a memory limit to what np.fromfile can read?

Deepak Dalakoti
  • 63
  • 1
  • 7
  • On what OS are you? On older versions of OS X there was a known bug about this: https://github.com/numpy/numpy/issues/2806 – ChatterOne Oct 27 '19 at 05:29
  • Have you happened to see this thread [Max limit for filesize with np.fromfile?](https://stackoverflow.com/questions/13769545/max-limit-for-file-size-with-np-fromfile)! Are you working with macOS? – Milad Sikaroudi Oct 27 '19 at 05:31
  • I am on Ubuntu 16.04.6. I am using python3.6 and numpy version 1.17.2. I seem to be having a similar problem as mentioned in the link you shared. I will have a look. Thanks! – Deepak Dalakoti Oct 27 '19 at 05:35

0 Answers0