1

I use Eigen's matrix format to read a previously acquired multi-dimensional data:

Eigen::Matrix<unsigned long long, Eigen::Dynamic, 12> dummyData;

and later on after knowing my data size:

dummyData.resize(PackSize, 12);

PackSize could be in order of 6e08. To avoid integer overflow when computing buffer size, I used unsigend long long to be able to address (PackSize* 12*8) mod (2^32) on win32. Yet, I come up with: Unhandled exception at 0x75362F71 in DataRead.exe: Microsoft C++ exception: std::bad_alloc at memory location 0x004CBCF0. Could someone please help me to handle this problem?

QuestionMark
  • 412
  • 1
  • 4
  • 16
  • Do you know how much memory you need in total, to me it seems a lot and if your system can't handle that amount you should recieve a `std::bad_alloc`. – hetepeperfan Jan 09 '15 at 15:21
  • Er...that's roughly 54 GB you're trying to allocate there. That's more than the virtual address space in win32. – Wintermute Jan 09 '15 at 15:21
  • Have you considered foregoing 32-bit and instead create a 64-bit application? – PaulMcKenzie Jan 09 '15 at 15:21
  • @Wintermute I know I am allocating a big chunk of memory! Actually I do not know any alternative solutions. Could you please give me a hint about how to handle this problem? – QuestionMark Jan 09 '15 at 15:26
  • 1
    Get a 64 bit operating system and more than 54 GB of RAM. There's really nothing you can do to get 54 GB allocated into an address space that's 4 GB large (of which half is reserved for the operating system). Or restate your problem in a way that requires less memory. Since you didn't state what your problem is, I cannot help you with that. Oh, and note that most consumer-grade CPUs can only address up to 32 GB today, so you'll have to get a server CPU (such as a Xeon). – Wintermute Jan 09 '15 at 15:28
  • @Wintermute Thanks for the prompt reply. The data are raw information from different channels of a sensor that I need to save for another application. – QuestionMark Jan 09 '15 at 15:41

1 Answers1

1

I'm not completely sure you want to have unsigned long long as scalar type of your Matrix; mathematically speaking, Matrices should be defined over fields, and you have to be aware that ring theory won't go easy on you if you try to find the multiplicative inverse to a positive integer (ie. the unsigned integer you have to multiply your unsigned integer with to get 1).

However, this is legal in Eigen, so we'll just stick with it -- maybe you don't want to do operations on the matrix that require these properties from your field.

So, you're saying your using win32 (which is the windows API), but not really whether your operating system is 32- or 64-bitty. If you're running a 32bit windows, no process can have more than 2GB of virtual address space, and allocating more than 2GB/sizeof(long long) unsigned long longs won't work. Now, long long is 64bit=8Byte, so the maximum number of uint64_t's you can have per 2GB is 134217728; now, you want to have them in rows of 12 columns, leaving you with a maximum of 11,184,810 rows (neglecting the fact that your numbers are not the only thing in your process' memory). Now, 11e6 < 6e8, and you have to account for the fact that you don't know which type of allocator Eigen tries to use, which could actually try to allocate more than immediately necessary.

Most likely, though, is that your 12-column format also gets padded to something that is better aligned. The Eigen documentation is not too specific on that, and I think the actual implementation depends on how your Eigen library was compiled, so I can't generally well-advise you. You can try with the DontAlign Option in the Eigen::Matrix template.

Marcus Müller
  • 34,677
  • 4
  • 53
  • 94
  • 11,184,810 is roughly 11e6, not 11e8. – Wintermute Jan 09 '15 at 16:02
  • By the way, if you're trying to load 54GB of data, you might be doing something that screams "supercomputing", or you might not actually want to do that in-RAM at all. This will get algorithmically complicated, but you just couldn't stick with a smaller problem, could you ;P ? – Marcus Müller Jan 09 '15 at 16:15
  • @MarcusMüller What I'm trying to store is the data in Batch. I'm thinking of reading the data in several chunks, though it makes synchronization of data from different sources a bit painful! – QuestionMark Jan 09 '15 at 16:27