I'm not completely sure you want to have unsigned long long
as scalar type of your Matrix; mathematically speaking, Matrices should be defined over fields, and you have to be aware that ring theory won't go easy on you if you try to find the multiplicative inverse to a positive integer (ie. the unsigned integer you have to multiply your unsigned integer with to get 1).
However, this is legal in Eigen, so we'll just stick with it -- maybe you don't want to do operations on the matrix that require these properties from your field.
So, you're saying your using win32 (which is the windows API), but not really whether your operating system is 32- or 64-bitty. If you're running a 32bit windows, no process can have more than 2GB of virtual address space, and allocating more than 2GB/sizeof(long long) unsigned long longs won't work. Now, long long is 64bit=8Byte, so the maximum number of uint64_t's you can have per 2GB is 134217728; now, you want to have them in rows of 12 columns, leaving you with a maximum of 11,184,810 rows (neglecting the fact that your numbers are not the only thing in your process' memory). Now, 11e6 < 6e8, and you have to account for the fact that you don't know which type of allocator Eigen tries to use, which could actually try to allocate more than immediately necessary.
Most likely, though, is that your 12-column format also gets padded to something that is better aligned. The Eigen documentation is not too specific on that, and I think the actual implementation depends on how your Eigen library was compiled, so I can't generally well-advise you. You can try with the DontAlign
Option in the Eigen::Matrix template.