2

I am using std::bitset and I try to create two arrays std::bitset with size 100,000,000,000. As a result, program fills only 298 MB of my RAM but must to fill ~24 GB. I have 32 GB RAM and now 26 GB are free. When I build my code for x86, it compiles and can start but for x64 it shows me the following: error C2148: total size of array must not exceed 0x7fffffff bytes. How to fix it and do not reduce size of bitset arrays?

I have tried to make 2 global arrays. Also, I have set in Microsoft visual studio -> project -> name_project properties -> configuration properties -> linker -> system -> Stack Reserve Size to 25,000,000 (i think that there must be KB, so i think i have set to ~25 GB).

... // other libraries
#include <bitset>

std::bitset <100000000000>mas;
std::bitset <100000000000>a1;

int main() {.../* work with the arrays */...}

I want to run the code with the huge std::bitset arrays.

UPD: for x86 ok, but what for x64? my code checks whole arrays and at one moment it stops.

kileatb
  • 35
  • 6
  • 1
    Regarding 32 bit build. 32 bit addressing only allows 2 GiB RAM. This is a non-starter given the size of your data. If this is running, something really odd is afoot. – user4581301 Sep 11 '19 at 18:13
  • Are you sure that's the size? Check with `sizeof` on smaller allocations. – tadman Sep 11 '19 at 18:14
  • 2
    `mas` and `a1` are not on the stack, so stack size is not the problem. I don't know what will happen if you reserve 25 GB of stack but it sounds like a terrible idea. – François Andrieux Sep 11 '19 at 18:15
  • 1
    The value `100000000000` won't fit a 32 bit unsigned integer. So the 32 bit build is probably overflowing, giving you a much smaller size of `1215752192`. – François Andrieux Sep 11 '19 at 18:19
  • ok, but what about x64 ? – kileatb Sep 11 '19 at 18:19
  • How are you measuring memory usage? Note that you can request insane amounts of memory but the system might not actually allocate portions of it until they are used. If you make an array of 1 billion elements and only use the first few hundred, you may find your program only consuming a few hundred elements worth of RAM. If you only use the first hundred and the last hundred, you might find the system only providing actual storage for the first and last hundred. – user4581301 Sep 11 '19 at 18:20
  • @kileatb The error is pretty clear, the largest allowed array is `0x7fffffff` bytes on that platform. Don't make arrays larger than that. Maybe make a larger quantity of arrays but with a smaller size. Currently, you are simply asking more than your OS can physically handle. – François Andrieux Sep 11 '19 at 18:20
  • By the way, the global variables are not stored at the stack, so modifying the stack size is not relevant – Unlikus Sep 11 '19 at 18:24
  • Can't speak for the GUI, but on the command line `/STACK:Reserve` is in bytes. Bytes makes more sense because stack is a controlled resource and usually doled out in small bunches of a few K to a few MB. – user4581301 Sep 11 '19 at 18:30
  • @user4581301 Thank you, i'll adapt in bytes – kileatb Sep 11 '19 at 18:32

1 Answers1

5

std::bitset must, at a minimum (and most implementations use the minimum), use one byte per eight bits to be stored. For 100 billion bits, that means you need ~12.5 GB of memory per bitset. Problem is, on a 32 bit system, your maximum virtual memory size for the whole program is at most 4 GB. Some of that is eaten by the kernel memory reservation for the process, so odds are you only have 2 GB of virtual address space to use; you're trying to use six times that much.

Your program cannot run on a 32 bit system without shrinking the bitset. If it claims to run, it's likely a different error; it would be truncating 100_000_000_000 to fit in a 32 bit size_t, creating a std::bitset<1215752192> instead, which would only require ~150 MB of memory, causing no problems. That would explain your observed memory usage of 298 MB; the memory usage display uses "mebibytes" (base-2, not base-10, so KiB == 1024 and MiB == 1048576), which makes each array consume just under 145 MiB, 290 MiB total for two of them, leaving 8 MiB for the rest of your program (which seems reasonable).

If it is in fact dying on x64 with that error, you're stuck; whoever implemented your std::bitset (or whatever data structure backs it, e.g. std::array) limited it to 0x7fffffff bytes even on a 64 bit system, which would limit you to std::bitsets of around 17 billion bits or less. Your only option would be to find a different provider for your standard library, or reimplement it yourself.

Update: Apparently you're using Windows, and the limit on static data size is 2GB (0x7fffffff bytes), even on 64 bit Windows; the Windows Portable Executable file format (used by .exe and .dll files) uses 32 bit offsets and lengths for each section, even for 64 bit executables. By using global std::bitsets, you're trying to store 25 GB of static data in the image, which won't work. Moving them to the stack (declaring them non-static inside the body of main) might work if you increase the Stack Reserve Size like you did, but it's still a bad idea to rely on a stack that large. I'd suggest simply dynamically allocating the bitset (e.g. auto mas = std::make_unique<std::bitset<100000000000>>()), or finding a better way to do this with a smaller bitset.

ShadowRanger
  • 143,180
  • 12
  • 188
  • 271
  • @kileatb I think we'll need a [mcve] ir order to say much more. I'll have to pass though as I've only got 16 GB at my disposal. Left my Cray in my other pants. – user4581301 Sep 11 '19 at 18:23
  • @kileatb: Your error message tells you the problem there; looks like your standard library implementer put a strict limit of `0x7fffffff` bytes on `std::bitset` or the storage implementation it uses, so you're stuck; you need a different implementation, either from a different compiler writer, or reimplementing it yourself. – ShadowRanger Sep 11 '19 at 18:26
  • @ShadowRanger Can I change the max size for my platform or i have to create a different implementation? – kileatb Sep 11 '19 at 18:29
  • @kileatb VC++ has a hard compiler limit for the size of arrays. This isn't an error produced by the standard library, it's the actual compiler complaining. I believe this is due to a Windows technical limitation to how processes are loaded. I doubt you'll get around this without changing platform entirely. – François Andrieux Sep 11 '19 at 18:33
  • 1
    @kileatb Apparently, [static data is capped at 2GB for an image](https://software.intel.com/en-us/articles/memory-limits-applications-windows). Even breaking up your `bitset`s into smaller ones won't help you. You could try dynamically allocating them (up to 8TB). – François Andrieux Sep 11 '19 at 18:37
  • [MSDN page for the error](https://learn.microsoft.com/en-us/cpp/error-messages/compiler-errors-1/compiler-error-c2148?view=vs-2019). – user4581301 Sep 11 '19 at 18:42
  • @FrançoisAndrieux: I've added that info to the answer; thanks for finding it. – ShadowRanger Sep 11 '19 at 18:44
  • How can I create std::bitset array in dynamic data? – kileatb Sep 11 '19 at 18:46
  • @kileatb: Just added that to the end of the answer; `std::unique_ptr` is probably the way to do this (low overhead, no risk of memory leak). – ShadowRanger Sep 11 '19 at 18:48
  • @ShadowRanger Note that even with dynamic allocations, the size of any given individual array is still limited to `0x7fffffff` on Windows even in 64 bit but you can allocate many of them. The technical reasons are not clear to me. – François Andrieux Sep 11 '19 at 18:51
  • @FrançoisAndrieux: Ah. So the original error is because arrays are length limited, but even if that error didn't occur, it would have failed due to the static data cap. But it also means there is no way around the issue without cobbling together multiple `bitset`s. That's... so dumb. I assume the limit doesn't apply to stuff that isn't backed by a proper array, so maybe `std::vector(100000000000)` or maybe `boost::dynamic_bitset`? might do the trick by not declaring a statically sized array? The latter should have the same features (but requires boost), the former would be more limited. – ShadowRanger Sep 11 '19 at 19:41
  • @ShadowRanger `std::vector` *might* work, depending on how it's implemented. But I imagine it may still be backed by a single array. You could write your own class to fake the contiguity. In reality I don't know why you actually would need hundreds of billions of contiguous individual bits. But I agree that in principal, it's an unfortunate limitation. This is likely a casualty of Microsoft's excellent backwards compatibility. I found some blog posts about C# and VB.NET about how they had to work around this problem for large arrays, so it's likely not fixable at the user level. – François Andrieux Sep 11 '19 at 19:53