1

I have a Problem. I have a huge c++-project that I change at a few points to meet my requirements. I load more data than expected and at some point in this program there is a new vector allocated with the size of the number of data multiplied by another number. 

vector = new real[data.size()*28];

Here I get the error message:

terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc

because I can't allocate that much space . I can't change this vector because it is used in many different parts in the program and it would be very difficult and (for me) maybe impossible to fit the rest of the program to a new definition here.

What can I do that I can use this vector but get my large dataset into it?

Btw: I use eclipse, maybe I can increase the size of possible space to allocate in eclipse itself?

Thank you!

smaica
  • 723
  • 2
  • 11
  • 26
  • Are your software 32-bit? – Siyuan Ren Jun 25 '14 at 07:59
  • 1
    Don't name your variable "vector". It gives problems if you use `using namespace std;` (which I do not recommend). On top of that, name your variables as what they represent, not their storage type. – tillaert Jun 25 '14 at 07:59
  • How much data? What kind of device? How much memory does that device have? Some additional background info were nice. – Csq Jun 25 '14 at 08:00
  • What is the definition of `real` and the value of `data.size()`? – tillaert Jun 25 '14 at 08:00
  • I use linux (open suse 13.1) in a vertual box. I have about 16GB ram. The software is 64-bit. I didn't name my vector "vector" it's just to make it more easy for this post (; – smaica Jun 25 '14 at 08:03
  • @smaica Please edit the question to contain the additional details. Also, you did not answer the how much is `data.size()`, what is `real` and how much memory does your virtual machine have questions. (The memory of the host does not really matter.) – Csq Jun 25 '14 at 08:06
  • oh and the value of data.size() is about 42000000 – smaica Jun 25 '14 at 08:10
  • 1
    @smaica waht is `sizeof(real)`? Or supply your definition of `real`. Can you give a the smallest compilable example that reproduces your problem? – tillaert Jun 25 '14 at 08:16
  • 1
    Just to be sure: this variable of the name 'vector' is not really an std::vector, but a dynamic array? Is this correct? Because you got me confused by writing that the new 'vector' is being created. – KjMag Jun 25 '14 at 08:41
  • no. my vector is called 'datavec' - sizeof(real) is 8 and my problem arises when I try datavec = new real[4323388*28]; – smaica Jun 25 '14 at 09:15
  • You are trying to allocate 4323388* 28 * 8 = 968438912 bytes in one contiguous block. The new operator is telling you "I don't have a block that large". This is probably due to memory fragmentation. You have to put your data in smaller blocks. A `std::deque` as Exceptyon suggested, will allocate your data in smaller blocks, while you can still access it easily. – tillaert Jun 25 '14 at 09:31

2 Answers2

2

As Encryptyon pointed out (and he should get the credit), you need to allocate your memory as a non-contiguous block. You can do this using a std::deque. 

std::deque<float> v( data.size() * 28 );

You can access any member using the operator[]. 

real x = v[1000000];

You van also iterate over (parts of the) deque, as if it was a std::vector as the interface of a std::deque is very similar to a std::vector. However, what you cannot do, is &v[0] (or v.data() in c++11) since the internal storage of the container most probably is non-contiguous.

tillaert
  • 1,797
  • 12
  • 20
1

you can't allocate that much contiguous ram, use a deque

Exceptyon
  • 1,584
  • 16
  • 23