0

I need a large map of sets of numbers. So i'm using stxxl and it's basically something like map<std::string, std::vector<int>>. The key is 3-letters identifier. There are about 3k enteries in the map, and each of them can contain up to 500K numbers. It is important to use vector in the value since i need to change it (add/remove numbers).

The problem is that after i insert all my data, when i try to access an element in the map, the vector is currupted.

#define MAX_STRING_IN_MAP    ("ZZZZZZZ")

// String comparator.
struct CompareGreaterString
{
    bool operator () (const std::string& a, const std::string& b) const {
        return a > b; 
    }
    static std::string max_value() {
        return MAX_STRING_IN_MAP; 
    }
};
typedef stxxl::map<std::string, std::vector<int>, CompareGreaterString, DATA_NODE_BLOCK_SIZE, DATA_LEAF_BLOCK_SIZE> myMap;

...
ifstream input_file;
myMap map((mapBacteria::node_block_type::raw_size) * 5, (mapBacteria::leaf_block_type::raw_size) * 5);
...

unsigned int count = 0;
while (count < max_number) {
    count++;
    std::string name;
    unsigned int length;
    input_file >> name;
    input_file >> length;
    std::vector<int> iterationGroup = readNumbers(input_file);
    myMap.insert(std::pair<std::string, std::vector<int>>(name, iterationGroup));
    // If i am trying to access the data here, like this:
    // auto result = myMap[name];
    // It seems perfectely fine.
}

For example, if after the above insertion i will try to access the map: 

auto result = myMap["aaa"];

Then i get a vector with the right size, but all it's elements have a value of 0xfeeefeee instead of the numbers that were originally in the inserted vector.

flyman
  • 210
  • 4
  • 15
  • 1
    What's with people using a library without actually reading [its documentation](http://stxxl.sourceforge.net/tags/master/classstxxl_1_1map.html), which clearly requires the key and value types to be PODs? – T.C. Sep 26 '15 at 09:07
  • @T.C.: probably the influence of fail-friendly languages, which allow you to basically faceroll the keyboard and still get a usable error message at the point the runtime notices you're trolling it ;-) When I were a lad any error locked the hardware solid, you spent a week reading the docs before they let you *touch* a console, kids today don't know they're born etc. etc. – Steve Jessop Sep 26 '15 at 09:09
  • :( oh no. Then what should i do? i mean i could write some POD string easly, but what about the vecor? or maybe alternatives to stxxl? help please :( – flyman Sep 26 '15 at 09:12
  • @flyman: supposing you have a maximum size, you could put a `std::array` and a size field into a struct and you have your POD thing-that's-a-bit-like-a-vector. There might be better ways to structure your data, especially given that the maximum size will actually be occupied by every value, but it's a start. – Steve Jessop Sep 26 '15 at 09:20
  • it'll work great for the key, but for the values i'm afraid it might be too inefficient for my needs. You still haven't told me that think you know that will help me very much. right? please? – flyman Sep 26 '15 at 09:28
  • What is the point of using stxxl::map for a job that would be easy and efficient with std::map? – JSF Sep 26 '15 at 10:01
  • Because the data is large, and mainly because it'll get much larger. now it's about 100MB. and i predict it'll get to 4GB. – flyman Sep 26 '15 at 10:20
  • Do you really have so little ram (or a 32 bit architecture) that using swapping (if necessary) of ordinary structures isn't a better way to deal with 4GB? If you had said 400GB, I would get the point. But 4GB !? – JSF Sep 26 '15 at 11:25
  • 1
    Also, you seem to be describing a reason to use `std::map< ..., stxxl::vector>` rather than a reason to use `stxxl::map< ..., std::vector>`. Why did you invert that? – JSF Sep 26 '15 at 11:30
  • The idea is that it should be scalable and efficient. I think using map with stxxl vector is annoying (no erasing?) but maybe it's the best solution. Thank you! – flyman Sep 26 '15 at 13:22
  • Can you put in a POD array, take it out, modify (perhaps by constructing a vector from it), extract a new array, put that in instead? – Davislor Sep 26 '15 at 22:00
  • If the key is 3 letters identifier - you can use `std::array` – PiotrNycz Oct 01 '15 at 00:44

0 Answers0