QList crashes when size is large

Question

I am using a QList to store the data read from a SQL Table. The table has more than a million records. I need to get them in a list and then do some processing on the list.

QList<QVariantMap> list;

QString selectNewDB = QString("SELECT * FROM newDatabase.M106SRData");
QSqlQuery selectNewDBQuery = QSqlDatabase::database("CurrentDBConn").exec(selectNewDB);
while (selectNewDBQuery.next())
{
    QSqlRecord selectRec = selectNewDBQuery.record();
    QVariantMap varMap;
    QString key;
    QVariant value;
    for (int i=0; i < selectRec.count(); ++i)
    {
        key = selectRec.fieldName(i);
        value = selectRec.value(i);
        varMap.insert(key, value);
    }
    list << varMap;
}

I get "qvector.h, line 534: Out of memory" error.

The program crashes when the list reaches the size of <1197762 items>. I tried using reserve() but it didn't work. Is QList limited to a specific size?

Can you see if the memory used by your program is exceeding the memory installed on your machine. — Venom, Apr 10 '17 at 15:40
"QList crashes..." No, it didn't. Classes are not capable of crashing; what crashed was the application. — MrEricSir, Apr 10 '17 at 15:57
Besides, from the looks of it, it seems that QVector is crashing ;) — dtech, Apr 10 '17 at 16:02

score 3 · Accepted Answer · answered Apr 10 '17 at 15:43

You've ran out of memory because the C++ runtime has reported that it cannot allocate any more memory. It's not a problem with Qt containers. The containers are limited to 2^31-1 items due to the size of int the use for the index. You're nowhere near that.

At the very least:

Use a QVector instead of QList as it has much lower overhead for the QVariantMap element.
Attempt to reserve the space if the query allows it: this will almost halve the memory requirements!
Compile for a 64 bit target if you can.

QVector<QVariantMap> list;

QString selectNewDB = QString("SELECT * FROM newDatabase.M106SRData");
QSqlQuery selectNewDBQuery = QSqlDatabase::database("CurrentDBConn").exec(selectNewDB);
auto const size = selectNewDBQuery.size();
if (size > 0) list.reserve(size);
while (selectNewDBQuery.next())
{
    auto selectRec = selectNewDBQuery.record();
    QVariantMap varMap;
    for (int i=0; i < selectRec.count(); ++i)
    {
        auto const key = selectRec.fieldName(i);
        auto const value = selectRec.value(i);
        varMap.insert(key, value);
    }
    list.append(varMap);
}

Thanks for the suggestions. Using QVector didn't help. Yes, I am building using a 32-bit Qt. I will try building using a 64-bit now. — medasumanth, Apr 10 '17 at 15:58
Do you need to store a `QVariantMap` and variants themselves? Do you need to store the redundant field names? No. Create a `struct` to represent the row of the table, and store that in the `QVector`. It will likely work then. Also ask yourself: why are you moving all that data from the database to RAM? Just query the database when you need it. — Kuba hasn't forgotten Monica, Apr 10 '17 at 16:13
Yes, what you said made sense. I did some code refactor and I am not saving the entire result set in the QList/QVector now. Instead I am working on one record at a time. Thanks for your advice. — medasumanth, Apr 10 '17 at 19:03

dtech · Answer 2 · 2017-04-10T16:12:22.503

You either don't have enough ram, or more likely are using a 32bit Qt build, which cannot utilize more than 4 GB of ram. Or maybe both. Size wise the container itself should be able to handle more than 2 billion elements.

QList ain't helping either, as in your case it will likely store every element as a pointer and do an additional heap allocation for the actual variant map. So you end up with a sizeable additional heap allocation overhead.

And since the query already contains a significant amount of data, it probably eats a decent amount of ram itself.

Unless you have disabled pagefile, running out of ram on its own should not result in a crash, as it would just start paging and ruin performance, but keep running, so you are likely hitting the memory limit for a 32 bit process, which may be as low as a mere 2 GB.

Aside from doing the things Kuba suggested in his answer, you might want to split your query into smaller pieces, and get the results in a few queries rather than one if possible, and process them one at a time, reducing the memory used by the query results and freeing the memory for a query once you are done with it.

There is also the option of saving on RAM from QString, in case you have a lot of repeating strings. As it is implicitly shared, you can have a bunch of identical strings that all use the same underlying data. You can take advantage of this, by using a QSet to keep a collection of unique strings and a quick check if a string is already present. Then instead of using the string from the query result, use the one from the set. All identical strings copied by value from the set will reuse the same string data. In contrast, your current approach will use n amount of space for every n duplicated strings.

Thanks for the response. I will try with the options you suggested. The target computer has only 2 GB of RAM space and also the application is a 32-bit one. — medasumanth, Apr 10 '17 at 16:05
@medasumanth - check also the last addition I made on the strings, that may be able to save you extra memory. — dtech, Apr 10 '17 at 16:12
"You either don't have enough ram, or more likely are using a 32bit Qt build […]” - or just not a big enough block of *consecutive* memory, which one usually runs into much quicker. — Frank Osterfeld, Apr 11 '17 at 07:34
@FrankOsterfeld - there is still the option to use virtual memory. If it crashes, then most likely the process has ran out of addressing space. — dtech, Apr 11 '17 at 09:35

QList crashes when size is large

2 Answers2