How can I *directly* append to a QVariantList, stored in a QVariantMap?

Question

I have a collection of QVariantMaps which contains QVariantLists in SOME of their entries.

I need to append to the lists. These lists can grow rather large, and I have to do this for many of them, many times per second, basically the whole time the program is running. So, efficiency is pretty important here..

I'm not arriving at a way to avoid making copies of these lists though! They might as well be immutable. :(

This works:

map[ key ] = QVariantList() << map[ key ].toList() << someOtherVarList;

But, that's doing a whole lot more work (under the hood) than I want...

Here's something I've tried, but ran into a wall:

I can get a non-const pointer to the variant I want to modify directly by using this macro I wrote utilizing a QMutableMapIterator:

#define getMapValuePtr( keyType, valueType, map, keyValue, ptrName ) \
    valueType *ptrName( nullptr ); \
    for( QMap<keyType, valueType>::iterator it = map.begin(); \
         it != map.end(); ++it ) \
        { if( it.key()==keyValue ) { ptrName = &(*it); break; } }

Example:

const QString key( "mylist" );
QVariantMap map;
map[ key ] = QVariantList( {"a","b","c"} );           
getMapValuePtr( QString, QVariant, map, key, valuePtr )

So now valuePtr could give me the ability to directly modify the value in the map, but the fact it's a QVariant is still in the way...

I have next tried the following without Qt Creator highlighting anything to indicate an error, but it does NOT quite compile despite that...

QVariantList *valListPtr( qvariant_cast<QVariantList *>(*valuePtr) );    
valListPtr->append( QVariantList( {"x","y","z"} ) );

Ideas?

Note: Not all entries in these maps are list. I CANNOT therefore change my map type to QMap<QString,QVariantList>. I tried, and the many client classes to this code all threw fits... That would have simplified matters, had it worked, by implicitly eliminating this final sticking point.

Answer 1

I have a collection of QVariantMaps which contains QVariantLists in SOME of their entries.

I need to append to the lists.

Don't. Obviously you have some schema to these QVariants, otherwise you wouldn't know if they are a list or not. Do you operations against a strongly typed model of that, then at the very last moment, transform that into a QVariant (or whatever)

map[ key ] = QVariantList() << map[ key ].toList() << someOtherVarList;

That's copying every element at least twice, and looking up the variant twice, so you can shave an immediate 50% by dropping it to

auto & var = map[ key ]; 
var = (var.toList() << someOtherVarList);

The main problem is that QVariant is garbage for performance. Contrast std::any which lets you have reference access to the stored value.

Answer 2

Got it!

I was on the right track, but had to perform a deep dive into the Qt source to figure out how to finish this off. I now have another macro to call upon, which takes me the rest of the way:

#define varPtrToListPtr( varPtr, ptrName ) \
    QVariantList *ptrName( varPtr->type() == QVariant::List ? \
        static_cast<QVariantList *>( static_cast<void *>( &(varPtr->d.data.c) ) ) \
        : nullptr );

So, here's the example from above with the rest of the story filled in:

const QString key( "mylist" );
QVariantMap map;
map[ key ] = QVariantList( {"a","b","c"} );           
getMapValuePtr( QString, QVariant, map, key, valuePtr )
if( !valuePtr ) return; // imagine this being inside a function...
varPtrToListPtr( valuePtr, valueListPtr )
if( valueListPtr ) *valueListPtr << QVariantList( {"x","y","z"} );

This achieves the same results as the straight forward one-liner I showed in the question, but now I'm avoiding all the overhead of creating and destroying temp copies of giant lists!

Admittedly, the initial search with the iterator might burn some time compared to using QMap::value( key ), but none of my maps here have many k/v pairs, so the overall gain of avoiding the list copies negates that.

Update:

• I ran some tests using this method vs the original and observed it literally shave MORE than 99% of the time required off the same workload. Like doing a task in 50 ms rather than the painful 7.5 seconds the old way would take.

• This was originally written on Windows, against MSVC. I discovered, however, that on other platforms/compilers this fails to build unless you make a 1 line tweak to the Qt base library:

  • Open the Qt header for class QVariant.
  • Search the file for the line containing: Private d;
  • Directly above that, add the line: public: // HACK!. That aligns other build contexts with the MSVC access to that class member.

Note: I fully acknowledge the "bad form", and the downsides, to this solution... BUT the payoff may be worth it for you. It was for my use case.