How to stream process a flatbuffer bigger than RAM?

Question

We have following scenario:

We are building a flatbuffer application in an environment with only static memory allocation allowed.
Our incomming flatbuffers contain software update images which are bigger than our whole available RAM on the processing unit.
We require to process the incomming flatbuffer partially and stream the images to another unit for storage.
The processing unit does not have persistant storage, we cannot dump the big flatbuffer and utilize mmap().
We want to utilize the flatbuffers::Verifier class to check that our received flatbuffer is correct

Our approach was to store meta information about the objects in [ObjectInfo] and binary concat all the image objects and put it into a ubyte vector.

But we have to manually track the pointer to the buffer and where we are within the objects vector. We do not profit from flatbuffer generated code to access the objects.

Schema example for problem:

root_type Container;

table Container {
    // Information about the big objects, has to fit into RAM,
    metaInfo:[ObjectInfo]
    // Does not fit into RAM, has to be streamed with arbitrary size buffer
    bigObjects:[Objects];
}

table ObjectInfo {
    type:int;  
    name:string;
    offset:double; // Offset within Objects
    length:double; // length of object
}

table Objects {
    objects:[ubyte] // Contains the objects, contatinated 
}

We need to partially process flatbuffers. What could we do?

score 3 · Accepted Answer · answered Apr 04 '19 at 17:56

There is no elegant way to do this with FlatBuffers, it would require a lot of hacks. You can't verify a partial buffer. You can't follow references to other tables safely, etc.

If you want to stream, put each piece of data in its own FlatBuffer that individually fits in ram, then stream a sequence of such buffers. You can use the SizePrefixed functionality to make these buffers easy to stream. You can use the file_identifier to recognize different kinds of buffers.

Also, don't use double for offset/length data, use an appropriately sized integer type.

How to stream process a flatbuffer bigger than RAM?

1 Answers1