I have a need to write > 2GB to a an instance of flatbufferBuilder class. The following question on StackOverflow briefly covers this topic. I am looking for options to bypass this by considering the advice mentioned in the linked question which is to Create a sequence of flatbuffers
//flatbuffer struct contains a vector that can grow ~3.5 million in size(which consequently exceeds 2GB)
struct ValidationReportT : public flatbuffers::NativeTable
{
std::vector<flatbuffers::unique_ptr<ValidationDefectT>> defects{};
}
//The code snippet below is my attempt to create a sequence of flatbufferBuilder instances.
void createReport(std::vector<flatbuffers::FlatBufferBuilder>& fbbDefectsVec)
{
ValidationReportT report; //report has all defects already populated
ValidationReportT partialReport; //Partial report captures blocks of max_elems, writes to flatbufferBuilder class, then the partialReport defects vector is cleared
if (report.defects.size())
{
int size = report.defects.size();
auto elem_size = sizeof(*report.defects[0]);
float32_t max_elems = FLATBUFFERS_MAX_BUFFER_SIZE / (float32_t)elem_size;
auto fbbsNeeded = size < max_elems ? 1 : (int)ceil(size / max_elems);
fbbDefectsVec.resize(fbbsNeeded);
int offset = 0;
int idx = 0;
size_t start = offset;
size_t end = offset + max_elems; //Loop over blocks of max elems
while (size > max_elems)
{
for (size_t i = start; i < end; i++)
{
partialReport.defects.push_back(std::move(report.defects[i]));
}
offset += max_elems;
size -= max_elems;
fbbDefectsVec[idx].Finish(
validation::ValidationReport::Pack(fbbDefectsVec[idx], &partialReport));
idx++;
start = offset;
end = offset + max_elems;
partialReport.defects.clear();
}
if(size > 0)
{
//Copy the remaining defects
//Set the loop bounds
if(max_elems >= report.defects.size()) //All defects can be fit into a flatbuffer vector
{
start = 0;
end = report.defects.size();
}
else
{
start = end;
end = report.defects.size();
}
for(size_t i = start; i < end; i++)
{
partialReport.defects.push_back(std::move(report.defects[i]));
}
fbbDefectsVec[idx].Finish(
validation::ValidationReport::Pack(fbbDefectsVec[idx], &partialReport));
}
}
}
Even though partialReport holds only the maximum allowed element count based on FLATBUFFERS_MAX_BUFFER_SIZE, I still get the size limit assertion failed:
Assertion `size() < FLATBUFFERS_MAX_BUFFER_SIZE' failed.
Aborted
On this particular line
fbbDefectsVec[idx].Finish(
validation::ValidationReport::Pack(fbbDefectsVec[idx], &partialReport));
Why is this so? And how do I bypass this?
