I'm seeing this:
Missing fields! Expected 8 fields but only got 8! Last field end 2876176 and serialize buffer end 2876175.
in my Hive log. It clearly does not make sense "expect 8 got 8", so I checked the source code:
// Missing fields?
if (!missingFieldWarned && lastFieldByteEnd > structByteEnd) {
missingFieldWarned = true;
LOG.warn("Missing fields! Expected " + fields.length + " fields but " +
"only got " + fieldId + "! " +
"Last field end " + lastFieldByteEnd + " and serialize buffer end " + structByteEnd + ". " +
"Ignoring similar problems.");
}
It seems the expected bytes range went over actual byte so the Serde thinks there might be missing fields, but field id and field length are the same.
Is there a bug in this code?
if (7 == (i%8)) {
if (lastFieldByteEnd < structByteEnd) {
nullByte = bytes[lastFieldByteEnd];
lastFieldByteEnd++;
} else {
// otherwise all null afterwards
nullByte = 0;
lastFieldByteEnd++;
}
}
The last lastFieldByteEnd++;
seems to be unnecessary.