Protoc failed to parse input if integer value more than 127 in Unreal Engine 5 c++

Question

Environment:

Unreal Engine 5
Windows 10
Protocol Buffers v3.18.0

I'm trying to decode serialized data in Unreal Engine 5 (c++) by using protoc. If the message contains the value of int var less than 127 everything is okay. But if the value more than 127 I catch the error: Failed to parse input.

player.proto:

syntax = "proto3";
package com.game;

message Player {
    string username = 1;
    int64 experience = 2;
}

Success case

C++ serialization and save to file:

com::game::Player MyPlayer;
MyPlayer.set_username("test user name");
MyPlayer.set_experience(127); // <-- PAY ATTENTION

// serialization
std::string MyPlayerString;
if(!MyPlayer.SerializeToString(&MyPlayerString))
{
    UE_LOG(LogGameInstance, Error, TEXT("Can't serialize MyPlayer to String"));
    return;
}
const FString MyPlayerFString(MyPlayerString.c_str());

// save to file
const FString File = *FPaths::Combine(FPaths::GameSourceDir(), FApp::GetProjectName(), TEXT("temp.dat"));
FFileHelper::SaveStringToFile(
    MyPlayerFString,
    *File,
    FFileHelper::EEncodingOptions::AutoDetect,
    &IFileManager::Get()
);

protoc --decode_raw < temp.dat

1: "test user name"
2: 127

Fail case

C++ serialization and save to file:

...
MyPlayer.set_experience(128); // <-- PAY ATTENTION
...

protoc --decode_raw < temp.dat

Failed to parse input.

I guess the problem occurs when I try to convert std::string -> FString. Any ideas?

127 is maximum signed character on almost all modern systems. Looks like you might be trying to serialize something to a signed `char` rather than an `int`. — user4581301, Sep 19 '21 at 06:38
This usually happens when the file has been created as a text file somehow, rather than a binary file. Below byte 128, you'll often get away with it, but above that: corruption is almost guaranteed (and varint encoding will end up writing a byte >= 128 in this case, although it will actually be a two byte value). Can you post the hex of the file contents, so we can see what happened? — Marc Gravell, Sep 19 '21 at 09:38
Success case: 0x0A, 0x0E, 0x74, 0x65, 0x73, 0x74, 0x20, 0x75, 0x73, 0x65, 0x72, 0x20, 0x6E, 0x61, 0x6D, 0x65, 0x10, 0x7F Fail case: 0x0A, 0x0E, 0x74, 0x65, 0x73, 0x74, 0x20, 0x75, 0x73, 0x65, 0x72, 0x20, 0x6E, 0x61, 0x6D, 0x65, 0x10, 0x3F, 0x01 — Vasyl Khvostyk, Sep 19 '21 at 10:24
My educated guess: In the failure case, `MyPlayerString` contains an embedded `\0` character. `FString MyPlayerFString(MyPlayerString.c_str());` then truncates the string at that character - I don't know what `FString` is, but it appears to assume a nul-terminated C-style string as its argument. Compare the length of `MyPlayerString` and `MyPlayerFString` in the failure case - I bet the former is longer. — Igor Tandetnik, Sep 19 '21 at 12:37
If not that, then some other transformation performed by `FString` or `SaveStringToFile` results in the bytes written to the file to be different from bytes in `MyPlayerString` — Igor Tandetnik, Sep 19 '21 at 12:43

score 1 · Accepted Answer · answered Sep 20 '21 at 17:58

I found a solution.

I'll keep it here, maybe it will help somebody. I changed the implementation to avoid string conversion.

Before:

...

// serialization
std::string MyPlayerString;
if(!MyPlayer.SerializeToString(&MyPlayerString))
{
    UE_LOG(LogGameInstance, Error, TEXT("Can't serialize MyPlayer to String"));
    return;
}
const FString MyPlayerFString(MyPlayerString.c_str());

// save to file
const FString File = *FPaths::Combine(FPaths::GameSourceDir(), FApp::GetProjectName(), TEXT("temp.dat"));
FFileHelper::SaveStringToFile(
    MyPlayerFString,
    *File,
    FFileHelper::EEncodingOptions::AutoDetect,
    &IFileManager::Get()
);

After:

// serialization
const auto MyPlayerSize = MyPlayer.ByteSizeLong();
const auto MyPlayerBytes = new uint8[MyPlayerSize]();
if(!MyPlayer.SerializeToArray(MyPlayerBytes, MyPlayerSize))
{
    UE_LOG(LogGameInstance, Error, TEXT("Can't serialize MyPlayer to Array"));
    return;
}
TArray64<uint8>* MyPlayerArray = new TArray<uint8, FDefaultAllocator64>(MyPlayerBytes, MyPlayerSize);

// save to file
const FString File = *FPaths::Combine(FPaths::GameSourceDir(), FApp::GetProjectName(), TEXT("temp.dat"));
FFileHelper::SaveArrayToFile(
    *MyPlayerArray,
    *File,
    &IFileManager::Get()
);

Protoc failed to parse input if integer value more than 127 in Unreal Engine 5 c++

Success case

Fail case

1 Answers1