Why are the results of Go and Rust serialization inconsistent？

Question

The results I want are as follows， but the result of Rust is another.

 [123 34 66 111 100 121 34 58 34 97 71 86 115 98 71 56 61 34 125]

golang sample:

type Response struct {
    Body    []byte
}
func Test_Marshal(t *testing.T)  {
    body := "hello"
    resp := &Response{
        Body:[]byte(body),
    }
    t.Log("resp:", resp)
    result, _ := json.Marshal(resp)
    t.Log("result: ", result)
}

go result:

    aa_test.go:17: resp: &{[104 101 108 108 111]}
    aa_test.go:19: result:  [123 34 66 111 100 121 34 58 34 97 71 86 115 98 71 56 61 34 125]

rust sample:

use serde_json::{Result, Value};
use serde::Serialize;

#[derive(Serialize, Debug)]
pub struct Response {
    pub body: ::std::vec::Vec<u8>,
}

fn main() {
    let body_str = "hello".to_string();
    let resp = Response {
        body: Vec::from(body_str)
    };

    println!("resp: {:?}", resp);

    let result = serde_json::to_vec(&resp).unwrap();
    println!("result: {:?}",result)
}

rust result

resp: Response { body: [104, 101, 108, 108, 111] }
result: [123, 34, 98, 111, 100, 121, 34, 58, 91, 49, 48, 52, 44, 49, 48, 49, 44, 49, 48, 56, 44, 49, 48, 56, 44, 49, 49, 49, 93, 125]

The gonalg json contains a field called `Body`, and the rust contains a field called `body`. The difference is the capital `B` in the rust version. — pigeonhands, Dec 27 '21 at 06:34
But what are you reading and comparing JSON as bytes instead of string ? There's absolutely no guarantee that different serializers would have the same formatting even if you did have the same field names. If you want a strict binary format, use another serialization format. — Denys Séguret, Dec 27 '21 at 06:36
What's exactly your question ? What's "inconsistent" ? Are you comparing the two `result` byte arrays ? Did you expect them to be equal ? Why ? — Denys Séguret, Dec 27 '21 at 06:42
Two background programs. The rust signs the response through np-256, and then golang needs to verify the signature of the response, so the [] bytes need to be equal. — bin zheng, Dec 27 '21 at 07:18

score 5 · Answer 1 · answered Dec 27 '21 at 06:47

If we convert both your outputs from byte arrays to ASCII strings (I've used a simple Javascript code - array.map(ch => String.fromCharCode(ch)).join("")), we'll get the following:

// for Rust
{"body":[104,101,108,108,111]}
// for Go
{"Body":"aGVsbG8="}

So, there seem to be two differences:

The first and the most obvious is the fields naming. Both serializers leave the field name in JSON as it was in the source code, so in Go it is Body (capitalized), while in Rust it is body (lower-case).
The second is that Rust encodes the value as-is - i.e. the byte array is treated as a vector of numbers (it doesn't get any special treatment). In Go, however, it is converted to Base64 string - this is an explicitly documented exception, probably because in most cases this is the intended behavior.

Now, what to do here? It depends on what behavior you want to use. I don't know how to change output with Go, so I assume that it is correct, and the Rust one needs to be tweaked.

To fix the key, you can simply annotate your field with serde::rename("Body").
To fix the value, the simplest way is probably to use serde::serialize_with attribute together with base64 crate - an example can be found here:

use serde::{Serialize, Deserialize};

#[derive(Serialize, Deserialize)]
pub struct Response {
    #[serde(with="base64")]
    pub body: Vec<u8>,
}

mod base64 {
    use serde::{Serialize, Deserialize};
    use serde::{Deserializer, Serializer};

    pub fn serialize<S: Serializer>(v: &Vec<u8>, s: S) -> Result<S::Ok, S::Error> {
        let base64 = base64::encode(v);
        String::serialize(&base64, s)
    }
    
    pub fn deserialize<'de, D: Deserializer<'de>>(d: D) -> Result<Vec<u8>, D::Error> {
        let base64 = String::deserialize(d)?;
        base64::decode(base64.as_bytes())
            .map_err(|e| serde::de::Error::custom(e))
    }
}

Through your method "#[serde (with =" Base64 ")]", it works. thank you — bin zheng, Dec 27 '21 at 07:10

Why are the results of Go and Rust serialization inconsistent？

1 Answers1