How do I map a C struct with padding over 32 bytes using serde and bincode?

Question

I'm mapping a binary structure using serde and bincode.

#[macro_use]
extern crate serde_derive;
extern crate serde;
extern crate bincode;

#[derive(Serialize, Deserialize)]
struct Superblock {
    magic: [u8; 16],
    //reserved: [u8; 492],
    crc: u32,
}

Things work as expected, but I can't map the reserved field. Apparently fixed size arrays are only defined for sizes up to 32 bytes.

How do I register my custom-sized array so that the padding gets deserialised?

Is serde+bincode the right approach? I need control over endianness (which bincode provides) and I like the declarative style.

I think I'll just drop the padding and grab the CRC separately using byteorder. — Gabriel, Jun 10 '17 at 01:01
If you do want to use a fixed-sized array, https://github.com/fizyk20/generic-array/ might be useful for this. It has a `serde` feature which allows serde deserialization. It's not exactly a fixed-sized array, it's implemented slightly differently, but the memory representation is exactly the same and it should allow for all the same uses. — daboross, Jun 10 '17 at 20:35

score 7 · Answer 1 · answered Jun 10 '17 at 07:38

serde_derive supports the field attributes #[serde(serialize_with="func")], #[serde(deserialize_with="func")] and #[serde(with="module")], which allows one to provide a custom serialization/deserialization routine:

#[derive(Serialize, Deserialize)]
struct Superblock {
    magic: [u8; 16],
    #[serde(with="array_492")]   // <--
    reserved: [u8; 492],
    crc: u32,
}

mod array_492 {
    use serde::*;

    pub fn serialize<S, T>(array: &[T; 492], ser: S) -> Result<S::Ok, S::Error> 
        where S: Serializer, T: Serialize
    {
        unimplemented!() // fill in yourself.
    }

    pub fn deserialize<'de, D, T>(de: D) -> Result<[T; 492], D::Error> { 
        where D: Deserializer<'de>, T: Deserialize<'de>
    {
        unimplemented!() // fill in yourself.
    }
}

For an actual implementation, see this gist: https://gist.github.com/kennytm/21403667b5a17172cfcd11f9df9365e2. Note that this is not optimized for deserializing the whole byte array at once.

Thanks! Since this is just padding, an almost-empty implementation is fine. — Gabriel, Jun 10 '17 at 22:58

Shepmaster · Accepted Answer · 2017-06-11T02:18:12.687

I'd recommend implementing Serialize and implementing Deserialize for a custom type.

The big thing to note is that you don't care about the data at all. This means there's no reason to take up memory! We can define a type Reserved that will serialize to a bunch of bytes and will deserialize from bytes, but doesn't actually require any space in our in-memory struct.

Then, it's just a matter of filling in the trait implementation:

#[macro_use]
extern crate serde_derive;
extern crate serde;
extern crate bincode;

use std::fmt;
use serde::ser::SerializeTuple;

#[derive(Debug, Serialize, Deserialize, PartialEq)]
struct Superblock {
    magic: [u8; 16],
    reserved: Reserved,
    crc: u32,
}

#[derive(Debug, PartialEq)]
struct Reserved;
const RESERVED_LENGTH: usize = 492;

impl serde::Serialize for Reserved {
    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
        where S: serde::Serializer
    {
        let mut tuple = serializer.serialize_tuple(RESERVED_LENGTH)?;
        for _ in 0..RESERVED_LENGTH {
            tuple.serialize_element(&0xA0_u8)?; // Just to see it easily in the output
        }
        tuple.end()
    }
}

impl<'de> serde::Deserialize<'de> for Reserved {
    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
        where D: serde::Deserializer<'de>
    {
        struct Visitor;
        impl<'de> serde::de::Visitor<'de> for Visitor {
            type Value = Reserved;

            fn expecting(&self, f: &mut fmt::Formatter) -> fmt::Result {
                write!(f, "{} bytes", RESERVED_LENGTH)
            }

            fn visit_seq<A>(self, mut tuple: A) -> Result<Self::Value, A::Error>
                where A: serde::de::SeqAccess<'de>,
            {
                for _ in 0..RESERVED_LENGTH {
                    tuple.next_element::<u8>()?;
                }
                Ok(Reserved)
            }
        }

        deserializer.deserialize_tuple(RESERVED_LENGTH, Visitor)
    }
}

fn main() {
    let block = Superblock {
        magic: [
            0x00, 0x01, 0x02, 0x03,
            0x04, 0x05, 0x06, 0x07,
            0x08, 0x09, 0x0a, 0x0b,
            0x0c, 0x0d, 0x0e, 0x0f,
        ],
        reserved: Reserved,
        crc: 0xffffffff,
    };

    let ser = bincode::serialize(&block, bincode::Infinite).expect("Couldn't serialize");
    println!("length: {}", ser.len());
    println!("{:?}", ser);

    let block2: Superblock = bincode::deserialize(&ser).expect("Couldn't deserialize");
    assert_eq!(block, block2);
    println!("{:?}", block2);

    println!("Takes: {} bytes", std::mem::size_of::<Superblock>());
    // prints "Takes: 20 bytes"
}

Thanks! With the length as a generic parameter, this might be a worthy addition to bincode. — Gabriel, Jun 11 '17 at 02:19
@Gabriel sadly, type-level numerics don't exist (yet), so you'd have to use something like [typenum](https://crates.io/crates/typenum) which adds a bit of inelegance. However, when they do land, I think it's something that would perfectly fit with bincode! — Shepmaster, Jun 11 '17 at 02:21

score 3 · Answer 3 · answered Jun 10 '17 at 11:33

3

As a workaround, you could construct a different object of the right size and ignore that instead. For example:

reserved: ([u64; 32], [u64; 29], u32) // 492 bytes

answered Jun 10 '17 at 11:33

Peter Hall

53,120
14
139
204

How do I map a C struct with padding over 32 bytes using serde and bincode?

3 Answers3

Linked