Storing the data representations of multiple, differently typed objects in a single Data instance

Question

Motivation

To my knowledge, Data is a struct that abstracts a byte buffer. It references a physical area in memory, in other words: a contiguous number of bytes. Now I want to efficiently store multiple values in memory (as raw data), where the values are not all of the same type.

My definition of efficient here ≔ Store all those values without any unused buffer / gap bytes.

Storing the raw data in memory

let a: UInt8 = 39
let b: Int32 = -20001
let string: String = "How awesome is this data?!"

Now I want to store the data of all those values sequentially in memory, without any type information.

let data = [a.asData, b.asData, string.asData].concatenated()

Imagine that the .asData property retrieves the byte representations of each instance as a [UInt8] array and then wraps those in a Data instance. The concetenated() method then just concatenates these 3 Data instances to a single Data instance as follows:

extension Collection where Element == Data {
    func concatenated() -> Data {
        reduce(into: Data()) { (result, nextDataChunk) in
            result.append(nextDataChunk)
        }
    }
}

Reading the data back from memory into the respective types

Let's assume this all worked great and I now have this single Data instance from which I want to restore the 3 original values (with their original types). This is what I do:

var cursor = 0

let a: UInt8 = data.withUnsafeBytes { pointer in
    pointer.load(fromByteOffset: cursor, as: UInt8.self)
}
cursor += MemoryLayout<UInt8>.size // +1

let b: Int32 = data.withUnsafeBytes { pointer in
    pointer.load(fromByteOffset: cursor, as: Int32.self)
}
cursor += MemoryLayout<Int32>.size // +4

let string: String = data.withUnsafeBytes { pointer in
    pointer.load(fromByteOffset: cursor, as: String.self)
}
cursor += MemoryLayout<String>.size // +16

The Problem

The problem is that this throws a runtime error:

Fatal error: load from misaligned raw pointer

and I know exactly why:

Int32 has an alignment of 4 (because it's 4 bytes long). In other words: When reading data with a raw pointer, the first byte of the Int32 must be at an index that is a multiple of 4. But as the first value is a UInt8 only, the data bytes for the Int32 start at index 1, which is not a multiple of 4. Thus, I get the error.

---

My question is this:

• Can I somehow use the raw Data that represents instances of different types to recreate such instances without alignment errors? How?

• And if this is not possible, is there a way to automatically align the Data chunks correctly when concatenating them in the first place?

Answer 1

The issue about misaligned data is that you need to use Data's subdata method. Besides that you can create some helpers to make your life easier as follow:

This would convert any numeric type to Data:

extension Numeric {
    var data: Data {
        var bytes = self
        return .init(bytes: &bytes, count: MemoryLayout<Self>.size)
    }
}

This would convert any type that conforms to String Protocol to Data (String/Substring)

extension StringProtocol {
    var data: Data { .init(utf8) }
}

This would convert any valid utf8 encoded sequence of bytes (UInt8) to string

extension DataProtocol {
    var string: String? { String(bytes: self, encoding: .utf8) }
}

This is a generic method to convert the bytes to object or to a collection (array) of objects:

extension ContiguousBytes {
    func object<T>() -> T { withUnsafeBytes { $0.load(as: T.self) } }
    func objects<T>() -> [T] { withUnsafeBytes { .init($0.bindMemory(to: T.self)) } }
}

and a simplified generic version to concatenate an array of data:

extension Collection where Element == DataProtocol {
    var data: Data { .init(joined()) }
}

---

Usage:

let a: UInt8 = 39
let b: Int32 = -20001
let string: String = "How awesome is this data?!"
let data = [a.data, b.data, string.data].data

// just set the cursor (index) at the start position
var cursor = data.startIndex
// get the subdata from that position onwards
let loadedA: UInt8 = data.subdata(in: cursor..<data.endIndex).object()  // 39
// advance your cursor for the next position
cursor = cursor.advanced(by: MemoryLayout<UInt8>.size)
// get your next object
let loadedB: Int32 = data.subdata(in: cursor..<data.endIndex).object()  // -20001
// advance your position to the start of the string data
cursor = cursor.advanced(by: MemoryLayout<Int32>.size)
// load the subdata as string
let loadedString = data.subdata(in: cursor..<data.endIndex).string  // "How awesome is this data?!"

---

edit/update: Of course loading the string only works because it is located at the end of your collection of bytes otherwise you would need to use 8 bytes to store its size:

let a: UInt8 = 39
let b: Int32 = -20001
let string: String = "How awesome is this data?!"
let c: Int = .max
let data = [a.data, b.data, string.count.data, string.data, c.data].data

var cursor = data.startIndex
let loadedA: UInt8 = data.subdata(in: cursor..<data.endIndex).object()  // 39
print(loadedA)
cursor = cursor.advanced(by: MemoryLayout<UInt8>.size)
let loadedB: Int32 = data.subdata(in: cursor..<data.endIndex).object()  // -20001
print(loadedB)
cursor = cursor.advanced(by: MemoryLayout<Int32>.size)
let stringCount: Int = data.subdata(in: cursor..<data.endIndex).object()
print(stringCount)
cursor = cursor.advanced(by: MemoryLayout<Int>.size)
let stringEnd = cursor.advanced(by: stringCount)

if let loadedString = data.subdata(in: cursor..<stringEnd).string {  // "How awesome is this data?!"
    print(loadedString)
    cursor = stringEnd
    let loadedC: Int = data.subdata(in: cursor..<data.endIndex).object()  // 9223372036854775807
    print(loadedC)
}

This would print

39
-20001
26
How awesome is this data?!
9223372036854775807