Deserialize part of a binary file

Question

Is it possible to deserialize part of a binary file?

Basically I have an object similar to below, which I serialize into a binary file.

public class MyObject
{
    public string Name { get; set; }

    public int Value { get; set; }

    public IList<MyOtherObject> { get; set; } // lots of data in here (order of kB-MB)
}

What I would like is to be able to deserialize only Name and Value by way of populating a ListView for file selection purposes and then deserialize the rest of the file when needed (i.e. the user chooses that file from the ListView).

As always, any help greatly appreciated and if any 3rd party libraries are suggested they would need to be able to be used freely in a commercial environment.

Might be worth mentioning how you have serialized the file in the first place — musefan, Nov 12 '12 at 10:49
Here's a similar question: http://stackoverflow.com/questions/1572999/c-sharp-partial-deserialization might contain the answer you are looking for. — Rafael Emshoff, Nov 12 '12 at 10:51
Just want to confirm, The list View Items represent sections of the Binary File, and you only want to load those parts that the user selects? — Derek, Nov 12 '12 at 10:56
@musefan I did say in the question, I will update to make it more clear (P.S. love Muse) — dav_i, Nov 12 '12 at 11:00
The problem I can see here is that : most serializers that support reading back partial data (into a different type) **will not** support `object` / `IList — Marc Gravell, Nov 12 '12 at 11:00
@RafaelCichocki Thanks, however, that question is for XML and I am using Binary — dav_i, Nov 12 '12 at 11:02
@Derek The `ListView` shows summary data, as in Explorer (i.e. name, file size, date created, but also some custom data specific to my program) then the ideal is to load the rest of the data when needed. — dav_i, Nov 12 '12 at 11:03
@dav_i the answer seems to hint at a solution for binary serialization, why I thought of linking it here, but it's not comprehensive. — Rafael Emshoff, Nov 12 '12 at 11:04
@MarcGravell Ah don't worry about that I thought that `IList — dav_i, Nov 12 '12 at 11:04
Technically, *all* serializers are (give-or-take the use of `Encoding`) "binary". I guess the real question is : are you tied to `BinaryFormatter` ? — Marc Gravell, Nov 12 '12 at 11:15
@MarcGravell Thankfully not (except for some legacy data (but I can handle that separately))! I'm currently researching "protobuf-net" which you suggested. Cheers. — dav_i, Nov 12 '12 at 11:26

score 6 · Accepted Answer · answered Nov 12 '12 at 11:13

protobuf-net can do that, because it is not tied to the specific type; for example:

using ProtoBuf;
using System.Collections.Generic;
using System.IO;

[ProtoContract]
public class MyOtherObject { }
[ProtoContract]
public class MyObject
{
    [ProtoMember(1)]
    public string Name { get; set; }
    [ProtoMember(2)]
    public int Value { get; set; }
    [ProtoMember(3)]
    public IList<MyOtherObject> Items { get; set; }
}

[ProtoContract]
public class MyObjectLite
{
    [ProtoMember(1)]
    public string Name { get; set; }
    [ProtoMember(2)]
    public int Value { get; set; }
}

static class Program
{
    static void Main()
    {
        var obj = new MyObject
        {
            Name = "abc",
            Value = 123,
            Items = new List<MyOtherObject>
            {
                new MyOtherObject(),
                new MyOtherObject(),
                new MyOtherObject(),
                new MyOtherObject(),
            }
        };
        using (var file = File.Create("foo.bin"))
        {
            Serializer.Serialize(file, obj);
        }
        MyObjectLite lite;
        using (var file = File.OpenRead("foo.bin"))
        {
            lite= Serializer.Deserialize<MyObjectLite>(file);
        }
    }
}

But if you don't want two different types, and/or you don't want to have to add attributes - that can be done too:

using ProtoBuf.Meta;
using System.Collections.Generic;
using System.IO;

public class MyOtherObject { }
public class MyObject
{
    public string Name { get; set; }
    public int Value { get; set; }
    public IList<MyOtherObject> Items { get; set; }
}
static class Program
{
    static readonly RuntimeTypeModel fatModel, liteModel;
    static Program()
    {
        // configure models
        fatModel = TypeModel.Create();
        fatModel.Add(typeof(MyOtherObject), false);
        fatModel.Add(typeof(MyObject), false).Add("Name", "Value", "Items");
        liteModel = TypeModel.Create();
        liteModel.Add(typeof(MyOtherObject), false);
        liteModel.Add(typeof(MyObject), false).Add("Name", "Value");
    }
    static void Main()
    {
        var obj = new MyObject
        {
            Name = "abc",
            Value = 123,
            Items = new List<MyOtherObject>
            {
                new MyOtherObject(),
                new MyOtherObject(),
                new MyOtherObject(),
                new MyOtherObject(),
            }
        };
        using (var file = File.Create("foo.bin"))
        {
            fatModel.Serialize(file, obj);
        }
        MyObject lite;
        using (var file = File.OpenRead("foo.bin"))
        {
            lite = (MyObject)liteModel.Deserialize(
                file, null, typeof(MyObject));
        }
    }
}

+1. This is great that you can get two different models out of the same byte array. But is there a way to deserialize the "rest" into `lite` when needed? If I understand this solution correctly, it'd require you to throw away the `lite` and deserialize a whole new object using the `fatModel` when you need the full object. — Eren Ersönmez, Nov 12 '12 at 11:29
So in your first example does it not care what object it is deserializing to as long as the property names and types and the `ProtoMember` values are the same? — dav_i, Nov 12 '12 at 11:34
@Eren in some cases, *yes* (protobuf is mergeable), but in the general case I would say that's just the wrong approach. — Marc Gravell, Nov 12 '12 at 11:35
@dav_i actually, protobuf never uses names - it is **just** the *key* (the 1,2,3) that matters. In the second example, that syntax is just a short-hand to "configure these as 1,2,3,..." — Marc Gravell, Nov 12 '12 at 11:36
+1,✓: Okay I haven't implemented this yet, but it does seem the best solution. You've done a great job with this library, @MarcGravell. — dav_i, Nov 12 '12 at 13:10
@MarcGravell With the second solution, is it possible to later add/remove properties in later versions and still be able to read previous? Or is it advised to use the attributes in that case? — dav_i, Nov 12 '12 at 13:11
@dav_i yes, the protobuf wire-format was designed (by Google, not me) to be extremely version tolerant. Much more version-friendly than `BinaryFormatter`, I should note. However! Don't re-use key-numbers to mean different things - that will cause problems. If you store `string Name` as key 5 in v1, then remove it in v2 - don't re-use key 5 for `decimal AccountBalance` in v3 - as trying to load old v1 data will cause craziness. — Marc Gravell, Nov 12 '12 at 13:12
@MarcGravell Just to confirm - if I say remove `Value` and add another called `Value2` then `fatModel.Add(typeof(MyObject), false).Add("Name", "Value2", "Items");` can't be used because `Value2` will be assigned `2` which conflicts with `Value`? — dav_i, Nov 12 '12 at 13:24
@dav_i the `Add("Name", "Value2", "Items")` is just a short-hand; there are a lot more methods available for defining the schema. — Marc Gravell, Nov 12 '12 at 13:26
@MarcGravell Sorry to keep bothering you - why does you have to do `fatModel.Add(typeof(MyOtherObject), false);` and would you have to do this if `MyObject` had a property of type `AnotherObject`? — dav_i, Nov 12 '12 at 16:17

score 0 · Answer 2 · answered Nov 12 '12 at 10:55

0

How about putting the Name and Valueinto a superclass and serializing them separately?

Alternatively, you could maintain a Dictionary and serialize that into one file.

answered Nov 12 '12 at 10:55

Davio

4,609
2
31
58

Deserialize part of a binary file

2 Answers2

Linked