Dynamic Object Serialization

Question

I tried to serialize a DynamicObject class with BinaryFormatter, but:

• Output file is too big, not exactly wire-friendly
• Circular References not handled (stuck while serializing)

Since serializing a DynamicObject means very little by itself, here's the class I tried to serialize:

[Serializable()]
class Entity
    : DynamicObject, ISerializable
{

    IDictionary<string, object> values = new Dictionary<string, object>();

    public Entity()
    {

    }

    protected Entity(SerializationInfo info, StreamingContext ctx)
    {
        string fieldName = string.Empty;
        object fieldValue = null;

        foreach (var field in info)
        {
            fieldName = field.Name;
            fieldValue = field.Value;

            if (string.IsNullOrWhiteSpace(fieldName))
                continue;

            if (fieldValue == null)
                continue;

            this.values.Add(fieldName, fieldValue);
        }

    }

    public override bool TryGetMember(GetMemberBinder binder, out object result)
    {
        this.values.TryGetValue(binder.Name, out result);

        return true;
    }

    public override bool TrySetMember(SetMemberBinder binder, object value)
    {
        this.values[binder.Name] = value;

        return true;
    }        

    void ISerializable.GetObjectData(SerializationInfo info, StreamingContext context)
    {            
        foreach (var kvp in this.values)
        {
            info.AddValue(kvp.Key, kvp.Value);                 
        }
    }

}

(I guess I could have used an ExpandoObject, but that's another story.)

Here's a simple test program:

    static void Main(string[] args)
    {
        BinaryFormatter binFmt = new BinaryFormatter();

        dynamic obj = new Entity();
        dynamic subObj = new Entity();
        dynamic obj2 = null;

        obj.Value = 100;
        obj.Dictionary = new Dictionary<string, int>() { { "la la la", 1000 } };

        subObj.Value = 200;
        subObj.Name = "SubObject";

        obj.Child = subObj;

        using (var stream = new FileStream("test.txt", FileMode.OpenOrCreate))
        {
            binFmt.Serialize(stream, obj);                
        }

        using (var stream = new FileStream("test.txt", FileMode.Open))
        {
            try
            {
                obj2 = binFmt.Deserialize(stream);                    
            }
            catch (Exception ex)
            {
                Console.WriteLine(ex);
            }                
        }

        Console.ReadLine();

    }

Putting some breakpoints here and there helped me have a look at obj2 contents and it looks like the original data is correctly deserialized, though with the above shortcomings if you get imaginative and move data around.

I had a look at Marc Gravell's protobuf-net, but I'm not really sure how to use it in such a context (I'm not even sure I picked up the right version from the repository, but hey).

I know it's more code than words, but I don't think I can explain the scenario any better. Please do tell if there's something I can add to make this question clearer.

Any help is much appreciated.

Answer 1

I'm 98% certain that this sequence will work for a dynamic object.

1. convert object to an Expando Object
2. cast expando object to be of type Dictionary
3. use ProtoBuf-net Serializer.Serialize / .Deserialize as per normal
4. convert dictionary to Expando Object

You can convert objects to a collection of name/value pairs for transfering.

That's just a small subset of what dynamic can do, but perhaps it is enough for you.

There's some custom code to handle some of the conversions above that I can show you if there's interest.

I don't have a solution for when dynamic is a placeholder to a class. For this case I'd suggest getting the type and using a switch statement to serialize / deserialize as you require. For this last case, you'd need to place a something to indicate which type of generic deserialization that you need (string / id / fully qualified type name / etc). Assumption is that you are dealing with a list of expected types.

Note: Expando implements IDictionary. An Expando is merely merely a list of key/value pairs. ie. the thing you dot into is the key, and the value is the return from whatever chain of functions implements that. There are a set of dynamic interfaces for customising the syntactic sugar experience, but most of the time you wont to look at them.

refs:

• Dictionary from IDictionary using the constructor -- http://msdn.microsoft.com/en-us/library/et0ke8sz(v=vs.110).aspx
• IDictionary/Dictionary to Expando -- http://theburningmonk.com/2011/05/idictionarystring-object-to-expandoobject-extension-method/

Answer 2

I am not sure if JSON would be acceptable in your senario, but if it is I have used Json.net (http://json.codeplex.com) to serialize a dynamic types. It works quite well, it is fast, and the output is small in size. While Json.net doesn't return dynamic objects directly, it is very easy to convert the deserialized output of Json.Net to any dynamic type. In the example below I am using ExpandoObject as my dynamic type. The code below is what I have used in the Facebook Graph Toolkit. See this link for the original source: http://facebookgraphtoolkit.codeplex.com/SourceControl/changeset/view/48442#904504

public static dynamic Convert(string s) {
            object obj = Newtonsoft.Json.JsonConvert.DeserializeObject(s);
            if (obj is string) {
                return obj as string;
            } else {
                return ConvertJson((JToken)obj);
            }
    }

    private static dynamic ConvertJson(JToken token) {
        // FROM : http://blog.petegoo.com/archive/2009/10/27/using-json.net-to-eval-json-into-a-dynamic-variable-in.aspx
        // Ideally in the future Json.Net will support dynamic and this can be eliminated.
        if (token is JValue) {
            return ((JValue)token).Value;
        } else if (token is JObject) {
            ExpandoObject expando = new ExpandoObject();
            (from childToken in ((JToken)token) where childToken is JProperty select childToken as JProperty).ToList().ForEach(property => {
                ((IDictionary<string, object>)expando).Add(property.Name, ConvertJson(property.Value));
            });
            return expando;
        } else if (token is JArray) {
            List<ExpandoObject> items = new List<ExpandoObject>();
            foreach (JToken arrayItem in ((JArray)token)) {
                items.Add(ConvertJson(arrayItem));
            }
            return items;
        }
        throw new ArgumentException(string.Format("Unknown token type '{0}'", token.GetType()), "token");
    }

Answer 3

First off, the size of your file depends on 2 things (if I understand how BinaryFormatter works, please correct me if I'm wrong):

1. The size of the actual values being serialized, and
2. The names used to serialize the object's values with the SerializationInfo.AddValue method, which are stored in the output file so values can be used during deserialization with the same name.

Obviously, #1 is going to cause your biggest slowdown, which can only be reduced by optimizing the objects you're trying to serialize.

Because you're using dynamic objects, the almost unnoticably small size increase caused by #2 is unavoidable. If you knew the types and names of the object's members ahead of time, you could just give each member of the object very short, sequentially-determined name ("1", "2", "3", etc.) as you iterated over the object's members, adding them via SerializationInfo.AddValue. Then, during deserialization, you could use SerializationInfo.GetValue with the same sequentially-determined name, and deserialization would work just fine, regardless of the actual names of the values being deserialized, as long as you iterated through the object's members in the same order they were added in. Granted, this might only save you an average of 4 or 5 bytes per member, but those little amounts can add up in large objects.

@Raine: (I guess I could have used an ExpandoObject, but that's another story.)

Not so; I changed your code sample to use ExpandoObject instead of your Entity class, and got a SerializationException thrown at me. ExpandoObject is not marked with a SerializableAttribute, and it doesn't have the appropriate constructors to be deserialized or serialized. However, this doesn't mean you can't use ExpandoObject if you really want to: it implements IDictionary<string, object>, which in turn implements ICollection<KeyValuePair<string, object>>. Thus, an ExpandoObject instance is a collection of KeyValuePair<string, object> instances, which are marked as serializable. So, you could serialize an ExpandoObject, but you'd have to cast it as ICollection<KeyValuePair<string, object>> and serialize each KeyValuePair<string, object> in it individually. This would pointless, though, in terms of optimizing your original code sample, because it takes just as much file space.

In summary, I really don't think there's any way you could optimize serializing a dynamic object- you have to loop through the object's members every time it's serialized, and you have no way to know the object's size beforehand (by definition of dynamic).

Answer 4

I don't know if SharpSerializer supports Dynamic Objects but it might be worth a try:

http://www.sharpserializer.com/en/index.html

Answer 5

One Suggestion for serializing dynamic object is convert them to String and then serialize, you can then deserialize back to your object, if applicable in your case.