Reflection is too slow while deserialising JSON strings into .NET objects

Question

I'm having some issues with System.Reflection in C#. I'm pulling data from a database and retrieving that data in a JSON string. I've made my own implementation of handling the data from JSON into my self declared objects using Reflection. However, since I ussually get a JSON string with an array of like 50 - 100 objects my program runs really slow because of the loops I'm using with reflection.

I've heard that reflection is slow but it shouldn't be this slow. I feel something is not right in my implementation since I have a different project where I use JSON.NET serializer and instantiate my objects a bit differently with reflection that runs just fine on the same output (less than a second) while my slow program takes about 10 seconds for 50 objects.

Below are my classses that I'm using to store data

class DC_Host
{
    public string name;

    public void printProperties()
    {
        //Prints all properties of a class usign reflection
        //Doesn't really matter, since I'm not usign this for processing
    }
}

class Host : DC_Host
{
    public string asset_tag;
    public string assigned;
    public string assigned_to;
    public string attributes;
    public bool? can_print;
    public string category;
    public bool? cd_rom;
    public int? cd_speed;
    public string change_control;
    public string chassis_type;
    //And some more properties (around 70 - 80 fields in total)

Below you'll find my methods for processing the information into the objects that are stored inside a List. The JSON data is stored inside a dictionairy that contains a another dictionairy for every array object defined in the JSON input. Deserialising the JSON happens in a matter of miliseconds so there shouldn't be a problem in there.

public List<DC_Host> readJSONTtoHost(ref Dictionary<string, dynamic> json)
{
    bool array = isContainer();

    List<DC_Host> hosts = new List<DC_Host>();

    //Do different processing on objects depending on table type (array/single)
    if (array)
    {
        foreach (Dictionary<string, dynamic> obj in json[json.First().Key])
        {
            hosts.Add(reflectToObject(obj));
        }
    }
    else
    {
        hosts.Add(reflectToObject(json[json.First().Key]));
    }

    return hosts;
}

private DC_Host reflectToObject(Dictionary<string,dynamic> obj)
{
    Host h = new Host();

    FieldInfo[] fields = h.GetType().GetFields();

    foreach (FieldInfo f in fields)
    {
        Object value = null;

        /* IF there are values that are not in the dictionairy or where wrong conversion is
         * utilised the values will not be processed and therefore not inserted into the 
         * host object or just ignored. On a later stage I might post specific error messages
         * in the Catch module. */

        /* TODO : Optimize and find out why this is soo slow */
        try
        {
            value = obj[convTable[f.Name]];
        }
        catch { }

        if (value == null)
        {
            f.SetValue(h, null);
            continue;
        }
        // Het systeem werkt met list containers, MAAAR dan mogen er geen losse values zijn dus dit hangt
        // zeer sterk af van de implementatie van Service Now.
        if (f.FieldType == typeof(List<int?>)) //Arrays voor strings,ints en bools dus nog definieren 
        {
            int count = obj[convTable[f.Name]].Count;
            List<int?> temp = new List<int?>();
            for (int i = 0; i < count; i++)
            {
                temp.Add(obj[convTable[f.Name]][i]);
                f.SetValue(h, temp);
            }
        }
        else if (f.FieldType == typeof(int?))
            f.SetValue(h, int.Parse((string)value));
        else if (f.FieldType == typeof(bool?))
            f.SetValue(h, bool.Parse((string)value));
        else
            f.SetValue(h, (string)value);
    }

    Console.WriteLine("Processed " + h.name);

    return h;
}

I'm not sure what JSON.NET's implementation is in the background for using reflection but I'm assumign they use something I'm missing for optimising their reflection.

To gain maximum performance one needs to generate new code to fill in the data. — CodesInChaos, Jul 27 '12 at 07:48
But even without that, making good use of dictionaries and caching should give quite a speedup over your code. — CodesInChaos, Jul 27 '12 at 07:50
@CodesInChaos Removed the empty catch, had to use it in an earlier implementation but realized my code now allows for a TryGetValue :). I'm not exactly sure what you mean by generating new code to fill in data? — Joey Dewd, Jul 27 '12 at 07:56
You can emit new methods at runtime, which fill the data into the objects. So you only need reflection to build those methods (`Reflection.Emit`, `Expression.Compile`,...), but not to hydrate the objects. — CodesInChaos, Jul 27 '12 at 07:59
@CodesInChaos That is quite a nice trick! Never thought about that. I've just removed the Try Catch structure for a TryGetValue and my program instantly read and processed the JSON. So that actually fixed my problem although I have no idea why removing the Try Catch statement increased my performance so drastically. Thanks for the help! :D — Joey Dewd, Jul 27 '12 at 08:01
@JoeyDewd, try running the slow version without attaching the debugger. You might be surprised by the performance. What kills your app is the many exceptions that occur. Handling exceptions is expensive, and much more so if the debugger is attached. Lesson learned: never use exceptions for common program flow control. — Lucero, Jul 27 '12 at 08:06
You're right. Running the program without the debugger did all the performance in a matter of miliseconds. I was a bit sceptic about using this empty catch statement but my previous implementation forced me to use that. I didn't knew it had such a great impact on performance, thanks for clearing that out for me. ( You can probably Guess one of the reasons why I had to improve my implementation ;) ) — Joey Dewd, Jul 27 '12 at 08:08

score 3 · Answer 1 · edited May 23 '17 at 12:00

3

Basically, high-performance code like this tends to use meta-programming extensively; lots of ILGenerator etc (or Expression / CodeDom if you find that scary). PetaPoco showed a similar example earlier today: prevent DynamicMethod VerificationException - operation could destabilize the runtime

You could also look at the code other serialization engines, such as protobuf-net, which has crazy amounts of meta-programming.

If you don't want to go quite that far, you could look at FastMember, which handles the crazy stuff for you, so you just have to worry about object/member-name/value.

edited May 23 '17 at 12:00

Community

1
1

answered Jul 27 '12 at 08:02

Marc Gravell

1,026,079
266
2,566
2,900

Yeah, I'm quite new to reflection to be honest and I know there are ways to program LowLevel with IL to achieve the best results and even create dynamic classes at runtime but for now I'm trying to avoid those paths since they are at the moment too difficult for me. I've fixed the problem by removing the Try Catch statement using TryGetvalue (although I have no idea why that seemeed to fix the performance issues). I'll look into FastMember though for future optimalisation issues :). Thank you – Joey Dewd Jul 27 '12 at 08:04

score 2 · Accepted Answer · answered Jul 27 '12 at 08:13

For people that are running into this article I'll post my solution to my problem in here. The issue wasn't really related to reflection. There are ways to improve the speed using Reflection like CodesInChaos and Marc Gravell mentioned where Marc even craeted a very usefull library (FastMember) for people with not too much experience in low level reflection.

The solution however was non related to reflection itself. I had a Try Catch statement to evaluate if values exist in my dictionary. Using try catch statements to handle program flow is not a good idea. Handling exceptions is heavy on performance and especially when you're running the debugger, Try Catch statements can drastically kill your performance.

//New implementation, use TryGetValue from Dictionary to check for excising values.
dynamic value = null;
obj.TryGetValue(convTable[f.Name], out value);

My program runs perfectly fine now since I omitted the TryCatch statement.

Reflection is too slow while deserialising JSON strings into .NET objects

2 Answers2