Performance monitoring/metrics in .NET app

Question

We want to gather performance data about our (say 80% WinApp) application, both internally in dev, as well as out at customer sites. Our objectives are the following:

• It should be very fast
• It should be lightweight
• It should allow for tracking/timing & counting many different kinds of events across our apps.
• It should be able to gather (efficiently) a fair amount of data such as Domain, ComputerName, User, OS, Memory, Etc.
• All data collected should be analyzable (once it's been transmitted to our in-house BI database) across all of the dimensions mentioned above.

It would also, ideally, be relatively configurable on a site-wide or user-by-user basis to control:

• If these performance metrics are even collected
• How verbosely to collect them
• Whether to report the results back to us

We have been planning to essentially write our own system for doing most of this. We are then looking at a number of different "persistence" models, including the Binary Formatting of "performance packets", XmlSerialization and the Trace Logging functionality that is already part of the .NET framework, just to name a few.

Before engaging in this though, I wanted to double check to see what recommendations "the crowd" suggests. I've described what we are trying to do, and essentially our current plan of action - but I'm looking for any and all suggestions about how to best solve the problem at hand.

Thanks in advance.

Answer 1

Have a look at Metrics.NET, a relatively new project that should cover most of these needs. It's a port of Java's 'metrics' project.

(Added March 2016:) The Metrics.NET project has been transferred to new people after a long period of no updates.

Answer 2

You could take a look at building something on top of StatsD:

https://github.com/etsy/statsd/

There's a .NET package for that can be found here:

https://github.com/robbihun/NStatsD.Client

It's lightweight as it's basically just firing off an UDP packet to a remote server. That remote server then aggregates it and stores it in Graphite which is good at taking this data and turning it into graphs and it manages things like data retention for you. For analysis, Graphite also allows you to output the data as JSON (which basically turns your metrics into an array of doubles) and has many functions and filters you can apply to recorded metrics which you could then feed into your other systems.

See here for some examples of what is possible with Graphite:

http://matt.aimonetti.net/posts/2013/06/26/practical-guide-to-graphite-monitoring

It doesn't meet all your requirements however, such as being able to record data about the client like computer names. But you can probably reach that goal by using namespacing in your metric names; so you would record your metric with a key like "client567.orders.loadtime" and "client567" would be an entry in some other database that stores that client567 uses IE11 on Windows 7, for example.

So it's not a complete out-of-the-box solution, but it forms a nice basis I think.

Another option is to use a commercial platform such as NewRelic:

http://newrelic.com/

It comes with performance monitoring for a lot of technologies (from ASP.NET to SQL Server to Solr). It does however require an agent process (a service on Windows) running in the background that handles the monitoring for you, which might or might not be an option for you. It's more or less meant for webservers and would probably not be very suitable to monitor a client-side WPF application.

Answer 3

Going with your plan I'd take this approach:

1. You're going to write all the performance data to a log file. Use one of the fast C# log libraries like log4net or nlog. These frameworks support binary appenders if necessary.
2. When the app starts write the available RAM, CPU, etc.
3. Use an aspect injector like PostSharp to automatically add a timer to every (non-inline) method in your app. The Stopwatch class won't make a significant performance hit. To make this configurable have the aspect injector change the log level according to the estimated priority of the method (or use your own attributes to specify that).
4. Prompt the user at exit (or start) to send the recorded performance data to your webserver.
5. Zip the data on its way out.

You may also want to consider writing your own lock class. Instead of using lock(blah) or Monitor.Enter(blah), wrap up Monitor.Enter/Exit in your own disposable class that logs how long you had to wait to enter the lock.

Answer 4

I would start with testing whether or not using the EventLog would solve my problem. EventLog messages contain a lot of information and you can add your own.

For some diagnostics I'd have a look at (custom) Performance Counters. My bet is that it is hard to create something that beats the performance of the windows Performance Counters.

Writing this all over again might not be worth it.

It might be required to create special filters in the event viewer to get the output you are looking for or to gather event across computers (you might even write your own viewer)