Generic Performance Testing Framework For .NET

Question

I have a client/server application written in C#/.NET 3.5 that I want to do a bit of performance testing on. I've been looking for a generic framework to help me but not had much luck. I would like something that can manage a set of clients and perform random actions for them based on some settings. I would also then like to record some data relating to this to help me work outsome rough thresholds for my system, e.g. I can support n users performing x actions per second.

I would write code specific to my application to perform tasks such as:

• Login/logout a client.
• Send messages to the server to perform various actions.
• Record acknowledgements and other messages from the server.
• Measure statistics specific to the system.

I'm hoping the framework will then be able to take a set of parameters to describe a testing scenario such as:

• Number of clients logged in at a given time.
• Perform a given number of actions per second for each client.

It would then run the scenario, manage and track all of the users and actions and collate all of the data. (This is the boring bit I'm trying to avoid coding myself...) Ideally it would have some general measurements built in, e.g. time between sending a message and receiving a response, but I could code them myself if not.

I don't want to do any profiling of my code; I can always attach a profiler whilst running these tests later on. Instead I want to make some rough conclusions about my system, i.e. how many users can I throw at it before it breaks. (If there is a better term for this than 'performance testing' please let me know... Stress testing maybe?)

I realise I'm not giving very many specifics about the system here. It strikes me as a fairly general situation - I'm sure there are lots of client/server systems out there that people need to do similar tests on. I've found lots of web based frameworks to do similar things but they seem to be pretty web ingrained and don't lend themselves easily to non-HTTP based systems.

Anyone know of anything that might help? My searching hasn't found anything yet. I should point out that I'm stuck with Visual Studio 2008 Professional for the foressable future so if 2010 can do this it's out of bounds for me. I guess it doesn't have to be a .NET framework provided I can still plugin my .NET code fairly easily.

EDIT To be clear my application isn't a website, it's a Windows Forms client application that connects via a custom protocol to a .NET service. I can write code to perform the relevant client actions, I just need a framework to put it in.

Answer 1

The keyword you are looking for is "Distributed testing".

Smart Bear have a product called TestComplete which supports distributed testing. I don't think it can run multiple instances of your client on a single machine though (maybe it can, but I guess it's not a good idea any way since it would impact the performance results).

They also have an open source project called LoadUI, it is built to integrate with SoapUI, however you might be able to hook it up to your own client-test tool. I have no idea how much effort that would cost.

These are the tools I know of, but there are many more distributed testing tools out there. While most are indeed for web-bases testing, they often are extensible enough to simply kick off a different (GUI-based) testing framework (my favorite is QAliber) which runs the tests on your client app.

Answer 2

To my knowledge, such a framework does not currently exist (and it is likely it won't exist because everyones scenarios are so different). Because "Performance" means different things to different people and different projects, you will usually have to roll your own. Thankfully, with the advent of .NET4 and the TPL this has become easier than ever before.

There are three aspects a Performance test needs to cover:

1. Define "Load"
2. Measure "Performance"
3. Evaluate the Results

As you can imagine, both of these are totally lax definitions. Load could be measured in Requests per Second, Logged In Users etc... Performance could be measured in Response Time, Memory Usage... you get the picture.

So your first step is to define a "unit load", that you can easily scale up. For example you could create a ClientUnitLoad class, that simulates a client and periodically performs actions against your service. The TPL makes it easy to scale your ClientUnitLoad (except you are bound by hardware constraints of course).

The next step is to measure Performance. How you measure heavily depends on the metric you want to collect: Simple Stopwatching is fine 90% of the cases but can only measure time. Building a custom tracing and metrics collection infrastructure is definitely worth it, as you will also need it in production to verify your system performs as expected in the real world.

The last step is to evaluate your Metrics. The largest issue with Performance Tests is that: a) they are slow, and b) unreliable. Theres no way around that. In general, you need large sample sizes (repeated test runs) and some sort of statistical analysis on your metrics to ignore outliers. The processed results should then be compared to a configurable performance baseline.

Depending on the desired level of sophistication, that performance baseline may have simple acceptance criteria (such as: Repsonse Time under 100ms) or advanced, specific criteria (such as: Repsonse Time does not decrease by more than 10% when the amount of logged in users is 10 vs. 1).

Answer 3

I've not tried it myself but this may be suitable for your needs.

https://browsermob.com/performance-testing

Answer 4

Can't really say if there is a testing framework for such a specific purpose, but maybe you could use a simple console application (since the main idea is to test server performance)? You could make multiple calls to the server in different threads and use, for example, Stopwatch class to measure response times.

Or, you could try overloading the service by just increasing the number of calls (likely the service will slow down before threadpool runs out of threads?).

Answer 5

I've done some performance tests a while ago in a client/server application using Microsoft Visual Studio Load Test

When you add unit tests to a load test, you exercise the performance of non-Web based server components

You have to code your load generators as "Visual Studio Unit Tests", so that later you can add them into the Load Test project.

Answer 6

Kindly check at the following link.This may help you. http://msdn.microsoft.com/en-us/library/aa337591.aspx

I guess you will get the feature even in VS2008 but for ultimate edition.

Answer 7

http://blog.maartenballiauw.be/post/2008/02/code-performance-analysis-in-visual-studio-2008.aspx

Available in Developer and Team-based Visual Studio 2008. I think the feature has been improved in Visual Studio 2010 though.

I should clarify - this doesn't provide support for all the test cases you want to run. You would have to write that logic yourself in a separate project, set that as the startup project, then run the Performance Analysis.