ASP.NET Background Threads

Question

I'm working on an ASP.NET application containing WCF services that is meant to act as a test double for a simulation production environment. It is standing in for a very complicated backend that can receive orders via it's web services and then hours, or days later send notifications (by calling web services on other systems).

One of the requirements for the first release of this test double application was for it to be stateless i.e. no database.

The design put in place by the original developers who are no longer on the project is:

1. Receive an order via the web service.
2. Spawn a background thread
3. Return a canned response to the client
4. The background thread then sleeps for 5 - 30 seconds, sends a notification, sleeps again, sends a notification, repeating perhaps three to four times and then finally terminating when there is nothing left to do

The thread sleeps of 5 - 30 seconds is meant to simulate the hours and days long delays of the production system. Each background thread would live for no longer than say about three minutes.

Currently, the design seems to work well for its purpose. The only problem encountered with this approach so far is that if something happens that causes the application to restart (web.config change, new DLLs deployed, restart IIS, etc), the background threads appear to be killed off and any pending notifications for the background threads never happen.

It has now been decided by the customer that the 5 - 30 seconds delays are too short, and that a delay of 5 minutes between notifications is more appropriate. This would mean that the lifetime of a background thread could be more than 20 minutes.

My gut feeling is that making the delays five minutes probably isn't a good idea for this design. I'm interested to see what peoples thoughts are on this.

How reliable will this design be with the increased delays? Has anybody had any experience with long running background threads in ASP.NET that could comment on this?

Answer 1

'Sleeping' like this should never be done as it has been done in this application; you've already seen why that is so. Increasing that delay will exponentially increase the times when the task is not completed, as the longer the time period, the greater the chance of an app pool recycle.

It may be worthwhile to create a separate Windows service which the web site/service can connect to and initiate these long-running tasks. A Windows service will not automatically recycle itself as an ASP.NET service will. You can use WCF to facilitate the communication between ASP.NET and the Win service.

Answer 2

You will need some sort of service to deal with this, regardless. Websites shouldn't do background processing, particularly IIS since it's quite aggressive about app pool recycles.

Essentially what you need is a spooler, or a queue. This often comes in the form of a separate service, or a scheduled task that runs. In the linux world, you would call these daemons or cron jobs.

If you go with a service, you will need to be able to interact with that service by passing it work orders or queue items, whatever you call them, and it will process them using some sort of notification delay rules that you build in. No database storage required, although you may find it easy to dump files into a folder that it picks up occasionally.

If you go with a scheduled task (maybe that runs every minute) it would be easy to store that in a sqlite database or again, on the file system, and let the task pick it up. You can write a simple command line app that picks up the files and does its thing.

Answer 3

if you can use MSMQ to Store the Orders and then run a thread to process the queue will be a good idea. This will ensure more reliability in case of upgrade/restart of IIS server.