Multiple instance performance issues

Question

I am using windows workflow 4.0 version in my project to manage workflow. One project gets created in the system which has 100 documents to be approved by users and each document should follow same workflow process. The document processing is parallel for 100 documents. Which ever finishes can go to the next workflow step. [Eg: Document Created - Send for Review - Reviewed - Approved].

Now I am creating 100 instances for 100 documents. But I am not sure about performance when the documents are getting increased phenomenally (Eg: 10000 documents). Also multiple project can be created in the system at any given point of time.

Is there any better way to handle this scenario?

The application is based on ASP.Net and running IIS.

Answer 1

Your workflow steps (Document Created - Send for Review - Reviewed - Approved) sound like they involve human interactions, meaning they'll proceed at human time scales. Minutes at a minimum, more likely hours or days. Workflow 4.0 is designed with this use case in mind, as instances that are not actively progressing will be persisted out to your database store. Scaling to the 10s of thousands of instances should not be a problem.

Answer 2

You could operate the items in a batch process or as you have in realtime. The issues to consider is how real-time does the process need to be. Can it be sufficient to execute a batch workflow every minute (near realtime) as opposed to the overhead of 100-1000 WF per second?

You definitely need to consider the overhead of your design (realtime scalability will be an issue), so you may want to reconsider the realtime approach you have if your volume should increase significantly. I would aim for a batched approach to poll at a preconfigured frequency and use a DelayActivity.

Answer 3

I cannot help you to optimize your scenario but you can optimize the performance by first caching the activities (instead of each time loading the xaml and creating an instance) and second by determining correctly when to unload the instances in memory.

Answer 4

If you are hosting your WF using AppFabric (windows server) in IIS then this is not a problem. The steps you describe mean your WFs are waiting for time delays or messages resulting from some human interaction. The process will be idle until those events, which means after a small idle timeout, they will be unloaded from memory. You can handle millions of concurrent processes like this.