How can I monitor the Task queues in the .NET TaskSchedulers (across AppDomain)

Question

As a developer, I would like to monitor the size (and progress) of work in the Task Queues in the TaskSchedulers so that I can evaluate whether an experienced slowdown under production load is due to the magnitude or perhaps stall of tasks scheduled.

Normally I would simply attach the debugger and inspect task sizes, but:

1. The application is running under mono and in production, so I cannot attach to it using visual studio
2. I would like to deliver output of analysis on this data as surveillance input for health-monitoring of the service

I have been through the docs and found

• TaskScheduler.GetScheduledTasks, which delivers this information to the debugger. It is protected (which I could circumvent), but also seem to require some guarantees about frozen threads which I cannot honor. However, I am willing to use inconsistent data.
• how to get a list of running tasks in .net 4.0. Which focuses on the running Tasks, which are not that interesting to me. I am interested in the size of the backlog and whether work is progressing.

I am willing to:

1. Use code designed for other purposes (such as for the debugger)
2. Accept that i get inconsistent data (this is for statistics and analysis)

Things that I do not want to do are:

1. Add tracking to each task created.
   • Some of the code is 3rd party and cannot be refactored
   • There is lots of code, and injecting this into that is worse than not being able to tell how it is progressing.
2. Use a custom TaskScheduler
   • Some of the code is 3rd party and cannot be refactored
   • Some of the code use TaskSchedulers which sets thread-priority and max-concurrency. Factoring in the monitoring aspect would be awful.

Answer 1

I am able to retrieve the information using reflection and the GetScheduledTasksForDebugger:

public static class ScheduledTaskAccess
{
    public static Task[] GetScheduledTasksForDebugger(this TaskScheduler ts)
    {
        var mi = ts.GetType().GetMethod("GetScheduledTasksForDebugger", BindingFlags.NonPublic | BindingFlags.Instance);
        if (mi == null)
            return null;
        return (Task[])mi.Invoke(ts, new object[0]);
    }
    public static TaskScheduler[] GetTaskSchedulersForDebugger()
    {
        var mi = typeof(TaskScheduler).GetMethod("GetTaskSchedulersForDebugger", BindingFlags.NonPublic | BindingFlags.Static);
        if (mi == null)
            return null;
        return (TaskScheduler[])mi.Invoke(null, new object[0]);
    }
}

using a custom IEqualityComparer I am able to maintain how long they have been queued. Very useful meta-information :)

Now I just wish there was some statistics on the the tasks that one could access, CPU-time spent, idle-time, task-switchcount, ...

Answer 2

Unfortunately the TaskScheduler.GetScheduledTasks is designed for debuggers only and there is no direct way to get scheduled tasks.

But MSDN says (source):

Developers who are implementing custom debuggers shouldn't call this method directly, but should use the internal wrapper method GetScheduledTasksForDebugger instead: internal Task[] GetScheduledTasksForDebugger(). This wrapper method returns an array of tasks instead of an enumerable. To retrieve a list of active schedulers, use the internal method internal static TaskScheduler[] GetTaskSchedulersForDebugger(). This static method returns an array of all active TaskScheduler instances. You can then use GetScheduledTasksForDebugger on each scheduler instance to retrieve its list of scheduled tasks.

So why not use reflection? This is a safe and easy way to achieve what you want because GetScheduledTasksForDebugger will exist in future and it's designed for debugging and probably for profiling.