Unexpected values for AsyncLocal.Value when mixing ExecutionContext.SuppressFlow and tasks

Question

In an application I am experiencing odd behavior due to wrong/unexpected values of AsyncLocal: Despite I suppressed the flow of the execution context, I the AsyncLocal.Value-property is sometimes not reset within the execution scope of a newly spawned Task.

Below I created a minimal reproducible sample which demonstrates the problem:

private static readonly AsyncLocal<object> AsyncLocal = new AsyncLocal<object>();
[TestMethod]
public void Test()
{
    Trace.WriteLine(System.Runtime.InteropServices.RuntimeInformation.FrameworkDescription);
    var mainTask = Task.Factory.StartNew(() =>
        {
            AsyncLocal.Value = "1";
            Task anotherTask;
            using (ExecutionContext.SuppressFlow())
            {
                anotherTask = Task.Run(() =>
                    {
                        Trace.WriteLine(AsyncLocal.Value); // "1" <- ???
                        Assert.IsNull(AsyncLocal.Value); // BOOM - FAILS
                        AsyncLocal.Value = "2";
                    });
            }

            Task.WaitAll(anotherTask);
        });

    mainTask.Wait(500000, CancellationToken.None);
}

In nine out of ten runs (on my pc) the outcome of the Test-method is:

.NET 6.0.2
"1"

-> The test fails

As you can see the test fails because within the action which is executed within Task.Run the the previous value is still present within AsyncLocal.Value (Message: 1).

My concrete questions are:

1. Why does this happen? I suspect this happens because Task.Run may use the current thread to execute the work load. In that case, I assume lack of async/await-operators does not force the creation of a new/separate ExecutionContext for the action. Like Stephen Cleary said "from the logical call context’s perspective, all synchronous invocations are “collapsed” - they’re actually part of the context of the closest async method further up the call stack". If that’s the case I do understand why the same context is used within the action.

Is this the correct explanation for this behavior? In addition, why does it work flawlessly sometimes (about 1 run out of 10 on my machine)?

2. How can I fix this? Assuming that my theory above is true it should be enough to forcefully introduce a new async "layer", like below:

private static readonly AsyncLocal<object> AsyncLocal = new AsyncLocal<object>();
[TestMethod]
public void Test()
{
    Trace.WriteLine(System.Runtime.InteropServices.RuntimeInformation.FrameworkDescription);
    var mainTask = Task.Factory.StartNew(() =>
        {
            AsyncLocal.Value = "1";
            Task anotherTask;
            using (ExecutionContext.SuppressFlow())
            {
                var wrapper = () =>
                    {
                        Trace.WriteLine(AsyncLocal.Value);
                        Assert.IsNull(AsyncLocal.Value); 
                        AsyncLocal.Value = "2";
                        return Task.CompletedTask;
                    };

                anotherTask = Task.Run(async () => await wrapper());
            }

            Task.WaitAll(anotherTask);
        });

    mainTask.Wait(500000, CancellationToken.None);
}

This seems to fix the problem (it consistently works on my machine), but I want to be sure that this is a correct fix for this problem.

Many thanks in advance

Answer 1

Why does this happen? I suspect this happens because Task.Run may use the current thread to execute the work load.

I suspect that it happens because Task.WaitAll will use the current thread to execute the task inline.

Specifically, Task.WaitAll calls Task.WaitAllCore, which will attempt to run it inline by calling Task.WrappedTryRunInline. I'm going to assume the default task scheduler is used throughout. In that case, this will invoke TaskScheduler.TryRunInline, which will return false if the delegate is already invoked. So, if the task has already started running on a thread pool thread, this will return back to WaitAllCore, which will just do a normal wait, and your code will work as expected (1 out of 10).

If a thread pool thread hasn't picked it up yet (9 out of 10), then TaskScheduler.TryRunInline will call TaskScheduler.TryExecuteTaskInline, the default implementation of which will call Task.ExecuteEntryUnsafe, which calls Task.ExecuteWithThreadLocal. Task.ExecuteWithThreadLocal has logic for applying an ExecutionContext if one was captured. Assuming none was captured, the task's delegate is just invoked directly.

So, it seems like each step is behaving logically. Technically, what ExecutionContext.SuppressFlow means is "don't capture the ExecutionContext", and that is what is happening. It doesn't mean "clear the ExecutionContext". Sometimes the task is run on a thread pool thread (without the captured ExecutionContext), and WaitAll will just wait for it to complete. Other times the task will be executed inline by WaitAll instead of a thread pool thread, and in that case the ExecutionContext is not cleared (and technically isn't captured, either).

You can test this theory by capturing the current thread id within your wrapper and comparing it to the thread id doing the Task.WaitAll. I expect that they will be the same thread for the runs where the async local value is (unexpectedly) inherited, and they will be different threads for the runs where the async local value works as expected.

Answer 2

If you can, I'd first consider whether it's possible to replace the thread-specific caches with a single shared cache. The app likely predates useful types such as ConcurrentDictionary.

If it isn't possible to use a singleton cache, then you can use a stack of async local values. Stacking async local values is a common pattern. I prefer wrapping the stack logic into a separate type (AsyncLocalValue in the code below):

public sealed class AsyncLocalValue
{
    private static readonly AsyncLocal<ImmutableStack<object>> _asyncLocal = new();

    public object Value => _asyncLocal.Value?.Peek();

    public IDisposable PushValue(object value)
    {
        var originalValue = _asyncLocal.Value;
        var newValue = (originalValue ?? ImmutableStack<object>.Empty).Push(value);
        _asyncLocal.Value = newValue;
        return Disposable.Create(() => _asyncLocal.Value = originalValue);
    }
}

private static AsyncLocalValue AsyncLocal = new();

[TestMethod]
public void Test()
{
    Console.WriteLine(System.Runtime.InteropServices.RuntimeInformation.FrameworkDescription);
    var mainTask = Task.Factory.StartNew(() =>
    {
        Task anotherTask;
        using (AsyncLocal.PushValue("1"))
        {
            using (AsyncLocal.PushValue(null))
            {
                anotherTask = Task.Run(() =>
                {
                    Console.WriteLine("Observed: " + AsyncLocal.Value);
                    using (AsyncLocal.PushValue("2"))
                    {
                    }
                });
            }
        }

        Task.WaitAll(anotherTask);
    });

    mainTask.Wait(500000, CancellationToken.None);
}

This code sample uses Disposable.Create from my Nito.Disposables library.