Cancel array of Tasks with RetryOptions using Azure Functions

Question

I'm using Azure Functions V4. In an orchestration (MyCustomOrchestration) I do a fan-out and call an ActivityTriggered-function (HandleIdFunction). The fan-out will usually contain around 20 tasks. The HandleIdFunction is called with a RetryOptions and can throw different exceptions. The retry should retry if any exception is thrown, except for MyCustomError. If MyCustomError is thrown all tasks should be aborted and stop retrying, no matter what exception they may throw. I don't seem to get it to work. The retries keep retrying.

I could not find a way to pass a CancellationToken. And if I set a variable in Handle it will be reset during replay.

Complete reproducable example below:

public class MyCustomError : Exception
{

}

public static class Function1
{
    [FunctionName("Function1")]
    public static async Task<IActionResult> Run(
        [HttpTrigger(AuthorizationLevel.Anonymous, "get", "post", Route = null)] HttpRequest req,
          [DurableClient] IDurableClient starter,
        ILogger log)
    {
        var result = await starter.StartNewAsync(nameof(MyCustomOrchestration));
        return new OkObjectResult(result);
    }

    [FunctionName(nameof(HandleIdFunction))]
    public static async Task HandleIdFunction([ActivityTrigger] IDurableActivityContext context)
    {
        var input = context.GetInput<int>();
        Console.WriteLine($"{nameof(HandleIdFunction)} - {input}.");
        await Task.Delay(1000);
        // Fake errors.
        if (input == 10)
            throw new MyCustomError();
        if (input % 2 == 0)
            throw new Exception();
    }

        [FunctionName(nameof(MyCustomOrchestration))]
    public static async Task MyCustomOrchestration([OrchestrationTrigger] IDurableOrchestrationContext context)
    {
        var parallelTasks = new List<Task<Guid>>();

        var retryOptions = new RetryOptions(TimeSpan.FromSeconds(5), 30)
        {
            BackoffCoefficient = 1.85,
            MaxRetryInterval = TimeSpan.FromSeconds(5),
            RetryTimeout = TimeSpan.FromMinutes(1),
            Handle = exception =>
            {
                Console.WriteLine("Retry handled called");
                return (exception.InnerException is not MyCustomError);
            }
        };

        int[] idBatch = Enumerable.Range(0, 20).ToArray();
        for (int i = 0; i < idBatch.Length; i++)
        {
            var id = idBatch[i];
            Task<Guid> task = context.CallActivityWithRetryAsync<Guid>(nameof(HandleIdFunction), retryOptions, id);
            parallelTasks.Add(task);
        }

        Guid[] result = null;

        try
        {
            result = await Task.WhenAll(parallelTasks);
        }
        catch (Exception ex) when (ex.InnerException is MyCustomError)
        {
            // Custom handling.
        }
        catch (Exception ex)
        {
            // Custom handling.
        }
    }
}