Timeout manager retuning timout immediately

Question

I have a very simple scheduler saga, that is supposed to send a specific message every day. It is implemented as a saga that requests a timeout. when the timeout is handled an action is executed (a message is sent) and a new timeout is requested the next day.

I have done the exact same thing before with success, but now the timeout seems to trigger immediately so matter what DateTime is requested.

The endpoint is self-hosting, and configured to use InMemoryPersistence. NServiceBus version is 6.4.3.

The saga is implemented as follows. I have removed all the logic, but still timeout messages are received immediately and infinitely.

public class SchedulerSaga: Saga<SchedulerState>,
    IAmStartedByMessages<StartSchedulerSagaCommand>,
    IHandleTimeouts<SchedulerTimeout>
{
    private readonly IConfigurationProvider _config;

    public SchedulerSaga(IConfigurationProvider config)
    {
        _config = config;
    }

    protected override void ConfigureHowToFindSaga(SagaPropertyMapper<SchedulerState> mapper)
    {
        mapper.ConfigureMapping<StartSchedulerSagaCommand>(_ => _.SchedulerName).ToSaga(_ => _.SchedulerName);
    }

    public async Task Handle(StartSchedulerSagaCommand message, IMessageHandlerContext context)
    {
        Data.SchedulerName = message.SchedulerName;
        await StartProcessAndScheduleNewTimeout(context);
    }

    public async Task Timeout(SchedulerTimeout state, IMessageHandlerContext context)
    {
        Data.Counter++;
        await StartProcessAndScheduleNewTimeout(context);
    }

    private async Task StartProcessAndScheduleNewTimeout(IMessageHandlerContext context)
    {
        await RequestTimeout(context, new DateTime(2018, 9, 16, 0, 0, 0, DateTimeKind.Utc), new SchedulerTimeout { Counter = Data.Counter });
    }
}

The endpoint is configured as follows:

    public static EndpointConfiguration ConfigureMsmqEndpoint(IWindsorContainer container)
    {
        var endpointConfiguration = new EndpointConfiguration(MsmqEndpointName);

        ConfigureRouting(endpointConfiguration);

        endpointConfiguration.UsePersistence<InMemoryPersistence>();

        endpointConfiguration.SendFailedMessagesTo($"{MsmqEndpointName}.error");
        endpointConfiguration.AssemblyScanner().ExcludeAssemblies("tools");
        endpointConfiguration.EnableInstallers();

        ConfigureUnobtrusiveMessageConvention(endpointConfiguration);

        endpointConfiguration.Recoverability().Delayed(DelayedSettings);

        endpointConfiguration.UseContainer<WindsorBuilder>(c => c.ExistingContainer(container));

        return endpointConfiguration;
    }

I also tried to use the built in Scheduling mechanism and the same thing happens, hundereds of timeouts are triggered every second.

await endpointInstance.ScheduleEvery(
        timeSpan: TimeSpan.FromMinutes(5),
        task: context=> context.SendLocal(new SomeMessage())
    )
.ConfigureAwait(false);

Update: Add repo with code reproducing the problem.

https://github.com/spinakr/nsb-scheduling-msmq

The problem only occur when the package "NServiceBus.Azure.Transports.WindowsAzureStorageQueues" is references in the project, even if it is not used!

The app in review has two endpoint hosted in the same process. Consuming messages from MSMQ and Azure storage queues. In the repo, it is only when adding the azure storage queues transport package that the issue start occurring.

Could you confirm that a single starting message is sent and incoming queue doesn't contain previously sent messages? — Sean Feldman, Jan 24 '18 at 14:40
Yes, definitely. I tried debugging, and breaking in both the message handler and timeout handler, only timeouts are received, this can also be observed in the journal of the msmq queue. — sp1nakr, Jan 24 '18 at 15:15
Also take note that the same behaviour is happening when using the `endpointInstance.ScheduleEvery()` function. This leads my to believe that the problem is isolated to the timeout manager, as both ScheduleEvery and RequestTimeout uses it. — sp1nakr, Jan 24 '18 at 15:15
Have tried it with NServiceBus 6.4.3 and it works as expected (timeout is sent and doesn't kick in for a large timeout). If modified for a shorter period of time, timeout is delivered as expected. Could you share the code you used to reproduce this issue as a standalone project on GitHub or similar? — Sean Feldman, Jan 24 '18 at 16:30
Thanks for the answer @SeanFeldman. I will test it out in isolation when I get back to work. I guess it must be something with the configuration or my environment. — sp1nakr, Jan 25 '18 at 13:51
The added minimal example has the same issue. @SeanFeldman did you test it using msmq transport? — sp1nakr, Jan 29 '18 at 12:45
yes. That's why I asked for a repro to be shared in a repo rather than copy/paste, to see if there's anything else that might get in a way. — Sean Feldman, Jan 29 '18 at 21:52
Added repository with the code. Also found that the issue is connected to another endpoint hosted in the same process. The issue only persist when the package "NServiceBus.Azure.Transports.WindowsAzureStorageQueues" is referenced. The example repo has the package referenced, without using it. — sp1nakr, Jan 30 '18 at 07:46
And that's why nothing beats a repo :) The answer provided by @KatoStoelen is correct. — Sean Feldman, Jan 30 '18 at 17:44

score 2 · Accepted Answer · answered Jan 30 '18 at 11:22

2

I assume that the endpoint you are describing are using MSMQ as transport (based on names of methods and fields) and that the saga is running on that endpoint.

MSMQ relies on the Timeout Manager to support delayed delivery. The Azure Storage Queues transport, on the other hand, seems to solve delayed delivery in a different way. It actually has a feature, enabled by default, that prevents deferred messages from being routed to the Timeout Manager. If your MSMQ endpoint were to scan the NServiceBus.Azure.Transports.WindowsAzureStorageQueues assembly, your deferred messages would not reach the Timeout Manager.

One solution to this issue would be to configure the assembly scanner in your MSMQ endpoint to exclude that assembly:

endpointConfiguration.AssemblyScanner().ExcludeAssemblies(
    "NServiceBus.Azure.Transports.WindowsAzureStorageQueues.dll");

answered Jan 30 '18 at 11:22

KatoStoelen

188
1
1
7

This seems to be the root cause of the problem. Excluding the azure table storage dll solves the issue. Question now is if this is the desired behavior, as the default scanner configuration will result in this problem, when hosting multiple endpoints in the same process. The result is quite critical as well, so should probably result in some kind of warning or similar? @SeanFeldman can maybe answer? – sp1nakr Jan 30 '18 at 11:56
@KatoStoelen is correct. When multi-hosting endpoints with different transports, side effects can take place. In this case disabling of a timeout manager as ASQ transport doesn't need it in the latest versions. You should either filter out assemblies that shouldn't be scanned per endpoint or move enpoint to its own host. Oh, and accept the answer, of course. – Sean Feldman Jan 30 '18 at 17:46

score 1 · Answer 2 · answered Jan 30 '18 at 18:00

Another consideration when co-hosting multiple endpoints in the same process - sharing of your handlers. You might want to have a look at the Endpoints multi hosting sample, which mentioned how to avoid undesired assembly loading using assembly scanning with blacklisting.

Timeout manager retuning timout immediately

2 Answers2