AccessViolationException during application exit while using MassTransit with Kafka and in-memory bus

Question

I'm using version 7.1.4 of MassTransit with .Net Core 3.1, ASP.NET Core Web Application. It happens when I close the application by Ctrl + C. When I press stop debugging button, it doesn't occur. The code responsible for registration goes as follows:

private void ConfigureMassTransit(IServiceCollection services)
{
    services.AddMassTransit(massTransit =>
    {
        massTransit.UsingInMemory((ctx, cfg)
            => cfg.ConfigureEndpoints(ctx, SnakeCaseEndpointNameFormatter.Instance));

        massTransit.ConfigureRider(
            _kafkaHost, _dbConnectionString);
            });

        services.AddMassTransitHostedService();
}

// The rider part.

rider.UsingKafka((ctx, kafka) =>
{
    kafka.Host(kafkaHost);

    kafka.TopicEndpoint<Null, IMessageName>(topicName, GroupIds.PushedDeals, cfg =>
    {
        cfg.CheckpointInterval = TimeSpan.FromMilliseconds(100);
        cfg.AutoOffsetReset = AutoOffsetReset.Earliest;
        cfg.ConfigureSaga<SagaState>(ctx);
    });                
});

The exception:

Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
   at Confluent.Kafka.Impl.NativeMethods.NativeMethods.rd_kafka_consumer_poll(IntPtr, IntPtr)
   at Confluent.Kafka.Impl.NativeMethods.NativeMethods.rd_kafka_consumer_poll(IntPtr, IntPtr)
   at Confluent.Kafka.Consumer`2[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Consume(Int32)
   at Confluent.Kafka.Consumer`2[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Consume(System.Threading.CancellationToken)
   at MassTransit.KafkaIntegration.Contexts.KafkaConsumerContext`2+<>c__DisplayClass15_0[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].<Consume>b__0()
   at MassTransit.Util.ChannelExecutor+SynchronousFuture`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Run()
   at MassTransit.Util.ChannelExecutor+<RunFromChannel>d__12.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[MassTransit.Util.ChannelExecutor+<RunFromChannel>d__12, MassTransit, Version=7.1.4.0, Culture=neutral, PublicKeyToken=b8e0e9f2f1e657fa]].ExecutionContextCallback(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[MassTransit.Util.ChannelExecutor+<RunFromChannel>d__12, MassTransit, Version=7.1.4.0, Culture=neutral, PublicKeyToken=b8e0e9f2f1e657fa]].MoveNext(System.Threading.Thread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[MassTransit.Util.ChannelExecutor+<RunFromChannel>d__12, MassTransit, Version=7.1.4.0, Culture=neutral, PublicKeyToken=b8e0e9f2f1e657fa]].MoveNext()
   at System.Threading.ThreadPoolGlobals+<>c.<.cctor>b__5_0(System.Object)
   at System.Threading.Channels.AsyncOperation`1[[System.Boolean, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].SetCompletionAndInvokeContinuation()
   at System.Threading.Channels.AsyncOperation`1[[System.Boolean, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].SignalCompletion()
   at System.Threading.Channels.SingleConsumerUnboundedChannel`1+UnboundedChannelWriter[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].TryWrite(System.__Canon)
   at System.Threading.Channels.SingleConsumerUnboundedChannel`1+UnboundedChannelWriter[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].WriteAsync(System.__Canon, System.Threading.CancellationToken)
   at MassTransit.Util.ChannelExecutor+<Run>d__11`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[MassTransit.Util.ChannelExecutor+<Run>d__11`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], MassTransit, Version=7.1.4.0, Culture=neutral, PublicKeyToken=b8e0e9f2f1e657fa]](<Run>d__11`1<System.__Canon> ByRef)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[MassTransit.Util.ChannelExecutor+<Run>d__11`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], MassTransit, Version=7.1.4.0, Culture=neutral, PublicKeyToken=b8e0e9f2f1e657fa]](<Run>d__11`1<System.__Canon> ByRef)
   at MassTransit.Util.ChannelExecutor.Run[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.Func`1<System.__Canon>, System.Threading.CancellationToken)
   at MassTransit.KafkaIntegration.Contexts.KafkaConsumerContext`2[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Consume(System.Threading.CancellationToken)
   at MassTransit.KafkaIntegration.Contexts.SharedConsumerContext`2[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Consume(System.Threading.CancellationToken)

When I've commented out topics registration or used RabbitMq as a bus, it has stopped occurring. I know that an in-memory bus is mainly for testing, but there is an existing Kafka infrastructure, so running another infrastructure makes no sense.

The difference between in-memory vs RabbitMQ is likely just timing. The error appears to be throw by the Confluent client, I'm not sure if it's an issue in their package or the way it's being called. — Chris Patterson, Feb 05 '21 at 13:09
@ChrisPatterson Do you have any advice on what to do about it? — Rafał Ryszkowski, Feb 05 '21 at 14:10
I don't see how that makes sense. And the Confluent recommendation is one minute, so... — Chris Patterson, Feb 05 '21 at 15:54
Taking into account Chris answer, you shouldn't commit so often. Your application is spending more time on committing than actual message handling — NoNameR, Feb 05 '21 at 22:13
@NoNameR According to Docker: yes, it is. Of course Kafka is dockerized, not the application. — Rafał Ryszkowski, Feb 06 '21 at 17:54
sometime it happens with Docker, I dunno why, previously it was happening even frequently — NoNameR, Feb 07 '21 at 18:17
Did you already try KafkaFlow instead of MassTransient? I think is more flexible and easier to use :) — Filipe Ferreira Esch, Feb 19 '21 at 17:21
@FilipeFerreiraEsch The solution you suggested doesn't have sagas. — Rafał Ryszkowski, Feb 20 '21 at 16:37

AccessViolationException during application exit while using MassTransit with Kafka and in-memory bus

0 Answers0