MassTransit StateMachine Saga - running behind LoadBalncer, How to stop consuming the same published message more than once

Question

In MassTransit Send and RequestClient will be mapped to exchange or queue, That will be handled by LoadBalanced Consumer. But for Publish Message, It will be consumed by all the instances that are running and waiting for the Message.

So, In StateMachine, Consumer has to publish the Events, That will make if more than once StateMachine Instance running it will be Picked by both StateMachine and Process will be duplicated? This is what happening at my work. So, We end up running Single StateMachine Instance.

await context.Publish(new { context.Message.OrderId, context.Message.Timestamp, context.Message.CustomerNumber, context.Message.PaymentCardNumber, context.Message.Notes }); This publishes the events to Saga, if Saga is running in LoadBalancer. Both Instance will be receiving the SameEvent. And Start Processing the Event and changing the Next State. If this is the Case, How to solve this. Only one StateMachine Should Pick the published message at once.

We end up running Single StateMachine Instance. So, the Published message wont be picked by both instance and will endup haivng duplicate process.

The Current Implmentation:

Have a REST Api - That receives the request to Start the Initial State.

var sendToUri =
                new Uri(
                    $"rabbitMq://{_rabbitMqConfig.Host}/{_rabbitMqConfig.VirtualHost}-{_rabbitMqConfig.WfSagaQueue}");
            var endPoint = await bus.GetSendEndpoint(sendToUri);
            var req = wfRequest;
await endPoint.Send<IWfExecRequest>(req);

In the StateMachine :

 services.AddMassTransit(x =>
            {
                x.AddConsumer<WfExecRequestConsumer>();
                x.AddConsumer<WfTaskCompletedConsumer>();
                x.UsingRabbitMq((context, cfg) =>
                {
                    var wfTaskExecHandler = context.GetRequiredService<IWfTaskExecHandler>();
                    var wfManagementClient = context.GetRequiredService<IWfManagementClient>();
                    var wfSagaStateMachine = new MsrAutomationStateMachine(wfTaskExecHandler, wfManagementClient);

                    cfg.Host(HostCredets);
                    cfg.ReceiveEndpoint(queueName: "msr-automation-wf-exec-request", configureEndpoint: e =>
                    {
                        e.PrefetchCount = 1;
                        e.ConfigureConsumer<WfExecRequestConsumer>(context);
                        e.StateMachineSaga(wfSagaStateMachine, repo);
                    });
                    cfg.ReceiveEndpoint(queueName: "WfTaskCompleted", configureEndpoint: e =>
                    {
                        e.PrefetchCount = 1;
                        e.ConfigureConsumer<WfTaskCompletedConsumer>(context);
                    });
                });
            });

This StateMachine Receives , WfExecRequest (Inital Event), TaskCompleted and TaskFaulted (From Muliple Consumer Saga/Consumer) - This was done at Consumer Side as Context.Publish.

So, What I see if we Run more than one Instance of the same StateMachine the TaskCompled Message getting Consumed by both Instances.

Thanks Again.

Answer 1

First, clearly something is wrong with your configuration. If the saga state machine is running on a single queue (receive endpoint, regardless of how many instances of your service are running) it will automatically load balance on that single queue across all running instances.

If you are running multiple instances of the saga state machine on different queues, well yeah, you're basically doing it wrong.

Second, I'm not sure what "LoadBalancer" is but typically something that unnecessary when using a message broker. If "LoadBalancer" is something for your HTTP/API endpoints, that's fine, but the broker and the queue are the scale out points in a message-based system.

If you had posted actual code, or shared some explicit details that would help as it is now it's entirely based on supposition.