SignalR + RabbitMQ Backplane: SignalR client receives same message repeatedly

Question

https://github.com/mdevilliers/SignalR.RabbitMq/issues/43

I'm having real difficulty getting this to work, can someone please glance at this and see if something is wrong in the setup?

Here's my test:

I spin up two self hosted servers, configured to the same RabbitMqScaleoutConfiguration

[Test]
    public void BasicBackplaneTest()
    {
        SubsciberTestServerNode nodeA = null;
        SubsciberTestServerNode nodeB = null;

        string messageA = null;
        string messageB = null;
        try
        {
            Log("Given I have a WorkCenter Dispatch Publisher");
            var publisher = new WcDispatchPublisher(ConnectionString);

            Log("And I have multiple server nodes subscribed to the backplane");
            nodeA = new SubsciberTestServerNode("nodeA").Start().Result;
            nodeB = new SubsciberTestServerNode("nodeB").Start().Result;

            Log("And I wait 5 seconds");
            Thread.Sleep(5000);

            Log("When I publish a message: {0}", TestPayload);
            publisher.Publish(TestPayload);

            Log("And I wait 60 seconds");
            Thread.Sleep(TimeSpan.FromSeconds(60));

            messageA = nodeA.Message;
            messageB = nodeB.Message;
        }
        catch (AggregateException exception)
        {
            Log("Exception Occurred: {0}", exception.Flatten().Message);
            Exception = exception;
        }
        catch (Exception exception)
        {
            Log("Exception Occurred: {0}", exception.Message);
            Exception = exception;
        }
        finally
        {
            nodeA?.Dispose();
            nodeB?.Dispose();

            Log("Then no exceptions should have been thrown.");
            Exception.Should().BeNull();

            Log("Then the message should have been added to the Message Queue");
            messageA.Should().NotBeNullOrWhiteSpace();
            messageB.Should().NotBeNullOrWhiteSpace();
        }

Server:

internal class SubsciberTestServerNode : IDisposable
{
    private readonly string _nodeName;
    private readonly string _url;
    private WcDispatchSubscriber _subscriber;
    private IDisposable _webApp;

    public SubsciberTestServerNode(string nodeName)
    {
        _nodeName = nodeName;
        _url = $"http://localhost:9999/{nodeName}";
        MessageList = new List<string>();
    }

    public string Message { get; set; }
    public List<string> MessageList { get; set; }

    public void Dispose()
    {
        if (_webApp != null)
        {
            _webApp.Dispose();
            _webApp = null;
            _subscriber.Dispose();
            _subscriber = null;
        }
    }

    public async Task<SubsciberTestServerNode> Start()
    {
        _webApp = WebApp.Start(_url, app =>
        {
            new Startup(_nodeName).Configuration(app);
            Thread.Sleep(TimeSpan.FromSeconds(5));
            //Place this code into your Application_Start() method.
            var factory = new ConnectionFactory
            {
                UserName = "guest",
                Password = "guest",
                HostName = "localhost"
            };

            var exchangeName = "WC_LeadDispatch_Exchange";

            var configuration = new RabbitMqScaleoutConfiguration(factory, exchangeName);
            GlobalHost.DependencyResolver.UseRabbitMq(configuration);
            GlobalHost.Configuration.TransportConnectTimeout = TimeSpan.FromSeconds(10);


            Thread.Sleep(TimeSpan.FromSeconds(5));
        });

        _subscriber = new WcDispatchSubscriber();
        await _subscriber.Subscribe(_url, msg =>
        {
            string message = $"Message received at Node: {_nodeName}. Message: {msg}.";
            Console.WriteLine(message);
            Message = message;
            MessageList.Add(message);
        });
        return this;
    }
}

Subscriber:

public class WcDispatchSubscriber : IDisposable
{
    private const string HubName = "DispatchUpdateHub";
    private const string MessageEventName = "addMessage";
    private readonly int _connectionLimitInt;
    private IDisposable _hubProxySubscription;

    public WcDispatchSubscriber()
    {
        string connectionLimit = ConfigurationManager.AppSettings.Get("SignalRConnectionLimit");
        int.TryParse(connectionLimit, out _connectionLimitInt);
        _connectionLimitInt = _connectionLimitInt == 0 ? 100 : _connectionLimitInt;
    }

    public void Dispose()
    {
        _hubProxySubscription.Dispose();
    }

    public async Task Subscribe(string hubConnectionString, Action<string> messageReceived)
    {
        var hubConnection = new HubConnection(hubConnectionString);
        IHubProxy dispatchHubProxy = hubConnection.CreateHubProxy(HubName);
        _hubProxySubscription = dispatchHubProxy.On(MessageEventName, messageReceived);
        ServicePointManager.DefaultConnectionLimit = _connectionLimitInt;
        await hubConnection.Start();
    }
}

Publisher:

public class WcDispatchPublisher
{
    private const string ExchangeName = "WC_LeadDispatch_Exchange";
    private readonly IHubContext _hubContext;

    public WcDispatchPublisher(string connectionString)
    {
        //actual string will look like this.  we may need to overload the other constructors in the Rabbit/SigR.
        //_rabbitConnectionString =
        //  "host=cprmqsrvt02vn01:5672;publisherConfirms=true;username=unittest;password=Un1t735t;virtualhost=UnitTest-NotificationService";
        var configuration = new RabbitMqScaleoutConfiguration(connectionString, ExchangeName);
        GlobalHost.DependencyResolver.UseRabbitMq(configuration);

        _hubContext = GlobalHost.ConnectionManager.GetHubContext<DispatchUpdateHub>();
    }

    /// <summary>
    /// </summary>
    /// <param name="payload"></param>
    public void Publish(string payload)
    {
        Task.Factory.StartNew(() =>
        {
            _hubContext.Clients.All.addMessage(payload);
        }).Wait();
    }
}

this will work every say 12th run or so, normally here's what I get:

Given I have a WorkCenter Dispatch Publisher
And I have multiple server nodes subscribed to the backplane
And I wait 5 seconds
When I publish a message: test-payload
And I wait 60 seconds
Message received at Node: nodeA. Message: test-payload.
Message received at Node: nodeB. Message: test-payload.
Message received at Node: nodeB. Message:         {"LeadId":37252,"DispatchId":153595,"NotificationTypeId":1,"NotificationTitle":"    Leads Dispatch","DispatchDateTime":"2016-06-  16T17:16:26.187","AlertMessage":"Lead number 37252 dispatched 6/16/2016 at 7:16   AM CST","DispatchedToFranchise":5576,"FranchiseOpsId":130}.
Message received at Node: nodeB. Message:  {"LeadId":37252,"DispatchId":153595,"NotificationTypeId":1,"NotificationTitle":"  Leads Dispatch","DispatchDateTime":"2016-06- 16T17:16:26.187","AlertMessage":"Lead number 37252 dispatched 6/16/2016 at 7:16  AM CST","DispatchedToFranchise":5576,"FranchiseOpsId":130}.
 Message received at Node: nodeB. Message:  {"LeadId":37252,"DispatchId":153595,"NotificationTypeId":1,"NotificationTitle":" Leads Dispatch","DispatchDateTime":"2016-06- 16T17:16:26.187","AlertMessage":"Lead number 37252 dispatched 6/16/2016 at 7:16   AM CST","DispatchedToFranchise":5576,"FranchiseOpsId":130}.
 Message received at Node: nodeB. Message:  {"LeadId":37252,"DispatchId":153595,"NotificationTypeId":1,"NotificationTitle":" Leads Dispatch","DispatchDateTime":"2016-06- 16T17:16:26.187","AlertMessage":"Lead number 37252 dispatched 6/16/2016 at 7:16  AM CST","DispatchedToFranchise":5576,"FranchiseOpsId":130}.
 Message received at Node: nodeB. Message: {"LeadId":37252,"DispatchId":153595,"NotificationTypeId":1,"NotificationTitle":"Leads Dispatch","DispatchDateTime":"2016-06-16T17:16:26.187","AlertMessage":"Lead number 37252 dispatched 6/16/2016 at 7:16 AM CST","DispatchedToFranchise":5576,"FranchiseOpsId":130}.
 Message received at Node: nodeB. Message: {"LeadId":37252,"DispatchId":153595,"NotificationTypeId":1,"NotificationTitle":"Leads Dispatch","DispatchDateTime":"2016-06-16T17:16:26.187","AlertMessage":"Lead number 37252 dispatched 6/16/2016 at 7:16 AM CST","DispatchedToFranchise":5576,"FranchiseOpsId":130}.

... CONTINUES LIKE THIS FOREVER

Kelso Sharp · Answer 1 · 2016-06-16T20:07:44.973

1

Off the top of my head, I would say try:

nodeA = new SubsciberTestServerNode("nodeA").Start().Wait();
nodeB = new SubsciberTestServerNode("nodeB").Start().Wait();

or

nodeA = await new SubsciberTestServerNode("nodeA").Start();
nodeB = await new SubsciberTestServerNode("nodeB").Start();

Result is a thread blocking call, so its likely blocking nodeA from firing.

edited Jun 16 '16 at 20:07

answered Jun 16 '16 at 19:16

Kelso Sharp

972
8
12

This change still presents the same behavior, although I've changed to your await code. Thank you – Chazt3n Jun 16 '16 at 20:39
Hmm... interesting, have you been able to step through the code at all? – Kelso Sharp Jun 20 '16 at 14:25
Oh yes, it just seems to be an issue running two servers in one test scenario – Chazt3n Jun 21 '16 at 15:26
The relevant backplane code actually works out in the environment – Chazt3n Jun 21 '16 at 15:27
Not this issue no because currently I'm unable to test on a single machine – Chazt3n Jun 22 '16 at 19:04
Is it possible that you are just not putting it under enough load to force the use of both servers? I would think that it needs to have a lot of clients hitting it at the same time, under load. I mean just a simple implementation of signalr can handle millions of messages a second. – Kelso Sharp Jun 22 '16 at 19:10
For sure, this will not be load based because it's a fanout rabbitMQ exchange type. That means one message will be delivered to all queues, and each server has a queue. It seems one is blocking the other on connect, like I said it will work every so often – Chazt3n Jun 23 '16 at 14:22
Then it sounds to me like it is a race condition, I don't think that its deadlocked, because its still processing message, but I am almost certain that there is something blocking at thread, how many threads do you have configured on your test box? – Kelso Sharp Jun 23 '16 at 16:27
Great questions, if you see the code above that's all the information I have. Should I be starting the services in different Tasks? – Chazt3n Jun 23 '16 at 18:16

SignalR + RabbitMQ Backplane: SignalR client receives same message repeatedly

1 Answers1