Mcollective-activemq load balancing

Question

We are facing couple of issues with ActiveMQ - MCollective load balancing. Can someone please help? Below is our setup

MCO has two ActiveMQ brokers configured in failover pool - Broker1 & Broker2. And we have set randomize property to true in server.cfg and client.cfg.

Broker1 & Broker2 together form an activemq cluster. They also have updateClientCluster and rebalanceClientCluster properties set to true (set on both activemq brokers).

Issue1: MCO clients are not equally load balanced and connected between broker1 and broker2. 700 MCO are connected to Broker1 whereas only 100 MCO connected to broker2. Are we missing any property for this loadbalancing ?

Issue2: When we add a new broker, say broker3 to activemq cluster, the MCO clients are not getting rebalanced or connecting to new broker. As per activemq documentation, if we enable updateClientCluster and rebalanceClientCluster properties, it should automatically notify MCO to rebalance. Can you please let us know any properties.

Attaching activemq.xml, server.cfg and client.cfg.

activemq.xml

    <transportConnectors>
    <transportConnector name="openwire" uri="tcp://0.0.0.0:61616?jms.prefetchPolicy.all=1000&amp;useQueueForAccept=false&amp;transport.closeAsync=false&amp;maximumConnections=2500&amp;wireFormat.maxFrameSize=104857600&amp;jms.watchTopicAdvisories=false" />

    <transportConnector name="stomp+ssl" uri="stomp+nio+ssl://0.0.0.0:61614?useQueueForAccept=false&amp;transport.closeAsync=false&amp;needClientAuth=true&amp;trace=true&amp;maximumConnections=2500&amp;wireFormat.maxFrameSize=104857600&amp;updateClusterClients=true&amp;rebalanceClusterClients=true&amp;updateClusterClientsOnRemove=true" />
    </transportConnectors>

server.cfg

    loglevel = info
    daemonize = 1

    # Plugins
    securityprovider = psk
    plugin.psk = unset

    connector = activemq
    plugin.activemq.heartbeat_interval = 60
    plugin.activemq.pool.size = 2
    plugin.activemq.pool.1.host = broker 1
    plugin.activemq.pool.1.port = port no.
    plugin.activemq.pool.1.user = user
    plugin.activemq.pool.1.password = password
    plugin.activemq.pool.1.ssl = 1
    plugin.activemq.pool.1.ssl.fallback = 0


    #classesfile = /var/lib/puppet/state/classes.txt

    # Registration:
    # We don't configure a listener, and only send these messages to keep the
    # Stomp connection alive. This will use the default "agentlist" registration
    # plugin.
    registerinterval = 600


    plugin.activemq.pool.2.host = broker 1
    plugin.activemq.pool.2.port = port no.
    plugin.activemq.pool.2.user = user
    plugin.activemq.pool.2.password = password
    plugin.activemq.pool.2.ssl = 1
    plugin.activemq.pool.1.ssl.fallback = 0

    securityprovider = ssl

     rpcauthorization = 1
    rpcauthprovider = action_policy
    plugin.actionpolicy.allow_unconfigured = 0
    plugin.actionpolicy.enable_default = 1
    plugin.actionpolicy.default_name = default
    \n




    plugin.activemq.initial_reconnect_delay = 0.01
    plugin.activemq.max_reconnect_delay = 30.0
    plugin.activemq.use_exponential_back_off = true
    plugin.activemq.back_off_multiplier = 2
    plugin.activemq.max_reconnect_attempts = 0
    plugin.activemq.randomize = true
    plugin.activemq.timeout = -1

client.cfg

    main_collective = mcollective
    libdir = /etc/mcollective
    logger_type = console
    loglevel = warn

    # Plugins
    securityprovider = psk
    plugin.psk = unset

    connector = activemq
    plugin.activemq.heartbeat_interval = 60
    plugin.activemq.pool.size = 2
    plugin.activemq.pool.1.host = borker 1
    plugin.activemq.pool.1.port = port no.
    plugin.activemq.pool.1.user = user
    plugin.activemq.pool.1.password = password
    plugin.activemq.pool.1.ssl = 1
    plugin.activemq.pool.1.ssl.fallback = 0

    plugin.activemq.pool.1.host = borker 2
    plugin.activemq.pool.1.port = port no.
    plugin.activemq.pool.1.user = user
    plugin.activemq.pool.1.password = password
    plugin.activemq.pool.2.ssl = 1
    plugin.activemq.pool.2.ssl.fallback = 0


    factsource = yaml

    securityprovider = ssl 

Answer 1

Issue 1. The balance will be close, but not perfect. As far as I know, it simply forces the client to use the randomize function after it initally connects, or is told to rebalance.

Issue 2: You are setting up the auto rebalance as part of the URI rather than an attribute of the transportConnector.

So you need something like:

    <transportConnector name="stomp+ssl" uri="stomp+nio+ssl://0.0.0.0:61614?useQueueForAccept=false&amp;transport.closeAsync=false&amp;needClientAuth=true&amp;trace=true&amp;maximumConnections=2500&amp;wireFormat.maxFrameSize=104857600" 
        updateClusterClients="true"
        rebalanceClusterClients="true"
        updateClusterClientsOnRemove="true" />
</transportConnectors>

Also know this will only apply to clients connecting over that one transport connector and not the other, which you only have a URI specified. Any clients connecting to the tcp transport on 61616 will have no knowledge of any of the settings on your Stomp transport.

Everything else looks good, although I've not set up any Stomp clients with that many properties, so you may want to double check they are all valid as well.