I'm using Wildfly 9.1 in domain mode.
I have three hosts: A , B and C.
All nodes have a test server with the same test.ear.
After starting the three nodes, all ClusterConnectionBridge are created correctly between them.
Each node send messages on commonTopic, and an MDB logs the received messages.
The problem is that what I see logged on the node A is only the messages sent by himself. On the other nodes i see logged all the messages sent by A , B , C . This is weird, since using tcpdump on node A i see the incoming messages on port 8080 from the other nodes.
Using cli to get metrics on node A:
[domain@MASTER:19990 /] /host=SLAVE.A/server=test/subsystem=messaging/hornetq-server=default/jms-topic=commonTopic:read-resource(recursive=true,proxies=false,include-runtime=true,include-defaults=true)
{
"outcome" => "success",
"result" => {
"delivering-count" => 0,
"durable-message-count" => 0,
"durable-subscription-count" => 1,
"entries" => [
"java:/jms/commonTopic",
"java:jboss/exported/jms/commonTopic"
],
"message-count" => 0L,
"messages-added" => 150L,
"non-durable-message-count" => 0,
"non-durable-subscription-count" => 3,
"subscription-count" => 4,
"temporary" => false,
"topic-address" => "jms.topic.commonTopic"
}
}
I see that the number of added messages is 3 times the messages sent by one node. So it seems messages are received through the bridge but not consumed by the MDB.
Could someone give me an idea where to look for this issue.
The hosts are on virtual machines , on the same LAN.
The same happens if I start a node D on my PC and join it to the cluster. Node D logs all messages from the other 3 nodes, but on the other nodes no D message are logged.
The following is a snippet from my domain.xml regarding HornetQ configuration:
<subsystem xmlns="urn:jboss:domain:messaging:3.0">
<hornetq-server>
<security-enabled>false</security-enabled>
<jmx-management-enabled>true</jmx-management-enabled>
<journal-file-size>102400</journal-file-size>
<connectors>
<http-connector name="http-connector" socket-binding="http">
<param key="http-upgrade-endpoint" value="http-acceptor"/>
</http-connector>
<http-connector name="http-connector-throughput" socket-binding="http">
<param key="http-upgrade-endpoint" value="http-acceptor-throughput"/>
<param key="batch-delay" value="50"/>
</http-connector>
<netty-connector name="netty" socket-binding="messaging"/>
<netty-connector name="netty-throughput" socket-binding="messaging-throughput">
<param key="batch-delay" value="50"/>
</netty-connector>
<in-vm-connector name="in-vm" server-id="0"/>
</connectors>
<acceptors>
<http-acceptor http-listener="default" name="http-acceptor"/>
<http-acceptor http-listener="default" name="http-acceptor-throughput">
<param key="batch-delay" value="50"/>
<param key="direct-deliver" value="false"/>
</http-acceptor>
<netty-acceptor name="netty" socket-binding="messaging"/>
<netty-acceptor name="netty-throughput" socket-binding="messaging-throughput">
<param key="batch-delay" value="50"/>
<param key="direct-deliver" value="false"/>
</netty-acceptor>
<in-vm-acceptor name="in-vm" server-id="0"/>
</acceptors>
<broadcast-groups>
<broadcast-group name="bg-group">
<jgroups-stack>udp</jgroups-stack>
<jgroups-channel>hq-cluster</jgroups-channel>
<connector-ref>
http-connector
</connector-ref>
</broadcast-group>
</broadcast-groups>
<discovery-groups>
<discovery-group name="dg-group">
<jgroups-stack>udp</jgroups-stack>
<jgroups-channel>hq-cluster</jgroups-channel>
</discovery-group>
</discovery-groups>
<cluster-connections>
<cluster-connection name="my-cluster">
<address>jms</address>
<connector-ref>http-connector</connector-ref>
<use-duplicate-detection>true</use-duplicate-detection>
<forward-when-no-consumers>false</forward-when-no-consumers>
<max-hops>1</max-hops>
<discovery-group-ref discovery-group-name="dg-group"/>
</cluster-connection>
</cluster-connections>
<security-settings>
<security-setting match="#">
<permission type="send" roles="guest"/>
<permission type="consume" roles="guest"/>
<permission type="createNonDurableQueue" roles="guest"/>
<permission type="deleteNonDurableQueue" roles="guest"/>
</security-setting>
</security-settings>
<address-settings>
<address-setting match="#">
<dead-letter-address>jms.queue.DLQ</dead-letter-address>
<expiry-address>jms.queue.ExpiryQueue</expiry-address>
<max-size-bytes>10485760</max-size-bytes>
<page-size-bytes>2097152</page-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<redistribution-delay>1000</redistribution-delay>
</address-setting>
</address-settings>
<jms-connection-factories>
<connection-factory name="InVmConnectionFactory">
<connectors>
<connector-ref connector-name="in-vm"/>
</connectors>
<entries>
<entry name="java:/jms/MessageConnectionFactory"/>
<entry name="java:jboss/jms/MessageConnectionFactory"/>
<entry name="java:/ConnectionFactory"/>
</entries>
<reconnect-attempts>-1</reconnect-attempts>
</connection-factory>
<connection-factory name="RemoteConnectionFactory">
<connectors>
<connector-ref connector-name="http-connector"/>
</connectors>
<entries>
<entry name="java:jboss/exported/jms/MessageConnectionFactory"/>
<entry name="java:jboss/exported/jms/RemoteConnectionFactory"/>
</entries>
<ha>true</ha>
<block-on-acknowledge>true</block-on-acknowledge>
<reconnect-attempts>-1</reconnect-attempts>
</connection-factory>
<pooled-connection-factory name="hornetq-ra">
<transaction mode="xa"/>
<connectors>
<connector-ref connector-name="in-vm"/>
</connectors>
<entries>
<entry name="java:/JmsXA"/>
<entry name="java:jboss/DefaultJMSConnectionFactory"/>
</entries>
</pooled-connection-factory>
</jms-connection-factories>
<jms-destinations>
<jms-queue name="ExpiryQueue">
<entry name="jms/queue/ExpiryQueue"/>
<entry name="java:jboss/exported/jms/queue/ExpiryQueue"/>
<durable>true</durable>
</jms-queue>
<jms-queue name="DLQ">
<entry name="jms/queue/DLQ"/>
<entry name="java:jboss/exported/jms/queue/DLQ"/>
<durable>true</durable>
</jms-queue>
<jms-topic name="commonTopic">
<entry name="java:/jms/commonTopic"/>
<entry name="java:jboss/exported/jms/commonTopic"/>
</jms-topic>
</jms-topic>
</jms-destinations>
</hornetq-server>
</subsystem>
I found out the problem was caused by message persistence. A 5 seconds time to live was not enough for node A to consume the message. I found the expired messages on expiredQueue. Each node sends one message every 10 seconds, so i don't understand the reason why the messages expire on node A.
