Apache ActiveMQ Artemis HA cluster deployment in Kubernetes with the Istio proxy injected into Artemis pods logs lot of the connection failure errors

Question

Apache ActiveMQ Artemis cluster in Kubernetes environment with the Istio proxy injected into the Artemis pods logs lot of errors showing connection loss between master(active) and slave (backup) instances. We also experience intermittent connection loss from JBoss EAP to Artemis which gets restored on the subsequent call (e.g., when sending a JMS message).

The tcpdump for the active (master) instance shows lots of RST for the connections between backup and active artemis instances (comparing to tcpdump for environment where Artemis pods do not have Istio proxy injected).

The allowed/used protocols for Artemis acceptor are CORE, AMQP. The default communication ports are preserved for the Artemis cluster.

The Artemis cluster is used for JMS messaging. JMS communication is performed from Jboss EAP that "sits" in another pod in the same kubernetes namespace.

The static connectors are used to form the Artemis cluster. The replication mechanism is used for data exchange. The static connectors are used in the Jboss configuration as well.

The TCP and HTTP connections idleTimeout was set to infinite (for both INBOUND and OUTBOUND) in the Istio proxy.

When Istio proxy is not injected, the Artemis logs do not show any error and no issues with JMS messaging observed in this case.

NOTE: There is no ping command available on the container where artemis/jboss are installed (in case it matters for performing "alive checks")

The stack trace from the active (master) Artemis pod (multiple snippets):

2023-04-10 20:56:19,436 WARN  [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure to <***artemis backup instance's DNS***>/<***artemis backup instance's IP***>:61616 has been detected: AMQ219011: Did not receive data from server for org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnection@6243c958[ID=fef52fc9, local= /<***artemis active instance's IP***>:55842, remote=<***artemis backup instance's DNS***>/<***artemis backup instance's IP***>:61616] [code=CONNECTION_TIMEDOUT]
2023-04-10 20:56:19,562 ERROR [org.apache.activemq.artemis.core.client] AMQ214016: Failed to create netty connection
java.lang.IllegalStateException: No ActiveMQChannelHandler has been found while connecting to <***artemis backup instance's DNS***>/<***artemis backup instance's IP***>:61616 from Channel with id = cf68f05a
    at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.createConnection(NettyConnector.java:954) ~[artemis-core-client-2.27.0.jar:2.27.0]
    at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.createConnection(NettyConnector.java:840) ~[artemis-core-client-2.27.0.jar:2.27.0]
    at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.createConnection(NettyConnector.java:822) ~[artemis-core-client-2.27.0.jar:2.27.0]
    at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.openTransportConnection(ClientSessionFactoryImpl.java:1105) ~[artemis-core-client-2.27.0.jar:2.27.0]
    at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createTransportConnection(ClientSessionFactoryImpl.java:1212) ~[artemis-core-client-2.27.0.jar:2.27.0]
    at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createTransportConnection(ClientSessionFactoryImpl.java:1146) ~[artemis-core-client-2.27.0.jar:2.27.0]
    at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.establishNewConnection(ClientSessionFactoryImpl.java:1375) ~[artemis-core-client-2.27.0.jar:2.27.0]
    at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnection(ClientSessionFactoryImpl.java:967) ~[artemis-core-client-2.27.0.jar:2.27.0]
    at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:858) ~[artemis-core-client-2.27.0.jar:2.27.0]
    at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.reconnectSessions(ClientSessionFactoryImpl.java:799) ~[artemis-core-client-2.27.0.jar:2.27.0]
    at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:656) ~[artemis-core-client-2.27.0.jar:2.27.0]
    at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:534) ~[artemis-core-client-2.27.0.jar:2.27.0]
    at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:527) ~[artemis-core-client-2.27.0.jar:2.27.0]
    at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:390) ~[artemis-core-client-2.27.0.jar:2.27.0]
    at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:58) ~[artemis-commons-2.27.0.jar:?]
    at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:33) ~[artemis-commons-2.27.0.jar:?]
    at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:69) ~[artemis-commons-2.27.0.jar:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
    at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) ~[artemis-commons-2.27.0.jar:?]

2023-04-11 01:22:33,180 WARN  [org.apache.activemq.artemis.core.server] AMQ222092: Connection to the backup node failed, removing replication now
org.apache.activemq.artemis.api.core.ActiveMQConnectionTimedOutException: AMQ229014: Did not receive data from /127.0.0.6:55653 within the 60000ms connection TTL. The connection will now be closed.
    at org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$FailureCheckAndFlushThread$2.run(RemotingServiceImpl.java:781) ~[artemis-server-2.27.0.jar:2.27.0]
    at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:58) ~[artemis-commons-2.27.0.jar:?]
    at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:33) ~[artemis-commons-2.27.0.jar:?]
    at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:69) ~[artemis-commons-2.27.0.jar:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
    at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) ~[artemis-commons-2.27.0.jar:?]
2023-04-11 01:22:33,405 ERROR [org.apache.activemq.artemis.core.client] AMQ214016: Failed to create netty connection
java.lang.IllegalStateException: No ActiveMQChannelHandler has been found while connecting to <***artemis backup instance's DNS***>/<***artemis backup instance's IP***>:61616 from Channel with id = f979331a
    at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.createConnection(NettyConnector.java:954) ~[artemis-core-client-2.27.0.jar:2.27.0]
    at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.createConnection(NettyConnector.java:840) ~[artemis-core-client-2.27.0.jar:2.27.0]
    at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.createConnection(NettyConnector.java:822) ~[artemis-core-client-2.27.0.jar:2.27.0]
    at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.openTransportConnection(ClientSessionFactoryImpl.java:1105) ~[artemis-core-client-2.27.0.jar:2.27.0]
    at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createTransportConnection(ClientSessionFactoryImpl.java:1212) ~[artemis-core-client-2.27.0.jar:2.27.0]
    at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createTransportConnection(ClientSessionFactoryImpl.java:1146) ~[artemis-core-client-2.27.0.jar:2.27.0]
    at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.establishNewConnection(ClientSessionFactoryImpl.java:1375) ~[artemis-core-client-2.27.0.jar:2.27.0]
    at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnection(ClientSessionFactoryImpl.java:967) ~[artemis-core-client-2.27.0.jar:2.27.0]
    at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:858) ~[artemis-core-client-2.27.0.jar:2.27.0]
    at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.reconnectSessions(ClientSessionFactoryImpl.java:799) ~[artemis-core-client-2.27.0.jar:2.27.0]
    at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:656) ~[artemis-core-client-2.27.0.jar:2.27.0]
    at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:534) ~[artemis-core-client-2.27.0.jar:2.27.0]
    at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:527) ~[artemis-core-client-2.27.0.jar:2.27.0]
    at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:390) ~[artemis-core-client-2.27.0.jar:2.27.0]
    at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:58) ~[artemis-commons-2.27.0.jar:?]
    at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:33) ~[artemis-commons-2.27.0.jar:?]
    at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:69) ~[artemis-commons-2.27.0.jar:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
    at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) ~[artemis-commons-2.27.0.jar:?]

The stack trace from the Jboss pod:

2023-04-10 20:56:10,649 AMQ214016: Failed to create netty connection: java.lang.IllegalStateException: No ActiveMQChannelHandler has been found while connecting to <***artemis backup instance's DNS***>/<***artemis backup instance's IP***>:61616 from Channel with id = 98ac7598
    at org.apache.activemq.artemis@2.16.0.redhat-00045//org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.createConnection(NettyConnector.java:970)
    at org.apache.activemq.artemis@2.16.0.redhat-00045//org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.createConnection(NettyConnector.java:856)
    at org.apache.activemq.artemis@2.16.0.redhat-00045//org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.createConnection(NettyConnector.java:838)
    at org.apache.activemq.artemis@2.16.0.redhat-00045//org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.openTransportConnection(ClientSessionFactoryImpl.java:1097)
    at org.apache.activemq.artemis@2.16.0.redhat-00045//org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createTransportConnection(ClientSessionFactoryImpl.java:1146)
    at org.apache.activemq.artemis@2.16.0.redhat-00045//org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.establishNewConnection(ClientSessionFactoryImpl.java:1378)
    at org.apache.activemq.artemis@2.16.0.redhat-00045//org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnection(ClientSessionFactoryImpl.java:952)
    at org.apache.activemq.artemis@2.16.0.redhat-00045//org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:841)
    at org.apache.activemq.artemis@2.16.0.redhat-00045//org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.reconnectSessions(ClientSessionFactoryImpl.java:779)
    at org.apache.activemq.artemis@2.16.0.redhat-00045//org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:638)
    at org.apache.activemq.artemis@2.16.0.redhat-00045//org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:525)
    at org.apache.activemq.artemis@2.16.0.redhat-00045//org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:518)
    at org.apache.activemq.artemis@2.16.0.redhat-00045//org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.access$100(ClientSessionFactoryImpl.java:74)
    at org.apache.activemq.artemis@2.16.0.redhat-00045//org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:381)
    at org.apache.activemq.artemis.journal//org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42)
    at org.apache.activemq.artemis.journal//org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
    at org.apache.activemq.artemis.journal//org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at org.apache.activemq.artemis.journal//org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)

Actions performed: We tried to set idleTimeout to infinite for the INBOUND and OUTBOUND connections for artemis and jboss Istio proxies using the EnvoyFilter configuration (the Envoy Filter is a configuration resource for the Istio proxy - helps to configure inbound/outbound traffic). We also set reconnect-attempts to "-1" and connection-ttl to "86400000" for the pooled-connection-factory in jboss (standallone-full.xml file).

We were expecting to keep the connections to Artemis alive at least for 24 hrs, but it did not happen.

I am wondering what the possible root cause is and what additional configuration needs to be applied to Artemis and Jboss installation to keep the connections alive. Is this issue related to the "keep alive" checks?

I'm not very familiar with this Istio proxy you're injecting. Can you elaborate on why you're injecting it in the first place? What purpose does it serve for the broker? Is it used by the broker, external clients, or both? Lastly, you didn't provide any details about what exactly you changed and where you changed it when adjusting the timeouts, etc. Can you elaborate on that? — Justin Bertram, Apr 13 '23 at 13:25
We use Istio Service Mesh for our containerized application - Istio Service Mesh helps, for example, to establish secure connection between pods without making changes in applications itself (in our case, Artemis and Jboss server) - all traffic goes through the proxies, and it is the proxies that take care for certificates. But in our test we did not enable the secure communication, the communication Jboss-Artemis, Artemis-Artemis is performed in a plain mode. More info - when we observed lot of RSTs in the TCPDump file, noticed that those calls are made every millisecond. — MYC, Apr 13 '23 at 16:57
So all the connections are still going through the Istio proxies even though the "secure communication" has not been enabled? — Justin Bertram, Apr 13 '23 at 17:29
If everything works _without_ the Istio proxies and then you see errors _with_ the Istio proxies it stands to reason that the problem is the Istio proxies. I'm not sure what can be done about this from the ActiveMQ Artemis or JBoss side of things. — Justin Bertram, Apr 13 '23 at 18:37
I agree that the problem might be in Istio proxy, I am just trying to understand what type of network traffic I need to enable on Istio proxy to make Artemis to work. Probably I need to ask more specific question like - how "keep alive" check is performed on artemis connections, is it pingRequest? why do we see RST records in tcpdump every millisecond - is it related to some setting on Artemis side because, that sends some tcp packages every millisecond? — MYC, Apr 13 '23 at 19:12
Generally speaking all the messaging protocols supported by ActiveMQ Artemis use TCP although there are some special circumstances where HTTP is used. — Justin Bertram, Apr 13 '23 at 19:14
On an Artemis "core" protocol connection the keep-alive is performed using a protocol-specific "ping" packet. I don't know what you mean by "pingRequest" specifically. Can you clarify that? — Justin Bertram, Apr 13 '23 at 19:20
Under pingRequest I meant just some sort of ping package. Justin, could you please point me to the source code that performs the Core protocol-specific "ping"? — MYC, Apr 13 '23 at 19:29
The class that's used to represent the "ping" is [here](https://github.com/apache/activemq-artemis/blob/main/artemis-core-client/src/main/java/org/apache/activemq/artemis/core/protocol/core/impl/wireformat/Ping.java). When a new protocol connection is established via [`ClientSessionFactoryImpl`](https://github.com/apache/activemq-artemis/blob/main/artemis-core-client/src/main/java/org/apache/activemq/artemis/core/client/impl/ClientSessionFactoryImpl.java#L1406) a Java `Runnable` is passed to a `java.util.concurrent.ScheduledExecutorService` according to the `clientFailureCheckPeriod`. — Justin Bertram, Apr 13 '23 at 20:27
Investigation is still going on, Justin. We might post a question to Istio Service Mesh community as well. And I might come with additional questions regarding Artemis. Do you want me to close this Question and raise another one when I need things to clarify around Artemis? — MYC, Apr 19 '23 at 05:13
This question seems to be going nowhere so that's probably best. — Justin Bertram, Apr 19 '23 at 05:19
The ArtemisCloud operator seems that partially supports Istio, see https://github.com/artemiscloud/activemq-artemis-operator/discussions/432 — Domenico Francesco Bruscino, Apr 19 '23 at 13:30
Justin, I think we found the solution, at least the original helm chart with artemis does not have that issue. I will need one more day for testing, just to finalize this question with a solution/explanation — MYC, Apr 19 '23 at 17:12
Justin, I think we can mark this question as answered. This issue does not seem to be related to Artemis, but to the configuration that we have in kubernetes for it. I still continue the investigation. When I have a clear answer what exactly was causing this issue, I will share here (unless it is related to our product specifics:) ). Thanks for you help! — MYC, Apr 21 '23 at 20:52

MYC · Answer 1 · 2023-05-02T14:43:48.627

0

This issue does not seem to be related to Artemis, but to the configuration that is used in kubernetes for the Artemis cluster.

UPDATE 5/2: additional testing was performed, and it seems that allow-direct-connections-only="true" as follows:

<static-connectors allow-direct-connections-only="true">

solves the issue till Artemis 2.24, but starting from 2.26 it still not working with Istio (and the logs shared above are still relevant).

edited May 02 '23 at 14:43

answered Apr 21 '23 at 21:00

MYC

3
3

As it’s currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Apr 30 '23 at 00:39
the answer has been updated – MYC May 02 '23 at 11:57

Apache ActiveMQ Artemis HA cluster deployment in Kubernetes with the Istio proxy injected into Artemis pods logs lot of the connection failure errors

1 Answers1