Eclipse ditto forwarding connection fails from time to time

Question

I have connected Eclipse hono with Eclipse ditto using the Connectivity api. When I set it up, this works fine. However, after some time the forwarding connection fails. When I retrieve the metrics, I'm getting following response:

{
    "?": {
        "?": {
            "type": "connectivity.responses:aggregatedResponse",
            "status": 200,
            "connectionId": "<connectionId>",
            "responsesType": "connectivity.responses:retrieveConnectionMetrics",
            "responses": {
                "connectivity-7cc7b5dc4c-6nn59": {
                    "type": "connectivity.responses:retrieveConnectionMetrics",
                    "status": 200,
                    "connectionId": "<connectionId>",
                    "connectionMetrics": {
                        "connectionStatus": "open",
                        "connectionStatusDetails": "Connected at 2019-03-19T08:28:53.211Z",
                        "inConnectionStatusSince": "2019-03-19T08:28:53.211Z",
                        "clientState": "CONNECTED",
                        "sourcesMetrics": [],
                        "targetsMetrics": [
                            {
                                "addressMetrics": {
                                    "gw/{{ thing:namespace }}/{{ thing:id }}": {
                                        "status": "failed",
                                        "statusDetails": "Producer closed at 2019-03-19T21:00:16.466Z",
                                        "messageCount": 2048,
                                        "lastMessageAt": "2019-03-19T21:00:05.361Z"
                                    }
                                },
                                "publishedMessages": 4070
                            }
                        ]
                    }
                }
            }
        }
    }
}

I've been checking the logs around the time mentioned, but I'm not getting any errors. The logs I'm posting here are the last one before and the first one after the mentioned timestamp (2019-03-19T21:00:16.466Z).

2019-03-19 21:00:11,771 DEBUG [ID:AMQP_NO_PREFIX:TelemetrySenderImpl-42872] o.e.d.s.c.m.a.AmqpPublisherActor akka://ditto-cluster/system/sharding/connection/7/tenant_aloxy_consumer-aloxy-forward/pa/$a/c1/amqpPublisherActor3
 - Message JmsTextMessage { org.apache.qpid.jms.provider.amqp.message.AmqpJmsTextMessageFacade@9bc051af } sent successfully.
2019-03-19 21:01:11,733 DEBUG [ID:AMQP_NO_PREFIX:TelemetrySenderImpl-42872] o.e.d.s.c.m.a.AmqpClientActor akka://ditto-cluster/system/sharding/connection/1/tenant_aloxy_consumer-aloxy/pa/$a/c1 - Inbound message: JmsInboundMessageDispatch { sequence = 38885, messageId = TelemetrySenderImpl-42873, consumerId = ID:a4925b59-1bb4-4cd8-9151-96ad422c36df:1:1:1 }

Although the log levels for all ditto services are set to debug, I'm not getting any useful logging.
Does any of you have any idea how I can get the loggging to investigate this problem or, even better, have any idea on what the problem might be and how to fix it?

When I delete the connection and recreate it, everything works as expected again. Maybe ditto can do this under the hood automatically?

UPDATE
When retrieving the connection via the API, I'm getting following response (including the failoverEnabled property which is set to true). This also indicates that the connection uses AMQP 1.0. The broker used is Enmasse.

{
    "?": {
        "?": {
            "type": "connectivity.responses:retrieveConnection",
            "status": 200,
            "connection": {
                "id": "<connectionId>",
                "name": null,
                "connectionType": "amqp-10",
                "connectionStatus": "open",
                "uri": "amqp://<consumer>:<password>@<amqp-host>:5672",
                "sources": [],
                "targets": [
                    {
                        "address": "gw/{{ thing:namespace }}/{{ thing:id }}",
                        "topics": [
                            "_/_/things/twin/events?filter=exists(features/alp)"
                        ],
                        "authorizationContext": [
                            "<auth-context>"
                        ]
                    }
                ],
                "clientCount": 1,
                "failoverEnabled": true,
                "validateCertificates": true,
                "processorPoolSize": 5,
                "tags": []
            }
        }
    }
}

score 2 · Accepted Answer · answered Mar 20 '19 at 13:14

2

Eclipse Ditto does an automatic failover if configured to so do (see https://www.eclipse.org/ditto/basic-connections.html - "failoverEnabled" property in the model). It could however be that this was improved since the release 0.8.0 you are using. The Ditto team is currently working towards a 0.9.0-M1 release which would contain an improved reconnection behavior.

Does the connection to Eclipse Hono automatically reconnect? You described that the "forwarding connection" fails from time to time. Which technology (broker, etc.) is as endpoint for that gw/{{ thing:namespace }}/{{ thing:id }} address?

answered Mar 20 '19 at 13:14

Thomas Jäckle

1,123
2
7
21

I've updated the question with the asked information. Failover was already enabled and the used technology is AMQP 1.0. I'm not having any problems with the eclipse hono connection. This keeps receiving the messages from hono – Bob Claerhout Mar 20 '19 at 14:59
Ok - did you try to build Ditto by yourself with the latest "master"? That should already improve failover - if that doesn't help maybe the Enmasse closes connections after an idle timeout? – Thomas Jäckle Mar 20 '19 at 15:29
I'll try to build Ditto with the latest master. I'll let you know if this fixed it. Although it's quite hard to check because it's not failing consequently. – Bob Claerhout Mar 21 '19 at 12:30

Eclipse ditto forwarding connection fails from time to time

1 Answers1