I am hoping someone has got this to work. I am the trying to connect RabbitMQ shovel to Azure Event hub or Service Bus. I found two tutorials on MS, (very short to set up) but both results in the shovel error. The request to Event hub or Service bus is success in the example.
My guess is that something is missing from the below tutorials or that some components have been changed and breaking changes has been introduced:
Importing Data from RabbitMQ into Azure Data Explorer via Event Hubs https://techcommunity.microsoft.com/t5/azure-data-explorer-blog/importing-data-from-rabbitmq-into-azure-data-explorer-via-event/ba-p/3777688 Shovel error
AMQP allows for interconnecting connecting brokers directly, for instance using routers like Apache Qpid Dispatch Router or broker-native "shovels" like the one of RabbitMQ. https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-amqp-overview
How to integrate Service Bus with RabbitMQ https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-integrate-with-rabbitmq
Version: RabbitMQ 3.10.7, Erlang 25.0
Case and steps: Adding a new Azure Service Bus Namespace rabbitmq01 basic public endpoint Creating our Azure Service Bus Queue from-rabbitmq01
Enabling the RabbitMQ Shovel Plugin
rabbitmq-plugins enable rabbitmq_shovel_management
Created queue in Rabbitmq telemetry01
Connecting RabbitMQ to Azure Service Bus Add SAS Policy, Manage now, tested with just send also. rabbitmq-shovel01 Primary Connection String
Connection String to AMQP https://red-mushroom-0f7446a0f.azurestaticapps.net/ amqps://rabbitmq-shovel01:SAS-KEY@rabbitmq01.servicebus.windows.net:5671/?sasl=plain
TNC
Test-NetConnection -ComputerName rabbitmq01.servicebus.windows.net -Port 5671 = True
Make the dynamic shovel with above properties. Shovel was done exactly like the tutorial The shovel status is just starting, never goes to running.
Log:
2023-07-08 19:09:56.385000+02:00 [error] <0.742.0> supervisor: {<0.742.0>,amqp10_client_connection_sup} 2023-07-08 19:09:56.385000+02:00 [error] <0.742.0> errorContext: start_error 2023-07-08 19:09:56.385000+02:00 [error] <0.742.0> reason: {badmatch,{error,closed}}
[...]
2023-07-08 19:09:56.386000+02:00 [error] <0.744.0> ** Stacktrace = 2023-07-08 19:09:56.386000+02:00 [error] <0.744.0> ** [{amqp10_client_frame_reader,init,1, 2023-07-08 19:09:56.386000+02:00 [error] <0.744.0> [{file,"amqp10_client_frame_reader.erl"}, 2023-07-08 19:09:56.386000+02:00 [error] <0.744.0> {line,109}]}, 2023-07-08 19:09:56.386000+02:00 [error] <0.744.0> {gen_statem,init_it,6,[{file,"gen_statem.erl"},{line,1001}]}, 2023-07-08 19:09:56.386000+02:00 [error] <0.744.0> {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]
[...]
2023-07-08 19:09:56.386000+02:00 [error] <0.688.0> Shovel 'rmq-2-sb' could not connect to destination 2023-07-08 19:09:56.387000+02:00 [error] <0.687.0> supervisor: {<0.687.0>,rabbit_shovel_dyn_worker_sup}
Shovel to Bus requests success
Tested
- I have tried to change TLS version in Service bus.
- Enabled rabbitmq_amqp1_0 plugin, https://github.com/rabbitmq/rabbitmq-amqp1.0.
- Tested a lot of combinations with the URI. After testing https://red-mushroom-0f7446a0f.azurestaticapps.net/ and URL enconding.
- Azure ServiceBus can be used with the AMQP 1.0 protocol. Since version 3.7 RabbitMQ supports shovels where either the source or destination (or both) uses AMQP 1.0. https://gist.github.com/kjnilsson/159c643fb34604f8ea20be336109261b
I have sent a mail to RabbitMQ community also. Thank you in advance
11.07.2023 Update New RabbitMQ 3.12.1 and Erlang 26.0
Today I did a new test with both tutorials. For the Event hub i followed it exactly. The shovel error is:
2023-07-11 10:01:57.638000+02:00 [error] <0.2006.0> supervisor: {<0.2006.0>,amqp10_client_connection_sup}
2023-07-11 10:01:57.638000+02:00 [error] <0.2006.0> errorContext: start_error
2023-07-11 10:01:57.638000+02:00 [error] <0.2006.0> reason: {options,incompatible,[{verify,verify_peer},{cacerts,undefined}]}
[...]
2023-07-11 10:09:27.788000+02:00 [error] <0.2476.0> Shovel 'shovel-07' could not connect to destination
For the Service bus it is the same.
First I tried with shovel Source AMQP 0.9.1, Destination, AMQP 1.0.
:
2023-07-11 10:44:41.042000+02:00 [error] <0.3627.0> supervisor: {<0.3627.0>,amqp10_client_connection_sup}
2023-07-11 10:44:41.042000+02:00 [error] <0.3627.0> errorContext: start_error
2023-07-11 10:44:41.042000+02:00 [error] <0.3627.0> reason: {options,incompatible,[{verify,verify_peer},{cacerts,undefined}]}
[...]
2023-07-11 10:44:41.043000+02:00 [error] <0.3610.0> Shovel 'shovel_01' could not connect to destination
Then I tried shovel Source AMQP 1.0, Destination, AMQP 1.0.
And as expected:
2023-07-11 11:08:18.155000+02:00 [error] <0.8999.0> amqp1_0_plugin_not_enabled
Enabled plugin amqp1_0
2023-07-11 11:14:58.438000+02:00 [error] <0.9613.0> supervisor: {<0.9613.0>,amqp10_client_connection_sup}
2023-07-11 11:14:58.438000+02:00 [error] <0.9613.0> errorContext: start_error
2023-07-11 11:14:58.438000+02:00 [error] <0.9613.0> reason: {options,incompatible,[{verify,verify_peer},{cacerts,undefined}]}
[...]
2023-07-11 11:14:58.438000+02:00 [error] <0.9582.0> Shovel 'shovel_03' could not connect to destination
After update to: RabbitMQ 3.12.1 and Erlang 26.0 .So it has do to with TLS, and that was not a step in any of the tutorials. This is gonna take time :) Any help is appreciated
I got in touch with MS, as per delivered test results, they did not find a issue with the test(s) done. TNC images validated connectivity. Further troubleshooting will be done by support. Looking forward to that. Collaboration is key.
