0

Could you please help me by providing some suggestions on consuming the Azure Service Bus streaming message using Python.

As I found there is no spark structured streaming source for Azure Service Bus then in this case can I read the Azure Service Bus message using provided Python client and from Python client I read the each message and write it into Kafka topic and on this Kafka topic I will apply the spark structured streaming programing.

My use case is to consume the Azure Service Bus streaming message and write each message by transforming it into a timestream database InfluxDb or Pramethoues and show the real time dashboard on business metrics in Grafana.

I am thinking of reading the Azure Service Bus streaming message using python kafka producer like program and write this data into Kafka topic and then consume this data into Spark structures streaming with Kafka topic.

Please suggest am I going in the right direction?Any suggestion will be appreciated....

Nicolas
  • 45
  • 4

1 Answers1

0

Looks like there is no readily available connector since Service Bus is not designed with this in mind, unlike Event Hubs (which provides the Kafka Protocol). But it should be possible to write your own receiver (like this one).

Another alternative would be to immediately forward messages from Service Bus to a compatible source like Event Hubs (or Kafka) using something simple like Azure Functions.

Azure Functions along with bindings for both Serice Bus and Event Hubs / Kafka, you could implement this forwarding service with almost no code. But if you prefer, using the Python SDK for both in your own client will do the trick as well which itself could be an Azure Function as well.

-- From my original answer on Microsoft Q&A

PramodValavala
  • 6,026
  • 1
  • 11
  • 30
  • 1
    Thanks a lot for your suggestions and hints.I would check all this options also.I am also thinking to using the apache Nifi ConsumeAMQP connector as Azure Service bus is AMQP message and Nifi will put this message into Kafka topic.Will check again with you after some R&D on all this as suggested. – Nicolas Mar 26 '21 at 14:32
  • @Nicolas Did you find a good approach in regards to consuming a service bus with DBR? – Kristian Schneider Sep 07 '21 at 06:11
  • 1
    @KristianSchneider - Unfortunately till now we have not reached on this Streaming phase of project and that is why not able to do much R&D on this. In a training cloudera suggested to use apache Nifi ConsumeAMQP processor you can try once this options but not sure as of now. – Nicolas Sep 08 '21 at 07:39