Questions tagged [azure-eventhub]

Microsoft Azure EventHubs is a highly-scalable high-throughput low-latency durable event ingestion system in Azure cloud. EventHubs serves as an event buffer for applications to process events with variable load profiles. Microsoft Azure Event Hubs is comparable to Apache Kafka or AWS Kinesis. This is a fully-managed service-offering built & run by the teams at Microsoft.

Microsoft Azure EventHubs is a highly scalable high throughput low latency durable event ingestion system in Azure cloud. EventHubs serves as an event buffer for applications to process events with variable load profiles.

Stream millions of events per second into multiple applications

Event Hubs is a highly scalable publish-subscribe ingestor that can intake millions of events per second so that you can process and analyze the massive amounts of data produced by your connected devices and applications. Once collected into Event Hubs you can transform and store data using any real-time analytics provider or with batching/storage adapters.

Enable applications to process events with variable load profiles

Today’s connected world is defined by big data. Big data originates from many sources such as connected cars and thermostats that produce telemetry data every few minutes, application performance counters that generate events every second or mobile apps that capture telemetry for every user’s individual action. In this connected world, you need a managed ingestor service capable of elastic scale in order to accommodate variable load profiles as well as the spikes caused by intermittent connectivity.

Connect millions of devices across platforms

The rapid proliferation of connected devices raises challenges due to the variety of platforms and protocols involved. Connecting these disparate data sources while handling the scale of the aggregate stream is a significant challenge. Event Hubs provide easy provisioning of capacity to ingest events from millions of devices while preserving event order on a per device basis. Support for AMQP and HTTP allow many platforms to work with Event Hubs. Native client libraries also exist for popular platforms.

1941 questions
4
votes
1 answer

Selecting Partition of Azure event hub when using rest api

I'm trying to send messages to Azure Event Hub using python and the rest API after some failed experiments i have found working code (see below) but i want to be able to select to which partition to send the event. Is this possible using the rest…
Srgrn
  • 1,770
  • 16
  • 30
4
votes
3 answers

Consuming messages from an Azure Event Hub using Python?

I can't find any documentation online using Python to subscribe and consume messages from an Azure Event Hub. I know it's possible in C, C#, and Java. I just need to know if it is possible to use Python. The Azure python SDK currently seems to only…
Phuc H Duong
  • 471
  • 3
  • 12
4
votes
1 answer

How can the serialized size of EventData be determined for batching?

In the documentation for the Azure EventHubClient there are two methods for sending a batch of data each of them has the remark below and will throw a MessageSizeExceededException if it is ignored. You should ensure that the total serialized size…
cacsar
  • 2,098
  • 1
  • 14
  • 27
4
votes
2 answers

Load testing Azure EventHub

I develop an application which uses Azure EventHub - which claims to be a very scalable solution for publishing and processing large number of events - and now I would like to load test the final solution to see if the whole system fits my…
Zsolt
  • 3,263
  • 3
  • 33
  • 48
4
votes
1 answer

When does Azure's EventProcessorHost call IEventProcessor.ProcessEventsAsync?

I'm writing a worker for Azure EventHub and I expect to be building on top of EventProcessorHost; however, I couldn't find some potentially relevant details either in the general documentation or the API documentation. When is IEventProcessor…
cacsar
  • 2,098
  • 1
  • 14
  • 27
4
votes
1 answer

What Azure .NET SDK EventHubClient instance methods are threadsafe?

I'm writing code that will be publishing messages from multiple threads to an Azure Event Hub in C# using the EventHubClient. The documentation for EventHubClient contains the fairly standard boiler plate. "Any public static (Shared in Visual…
cacsar
  • 2,098
  • 1
  • 14
  • 27
3
votes
1 answer

Azure Event grid delivery through private endpoint not possible - what are the alternatives?

We have an architecture in Azure sealed within a Vnet with private endpoints on all our services. We mostly work with transactional data, so quite moderate amounts of data throughput. Now, our initial plan was to communicate between Azure functions…
3
votes
2 answers

What is the logic behind scale controller in Azure Functions?

I am currently working on understanding Event Hub along with Azure Function. I have checked out event driven scaling which mentions about the scale controller. But none of the azure documents I referred gave out the logic behind scale controller as…
potterson11
  • 147
  • 7
3
votes
0 answers

How to handle requests larger than 1MB on EventHub

I'm working on a NestJS project that receives data from SAP MII and then send it to EventHub. Unfortunately, EventHub supports a maximum of 1MB (https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-quotas), and in my case, SAP MII sometimes…
Lucas
  • 491
  • 6
  • 21
3
votes
1 answer

How to write custom dataframe to eventhub from ADLS

I would like to write custom data frame to eventhub. val customDf = spark.read.json("path/to/json") EventHub ConnectionString val connectionString = new…
3
votes
1 answer

What are the differences between Azure Web PubSub and Event Grid/Event Hub?

It seems to me that Azure Web PubSub is a combination of both Event Grid and Event Hub using web sockets, instead of HTTP. However, I'm not 100% sure and could not find any articles that have a direct comparison between the 3 services.
TDao
  • 514
  • 4
  • 13
3
votes
3 answers

How to check the event received in Azure Event Hub

Perhaps a rather simple question, yet I couldn't find the answers after doing some searching. After I have send an event to an Azure event Hub (via the Java code), how could I check back in the Azure Portal about the event that I have sent ? (so…
once
  • 536
  • 7
  • 21
3
votes
0 answers

Azure Kafka Bootstrap broker disconnected

I try to create an Apache Kafka connection to Azure event hubs with reactor kafka in a spring boot application. At first I followed the official Azure tutorial to set up Azure event hubs and the spring backend:…
Rooky
  • 810
  • 3
  • 12
  • 20
3
votes
1 answer

Is there a way to store the azure Eventhub checkpoint to a remote bucket such as Google cloud bucket?

I want to store the checkpoints from Azure event hub to a google cloud platform bucket, owing to a use case, but I am unable to find a way to do this. As per my research on eventhub checkpointing, I see that a checkpoint_store object is created…
3
votes
1 answer

Azure Event Hubs Streaming: Does Checkpointing override setStartingPosition?

If we specify the starting position in EventHub conf like so: EventHubsConf(ConnectionStringBuilder(eventHubConnectionString).build) .setStartingPosition(EventPosition.fromStartOfStream) or …