Questions tagged [azure-eventhub]

Microsoft Azure EventHubs is a highly-scalable high-throughput low-latency durable event ingestion system in Azure cloud. EventHubs serves as an event buffer for applications to process events with variable load profiles. Microsoft Azure Event Hubs is comparable to Apache Kafka or AWS Kinesis. This is a fully-managed service-offering built & run by the teams at Microsoft.

Microsoft Azure EventHubs is a highly scalable high throughput low latency durable event ingestion system in Azure cloud. EventHubs serves as an event buffer for applications to process events with variable load profiles.

Stream millions of events per second into multiple applications

Event Hubs is a highly scalable publish-subscribe ingestor that can intake millions of events per second so that you can process and analyze the massive amounts of data produced by your connected devices and applications. Once collected into Event Hubs you can transform and store data using any real-time analytics provider or with batching/storage adapters.

Enable applications to process events with variable load profiles

Today’s connected world is defined by big data. Big data originates from many sources such as connected cars and thermostats that produce telemetry data every few minutes, application performance counters that generate events every second or mobile apps that capture telemetry for every user’s individual action. In this connected world, you need a managed ingestor service capable of elastic scale in order to accommodate variable load profiles as well as the spikes caused by intermittent connectivity.

Connect millions of devices across platforms

The rapid proliferation of connected devices raises challenges due to the variety of platforms and protocols involved. Connecting these disparate data sources while handling the scale of the aggregate stream is a significant challenge. Event Hubs provide easy provisioning of capacity to ingest events from millions of devices while preserving event order on a per device basis. Support for AMQP and HTTP allow many platforms to work with Event Hubs. Native client libraries also exist for popular platforms.

1941 questions
6
votes
1 answer

About checkpoint strategy in event hub processor

I use event hubs processor host to receive and process the events from event hubs. For better performance, I call checkpoint every 3 minutes instead of every time when receiving the events: public async Task ProcessEventAsync(context, messages) { …
Youxu
  • 1,050
  • 1
  • 9
  • 34
6
votes
1 answer

Azure Functions - Event Hub not triggering Functions

I have an Azure infrastructure: 2 HTTP Functions -> Event Hub -> 2 Functions -> Table Storage (so two http functions sending messages to event hub, and two functions triggered by messages in Event Hub, one of them saving message in table…
6
votes
1 answer

Azure C# EventHubClient mock for unit test

I am writing an event publisher for our application which internally uses Azure C# EventHubClient. I want to unit test that my event gets translated correctly into EventData object (Properties + Body) as well as some other features. Long story…
Richard Sirny
  • 61
  • 1
  • 4
6
votes
2 answers

Reading Event Hub Archive File in C#

Is there any sample code in C# for reading the Azure Event Hub Archive files (Avro format)? I am trying to use the Microsoft.Hadoop.Avro library. I dumped the schema out using a java avro tool which produces this: { …
Darren Ford
  • 251
  • 3
  • 13
6
votes
1 answer

Azure Stream Analytics is too slow - also time values are irrelavant

We want to migrate our dedicated servers to Azure platform for scaling easy and investigated a lot of Azure services for our needs. So one of the Azure service that we want to use is Azure Stream Analytics (ASA). We've added some Azure Platforms…
6
votes
2 answers

Exceeds the limit (262144 bytes) currently allowed on the link Exception on SendBatch() method of Azure service bus SDK

We are using SendBatch() method of Azure service bus SDK 3.1.7 to send data to Azure Event Hub. Following is the code snippet : foreach (var packet in transformedPackets) { EventData eventData = new…
Shallu
  • 127
  • 1
  • 5
6
votes
3 answers

How to set the EventProcessorHost to read events from now on (UTC)?

We are using the EventProcessorHost to receive events from Azure EventHubs. I've been unsuccessfully trying to configure it (through the EventProcessorOptions.InitialOffsetProvider) to read events from UTC now on but it always reads from the start…
Jose Parra
  • 877
  • 9
  • 23
6
votes
2 answers

High throughput send to EventHubs resulting into MessagingException / TimeoutException / Server was unable to process the request errors

We are experiencing lots of these exceptions sending events to EventHubs during peak traffic: "Failed to send event to EventHub. Exception : Microsoft.ServiceBus.Messaging.MessagingException: The server was unable to process the request; please…
Jose Parra
  • 877
  • 9
  • 23
6
votes
2 answers

How to get events count from Microsoft Azure EventHub?

I want to get events count from Microsoft Azure EventHub. I can use EventHubReceiver.Receive(maxcount) but it is slow on big number of big events. There is NamespaceManager.GetEventHubPartition(..).EndSequenceNumber property that seems to be doing…
Val
  • 61
  • 1
  • 3
5
votes
2 answers

Delta Live Tables with EventHub

I am trying to create streaming from eventhub using delta live tables, but I am having trouble installing the library . Is it possible to install maven library using Delta Live tables using sh /pip? I would like to…
repcak
  • 113
  • 8
5
votes
2 answers

EventHubProducerClient unit testing in c#

I find that EventHubProducerClient is not unit test friendly (not mockable) in c# and how do you go about testing it, like SendEventsAsync method?
user1229441
  • 344
  • 4
  • 11
5
votes
2 answers

Azure eventhub : Offset vs Sequence number

I see this question being asked on a lot of forums but none of them are solving my confusion. This documentation seems to suggest that both offset and sequence number are unique within a…
JavCray
  • 77
  • 1
  • 7
5
votes
1 answer

Azure functions with Event Hub trigger writes duplicate messages

I have an Azure function with an Event Hub trigger. This hub receives messages from devices and stores them in Blob. Recently, I noticed that duplicate messages were stored in blob. Files in blob store are order by last modified date and if you…
MAK
  • 1,250
  • 21
  • 50
5
votes
0 answers

.Net Core Event Hub client producer code executed without exception, but messages were not delivered

I have started work with EventHub recently and meet the problem: code of message producing executes without any exception, however message wasn't published to EventHub. I'm verifying that message wasn't published via simple consumer in console…
WhiteRom
  • 73
  • 6
5
votes
4 answers

Could not instantiate EventHubSourceProvider for Azure Databricks

Using the steps documented in structured streaming pyspark, I'm unable to create a dataframe in pyspark from the Azure Event Hub I have set up in order to read the stream data. Error message is: java.util.ServiceConfigurationError:…
KMM
  • 135
  • 1
  • 8