Questions tagged [azure-eventhub]

Microsoft Azure EventHubs is a highly-scalable high-throughput low-latency durable event ingestion system in Azure cloud. EventHubs serves as an event buffer for applications to process events with variable load profiles. Microsoft Azure Event Hubs is comparable to Apache Kafka or AWS Kinesis. This is a fully-managed service-offering built & run by the teams at Microsoft.

Microsoft Azure EventHubs is a highly scalable high throughput low latency durable event ingestion system in Azure cloud. EventHubs serves as an event buffer for applications to process events with variable load profiles.

Stream millions of events per second into multiple applications

Event Hubs is a highly scalable publish-subscribe ingestor that can intake millions of events per second so that you can process and analyze the massive amounts of data produced by your connected devices and applications. Once collected into Event Hubs you can transform and store data using any real-time analytics provider or with batching/storage adapters.

Enable applications to process events with variable load profiles

Today’s connected world is defined by big data. Big data originates from many sources such as connected cars and thermostats that produce telemetry data every few minutes, application performance counters that generate events every second or mobile apps that capture telemetry for every user’s individual action. In this connected world, you need a managed ingestor service capable of elastic scale in order to accommodate variable load profiles as well as the spikes caused by intermittent connectivity.

Connect millions of devices across platforms

The rapid proliferation of connected devices raises challenges due to the variety of platforms and protocols involved. Connecting these disparate data sources while handling the scale of the aggregate stream is a significant challenge. Event Hubs provide easy provisioning of capacity to ingest events from millions of devices while preserving event order on a per device basis. Support for AMQP and HTTP allow many platforms to work with Event Hubs. Native client libraries also exist for popular platforms.

1941 questions
3
votes
2 answers

Azure Kubernetes Container Logs to Event Hub

How to logs (stdout / stderr) from all container pods azure Kubernetes to the event hub. I can able to see all logs by Log Analytics workspaces >> Logs using an Azure query language. I want to send all logs to the event hub. Can anyone suggest on…
3
votes
0 answers

Spark Kafka - Cannot fetch record for offset in 120000 milliseconds

I'm using Spark to read from Kafka topic. This is my code: val df = spark.readStream.format("kafka"). option("kafka.bootstrap.servers", "myendpoint.servicebus.windows.net:9093"). option("kafka.security.protocol", "SASL_SSL"). …
xzk
  • 827
  • 2
  • 18
  • 43
3
votes
1 answer

Using Azure Function Deployment Slots with EventHub

I have an Azure function app that processes data from an EventHub. My current CI/CD process uses deployment slots to deploy: Deploy function to staging slot Validate staging slot with a smoke test Swap staging with production The CI/CD flow itself…
Tyler Murry
  • 2,705
  • 3
  • 29
  • 46
3
votes
1 answer

Databricks Connect with Azure Event Hubs

I'm facing issues while trying to run some Python code on Databricks using databricks-connect and depending on a Maven installed extension (in this case com.microsoft.azure:azure-eventhubs-spark_2.11:2.3.17 found on Databricks official documentation…
dernat71
  • 365
  • 4
  • 16
3
votes
2 answers

Is it possible to execute Event Hub triggered Azure Functions locally without using an actual Event Hub?

I just want to ask if it is possible to execute an Azure Function (Event Hub trigger) purely on my local machine without depending on any Azure Event Hubs resource? I've been following Microsoft's process in developing Azure Functions locally (Link)…
TheNinong
  • 48
  • 1
  • 4
3
votes
1 answer

Is there a way to get the headers data of events (EventHub) using @EventHubTrigger of Azure Functions in Java?

I have an Azure Function implemented with azure-functions-java-library that receives events from one EventHub and I'm using @EventHubTrigger, the problem is that I need the header data from an event but I don't see any way to get this, I have…
3
votes
0 answers

How to assure reliability of log-to-eventhub policies?

Problem / Background For logging every request and response out of API Management, I use a log-to-eventhub policy. My goal is to assure that all logged data of every single request / response is transmitted to Event Hubs. Example Code for…
3
votes
1 answer

kafkacat for event hub

hi I tried to check connection to eventhub using kafkacat in one of my VM in azure I gave the following parametrs(fill my hab name and all): kafkacat \ -b .servicebus.windows.net:9092 \ -X security.protocol=sasl_ssl \ -X…
NoamiA
  • 521
  • 4
  • 19
3
votes
1 answer

How to return Azure Eventhub primary and secondary connection keys as ARM template output?

I have prepared an ARM template for deploying an Azure Eventhub instance and wonder how to access the both connection keys for returning them as output? I would like to return a string in the…
Alexander Farber
  • 21,519
  • 75
  • 241
  • 416
3
votes
0 answers

Stream Azure EventHub data to ADLS/Blob using Spark Structured Streaming on a daily basis

I've a requirement to stream events from Azure EventHub(EH) to blob on a daily basis. I've used Spark structured streaming with Trigger.once() option along with spark check-pointing to achieve this, this work absolutely fine. But what i need further…
3
votes
1 answer

What are the specific difference between event hubs and Hdinsight in azure for Kafka?

Im working on an application where I need to choose between Hdinsight and event hubs. I have completely searched the internet but could not find any specific comparison between them. I would like to know it from some experienced azure users. P.S…
Anonymous
  • 151
  • 1
  • 1
  • 6
3
votes
0 answers

Outgoing messages are more than incoming messages in Event Hub when using EventProcessorHost

I am using Azure Event hub. Outgoing messages from Event hub are processed by application using EventProcessorHost. Event hub has one consumer group i.e. $Default. I am seeing outgoing messages are more than incoming messages. Not sure why outgoing…
Amit Agrawal
  • 51
  • 2
  • 16
3
votes
2 answers

Create Event Grid Subscription with Event Hub Topic type in ARM

I would like to deploy the same architecture pattern like this one using ARM templates. I have achieved almost the whole model, but I have a challenge connecting Event Hub with Event Grid. In the 4th point you can see how to create a subscription in…
3
votes
1 answer

Azure Event Hub diagnostic set up for User Errors

I am running a set up, where Stream Analytics job is reading from Azure Event Hub. Event Hub is contained within a namespace and there is consumer group created for SA specifically. Recently I noticed User Error on Event Hub metrics. Event Hub…
3
votes
3 answers

Directing messages to consumers

My client is attempting to send messages to the receiver. However I noticed that the receiver sometimes does not receive all the messages sent by the client thus missing a few messages (not sure where the problem is ? Client or the receiver). Any…
MistyD
  • 16,373
  • 40
  • 138
  • 240