3

I have a requirement of sending 1,00,000 batch of records per second. What i got after experimenting is azure event hub has limit of 10,00,000 events in bytes. My each record is of 145 bytes and total records i have to send are 1,00,000 as i already mentioned above. So, mathematically, 145 * 1,00,000 = 14500000 bytes of data i want to send per second.

Can somebody help me with this ?

Now, to send this data i am using azure event hub SDK and using following method i am trying to send data : await eventHubClient.SendAsync(ed); (ed is the object of EventData which has records in bytes UTf 8 format.)

But, when i am running the code and executing the above line (SendAsync () ) i am getting the following error :

$exception  
{Microsoft.Azure.EventHubs.MessageSizeExceededException: The received message (delivery-id:0, size:3922220 bytes) exceeds the limit (1046528 bytes) currently allowed on the link.
Ivan Glasenberg
  • 29,865
  • 2
  • 44
  • 60
Tanmay Sharma
  • 71
  • 1
  • 5

1 Answers1

3

You can not change it. The max size 1M is by design, you can consider reduce the size of your message.

You can take a look at reference here and here.

Ivan Glasenberg
  • 29,865
  • 2
  • 44
  • 60
  • 1
    1M in bytes correct ? i have also researched on Microsoft docs and found that Batch size can increase up to 4MB . check out this --> https://learn.microsoft.com/en-us/dotnet/api/microsoft.servicebus.messaging.eventdatabatch.tryadd?view=azure-dotnet#remarks – Tanmay Sharma Apr 26 '19 at 10:52
  • @TanmaySharma, we're now using this nuget package [Microsoft.Azure.EventHubs](https://www.nuget.org/packages/Microsoft.Azure.EventHubs/). so for tryAdd, I'm looking at this [one](https://learn.microsoft.com/en-us/dotnet/api/microsoft.azure.eventhubs.eventdatabatch.tryadd?view=azure-dotnet#remarks) – Ivan Glasenberg Apr 27 '19 at 00:58
  • Thanks for clearing my confusion. So, what can be the solution to above question i.e., to send 14500000 bytes of data per second ? . Should i go for batching them and then use EventHubClient.SendBatch(IEnumerable) Method or try to increase throughput units and Partitions of Event Hub Namespace ? – Tanmay Sharma May 03 '19 at 12:55
  • You can increase the size by increasing Throughput Unit (TU). – Ashish Deora Apr 08 '22 at 13:42