2

We are using stream ingestion from Event Hubs to Azure Data Explorer. The Documentation states the following:

The streaming ingestion operation completes in under 10 seconds, and your data is immediately available for query after completion.

I am also aware of the limitations such as

Streaming ingestion performance and capacity scales with increased VM and cluster sizes. The number of concurrent ingestion requests is limited to six per core. For example, for 16 core SKUs, such as D14 and L16, the maximal supported load is 96 concurrent ingestion requests. For two core SKUs, such as D11, the maximal supported load is 12 concurrent ingestion requests.

But we are currently experiencing ingestion latency of 5 minutes (as shown on the Azure Metrics) and see that data is actually available for querying 10 minutes after ingestion.

Our Dev Environment is the cheapest SKU Dev(No SLA)_Standard_D11_v2 but given that we only ingest ~5000 Events per day (per metric "Events Received") in this environment this latency is very high and not usable in the streaming scenario where we need to have the data available < 1 minute for queries.

Is this the latency we have to expect from the Dev Environment or are the any tweaks we can apply in order to achieve lower latency also in those environments?

How will latency behave with a production environment like Standard_D12_v2? Do we have to expect those high numbers there as well or is there a fundamental difference in behavior between Dev/test and Production Environments in this concern?

marc_s
  • 732,580
  • 175
  • 1,330
  • 1,459
Markus S.
  • 2,602
  • 13
  • 44

1 Answers1

4

Did you follow the two steps needed to enable the streaming ingestion for the specific table, i.e. enabling streaming ingestion on the cluster and on the table?

In general, this is not expected, the Dev/Test cluster should exhibit the same behavior as the production cluster with the expected limitations around the size and scale of the operations, if you test it with a few events and see the same latency it means that something is wrong.

If you did follow these steps, and it still does not work please open a support ticket.

Avnera
  • 7,088
  • 9
  • 14
  • 1
    Please also have a look at setting the https://learn.microsoft.com/en-us/azure/data-explorer/kusto/management/batchingpolicy for your tables/databases. It affects the internal batching performed by the ingestion service when streaming avenue is not available. You can set the batching policy as low as 10 seconds. This policy takes 5-10 minutes to take effect. – Vladik Branevich Jun 15 '21 at 09:30
  • I did enable streaming on the cluster but not explicitly on the table. And we did not configure an IngestionBatching Policy. So this means, if we don't specify it and have a low volume of data, ADX waits for 5 minutes by default until it performs the writes? – Markus S. Jun 15 '21 at 10:34
  • We just tested it. Streaming was only activated on the cluster but not on the table itself. Just running .alter table policy streamingingestion enable was sufficient to fix the issue. Thank you for pointing this out. – Markus S. Jun 15 '21 at 12:32
  • Thanks Markus for the confirmation. This is correct, the default batching policy for the queued ingestion is five minutes. – Avnera Jun 15 '21 at 20:18