Azure Application Insight Custom Telemetry Failures

Question

I am logging the custom telemetry events using Telemetry Client in azure function. It logs the custom event almost every time certain event is triggered but it fails occasionally say once in a thousandth time. I can ascertain that no exception is thrown and event is processed successfully without any error. Here is an example of how I am logging the custom telemetry.

        try
            {
                _telemetryClient.TrackEvent("MyCustomEventName",
                    new KeyValuePair<string, string>("CustomEvent",
                        JsonSerializer.Serialize(customEventObject)));
            }
            catch (Exception e)
            {
                _logger.LogError(e,"Failed to log CustomEventObject");
            }

I am logging the exception if telemetry client fails and logger shows no exception. Is there any reason Telemetry Client could fail to dispatch the event? If yes, how to diagnose and handle it?

I am on .net core 3.1 with Azure Function Runtime 3.1.3.0

Could be caused by [sampling](https://learn.microsoft.com/en-us/azure/azure-monitor/app/sampling) — Peter Bons, Aug 26 '21 at 07:03

score 1 · Accepted Answer · answered Aug 27 '21 at 04:03

You can use the following two ways of sampling to diagnose and handle the failure of dispatching the event:

Adaptive sampling automatically adjusts the volume of telemetry sent from the SDK in your ASP.NET/ASP.NET Core app, and from Azure Functions. This is the default sampling when you use the ASP.NET or ASP.NET Core SDK. Adaptive sampling is currently only available for ASP.NET server-side telemetry, and for Azure Functions.
Fixed-rate sampling reduces the volume of telemetry sent from both your ASP.NET or ASP.NET Core or Java server and from your users' browsers. You set the rate. The client and server will synchronize their sampling so that, in Search, you can navigate between related page views and requests.

There are two AdaptiveSamplingTelemetryProcessor nodes added by default, and one includes the Event type in sampling, while the other excludes the Event type from sampling.

<TelemetryProcessors>
    <Add Type="Microsoft.ApplicationInsights.WindowsServer.TelemetryChannel.AdaptiveSamplingTelemetryProcessor, Microsoft.AI.ServerTelemetryChannel">
        <MaxTelemetryItemsPerSecond>5</MaxTelemetryItemsPerSecond>
        <ExcludedTypes>Event</ExcludedTypes>
    </Add>
    <Add Type="Microsoft.ApplicationInsights.WindowsServer.TelemetryChannel.AdaptiveSamplingTelemetryProcessor, Microsoft.AI.ServerTelemetryChannel">
        <MaxTelemetryItemsPerSecond>5</MaxTelemetryItemsPerSecond>
        <IncludedTypes>Event</IncludedTypes>
    </Add>
</TelemetryProcessors>

You can refer to Types of Sampling, How can I track telemetry that's not automatically collected?, Telemetry sampling without affecting the errors/failures and open GitHub issue at How do you correctly get TelemetryClient dependency injected in ASP.NET Core?

Thanks Deep. Adaptive sampling was real culprit. It worked all fine after disabling it. Appreciated! — Prasad Bhokare, Sep 15 '21 at 20:09
Hello Again, earlier solution worked but now I am back to square one somehow. I have disabled the Adaptive Sampling. Never enabled fixed rate sampling and ingestion is set to 100% meaning no data is sampled there. Also, my subscription shows that I never reached my daily cap of ingestion. Still, lot of events are missing from my telemetry. Do I need to configure anything beyond this? Thanks in advance! — Prasad Bhokare, Sep 27 '21 at 17:13
You can report it to Microsoft via [raise support ticket](https://azure.microsoft.com/en-in/support/create-ticket/), [Twitter @AzureSupport](https://twitter.com/AzureSupport) and also ask the same question on [Microsoft Q&A](https://learn.microsoft.com/en-us/answers/products/) — Ecstasy, Sep 28 '21 at 04:01
You can also open an issue on GitHub: https://github.com/microsoft/ApplicationInsights-dotnet/issues — Ecstasy, Sep 28 '21 at 06:34
Looks like issue is already opened for very long time here. https://github.com/microsoft/ApplicationInsights-dotnet/issues/1808 — Prasad Bhokare, Sep 29 '21 at 12:47
As [cijothomas](https://github.com/microsoft/ApplicationInsights-dotnet/issues/1808#issuecomment-803043385) commented "Please raise Azure functions issue in Azure functions runtime repo, as Functions uses a custom enablement of application insights SDK." You can open an issue on [Azure Function runtime repo](https://github.com/Azure/azure-functions-host/issues) — Ecstasy, Sep 29 '21 at 12:54

Azure Application Insight Custom Telemetry Failures

1 Answers1