Azure Application Insights - Not recording all requests on high traffic situations

Question

It seems "Azure Application Insights" not recording all the requests on high traffic environment.

For example when we testing the .Net Core2.1 Web API app which is deployed on "Azure Service Fabric" with 10,000 requests for the period of 30 mins, can able to retrieve details of all the requests from "Azure Application Insights" via KQL using datetime stamp as filter, no problem.

When we increase the load to 100,000 requests for the period of 30 mins, only around 5-10% of the requests get recorded on "Azure Application Insights".

Why "Azure Application Insights" misses ingest/recording on high traffic environment which serves approximately 60 requests/second ?

Any extra configuration required? (or) Below one line code to ingest details of requests served by Azure Service Fabric service, is not enough? please clarify

SDK used,

Code used on Azure Service Fabric for ingestion

  return new WebHostBuilder().UseHttpSys()
                                .ConfigureServices(
                                    services => services
                                        .AddSingleton<StatelessServiceContext>(serviceContext)
                                        .AddSingleton<ServiceFabricAppContext>(new ServiceFabricAppContext(){
                                                NodeName = serviceContext.NodeContext.NodeName,
                                                ServiceHostIP=serviceContext.NodeContext.IPAddressOrFQDN,
                                                ServiceHostPort=FabricRuntime.GetActivationContext().GetEndpoints()[0].Port
                                        } )
                                            .AddSingleton<ITelemetryInitializer>((serviceProvider) => FabricTelemetryInitializerExtension.CreateFabricTelemetryInitializer(serviceContext))) // Azure Service Fabric Telemetry Initializer
                                .UseContentRoot(Directory.GetCurrentDirectory())
                                   .UseApplicationInsights()
                                .UseStartup<Startup>()
                                .UseEnvironment(environment)
                                .UseServiceFabricIntegration(listener, ServiceFabricIntegrationOptions.None)
                                .UseUrls(url)
                                .Build();

Sample query with item count

Sampling not enabled from Azure portal

It might be due to sampling, see https://learn.microsoft.com/en-us/azure/azure-monitor/app/sampling — Peter Bons, Oct 23 '20 at 11:29
But I haven't mentioned/configured sampling in my application. All I have done is included the one & only SDK (Microsoft.ApplicationInsights.ServiceFabric.Native v2.1.1), then one line of code(.AddSingleton((serviceProvider) => FabricTelemetryInitializerExtension.CreateFabricTelemetryInitializer(serviceContext)). Here where comes sampling? is sampling default behavior? Also sampling didn't come in to effect when I make 10,000 requests for the period of 30 mins.Only when I increase count to 100,000 requests for the period of 30 mins then record missing happening on Azure Application Insights — 191180rk, Oct 23 '20 at 12:13
From the docs: Adaptive sampling is enabled by default in all the latest versions of the Application Insights ASP.NET and ASP.NET Core Software Development Kits (SDKs) . Same applies to service fabric I think. On the page you can find info on how to detect if sampling is in use and how to disable/modify it. — Peter Bons, Oct 23 '20 at 12:29
10.000 requests in 30 sec. is about 5 rps. that probably falls within the range before sampling kicks in. — Peter Bons, Oct 23 '20 at 12:49
To check whether sampling is active - please check "itemCount" property in Kusto. If it is more than 1 then it means data is being sampled. — ZakiMa, Oct 23 '20 at 19:06
@ZakiMa, in my case "itemCount" property in Kusto.equals to 1, attached the query screenshot. Is sampling active or not?? — 191180rk, Oct 24 '20 at 11:19
It means sampling is not active. Overall AI SDK supports way higher throughput than 5 RPS / instance. It is possible to send 10k RPS without issues. One note here is that this is very old AI SDK (2.5 years old). On top of it there is 2.5.1 released month later. Usually it indicates that 2.5.0 has some issues. I'm not familiar with Service Fabric AI SDK but it doesn't seem to reference recent version. Is it possible to update AI SDK underneath to 2.15 manually? Is it possible to get repro outside of Service Fabric environment? — ZakiMa, Oct 25 '20 at 08:44
@ZakiMa, default adaptive sampling has been applied in my Application Insights Instance, attached the updated query screenshot. Now would like to know, how to disable default sampling in .Net core 2.1 web api application? also how to enable performance counter metric like CPU, memory etc? — 191180rk, Oct 25 '20 at 13:56
Have you checked if sampling is enabled from the app insight resource in the portal or not? — Hassan Raza, Oct 25 '20 at 18:30
How to check if sampling is enabled from the app insight resource in the portal ? — 191180rk, Oct 25 '20 at 18:35
@HassanRaza, Update the query with screenshot to confirm sampling not enabled on the app insight resource via the portal — 191180rk, Oct 25 '20 at 18:41
Please take a look at this link here for sampling on ingestion level from the portal itself, https://learn.microsoft.com/en-us/azure/azure-monitor/app/sampling#ingestion-sampling — Hassan Raza, Oct 25 '20 at 18:44
thanks @HassanRaza, attached screenshot of my app insight resource. Sampling is not appiled via portal explicitly but default sampling of .net core SDK comes to effect in my case. How to disable it in Service Fabric stateless service? — 191180rk, Oct 25 '20 at 18:49

score 1 · Accepted Answer · answered Nov 02 '20 at 07:40

You can disable adaptive sampling from code. Note instead of using deprecated .UseApplicationInsights() (removed that below), I am using .AddApplicationInsightsTelemetry below.

  return new WebHostBuilder().UseHttpSys()
                                .ConfigureServices(
                                    services => services
                                        .AddSingleton<StatelessServiceContext>(serviceContext)
                                        .AddSingleton<ServiceFabricAppContext>(new ServiceFabricAppContext(){
                                                NodeName = serviceContext.NodeContext.NodeName,
                                                ServiceHostIP=serviceContext.NodeContext.IPAddressOrFQDN,
                                                ServiceHostPort=FabricRuntime.GetActivationContext().GetEndpoints()[0].Port
                                        } )
                                       .AddSingleton<ITelemetryInitializer>((serviceProvider) => FabricTelemetryInitializerExtension.CreateFabricTelemetryInitializer(serviceContext)) // Azure Service Fabric Telemetry Initializer
                                       .AddApplicationInsightsTelemetry(o => 
                                            {
                                                o.EnableAdaptiveSampling = false; // disabling adaptive sampling
                                            }))
                                .UseContentRoot(Directory.GetCurrentDirectory())
                                .UseStartup<Startup>()
                                .UseEnvironment(environment)
                                .UseServiceFabricIntegration(listener, ServiceFabricIntegrationOptions.None)
                                .UseUrls(url)
                                .Build();

NOTE: I had to add nuget Microsoft.ApplicationInsights.AspNetCore

CAUTION: Disabling sampling might result throttling and too much network traffic in case of very high volume scenario. So you might want to look at Fixed rate sampling instead of adaptive. Refer Sampling in Application Insights.

Azure Application Insights - Not recording all requests on high traffic situations

1 Answers1