Azure Logs does not have all data

Question

I have an Azure Function and all calls I can see:

but when I go to "Logs" and try the following query:

traces 
| project
    timestamp,
    message,
    operation_Name,
    operation_Id,
    cloud_RoleName
| where cloud_RoleName =~ 'FunctionDeviceManager' and operation_Name =~ 'FunctionAlertServiceCallback'
| order by timestamp desc
| take 2000

I see the following result:

as we can see, many calls (for example, with id: 95ecc6d554d78fa34534813efb82abba, 29b613056e582666c132de6ff73b2c2e, 29b613056e582666c132de6ff73b2c2e and many others, most of them) are not displayed in the result.

What is wrong?

Peter Bons · Accepted Answer · 2022-09-13T09:10:38.860

The invocation log is not based on data in the traces collection. Instead, it is based on request data. You can easily see so by choosing Run query in Application Insights

It runs this query

requests
| project
    timestamp,
    id,
    operation_Name,
    success,
    resultCode,
    duration,
    operation_Id,
    cloud_RoleName,
    invocationId=customDimensions['InvocationId']
| where timestamp > ago(30d)
| where cloud_RoleName =~ 'xxx' and operation_Name =~ 'yyy'
| order by timestamp desc
| take 20

So that explains the difference in the result.

Now, regarding the cause of why the traces collection doesn't always contain data related to the request: per default, all types of telemetry are subject to sampling if not specified in the host.json file, see the docs.

For example, when you create a new http triggered function using Visual Studio 2022 the following host.json is added

{
    "version": "2.0",
    "logging": {
        "applicationInsights": {
            "samplingSettings": {
                "isEnabled": true,
                "excludedTypes": "Request"
            }
        }
    }
}

As you can see request telemetry is excluded from the types of telemetry being sampled. This can cause the issue you are experiencing: the request collection is not sampled, the traces collection is. Hence some data is not in the result list of your query.

score 1 · Answer 2 · answered Sep 10 '22 at 10:03

Most likely this is the effect of sampling. Unless you have tweaked your Function App config in host.json some executions are skipped in log. As per MS documentation:

Application Insights has a sampling feature that can protect you from producing too much telemetry data on completed executions at times of peak load. When the rate of incoming executions exceeds a specified threshold, Application Insights starts to randomly ignore some of the incoming executions. The default setting for maximum number of executions per second is 20 (five in version 1.x). You can configure sampling in host.json. Here's an example:

See also: https://learn.microsoft.com/en-us/azure/azure-monitor/app/sampling

Azure Logs does not have all data

2 Answers2