1

How to optimize Application Insight costs for Azure Functions?

I seems to log too much and Application Insight costs are huge. Almost all costs are coming from "messages". Not from metrics.

I'm trying to modify host.json, but in local debugging my new host.json does not provide info I need for debugging.

When I modify logging I cannot see any of these logging in the command line:

   logging.info(f" Calling Activity Function")

I see lot of non relevant noices like (I'm developing Azure Durable Function)

 testhubname-control-03: Starting lease renewal with token 2c336948-1daa-49db-ae7e-9989cc340461

Original (logs too much to Application Insight)

{
  "version": "2.0",
  "logging": {
    "applicationInsights": {
      "samplingSettings": {
        "isEnabled": true,
        "excludedTypes": "Request"
      }
    }
  },
  "extensionBundle": {
    "id": "Microsoft.Azure.Functions.ExtensionBundle",
    "version": "[2.*, 3.0.0)"
  }
}

New (I cannot see logging.info(f" Calling Activity Function") result in command line)

{
  "version": "2.0",
  "logging": {
    "applicationInsights": {
      "samplingSettings": {
        "isEnabled": true,
        "excludedTypes": "Request"
      }
    },
    "logLevel": {
      "default": "Information",
      "Host.Results": "Error",
      "Function": "Error",
      "Host.Aggregator": "Information"
    }
    
  },
  "extensionBundle": {
    "id": "Microsoft.Azure.Functions.ExtensionBundle",
    "version": "[2.*, 3.0.0)"
  }
}

https://learn.microsoft.com/en-us/azure/azure-functions/configure-monitoring?tabs=v2

Also tried to Add LogLevel under applicationInsight settings, but "LogLevel" was not accepted.

{
  "version": "2.0",
  "logging": {
    "applicationInsights": {
      "samplingSettings": {
        "isEnabled": true,
        "excludedTypes": "Request"
      },
      "LogLevel": {
        "Default": "Warning"
      }
    }
  },
  "extensionBundle": {
    "id": "Microsoft.Azure.Functions.ExtensionBundle",
    "version": "[2.*, 3.0.0)"
  }
}

Also tried without success: (Property not allowed error occurs)

{
  "version": "2.0",
  "ApplicationInsights": {
    "InstrumentationKey": "my-instrumentation-key",
    "EnableAdaptiveSampling": false,
    "EnablePerformanceCounterCollectionModule": false
  },
  "Logging": {
    "LogLevel": {
      "Default": "Warning"
    },
    "ApplicationInsights": {
      "LogLevel": {
        "Default": "Information"
      }
    }
  },
  "extensionBundle": {
    "id": "Microsoft.Azure.Functions.ExtensionBundle",
    "version": "[2.*, 3.0.0)"
  }
}
Kenny_I
  • 2,001
  • 5
  • 40
  • 94
  • The loglevel for AI needs to be set at a different level, see https://stackoverflow.com/a/69418266/932728 – Peter Bons Jan 14 '22 at 15:32
  • What is the namespace/class of which has `logging.info(f" Calling Activity Function")` log? Is it part of `Function` namespace? – user1672994 Jan 15 '22 at 16:40
  • @PeterBons I tried to add LogLevel under "applicationInsights", but did not work. See updated questions. – Kenny_I Jan 17 '22 at 08:52
  • @Kenny_I `ApplicationInsights` shoud not be nested under `logging`, it is a seperate sections as seen in https://stackoverflow.com/a/69418266/932728 – Peter Bons Jan 17 '22 at 09:05
  • See updated question. I tried latter host.json version in stackoverflow.com/a/69418266/932728 – Kenny_I Jan 17 '22 at 09:27

1 Answers1

1

One of the workaround is to resolve this issue:

  1. Created Azure Functions (Python Stack) in VS Code.
  2. Added this code logging.info(f" Calling Activity Function") in Activity Function Code like below: enter image description here

By default, this is the host.json code:

"logging": {
"applicationInsights": {
"samplingSettings": {
"isEnabled": true,
"excludedTypes": "Request"
        }
    }
}

When I run this durable function using local storage emulator, it is logging the information level:

enter image description here

If I change the host.json code to include your code given in the question, then I'm unable to get the Calling Activity Function logging information in the terminal:

{
  "version": "2.0",
  "logging": {
    "applicationInsights": {
      "samplingSettings": {
        "isEnabled": true,
        "excludedTypes": "Request"
      }
    },
    "logLevel": {
      "default": "Information",
      "Host.Results": "Error",
      "Function": "Error",
      "Host.Aggregator": "Information"
    }   
  }

enter image description here

Modified the host.json information level logging code attribute values and then run the function, it is logging the information specified in the activity function code as you can see below:

enter image description here

Also, same result can be get when Host.Results value is Error and Function value is Information level in logging code in the host.json:

enter image description here

There are difference in values of logging level attributes defined in the host.json like:

  1. For logs of Host.Results or Function, only log events at Error or a higher level.
  2. For logs of Host.Aggregator, log all generated metrics (Trace).
  3. For all other logs, including user logs, log only Information level and higher events.

Your Code:

"logLevel": {
      "default": "Information",
      "Host.Results": "Error",
      "Function": "Error",
      "Host.Aggregator": "Information"
    }
  • If Host.Results category is set to Error log level, it will only gather host execution telemetry events in the requests table for failed function executions.
  • If Function category is set to Error log level, it will stop gathering function telemetry data related to dependencies, customMetrics, and customEvents for all the functions, preventing to see any of this data in Application Insights. It will only gather traces logged with Error level.

To optimize the cost of Application Insights, you can set the host.json logging levels up to the collected enough data to understand your function behavior using different values set to each attribute defined in the log levels.

There are few techniques provided for controlling the impact of potential costs of monitoring your function app like Sampling, daily cap setting, Pre-aggregate metrics, customizing the logs collection in the Microsoft documentation.

Reference: Log Level Configuration and Potential Costs Impact Techniques in Azure Functions

Updated Answer:

How to configure host.json so that I can set different log levels to AI and the local environment? I need to optimize the number of logs in AI. Could you provide an example?

To minimize the number of logs, you can use the higher logging level of logs in host.json as you can see in the below screenshot:

As you can see the minimization of logs here function information log is not produced and the output log is shown in the browser, only the manual logging is shown in the logs/terminal.

enter image description here