9

AKS Insights

So, I can plainly see per container stats in the insights tab of AKS. These must come from somewhere, but I can only find per node stats when querying logs/metrics. How can I query this (in order to build a workbook).

neManiac
  • 181
  • 1
  • 7

1 Answers1

8

That data is in the Perf table in the LogManagement section: screenshot of log analytics query showing raw performance data

The documentation page on How to query logs from Azure Monitor for containers has example queries you can start with: screenshot of documentation page showing sample query

Querying this data takes a bit of parsing, because the Computer field always shows the name of the node the data was gathered from, not the pod. In order to get pod/container specific data, you have to look at records with ObjectName == 'K8SContainer' and parse the InstanceName field, which contains the data you need. InstanceName is built like this: /subscriptions/SUBSCRIPTIONID/resourceGroups/RESOURCEGROUPNAME/providers/Microsoft.ContainerService/managedClusters/CLUSTERNAME/PODUID/CONTAINERNAME. Given that data, we can parse out the PodUid and join with KubePodInventory to get the identifying information for that Pod.

Here's an example query:

Perf
| where ObjectName == 'K8SContainer' and TimeGenerated > ago(1m)
| extend PodUid = tostring(split(InstanceName, '/', 9)[0]), Container = tostring(split(InstanceName, '/', 10)[0])
| join kind=leftouter (KubePodInventory | summarize arg_max(TimeGenerated, *) by PodUid) on PodUid
| project TimeGenerated, ClusterName, Namespace, Pod = Name, Container, PodIp, Node = Computer, CounterName, CounterValue

This query produces a result like this, which should contain the data you need: performance query results table

As a side note - the Computer field always shows the node name because that's where the OMS agent is running. It's gathering statistics at the node level, but those stats include the memory and CPU usage for each cgroup, which is the backing CPU/memory isolation and limiting technology behind containers in general, just like how namespaces are used to separate networking, filesystems, and process IDs.

Patrick Healy
  • 166
  • 5
  • Thanks for the response! I did look at this table, but to my understanding - it stores metrics per computer/node not per container (or at the very least - pod). I have a couple of deployments in the same cluster, which means that the above data averages different applications and different instances of the same application, which is not what I want. – neManiac Feb 23 '21 at 09:39
  • 1
    The data in that table is actually both node level (ObjectName == 'K8SNode') and container level (ObjectName == 'K8SContainer'). I've edited my answer to provide a sample query showing how to parse out the container level detail and join with KubePodInventory to get the namespace, pod name, and container name. – Patrick Healy Feb 23 '21 at 13:33
  • The key insight was that I needed to join the Perf table (performance metrics ) with the KubePodInventory (to find out what container the data relates to). Thanks mate! Only one suggestion - make the join a left outer join :) – neManiac Feb 26 '21 at 09:21
  • 1
    Updated - you're correct. My actual production query does have that join type - I just didn't have it handy and recreated this quickly from memory. Good luck! – Patrick Healy Feb 27 '21 at 12:29
  • @PatrickHealy Is there a way to get the limit of each container resource as well? – Vijay Nirmal May 09 '22 at 16:31