AKS Container Insights: How to list not ready pods?

Question

I'm using Azure Container Insights for an AKS cluster and want to filter some logs using Log Analytics and Kusto Query Language. I do it to provide a convenient dashboard and alerts.

What I'm trying to achieve is list only not ready pods. Listing the ones not Running is not enough. This can be easily filtered using kubectl e.g. following this post How to get list of pods which are "ready"? However this data is not avaiable when querying in Log analytics with Kusto as the containerStatuses seems to be only a string

It should be somehow possible because Container Insights allow this filtering in Metrics section. However it's not fully satisfying because with metrics my filtering capabilities are much smaller.

score 1 · Answer 1 · answered May 31 '21 at 14:22

1

You can do it for pods as below for last 1h.

let endDateTime = now();
let startDateTime = ago(1h);
 
KubePodInventory
| where TimeGenerated < endDateTime
| where TimeGenerated >= startDateTime
| where PodStatus != "Running"
| distinct Computer, PodUid, TimeGenerated, PodStatus

answered May 31 '21 at 14:22

efdestegul

617
3
6

Thank you efdestegul, it was a helpful answer. As mentioned in the question, "Running" is not enough as I needed to know which pods are not ready. However your answer led me to a solution I actually implemented and it is possible. I'm sharing my solution in another answer. – Tomasz Chudzik Jun 09 '21 at 12:39
When i run a query in a log analytics workspace logs it shows ``` 'where' operator: Failed to resolve column or scalar expression named 'endDateTime' If issue persists, please open a support ticket. Request id: 5b8c50ce-1188-4324-8770-c5c591244799 ``` Can you give me modified query – jayaprakash R Mar 21 '22 at 11:48

score 0 · Accepted Answer · answered Jun 09 '21 at 12:47

The efdestegul's answer was only listing not "Running" pods and I was looking for not ready ones. However this answer led me to a query which I actually needed and thank you for that. Maybe this will help others.

let timeGrain=1m;

KubePodInventory
// | where Namespace in ('my-namespace-1', 'my-namespace-2')
| summarize countif(ContainerStatus == 'waiting') by bin(TimeGenerated,timeGrain)
| order by countif_ desc
| render timechart

With this query I'm able to render a chart that displays all not ready pods in time. And actually in a very useful way, only the pods that were not ready for more than expected and they needed to be restarted. You can always filter your results for any namespaces you need.

When i run this query in log analytics workspace it shows as '''summarize' operator: Failed to resolve scalar expression named 'timeGrain' If issue persists, please open a support ticket. Request id: 10e034f6-2612-4329-8c0f-fda55a91ef4'' — jayaprakash R, Mar 21 '22 at 11:50

AKS Container Insights: How to list not ready pods?

2 Answers2