7

I want to know the time it takes to start my pod in Kubernetes. My pod has 3 Init Containers and 9 actual Containers. Here's what I have tried so far:

  • Approach 1: Start the pod and monitor until there are 9/9 containers running. The AGE in kubectl describe pod <pod_name> will give an idea for startup time.
  • Approach 2: Refer to the events for the pod using kubectl events or kubectl describe command. The problem is events are not there for all containers. Also, there is no clear way to understand the time taken by pod to be in READY state.

enter image description here

Question: Does Kubernetes provide any way to determine the whole pod startup time?

Ratish Bansal
  • 1,982
  • 1
  • 10
  • 19

4 Answers4

8

As things begin to set up, your pod will progress through a set of PodConditions.

You could use:

kubectl get po <pod_name> -o yaml

to observe the progression through these various conditions:

...
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2020-09-09T08:47:11Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2020-09-09T08:47:12Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2020-09-09T08:47:12Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2020-09-09T08:47:11Z"
    status: "True"
    type: PodScheduled
...

If you want to automatically parse all of this in a shell, I'd recommend yq for the actual YAML parsing.

Jedi
  • 3,088
  • 2
  • 28
  • 47
2

In case this can help anyone out there, here is an example using kubectl, yq and bash

$ kubectl -n ${NAMESPACE} get pods -o yaml | yq e '.items[].status.conditions[] | select('.type' == "PodScheduled" or '.type' == "Ready") | '.lastTransitionTime'' - | xargs -n 2 bash -c 'echo $(( $(date -d "$0" "+%s") - $(date -d "$1" "+%s") ))'
67
70
70
72
...
Bruno Chauvet
  • 196
  • 2
  • 7
1

While there is no exact mechanism for that in Kubernetes you may want to check Kube-state-metrics:

kube-state-metrics is a simple service that listens to the Kubernetes API server and generates metrics about the state of the objects. (See examples in the Metrics section below.) It is not focused on the health of the individual Kubernetes components, but rather on the health of the various objects inside, such as deployments, nodes and pods.

With it you can export object creation time. The metric name in that case would be kube_<OBJECT>_created. This article about metrics in section 6 explain more about kube state metrics and usage of object creation time.

Note that if you are about to measure your pod start time you have assume that all images needed to run that pod are already pre-pulled on the machine. Otherwise your measurement will not be correct as it will include variables that influence Kubernetes performance such as network or image size.

acid_fuji
  • 6,287
  • 7
  • 22
0

Building on Jedi's answer, I wrote the below tool to do this using https://stedolan.github.io/jq/. I run it like this: for pod in $(kubectl get --no-headers=true pods -o name | awk -F "/" '{print $2}'); do ./get-pod-boot-time $pod ; done

#!/usr/bin/env bash

pod="$1"

fail() {
    echo "$@" 1>&2
    exit 1
}

# https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-conditions
get_condition_time() {
    condition="$1"

    iso_time=$(kubectl get pod "$pod" -o json | jq ".status.conditions[] | select(.type == \"$condition\" and .status == \"True\") | .lastTransitionTime" | tr -d '"\n')

    test -n "$iso_time" || fail "Pod $pod is not in $condition yet"

    date -d $iso_time +%s
}

initialized_time=$(get_condition_time PodScheduled)
ready_time=$(get_condition_time Ready)
duration_seconds=$(( $ready_time - $initialized_time ))

echo "It took $duration_seconds seconds for $pod to boot up"
Gabriel Deal
  • 935
  • 1
  • 8
  • 19