Do PodDisruptionBudgets take priority over HorizontalPodAutoscalers?

Question

If you have a PDB that specifies a higher minAvailable than the minReplicas of a HPA, will the number of pods ever reach the lower minReplicas?

Example configs:

PDB

apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: frontend-pdb
spec:
  minAvailable: 3 # HERE
  selector:
    matchLabels:
      app: frontend

HPA

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: frontend-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: frontend-deployment
  minReplicas: 2 # AND HERE
  maxReplicas: 20
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 65

I image putting the PDB to a % rather than an absolute would solve this potetial conflict, but I am curious how the two play with each other.

Have you seen official docs https://kubernetes.io/docs/concepts/workloads/pods/disruptions/#pod-disruption-budgets? From official documentation "PDBs cannot prevent involuntary disruptions from occurring, but they do count against the budget. Pods which are deleted or unavailable due to a rolling upgrade to an application do count against the disruption budget, but workload resources (such as Deployment and StatefulSet) are not limited by PDBs when doing rolling upgrades. Instead, the handling of failures during application updates is configured in the spec for the specific workload resource." — Malgorzata, Feb 01 '21 at 10:45
@Malgorzata I guess I am confused about this part: "Pods which are deleted or unavailable due to a rolling upgrade to an application do count against the disruption budget, but workload resources (such as Deployment and StatefulSet) are not limited by PDBs when doing rolling upgrades." I am not sure which the HPA rolling update would fall under. — danthegoodman, Feb 01 '21 at 14:12
Personally I am noticing that when I update a deployment, and I have the PDB less than the HPA, the amount of available nodes never goes below the HPA. — danthegoodman, Feb 01 '21 at 17:55

score 2 · Accepted Answer · answered Feb 11 '21 at 14:03

In your case both objects will be created and value minAvailable: 3 defined in PodDisruptionBudget will have higher priority than minReplicas: 2 defined in Deployment. Conditions defined in PDB are more important. In such case conditions for PDB are met but if autoscaler will try to decrease number of replicas it will be blocked because condition in PDB of minAvailable value will be not met. You have wrote that you have noticed that when you update a deployment, and you have the PDB minAvailable less than the minReplicas in HPA, the amount of available pods never goes below the HPA. And it is normal because autoscaler take under consideration PDB and if

minReplicas > minAvailable

so the conditions for PDB are met (minAvailable is critical line so higher value is also proper) so HPA can then take under consideration his conditions and will hit at best minReplicas.

Overall idea of defining Pod Disruption Budget is to make sure minimum replicas are always working.

While node operation, the drain will halt and wait that all PDBs condition are respected when performing pod eviction. If pods that was evicted would cause a PDB to be invalid would wait till the condition would be valid.

NOTE:

A PDB for a deployment with a single replica will most likely cause the node operation to never succeed.

Read more: downscaling-kubernetes, hpa-pdb-kubernetes.

Do PodDisruptionBudgets take priority over HorizontalPodAutoscalers?

1 Answers1