Multi-Replica Deployments Should Have A PodDisruptionBudget

More Info:

Advisory: define a PodDisruptionBudget for each multi-replica Deployment so node drains and rollouts keep a minimum number of pods available.

Risk Level

Informational

Address

Security

Compliance Standards

• Cloudanix Best Practice

Triage and Remediation

Remediation

Remediation

Manual Steps

1. On any machine with kubectl access, list non-compliant multi‑replica Deployments so you know what needs a PodDisruptionBudget:

   { kubectl get poddisruptionbudgets --all-namespaces -o json \
     kubectl get deployments --all-namespaces -o json; } | jq -rs '
     .[0] as $pdbs | .[1] |
     [ .items[]
     | select((.spec.replicas // 1) > 1)
     | .metadata as $m
     | (.spec.template.metadata.labels // {}) as $podLabels
     | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
     | ([ $pdbs.items[]
          | select(.metadata.namespace == $m.namespace)
          | select((.spec.selector.matchLabels // {}) | length > 0)
          | select([ (.spec.selector.matchLabels | to_entries)[]
                     | $podLabels[.key] == .value ] | all)
        ] | length) as $count
     | select($count == 0)
     | {namespace: $m.namespace, name: $m.name, labels: $podLabels, replicas: .spec.replicas}
     ]'
   

2. For one non‑compliant Deployment, capture its pod template labels and replica count (needed for the PDB selector and minAvailable / maxUnavailable):

   kubectl get deployment <deployment-name> -n <namespace> -o json \
     | jq '.spec.replicas as $r
           | {replicas: $r,
              labels: .spec.template.metadata.labels}'
   

3. On any machine with kubectl access, create a PodDisruptionBudget manifest file for that Deployment, using the pod template labels in spec.selector.matchLabels. For example, save as pdb-<deployment-name>.yaml:

   apiVersion: policy/v1
   kind: PodDisruptionBudget
   metadata:
     name: <deployment-name>-pdb
     namespace: <namespace>
   spec:
     minAvailable: 1
     selector:
       matchLabels:
         app: <value-from-deployment-labels>
         # include any other identifying labels used on the pod template
   

   Adjust minAvailable (or alternatively use maxUnavailable) according to how many replicas must remain available during disruptions.

4. Apply the PodDisruptionBudget:

   kubectl apply -f pdb-<deployment-name>.yaml
   

5. Repeat steps 2–4 for each remaining non‑compliant multi‑replica Deployment identified in step 1, ensuring each PDB’s spec.selector.matchLabels exactly matches the Deployment’s pod template labels you want protected.

6. Verify all multi‑replica Deployments now have at least one matching PodDisruptionBudget by rerunning the audit command on any machine with kubectl access:

   { kubectl get poddisruptionbudgets --all-namespaces -o json \
     kubectl get deployments --all-namespaces -o json; } | jq -rs '
     .[0] as $pdbs | .[1] |
     [ .items[]
     | select((.spec.replicas // 1) > 1)
     | .metadata as $m
     | (.spec.template.metadata.labels // {}) as $podLabels
     | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
     | ([ $pdbs.items[]
          | select(.metadata.namespace == $m.namespace)
          | select((.spec.selector.matchLabels // {}) | length > 0)
          | select([ (.spec.selector.matchLabels | to_entries)[]
                     | $podLabels[.key] == .value ] | all)
        ] | length) as $count
     | "kind=Deployment ns=\($m.namespace) name=\($m.name) replicas=\(.spec.replicas) podDisruptionBudgets=\($count) is_compliant=\(if $count > 0 then "true" else "false" end)"
     ]'
   

   Confirm that every listed multi‑replica Deployment now shows is_compliant=true and podDisruptionBudgets greater than 0.

Using kubectl

# 1) Identify non-compliant multi-replica Deployments
# Run on: any machine with kubectl access
kubectl get deployments --all-namespaces -o json | jq -r '
  .items[]
  | select((.spec.replicas // 1) > 1)
  | .metadata as $m
  | (.spec.template.metadata.labels // {}) as $podLabels
  | "NAMESPACE=\($m.namespace) NAME=\($m.name) LABELS=\($podLabels | to_entries | map("\(.key)=\(.value)") | join(","))"
'

# 2) For each multi-replica Deployment, create a matching PodDisruptionBudget manifest.
# Example: Deployment "web" in namespace "prod" with pod label app=web

cat <<'EOF' > pdb-web-prod.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web-pdb
  namespace: prod
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: web
EOF

# 3) Apply the PDB
kubectl apply -f pdb-web-prod.yaml

Repeat step 2–3 for each multi-replica Deployment, ensuring:

• metadata.namespace matches the Deployment’s namespace.
• spec.selector.matchLabels exactly matches the labels on .spec.template.metadata.labels of the Deployment’s pods (at least one stable, unique label such as app: <name>).

You can bulk-generate skeleton manifests to edit by hand, for example:

# Generate minimal PDB stubs for all multi-replica Deployments (edit before applying!)
kubectl get deployments --all-namespaces -o json | jq -r '
  .items[]
  | select((.spec.replicas // 1) > 1)
  | .metadata as $m
  | (.spec.template.metadata.labels // {}) as $podLabels
  | select(($podLabels | length) > 0)
  | "---"
  + "\napiVersion: policy/v1"
  + "\nkind: PodDisruptionBudget"
  + "\nmetadata:"
  + "\n  name: \($m.name)-pdb"
  + "\n  namespace: \($m.namespace)"
  + "\nspec:"
  + "\n  minAvailable: 1"
  + "\n  selector:"
  + "\n    matchLabels:"
  + (
      $podLabels
      | to_entries
      | map("\n      \(.key): \(.value)")
      | join("")
    )
' > generated-pdbs.yaml

# Review and edit generated-pdbs.yaml carefully, then:
kubectl apply -f generated-pdbs.yaml

# Verification: rerun the benchmark-style check
# Run on: any machine with kubectl access
{ kubectl get poddisruptionbudgets --all-namespaces -o json \
  kubectl get deployments --all-namespaces -o json; } | jq -rs '
  .[0] as $pdbs | .[1] |
  [ .items[]
  | select((.spec.replicas // 1) > 1)
  | .metadata as $m
  | (.spec.template.metadata.labels // {}) as $podLabels
  | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
  | ([ $pdbs.items[]
       | select(.metadata.namespace == $m.namespace)
       | select((.spec.selector.matchLabels // {}) | length > 0)
       | select([ (.spec.selector.matchLabels | to_entries)[]
                  | $podLabels[.key] == .value ] | all)
     ] | length) as $count
  | "kind=Deployment ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=apps/v1"
    + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
    + (if $labels == "" then "" else " labels=\($labels)" end)
    + " replicas=\(.spec.replicas) podDisruptionBudgets=\($count)"
    + " is_compliant=\(if $count > 0 then "true" else "false" end)"
  ] as $rows
  | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end'

Automation

#!/usr/bin/env bash
# Automation: Ensure each multi-replica Deployment in the cluster has a matching PodDisruptionBudget
# Applies to: any machine with kubectl access to the EKS cluster
#
# Requirements:
#   - kubectl configured to talk to the EKS cluster
#   - jq available locally
#
# Behavior:
#   - For every Deployment with replicas > 1:
#       * If at least one PDB in the same namespace has a selector.matchLabels
#         that is a subset of the pod template labels, it is considered covered
#       * Otherwise, a PDB is created with selector.matchLabels equal to the pod
#         template labels, and minAvailable set to floor(replicas/2)
#   - Safe to re-run: existing matching PDBs are left unchanged

set -euo pipefail

# Optional: narrow to specific namespaces (comma-separated list). Empty = all namespaces.
NAMESPACE_FILTER="${NAMESPACE_FILTER:-}"

# Helper: check if namespace is allowed by NAMESPACE_FILTER
ns_allowed() {
  local ns="$1"
  if [[ -z "${NAMESPACE_FILTER}" ]]; then
    return 0
  fi
  IFS=',' read -r -a ns_list <<< "${NAMESPACE_FILTER}"
  for n in "${ns_list[@]}"; do
    if [[ "${n}" == "${ns}" ]]; then
      return 0
    fi
  done
  return 1
}

# Fetch all Deployments and PDBs in one shot
echo "Fetching Deployments and PodDisruptionBudgets..."
DEPLOY_JSON="$(kubectl get deployments --all-namespaces -o json)"
PDB_JSON="$(kubectl get poddisruptionbudgets --all-namespaces -o json)"

# Process each multi-replica Deployment
echo "${DEPLOY_JSON}" | jq -r '
  .items[]
  | select((.spec.replicas // 1) > 1)
  | {
      namespace: .metadata.namespace,
      name: .metadata.name,
      replicas: (.spec.replicas // 1),
      podLabels: (.spec.template.metadata.labels // {}),
      deploymentLabels: (.metadata.labels // {})
    }
  | @base64
' | while read -r DEP_B64; do
  _jq() { echo "${DEP_B64}" | base64 --decode | jq -r "${1}"; }

  ns=$(_jq '.namespace')
  name=$(_jq '.name')
  replicas=$(_jq '.replicas')
  # podLabels may be empty; we still create a PDB if needed, but a selector with
  # zero labels matches nothing, so we skip such Deployments
  pod_labels_json=$(_jq '.podLabels')

  if ! ns_allowed "${ns}"; then
    continue
  fi

  # Skip if pod template has no labels; cannot construct a meaningful PDB selector
  if [[ "$(echo "${pod_labels_json}" | jq 'length')" -eq 0 ]]; then
    echo "Skipping Deployment ${ns}/${name}: pod template has no labels."
    continue
  fi

  # Determine if any PDB in this namespace covers this Deployment:
  # selector.matchLabels must be non-empty and a subset of podLabels
  covered_count="$(
    jq -n --argjson pdbs "${PDB_JSON}" --arg ns "${ns}" --argjson podLabels "${pod_labels_json}" '
      ($pdbs.items // [])
      | map(select(.metadata.namespace == $ns))
      | map(select((.spec.selector.matchLabels // {}) | length > 0))
      | map(
          if ((.spec.selector.matchLabels // {}) as $sel
              | [ $sel | to_entries[] | ($podLabels[.key] == .value) ] | all
             ) then 1 else 0 end
        )
      | add // 0
    '
  )"

  if [[ "${covered_count}" -gt 0 ]]; then
    echo "Deployment ${ns}/${name} already covered by ${covered_count} PodDisruptionBudget(s)."
    continue
  fi

  # No covering PDB: create one with selector.matchLabels == pod template labels
  # and minAvailable = floor(replicas / 2)
  min_available=$(( replicas / 2 ))
  if [[ "${min_available}" -lt 1 ]]; then
    min_available=1
  fi

  pdb_name="${name}-pdb"

  echo "Creating PodDisruptionBudget ${ns}/${pdb_name} for Deployment ${name} (replicas=${replicas}, minAvailable=${min_available})..."

  # Build selector.matchLabels YAML from pod_labels_json
  selector_yaml="$(echo "${pod_labels_json}" | jq -r '
    to_entries
    | map("      \(.key): \"\(.value)\"")
    | join("\n")
  ')"

  # Apply PDB manifest (idempotent with kubectl apply)
  cat <<EOF | kubectl apply -f -
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: ${pdb_name}
  namespace: ${ns}
spec:
  minAvailable: ${min_available}
  selector:
    matchLabels:
$(echo "${selector_yaml}")
EOF

done

echo
echo "Verification: re-running compliance-style check..."

# Re-run a simplified version of the audit to confirm coverage
{
  kubectl get poddisruptionbudgets --all-namespaces -o json
  kubectl get deployments --all-namespaces -o json
} | jq -rs '
  .[0] as $pdbs | .[1] |
  [ .items[]
  | select((.spec.replicas // 1) > 1)
  | .metadata as $m
  | (.spec.template.metadata.labels // {}) as $podLabels
  | ([ $pdbs.items[]
       | select(.metadata.namespace == $m.namespace)
       | select((.spec.selector.matchLabels // {}) | length > 0)
       | select(
           (.spec.selector.matchLabels // {}) as $sel
           | [ $sel | to_entries[] | ($podLabels[.key] == .value) ] | all
         )
     ] | length) as $count
  | "Deployment " + $m.namespace + "/" + $m.name + " podDisruptionBudgets=" + ($count|tostring)
    + " is_compliant=" + (if $count > 0 then "true" else "false" end)
  ][]'