Kubelet Streaming Connection Idle Timeout Not Set To Zero

More Info:

The kubelet streamingConnectionIdleTimeout should not be 0, which disables the timeout and leaves idle streaming connections open indefinitely.

Risk Level

High

Address

Security

Compliance Standards

• CIS AKS

Triage and Remediation

Remediation

Remediation

Manual Steps

1. Check current kubelet config (per worker node)
   Run on every worker node:

   ps -fC kubelet
   

   Note whether --config is used (config file) and whether --streaming-connection-idle-timeout=0 appears as a flag.

2. If using kubelet config file, set a non‑zero timeout
   Run on every worker node (only if /var/lib/kubelet/config.yaml is in use):

   sudo sed -i 's/^streamingConnectionIdleTimeout: 0$/streamingConnectionIdleTimeout: 5m/' /var/lib/kubelet/config.yaml
   

   If the key does not exist, append it:

   sudo sh -c 'echo "streamingConnectionIdleTimeout: 5m" >> /var/lib/kubelet/config.yaml'
   

3. If using command‑line flags, set a non‑zero timeout
   Run on every worker node (only if kubelet is configured via systemd flags):
   Edit the drop‑in unit:

   sudo nano /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
   

   In the KUBELET_SYSTEM_PODS_ARGS (or corresponding) environment variable, ensure this flag is present and not 0, for example:

   --streaming-connection-idle-timeout=5m
   

   Save and exit.

4. Reload systemd and restart kubelet
   Run on every worker node:

   sudo systemctl daemon-reload
   sudo systemctl restart kubelet.service
   

5. Verify kubelet restarted cleanly
   Run on every worker node:

   sudo systemctl status kubelet.service --no-pager
   

6. Verify non‑zero streaming idle timeout is in effect
   Run on every worker node:

   ps -fC kubelet
   

   Confirm that --streaming-connection-idle-timeout is either absent (and handled via config file) or present with a value other than 0, and if using /var/lib/kubelet/config.yaml, confirm:

   grep -E '^streamingConnectionIdleTimeout:' /var/lib/kubelet/config.yaml
   

   shows a non‑zero value (for example 5m).

Using kubectl

kubectl cannot modify kubelet process flags or its config file, so this finding cannot be fixed via the Kubernetes API. The streamingConnectionIdleTimeout setting must be changed directly on each worker node’s host configuration (for example /var/lib/kubelet/config.yaml or the kubelet systemd unit); see the Manual Steps section for how to do that.

Automation

#!/usr/bin/env bash
#
# Remediates CIS AKS 3.2.5 on every worker node:
# - Ensures kubelet streamingConnectionIdleTimeout is not 0
# - Sets it to 5m in /var/lib/kubelet/config.yaml when present
# - Falls back to adding --streaming-connection-idle-timeout=5m to
#   /etc/systemd/system/kubelet.service.d/10-kubeadm.conf when needed
#
# Run on: every worker node (as root)
# Safe to re-run: yes

set -euo pipefail

KUBELET_CONFIG="/var/lib/kubelet/config.yaml"
KUBELET_UNIT_DROPIN="/etc/systemd/system/kubelet.service.d/10-kubeadm.conf"
DESIRED_TIMEOUT="5m"

echo "=== CIS AKS 3.2.5 remediation: kubelet streamingConnectionIdleTimeout !== 0 ==="

backup_file() {
  local file="$1"
  if [ -f "$file" ]; then
    local ts
    ts="$(date +%Y%m%d%H%M%S)"
    cp -a "$file" "${file}.cis_backup_${ts}"
    echo "Backed up $file to ${file}.cis_backup_${ts}"
  fi
}

restart_kubelet() {
  echo "Reloading systemd and restarting kubelet..."
  systemctl daemon-reload
  systemctl restart kubelet.service
}

needs_restart=0

############################################
# 1. Prefer config file: /var/lib/kubelet/config.yaml
############################################
if [ -f "$KUBELET_CONFIG" ]; then
  echo "Detected kubelet config file: $KUBELET_CONFIG"

  backup_file "$KUBELET_CONFIG"

  # If key exists, update it; else append it at the end.
  if grep -qE '^[[:space:]]*streamingConnectionIdleTimeout[[:space:]]*:' "$KUBELET_CONFIG"; then
    # If already desired and not 0, no change
    if grep -qE "^[[:space:]]*streamingConnectionIdleTimeout[[:space:]]*:[[:space:]]*\"?$DESIRED_TIMEOUT\"?[[:space:]]*$" "$KUBELET_CONFIG"; then
      echo "streamingConnectionIdleTimeout already set to $DESIRED_TIMEOUT in $KUBELET_CONFIG"
    else
      echo "Updating streamingConnectionIdleTimeout to $DESIRED_TIMEOUT in $KUBELET_CONFIG"
      # Normalize to quoted value for safety
      sed -i \
        -E "s|^[[:space:]]*streamingConnectionIdleTimeout[[:space:]]*:.*$|streamingConnectionIdleTimeout: \"$DESIRED_TIMEOUT\"|" \
        "$KUBELET_CONFIG"
      needs_restart=1
    fi
  else
    echo "Adding streamingConnectionIdleTimeout: \"$DESIRED_TIMEOUT\" to $KUBELET_CONFIG"
    printf '\nstreamingConnectionIdleTimeout: "%s"\n' "$DESIRED_TIMEOUT" >> "$KUBELET_CONFIG"
    needs_restart=1
  fi
else
  echo "No kubelet config file at $KUBELET_CONFIG; falling back to systemd drop-in."

  ############################################
  # 2. Fallback to command-line flag in 10-kubeadm.conf
  ############################################
  if [ ! -f "$KUBELET_UNIT_DROPIN" ]; then
    echo "ERROR: $KUBELET_UNIT_DROPIN not found. Cannot configure kubelet flags automatically."
    echo "Please create or edit $KUBELET_UNIT_DROPIN to include:"
    echo "  --streaming-connection-idle-timeout=$DESIRED_TIMEOUT"
    exit 1
  fi

  backup_file "$KUBELET_UNIT_DROPIN"

  # Remove any existing flag occurrences to avoid duplicates/conflicts
  if grep -q -- '--streaming-connection-idle-timeout' "$KUBELET_UNIT_DROPIN"; then
    echo "Removing existing --streaming-connection-idle-timeout flags from $KUBELET_UNIT_DROPIN"
    sed -i 's/--streaming-connection-idle-timeout=[^[:space:]]*//g' "$KUBELET_UNIT_DROPIN"
  fi

  # Ensure flag is present in the KUBELET_SYSTEM_PODS_ARGS line if it exists,
  # otherwise append it to all ExecStart lines as a last resort.
  if grep -q '^Environment=.*KUBELET_SYSTEM_PODS_ARGS' "$KUBELET_UNIT_DROPIN"; then
    echo "Adding --streaming-connection-idle-timeout=$DESIRED_TIMEOUT to KUBELET_SYSTEM_PODS_ARGS"
    sed -i \
      "s|^\(Environment=.*KUBELET_SYSTEM_PODS_ARGS=\"[^\"]*\)\"|\1 --streaming-connection-idle-timeout=$DESIRED_TIMEOUT\"|" \
      "$KUBELET_UNIT_DROPIN"
  elif grep -q '^ExecStart=' "$KUBELET_UNIT_DROPIN"; then
    echo "Adding --streaming-connection-idle-timeout=$DESIRED_TIMEOUT to ExecStart command(s)"
    sed -i \
      "s|^ExecStart=\(.*kubelet.*\)|ExecStart=\1 --streaming-connection-idle-timeout=$DESIRED_TIMEOUT|" \
      "$KUBELET_UNIT_DROPIN"
  else
    echo "ERROR: Could not locate KUBELET_SYSTEM_PODS_ARGS or ExecStart in $KUBELET_UNIT_DROPIN."
    echo "Please manually add:"
    echo "  --streaming-connection-idle-timeout=$DESIRED_TIMEOUT"
    exit 1
  fi

  needs_restart=1
fi

if [ "$needs_restart" -eq 1 ]; then
  restart_kubelet
else
  echo "No kubelet restart needed; configuration already compliant."
fi

############################################
# 3. Verification (CIS audit adaptation)
############################################
echo "Verifying kubelet process flags and config..."
/bin/ps -fC kubelet || {
  echo "ERROR: kubelet process not found."
  exit 1
}

echo
echo "Current kubelet process line:"
/bin/ps -fC kubelet

echo
if [ -f "$KUBELET_CONFIG" ]; then
  echo "Current streamingConnectionIdleTimeout in $KUBELET_CONFIG:"
  grep -E '^[[:space:]]*streamingConnectionIdleTimeout[[:space:]]*:' "$KUBELET_CONFIG" || \
    echo "streamingConnectionIdleTimeout not found in $KUBELET_CONFIG"
fi

echo
echo "Checking that streamingConnectionIdleTimeout is not 0 and/or flag is present..."
if [ -f "$KUBELET_CONFIG" ]; then
  if grep -Eq '^[[:space:]]*streamingConnectionIdleTimeout[[:space:]]*:[[:space:]]*"?0"?[[:space:]]*$' "$KUBELET_CONFIG"; then
    echo "NON-COMPLIANT: streamingConnectionIdleTimeout is still 0 in $KUBELET_CONFIG"
    exit 1
  fi
fi

if /bin/ps -fC kubelet | grep -q -- '--streaming-connection-idle-timeout=0'; then
  echo "NON-COMPLIANT: kubelet still running with --streaming-connection-idle-timeout=0"
  exit 1
fi

echo "Verification passed: kubelet streamingConnectionIdleTimeout is not 0 (CIS AKS 3.2.5)."