Kubelet Event Record QPS Set For Appropriate Event Capture

More Info:

The kubelet eventRecordQPS should be set to a level that ensures appropriate event capture for monitoring without dropping important events.

Risk Level

Medium

Address

Security

Compliance Standards

• CIS AKS

Triage and Remediation

Remediation

Remediation

Manual Steps

1. On every worker node, open the kubelet config file and set eventRecordQPS to an appropriate value (example: 5):

   sudo sed -i 's/^[[:space:]]*eventRecordQPS:.*$/eventRecordQPS: 5/' /var/lib/kubelet/config.yaml || \
   echo "eventRecordQPS: 5" | sudo tee -a /var/lib/kubelet/config.yaml
   

2. On every worker node, ensure no conflicting --eventRecordQPS flag is present in the kubelet systemd drop-in; remove or adjust any such flag:

   sudo sed -i 's/--eventRecordQPS=[0-9]\+//g' /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
   

   Then, if you need to set it via args instead of config file, append it explicitly:

   sudo sed -i 's#^\(KUBELET_KUBECONFIG_ARGS=.*\)$#\1 --eventRecordQPS=5#' /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
   

3. On every worker node, reload systemd and restart kubelet to apply the changes:

   sudo systemctl daemon-reload
   sudo systemctl restart kubelet.service
   

4. On every worker node, confirm kubelet is healthy after restart:

   sudo systemctl status kubelet -l
   

5. On every worker node, verify the running kubelet process reflects the desired eventRecordQPS setting:

   /bin/ps -fC kubelet
   

   Check the command output for --eventRecordQPS=5 if using flags, or rely on the config file if the flag is absent and eventRecordQPS: 5 is present in /var/lib/kubelet/config.yaml.

Using kubectl

Kubectl cannot modify kubelet host-level configuration such as /var/lib/kubelet/config.yaml or the systemd unit files on worker nodes. To remediate this finding, make the changes directly on each worker node’s filesystem and systemd configuration as described in the Manual Steps section.

Automation

#!/usr/bin/env bash
#
# Remediate CIS AKS 3.2.7 on every worker node:
# Ensure kubelet eventRecordQPS is set to an appropriate value (default here: 5).
#
# Run this script ON EACH WORKER NODE as root.
#

set -euo pipefail

EVENT_QPS_VALUE="5"
KUBELET_CONFIG="/var/lib/kubelet/config.yaml"
SYSTEMD_DROPIN_DIR="/etc/systemd/system/kubelet.service.d"
SYSTEMD_DROPIN_FILE="${SYSTEMD_DROPIN_DIR}/10-kubeadm.conf"

echo "==> Ensuring kubelet eventRecordQPS is set to ${EVENT_QPS_VALUE}"

# 1. Ensure kubelet config file exists
if [[ ! -f "${KUBELET_CONFIG}" ]]; then
  echo "ERROR: ${KUBELET_CONFIG} not found on this node."
  echo "This script expects kubelet to be configured via ${KUBELET_CONFIG}."
  exit 1
fi

# 2. Backup kubelet config
if [[ ! -f "${KUBELET_CONFIG}.cisbak" ]]; then
  cp "${KUBELET_CONFIG}" "${KUBELET_CONFIG}.cisbak"
  echo "Backup created at ${KUBELET_CONFIG}.cisbak"
fi

# 3. Update eventRecordQPS in kubelet config.yaml (idempotent)
if grep -qE '^\s*eventRecordQPS:' "${KUBELET_CONFIG}"; then
  # Replace existing setting
  sed -i -E "s/^\s*eventRecordQPS:.*/eventRecordQPS: ${EVENT_QPS_VALUE}/" "${KUBELET_CONFIG}"
else
  # Add setting under top-level (no strict YAML reformatting)
  echo "eventRecordQPS: ${EVENT_QPS_VALUE}" >> "${KUBELET_CONFIG}"
fi
echo "Updated ${KUBELET_CONFIG} with eventRecordQPS: ${EVENT_QPS_VALUE}"

# 4. Ensure no conflicting --eventRecordQPS in kubelet systemd drop-in
if [[ -d "${SYSTEMD_DROPIN_DIR}" && -f "${SYSTEMD_DROPIN_FILE}" ]]; then
  if grep -q -- "--eventRecordQPS" "${SYSTEMD_DROPIN_FILE}"; then
    cp "${SYSTEMD_DROPIN_FILE}" "${SYSTEMD_DROPIN_FILE}.cisbak" || true
    # Remove any --eventRecordQPS=... occurrences from KUBELET_KUBEADM_ARGS / KUBELET_ARGS lines
    sed -i -E 's/--eventRecordQPS=[^ "]*//g' "${SYSTEMD_DROPIN_FILE}"
    # Clean up double spaces left behind
    sed -i -E 's/  +/ /g' "${SYSTEMD_DROPIN_FILE}"
    echo "Removed conflicting --eventRecordQPS flags from ${SYSTEMD_DROPIN_FILE}"
  fi
else
  echo "Note: ${SYSTEMD_DROPIN_FILE} not found; no systemd kubelet drop-in to clean."
fi

# 5. Reload and restart kubelet
echo "==> Restarting kubelet"
systemctl daemon-reload
systemctl restart kubelet.service

# Wait briefly for kubelet to restart
sleep 5

# 6. Verification: show kubelet process flags and confirm effective configuration
echo "==> Verification: kubelet process (ps)"
/bin/ps -fC kubelet || {
  echo "ERROR: kubelet process not found after restart."
  systemctl status kubelet -l || true
  exit 1
}

echo "==> Verification: checking effective kubelet config for eventRecordQPS"
if grep -qE "^\s*eventRecordQPS:\s*${EVENT_QPS_VALUE}\b" "${KUBELET_CONFIG}"; then
  echo "PASS: ${KUBELET_CONFIG} has eventRecordQPS: ${EVENT_QPS_VALUE}"
else
  echo "WARN: ${KUBELET_CONFIG} does not show eventRecordQPS: ${EVENT_QPS_VALUE}"
fi

# Extra check: ensure no --eventRecordQPS in kubelet args (to avoid overriding file)
if /bin/ps -fC kubelet | grep -q -- "--eventRecordQPS"; then
  echo "WARN: kubelet process still has a --eventRecordQPS flag."
  echo "Inspect systemd unit and environment for overrides:"
  echo "  systemctl cat kubelet"
  echo "  systemctl show kubelet -p Environment"
else
  echo "PASS: kubelet process has no overriding --eventRecordQPS flag."
fi

echo "==> Completed CIS AKS 3.2.7 remediation on this worker node."