Kubelet Client CA File Set As Appropriate

More Info:

The kubelet —client-ca-file should point to a client CA file so that certificate-based client authentication is enforced. Without it, clients cannot be verified via x509 certificates.

Risk Level

High

Address

Security

Compliance Standards

• CIS AKS

Triage and Remediation

Remediation

Remediation

Manual Steps

1. On every worker node, confirm the kubelet is running and note how it is started:

   ps -fC kubelet
   

   If you see --config=/var/lib/kubelet/config.yaml (or similar), it is using a config file. If you see --client-ca-file directly in the command, it is using command line arguments via systemd.

2. If using a kubelet config file on this worker node, edit it to set the client CA file (adjust the path only if your CA is elsewhere):

   sudo vi /var/lib/kubelet/config.yaml
   

   Ensure the following block exists and is populated (create sections if missing):

   authentication:
     x509:
       clientCAFile: "/etc/kubernetes/certs/ca.crt"
   

   Save and exit.

3. If using command line arguments on this worker node, edit the kubelet systemd drop-in to add the --client-ca-file flag:

   sudo vi /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
   

   In the Environment= line that defines KUBELET_AUTHZ_ARGS, ensure it contains:

   Environment="KUBELET_AUTHZ_ARGS=--authentication-token-webhook=true --authorization-mode=Webhook --client-ca-file=/etc/kubernetes/certs/ca.crt"
   

   Preserve any other existing options; just add --client-ca-file=/etc/kubernetes/certs/ca.crt if it is missing. Save and exit.

4. Reload systemd configuration and restart the kubelet on this worker node (this will temporarily disrupt workloads scheduled here):

   sudo systemctl daemon-reload
   sudo systemctl restart kubelet.service
   

5. Verify on this worker node that the kubelet process now includes the correct --client-ca-file (or that the config file is in use and contains the setting):

   ps -fC kubelet
   

   Confirm either:

   • The command line shows --client-ca-file=/etc/kubernetes/certs/ca.crt, or
   • The command line shows --config=/var/lib/kubelet/config.yaml and the file contains:
     grep -A2 'authentication:' /var/lib/kubelet/config.yaml
     
     with clientCAFile: "/etc/kubernetes/certs/ca.crt" present.

Using kubectl

kubectl cannot configure kubelet process flags or host-level files, so it cannot be used to set --client-ca-file or edit /etc/kubernetes/certs/ca.crt. This must be fixed directly on each worker node’s OS (systemd unit and/or kubelet config file); see the Manual Steps section for exact remediation commands.

Automation

#!/usr/bin/env bash
#
# Configure kubelet --client-ca-file on every worker node.
# Usage: ./set-kubelet-client-ca.sh node1 node2 ...
# Requires: SSH access as a user with sudo on each node.

set -euo pipefail

CA_FILE="/etc/kubernetes/certs/ca.crt"
SYSTEMD_DROPIN="/etc/systemd/system/kubelet.service.d/10-kubeadm.conf"
SSH_USER="ubuntu"   # adjust if needed

run_remote() {
  local node="$1"; shift
  ssh -o StrictHostKeyChecking=no -o BatchMode=yes "${SSH_USER}@${node}" "$@"
}

configure_node() {
  local node="$1"
  echo "=== Configuring node: ${node}"

  # 1. Verify CA file exists
  run_remote "${node}" "sudo test -f '${CA_FILE}'" || {
    echo "ERROR: ${CA_FILE} not found on ${node}; skipping"
    return 1
  }

  # 2. Ensure systemd drop-in exists
  run_remote "${node}" "sudo mkdir -p /etc/systemd/system/kubelet.service.d"

  # 3. Ensure KUBELET_AUTHZ_ARGS is present and includes --client-ca-file
  run_remote "${node}" "sudo bash -s" <<'REMOTE_EOF'
set -euo pipefail

CA_FILE="/etc/kubernetes/certs/ca.crt"
SYSTEMD_DROPIN="/etc/systemd/system/kubelet.service.d/10-kubeadm.conf"

touch "${SYSTEMD_DROPIN}"
if ! grep -q '^\[Service\]' "${SYSTEMD_DROPIN}"; then
  echo '[Service]' | cat - "${SYSTEMD_DROPIN}" > "${SYSTEMD_DROPIN}.tmp"
  mv "${SYSTEMD_DROPIN}.tmp" "${SYSTEMD_DROPIN}"
fi

if grep -q '^Environment="KUBELET_AUTHZ_ARGS=' "${SYSTEMD_DROPIN}"; then
  # Update existing line
  sed -i 's/^Environment="KUBELET_AUTHZ_ARGS=.*$/Environment="KUBELET_AUTHZ_ARGS="/' "${SYSTEMD_DROPIN}"
fi

if ! grep -q '^Environment="KUBELET_AUTHZ_ARGS=' "${SYSTEMD_DROPIN}"; then
  echo 'Environment="KUBELET_AUTHZ_ARGS="' >> "${SYSTEMD_DROPIN}"
fi

# Ensure we have exactly one --client-ca-file=... in the value
# and no conflicting occurrences.
python3 - "$SYSTEMD_DROPIN" "$CA_FILE" <<'PYEOF'
import sys, re
path, ca = sys.argv[1], sys.argv[2]
lines = open(path).read().splitlines()
out = []
for line in lines:
    if line.startswith('Environment="KUBELET_AUTHZ_ARGS='):
        prefix = 'Environment="KUBELET_AUTHZ_ARGS='
        val = line[len(prefix):-1]
        # remove any existing --client-ca-file entries
        parts = [p for p in val.split() if not p.startswith('--client-ca-file=')]
        # append desired flag
        parts.append(f'--client-ca-file={ca}')
        new_val = ' '.join(sorted(set(parts)))
        out.append(f'{prefix}{new_val}"')
    else:
        out.append(line)
open(path, 'w').write('\n'.join(out) + '\n')
PYEOF

REMOTE_EOF

  # 4. Reload systemd and restart kubelet
  run_remote "${node}" "sudo systemctl daemon-reload && sudo systemctl restart kubelet.service"

  # 5. Verification: confirm kubelet is running with --client-ca-file=<CA_FILE>
  echo "--- Verifying on ${node}"
  run_remote "${node}" "/bin/ps -fC kubelet || echo 'kubelet process not found'"
  run_remote "${node}" "ps -o args= -C kubelet | tr ' ' '\n' | grep -E '^--client-ca-file=' || echo 'MISSING --client-ca-file FLAG'"
}

if [ "$#" -lt 1 ]; then
  echo "Usage: $0 worker-node-1 [worker-node-2 ...]"
  exit 1
fi

for node in "$@"; do
  configure_node "${node}" || echo "Node ${node} had errors; see messages above."
done