Prefer Using Secrets As Files Over Secrets As Environment Variables

More Info:

Secrets exposed as environment variables are more easily leaked via process listings, logs, and child processes. Mounting secrets as files reduces this exposure.

Risk Level

Medium

Address

Security

Compliance Standards

• CIS AKS

Triage and Remediation

Remediation

Remediation

Manual Steps

1. Identify pods using secrets as environment variables

   • Run on: any machine with kubectl access
   kubectl get all --all-namespaces -o jsonpath='{range .items[?(@..secretKeyRef)]}{.kind}{";"}{.metadata.namespace}{";"}{.metadata.name}{"\n"}{end}'
   
   • Note each NAMESPACE and NAME you need to fix (usually Deployments, StatefulSets, DaemonSets, Jobs, CronJobs, Pods).

2. Inspect one affected workload’s manifest

   • Run on: any machine with kubectl access
     Replace NAMESPACE and NAME with values from step 1:
   kubectl get deployment NAME -n NAMESPACE -o yaml > /tmp/NAME.yaml
   

   (If the resource is not a Deployment, change deployment to its kind in lowercase, e.g. statefulset, daemonset, job, pod.)

3. Edit the manifest to mount secrets as files instead of env variables

   • Run on: any machine with kubectl access
   • Open the file:
   vi /tmp/NAME.yaml
   
   • In each container spec:
     • Remove env and envFrom entries that use secretKeyRef. Example to delete:
       env:
         - name: DB_PASSWORD
           valueFrom:
             secretKeyRef:
               name: db-secret
               key: password
       envFrom:
         - secretRef:
             name: db-secret
       
     • Add a volumeMounts entry pointing to a new secret-backed volume, and add that volume under spec.template.spec.volumes (or spec.volumes for a Pod). Example pattern to add:
       spec:
         containers:
           - name: app
             volumeMounts:
               - name: db-secret-volume
                 mountPath: /var/run/secrets/db
                 readOnly: true
         volumes:
           - name: db-secret-volume
             secret:
               secretName: db-secret
       
   • Coordinate with the application team so the application reads from the mounted files (e.g. /var/run/secrets/db/password) instead of environment variables before applying.

4. Apply the updated manifest

   • Run on: any machine with kubectl access
   kubectl apply -f /tmp/NAME.yaml
   
   • Repeat steps 2–4 for each affected resource from step 1.

5. (Optional) Confirm pods are using secret volumes and not env-based secrets

   • Run on: any machine with kubectl access
   kubectl get deployment NAME -n NAMESPACE -o yaml | grep -n "secretKeyRef" || echo "No secretKeyRef in env for this resource"
   kubectl get deployment NAME -n NAMESPACE -o yaml | grep -n "secret:" -n
   

6. Verify cluster-wide that env-based secret references are removed

   • Run on: any machine with kubectl access
   output=$(kubectl get all --all-namespaces -o jsonpath='{range .items[?(@..secretKeyRef)]} {.kind} {.metadata.name} {"\n"}{end}')
   if [ -z "$output" ]; then echo "NO_ENV_SECRET_REFERENCES"; else echo "ENV_SECRET_REFERENCES_FOUND"; echo "$output"; fi
   

Using kubectl

On any machine with kubectl access:

1. Identify the affected workloads

kubectl get all --all-namespaces -o jsonpath='{range .items[?(@..secretKeyRef)]} {.kind} {.metadata.namespace} {.metadata.name} {"\n"}{end}'

For each listed Deployment/StatefulSet/DaemonSet/Pod, get its full manifest:

# Example for a Deployment
kubectl get deployment -n <NAMESPACE> <NAME> -o yaml > /tmp/<NAME>.yaml

2. Edit the manifest to use secrets as files

In the downloaded YAML:

• Under each affected container, remove any env or envFrom entries that reference secretKeyRef.
• Add a volume that sources the Secret.
• Add a volumeMount to mount that volume into the container.

Example transformation (before):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  namespace: default
spec:
  template:
    spec:
      containers:
        - name: app
          image: my-image:latest
          env:
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: db-secret
                  key: password
          envFrom:
            - secretRef:
                name: other-secret

After refactor:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  namespace: default
spec:
  template:
    spec:
      containers:
        - name: app
          image: my-image:latest
          volumeMounts:
            - name: db-secret-vol
              mountPath: /var/run/secrets/db
              readOnly: true
            - name: other-secret-vol
              mountPath: /var/run/secrets/other
              readOnly: true
      volumes:
        - name: db-secret-vol
          secret:
            secretName: db-secret
        - name: other-secret-vol
          secret:
            secretName: other-secret

Application code must be updated to read from the mounted files, e.g. /var/run/secrets/db/password instead of an environment variable.

3. Apply the updated manifest

kubectl apply -f /tmp/<NAME>.yaml

Repeat for each affected Deployment/StatefulSet/DaemonSet/Pod.

4. Verification

Run the same audit command and confirm it reports no remaining references:

output=$(kubectl get all --all-namespaces -o jsonpath='{range .items[?(@..secretKeyRef)]} {.kind} {.metadata.name} {"\n"}{end}')
if [ -z "$output" ]; then echo "NO_ENV_SECRET_REFERENCES"; else echo "ENV_SECRET_REFERENCES_FOUND"; fi

Automation

#!/usr/bin/env bash
#
# Refactor pods/workloads to mount Secrets as files instead of using them
# as environment variables (env/envFrom with secretKeyRef).
#
# REQUIREMENTS:
#   - Run on any machine with kubectl access and correct kubeconfig
#   - kubectl v1.20+ (for strategic-merge patches)
#
# LIMITATIONS (IMPORTANT):
#   - This script CANNOT safely infer how your application reads secrets.
#   - It ONLY reports and proposes changes via YAML patches.
#   - You MUST review and apply the generated patches manually.
#
# BEHAVIOR:
#   - Detects current use of env/envFrom.secretKeyRef
#   - For each offending workload, emits a patch file that:
#       * removes env/envFrom entries that reference Secrets
#       * adds secret volume + mount at /var/run/secrets/<secret-name>
#   - Safe to re-run: it overwrites previous patch files for same object.
#   - At the end, shows remaining offending resources.
#
# NOTE:
#   - After you APPLY a patch, your pods will be recreated/rolled out
#     by the controller (Deployment/StatefulSet/DaemonSet/Job/CronJob).
#   - You must adapt your application to read secrets from files.

set -euo pipefail

OUT_DIR="${OUT_DIR:-./secret-env-to-file-patches}"
mkdir -p "${OUT_DIR}"

echo "Discovering Kubernetes resources that use Secret env refs..."
# Resources likely to have pods
KINDS=(
  Deployment
  StatefulSet
  DaemonSet
  ReplicaSet
  ReplicationController
  Job
  CronJob
  Pod
)

# JSONPath to list any item that has a secretKeyRef in env or envFrom
JSONPATH='{range .items[?(@..secretKeyRef)]}{.kind}{" "}{.metadata.namespace}{" "}{.metadata.name}{"\n"}{end}'

# Get all offending objects
OFFENDERS=$(kubectl get "${KINDS[@]}" --all-namespaces -o jsonpath="$JSONPATH" 2>/dev/null || true)

if [ -z "${OFFENDERS}" ]; then
  echo "NO_ENV_SECRET_REFERENCES"
  exit 0
fi

echo "Found the following resources using secrets as environment variables:"
echo "${OFFENDERS}" | sort -u

echo
echo "Generating patch files under: ${OUT_DIR}"
echo "You MUST review and apply them manually with kubectl patch -f or kubectl apply -f."
echo

# Function: build a patch skeleton for workloads with pod template
generate_workload_patch() {
  local kind="$1" ns="$2" name="$3"
  local jsonpath_template
  case "${kind}" in
    CronJob)
      jsonpath_template='{.spec.jobTemplate.spec.template.spec}'
      ;;
    *)
      jsonpath_template='{.spec.template.spec}'
      ;;
  esac

  echo "  - Inspecting ${kind} ${ns}/${name}"

  # Extract full YAML to work on locally
  tmp_yaml="$(mktemp)"
  kubectl get "${kind}" "${name}" -n "${ns}" -o yaml > "${tmp_yaml}"

  # We create a generic patch that:
  #   * appends secret volumes for each secretKeyRef
  #   * removes offending env/envFrom entries
  # Because exact selective removal is app-specific, we give commented hints.

  # Discover distinct secret names used in env/envFrom for this object
  mapfile -t secrets < <(kubectl get "${kind}" "${name}" -n "${ns}" -o json \
    | jq -r '.. | objects | select(has("secretKeyRef")) | .secretKeyRef.name' \
    | sort -u)

  if [ "${#secrets[@]}" -eq 0 ]; then
    rm -f "${tmp_yaml}"
    return
  fi

  patch_file="${OUT_DIR}/${ns}__${kind}__${name}.patch.yaml"

  {
    echo "# Patch for ${kind} ${ns}/${name}"
    echo "# PURPOSE:"
    echo "#   - Mount the listed Secrets as volumes under /var/run/secrets/<secret-name>"
    echo "#   - Replace env/envFrom.secretKeyRef usage in containers to read from files."
    echo "# IMPORTANT:"
    echo "#   - You MUST manually edit this patch:"
    echo "#       * Remove/replace env and envFrom entries that reference these Secrets."
    echo "#       * Optionally adjust mountPath and keys to match your application."
    echo "#   - After editing, apply via one of:"
    echo "#       kubectl patch ${kind} ${name} -n ${ns} --type=merge --patch-file ${patch_file}"
    echo "#       kubectl apply -f ${patch_file}"
    echo
    echo "apiVersion: $(yq '.apiVersion' "${tmp_yaml}")"
    echo "kind: ${kind}"
    echo "metadata:"
    echo "  name: ${name}"
    echo "  namespace: ${ns}"
    case "${kind}" in
      CronJob)
        echo "spec:"
        echo "  jobTemplate:"
        echo "    spec:"
        echo "      template:"
        echo "        spec:"
        indent="          "
        ;;
      *)
        echo "spec:"
        echo "  template:"
        echo "    spec:"
        indent="      "
        ;;
    esac

    echo "${indent}volumes:"
    echo "${indent}  # Existing volumes are not shown here; this is a strategic merge patch."
    for s in "${secrets[@]}"; do
      echo "${indent}  - name: secret-${s}"
      echo "${indent}    secret:"
      echo "${indent}      secretName: ${s}"
    done

    echo
    echo "${indent}containers:"
    echo "${indent}  # For EACH container, add volumeMounts for the above Secrets."
    echo "${indent}  # Then, MANUALLY remove or adjust env/envFrom entries that use secretKeyRef."
    echo "${indent}  # Example:"
    echo "${indent}  # - name: my-container"
    echo "${indent}  #   volumeMounts:"
    for s in "${secrets[@]}"; do
      echo "${indent}  #   - name: secret-${s}"
      echo "${indent}  #     mountPath: /var/run/secrets/${s}"
      echo "${indent}  #     readOnly: true"
    done
    echo "${indent}  #   env:"
    echo "${indent}  #   # BEFORE (remove this):"
    echo "${indent}  #   # - name: DB_PASSWORD"
    echo "${indent}  #   #   valueFrom:"
    echo "${indent}  #   #     secretKeyRef:"
    echo "${indent}  #   #       name: ${secrets[0]}"
    echo "${indent}  #   #       key: password"
    echo "${indent}  #   # AFTER (example of file-based usage, if your app supports it):"
    echo "${indent}  #   # - name: DB_PASSWORD_FILE"
    echo "${indent}  #   #   value: /var/run/secrets/${secrets[0]}/password"
  } > "${patch_file}"

  rm -f "${tmp_yaml}"
}

# Function: handle plain Pods (no controller)
generate_pod_patch() {
  local kind="$1" ns="$2" name="$3"
  echo "  - Inspecting Pod ${ns}/${name}"

  tmp_yaml="$(mktemp)"
  kubectl get pod "${name}" -n "${ns}" -o yaml > "${tmp_yaml}"

  mapfile -t secrets < <(kubectl get pod "${name}" -n "${ns}" -o json \
    | jq -r '.. | objects | select(has("secretKeyRef")) | .secretKeyRef.name' \
    | sort -u)

  if [ "${#secrets[@]}" -eq 0 ]; then
    rm -f "${tmp_yaml}"
    return
  fi

  patch_file="${OUT_DIR}/${ns}__Pod__${name}.patch.yaml"

  {
    echo "# Patch for Pod ${ns}/${name}"
    echo "# NOTE: This only affects this specific Pod; consider patching the controller instead (Deployment, Job, etc.)."
    echo "# See workload patches in ${OUT_DIR} for a more durable fix."
    echo
    echo "apiVersion: $(yq '.apiVersion' "${tmp_yaml}")"
    echo "kind: Pod"
    echo "metadata:"
    echo "  name: ${name}"
    echo "  namespace: ${ns}"
    echo "spec:"
    echo "  volumes:"
    for s in "${secrets[@]}"; do
      echo "  - name: secret-${s}"
      echo "    secret:"
      echo "      secretName: ${s}"
    done
    echo
    echo "  containers:"
    echo "  # Add volumeMounts and refactor env/envFrom as in workload examples."
    echo "  # This is a strategic-merge patch; only specified fields are merged."
  } > "${patch_file}"

  rm -f "${tmp_yaml}"
}

# Generate patches per offending object
echo
echo "Creating patch templates..."
echo "${OFFENDERS}" | sort -u | while read -r kind ns name; do
  [ -z "${kind}" ] && continue
  case "${kind}" in
    Deployment|StatefulSet|DaemonSet|ReplicaSet|ReplicationController|Job|CronJob)
      generate_workload_patch "${kind}" "${ns}" "${name}"
      ;;
    Pod)
      generate_pod_patch "${kind}" "${ns}" "${name}"
      ;;
    *)
      echo "  - Skipping unsupported kind: ${kind} ${ns}/${name}"
      ;;
  esac
done

echo
echo "Patch templates generated under: ${OUT_DIR}"
echo "Next steps:"
echo "  1) Review and edit each *.patch.yaml file to:"
echo "       - Add volumeMounts under each container."
echo "       - Remove env/envFrom entries that reference Secrets."
echo "       - Optionally add *_FILE env vars pointing to mounted paths."
echo "  2) Apply each patch, for example:"
echo "       kubectl apply -f ${OUT_DIR}/<namespace>__<Kind>__<name>.patch.yaml"
echo
echo "Re-running audit to verify remaining secretKeyRef environment usage..."

output=$(kubectl get all --all-namespaces -o jsonpath='{range .items[?(@..secretKeyRef)]} {.kind} {.metadata.name} {"\n"}{end}')
if [ -z "$output" ]; then
  echo "NO_ENV_SECRET_REFERENCES"
else
  echo "ENV_SECRET_REFERENCES_FOUND"
  echo "Remaining offending objects:"
  echo "${output}" | sort -u
fi