Consider External Secret Storage

More Info:

Storing secrets in an external, dedicated secrets management system reduces the risk of exposure through the Kubernetes API and etcd. Evaluate cloud provider or third-party secret stores.

Risk Level

Medium

Address

Security

Compliance Standards

• CIS AKS

Triage and Remediation

Remediation

Remediation

Manual Steps

1. Inventory current Kubernetes Secret usage

   • Run on: any machine with kubectl access
   • List all Secrets by type to understand what data is stored in-cluster:
     kubectl get secrets -A -o go-template='{{range .items}}{{.metadata.namespace}} {{.metadata.name}} {{.type}}{{"\n"}}{{end}}' | sort
     
   • Spot candidates for externalization (e.g., Opaque, kubernetes.io/basic-auth, kubernetes.io/dockerconfigjson).

2. Identify workloads dependent on sensitive Secrets

   • Run on: any machine with kubectl access
   • Find which Pods reference which Secrets (env vars and volume mounts):
     kubectl get pods -A -o yaml | grep -E 'name:|secretKeyRef:|secretName:' -n
     
   • Prioritize Secrets used by internet-facing, privileged, or critical workloads for potential migration to an external store.

3. Review existing external secret store integrations

   • Run on: any machine with kubectl access
   • Check for common integrations (adapt namespace if needed):
     kubectl get pods -A | egrep -i 'vault|secrets-store|external-secrets|secret-manager'
     kubectl get crd | egrep -i 'secret|vault|externalsecret|secretstore'
     
   • If such components exist, inspect their configuration/manifests to confirm which Secrets are already backed by an external provider.

4. Evaluate cloud/third‑party options and select a standard

   • Out-of-band review (console/IaC)
   • For your cloud (e.g., AWS, Azure, GCP, OCI) or third‑party (e.g., Vault), review available services (KMS-integrated secret stores, dedicated secret managers) and Kubernetes integrations (CSI driver, External Secrets Operator, Vault Agent, etc.).
   • Decide on:
     • A primary external secret system for new workloads.
     • An access model (service accounts, IAM roles, workload identity).
     • A rollout approach (new apps first vs. migrating existing Secrets).

5. Pilot configuration using manifests / IaC

   • Run on: any machine with kubectl access
   • Deploy the chosen integration in a non‑production namespace using manifests or existing IaC, then create one test Secret through the external system and expose it to a test Pod. Example (generic pattern, adapt to chosen tool):
     # Example: verify external secret CRDs installed (if using an operator)
     kubectl get crd | grep -i externalsecret || echo "No ExternalSecret CRD found"
     
   • Confirm the Pod can read the secret value at runtime (e.g., via kubectl exec into the test Pod).

6. Plan and verify migration away from in‑cluster Secrets

   • Run on: any machine with kubectl access
   • For each high‑priority Kubernetes Secret selected in step 2:
     • Create the equivalent secret entry in the external store.
     • Update Pod/Deployment manifests (or Helm/IaC) to consume it via the chosen integration instead of a native Secret.
   • Verify you are reducing in‑cluster Secrets over time:
     kubectl get secrets -A | wc -l
     
   • Track this count and the list from step 1 periodically to confirm that sensitive data is increasingly sourced from the external store rather than native Secret objects.

Using kubectl

# 1) List all namespaces
# Run on: any machine with kubectl access
kubectl get namespaces

Look for non-system namespaces where applications run (exclude kube-system, kube-public, kube-node-lease by default).

# 2) For each application namespace, list Secrets and how many consumers they have
# Replace <NAMESPACE> with a real namespace, e.g. "default" or "prod-app"
kubectl get secrets -n <NAMESPACE> -o wide
kubectl get pods -n <NAMESPACE> -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.volumes[*].secret.secretName}{"\t"}{range .spec.containers[*].env[*]}{.valueFrom.secretKeyRef.name}{" "}{end}{"\n"}{end}' | sort

Problems indicated by:

• Large numbers of Secrets in a namespace.
• Many Pods consuming the same sensitive-looking Secrets (db-password, api-key, tls-*, jwt-secret, etc.), showing widespread in-cluster secret reliance.

# 3) Inspect Secret types, focusing on sensitive or long-lived credentials
kubectl get secrets -n <NAMESPACE> -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.type}{"\n"}{end}' | sort
kubectl describe secret <SECRET_NAME> -n <NAMESPACE>

Problems indicated by:

• Generic Opaque secrets holding passwords, API tokens, or database credentials.
• Long-lived, non-rotated credentials (age in kubectl get secrets is very large).
• Application credentials stored as Secrets instead of being sourced from an external manager.

# 4) Detect secrets mounted as environment variables or volumes
kubectl get deploy,sts,ds -n <NAMESPACE> -o yaml | grep -nE 'secretKeyRef|secretName' -C2

Problems indicated by:

• Many workloads directly depending on Kubernetes Secrets without any reference to an external secret provider.
• Secrets used broadly across multiple deployments, suggesting central, static credentials.

# 5) Check for use of SecretProviderClass (indicates external secret stores via CSI)
kubectl api-resources | grep -i secretproviderclass || echo "No SecretProviderClass API found"
kubectl get secretproviderclass -A 2>/dev/null

Problems indicated by:

• No secretproviderclass resource available or none defined in application namespaces.
• No other custom resources related to external secret integrations (e.g., ExternalSecret, VaultSecret).

# 6) Look for signs of external secret tooling in use
kubectl get crd | grep -iE 'secret|external|vault|aws|gcp|azure'
kubectl get pods -A | grep -iE 'vault|secrets-store|external-secrets|keyvault|secretsmanager|secret-manager' || echo "No obvious external secret operators/pods"

Problems indicated by:

• No CRDs or Pods related to any external secret operator / CSI driver / vault agent.
• All secret handling apparently done via native Kubernetes Secrets only.

# 7) Sample: inspect one application deployment end-to-end
kubectl get deploy <DEPLOYMENT_NAME> -n <NAMESPACE> -o yaml

In that YAML, look for:

• env.valueFrom.secretKeyRef and volumes.secret.secretName referencing Kubernetes Secrets.
• Absence of environment variables or volumes that would be populated via an external provider (e.g., csi driver volumes, vault agent sidecars, external-secrets annotations).

Problems indicated by:

• Critical app credentials (database, payment gateway, upstream APIs) all coming from Kubernetes Secrets.
• No integration pattern (sidecar, CSI, operator) that suggests use of a cloud/third-party secret manager.

These commands only surface the current state. A human must decide whether the sensitivity and spread of in-cluster Secrets justify adopting an external secrets management solution per the benchmark guidance.

Automation

#!/usr/bin/env bash
#
# Report Kubernetes secret usage patterns to help evaluate external secret storage
# Run on: any machine with kubectl access and cluster-wide read perms
# Usage: ./secrets-inventory.sh > secrets-report.txt

set -euo pipefail

echo "=== Cluster-wide Secret Usage Inventory ==="
echo "Timestamp: $(date -Iseconds)"
echo

# 1) Summary: count of Secrets by type per namespace
echo "== 1) Secret counts by type and namespace =="
kubectl get secrets --all-namespaces -o json \
  | jq -r '
    .items[]
    | [.metadata.namespace, .type]
    | @tsv' \
  | sort \
  | uniq -c \
  | sort -k2,2 -k3,3 \
  | awk 'BEGIN {printf "%-8s %-40s %s\n", "COUNT", "NAMESPACE", "TYPE"} {printf "%-8s %-40s %s\n", $1, $2, $3}'
echo

# 2) Namespaces and ServiceAccounts that use imagePullSecrets
echo "== 2) imagePullSecrets (candidates for registry integration with external secrets) =="

echo "-- Namespaces with imagePullSecrets configured --"
kubectl get namespaces -o json \
  | jq -r '
    .items[]
    | select(.spec."imagePullSecrets" != null)
    | .metadata.name as $ns
    | .spec.imagePullSecrets[].name
    | [$ns, .]
    | @tsv' 2>/dev/null || true

echo
echo "-- ServiceAccounts with imagePullSecrets configured --"
kubectl get sa --all-namespaces -o json \
  | jq -r '
    .items[]
    | select(.imagePullSecrets != null)
    | .metadata.namespace as $ns
    | .metadata.name as $sa
    | .imagePullSecrets[].name
    | [$ns, $sa, .]
    | @tsv' 2>/dev/null || true
echo

# 3) Workloads that mount Secrets as volumes or env
echo "== 3) Workloads referencing Secrets (pods, deployments, statefulsets, daemonsets, cronjobs, jobs) =="

WORKLOAD_KINDS=(
  pods
  deployments
  statefulsets
  daemonsets
  cronjobs
  jobs
)

for kind in "${WORKLOAD_KINDS[@]}"; do
  echo "-- $kind --"
  kubectl get "$kind" --all-namespaces -o json 2>/dev/null \
    | jq -r '
      .items[]
      | .metadata.namespace as $ns
      | .metadata.name as $name
      | .kind as $kind
      | [
          # volumes using secrets
          ( .spec.template.spec.volumes[]? // .spec.volumes[]? | select(.secret != null) | "volume:" + .secret.secretName ),
          # envFrom using secrets
          ( .. | objects | select(has("secretRef") and .secretRef.name != null) | "envFrom:" + .secretRef.name ),
          # env using valueFrom.secretKeyRef
          ( .. | objects | select(has("valueFrom") and .valueFrom.secretKeyRef.name != null) | "env:" + .valueFrom.secretKeyRef.name )
        ]
      | unique
      | select(length > 0)
      | [$kind, $ns, $name, (join(","))]
      | @tsv' 2>/dev/null \
    || true
  echo
done

# 4) Secrets containing literal ("stringData") definitions in manifests (best-effort heuristic)
#    This checks current objects; you still need to review IaC repos for plaintext secrets.
echo "== 4) Secrets with large data blobs (heuristic for potentially sensitive inline values) =="

kubectl get secrets --all-namespaces -o json \
  | jq -r '
    .items[]
    | .metadata.namespace as $ns
    | .metadata.name as $name
    | .type as $type
    | (.data // {}) as $d
    | ($d | to_entries | map(.key + ":" + ( ( .value | @base64d ) | length | tostring )) ) as $entries
    | [$ns, $name, $type, ( $entries | join(",") )]
    | @tsv' \
  | awk 'BEGIN {OFS="\t"; print "NAMESPACE","NAME","TYPE","KEY:PLAINTEXT_LENGTH"}1'
echo

# 5) Detect presence of common external secret integrations (helps understand current posture)
echo "== 5) Detection of common external secret controllers =="

echo "-- Installed CustomResourceDefinitions related to external secrets --"
kubectl get crds \
  | grep -Ei 'secret|vault|externalsecret|externalsecret|secrets-store' \
  || echo "None of the common external-secret-related CRDs detected."

echo
echo "-- Controllers in kube-system and other namespaces with names suggesting secret management --"
kubectl get pods -A \
  | grep -Ei 'vault|secret|externalsecret|secrets-store' \
  || echo "No obvious external secret controllers detected (name-based heuristic only)."
echo

echo "=== Interpretation Guidance ==="
cat <<'EOF'
Items to review as potential problems (i.e., opportunities to move to external secret storage):

1) High counts of generic "Opaque" secrets:
   - Output section 1: If many namespaces have large numbers of "Opaque" secrets,
     especially in application namespaces, these are prime candidates for migration
     to an external secrets manager.

2) imagePullSecrets:
   - Output section 2: Any namespace or ServiceAccount with imagePullSecrets likely
     contains container registry credentials stored as Kubernetes Secrets.
     These can often be moved to cloud-native registry integrations or external
     secret managers.

3) Workloads referencing secrets:
   - Output section 3: Any workload listing many "volume:", "envFrom:", or "env:"
     secret references indicates sensitive runtime configuration being supplied
     via Kubernetes Secrets. These secrets should be reviewed to decide whether
     they should come from an external secret store (e.g., via CSI driver or
     external-secret controller) instead of native Secrets.

4) Large plaintext values:
   - Output section 4: Very large PLAINTEXT_LENGTH values may indicate certificates,
     keys, or other high-value data. Focus on these for externalization.

5) Absence of external-secret integrations:
   - Output section 5: If there are no CRDs or controllers related to external
     secrets, the cluster likely relies entirely on native Kubernetes Secrets.
     That is not inherently wrong, but you should explicitly decide whether to
     adopt a cloud provider or third-party secrets manager for higher assurance.

This script does NOT change any configuration; it only provides data to support
a manual decision about adopting external secret storage, in line with the
benchmark requirement for review and consideration.
EOF