Prefer Bound Projected ServiceAccount Tokens Over Secret Tokens

More Info:

Advisory: avoid long-lived ServiceAccount token Secrets; use projected (TokenRequest) tokens with an audience and expiry instead.

Risk Level

Informational

Address

Security

Compliance Standards

• Cloudanix Best Practice

Triage and Remediation

Remediation

Remediation

Manual Steps

1. On any machine with kubectl access, list all legacy ServiceAccount token Secrets and capture details for review:

   kubectl get secrets -A --field-selector type=kubernetes.io/service-account-token \
     -o wide
   kubectl get secrets -A --field-selector type=kubernetes.io/service-account-token \
     -o json > sa-token-secrets.json
   

2. Identify which applications still rely on these Secrets by searching for references in Pods, Deployments, and other workload specs:

   kubectl get pods -A -o json | jq -r '
     .items[] |
     {ns: .metadata.namespace, pod: .metadata.name,
      sa: .spec.serviceAccountName,
      secrets: ([.spec.volumes[]? | select(.secret!=null) | .secret.secretName] // [])}
   ' | grep -Ff <(jq -r '.items[].metadata.name' sa-token-secrets.json | sort -u)
   

   Also inspect any custom manifests/IaC (Helm charts, CD configs, etc.) for explicit mounting of these Secrets.

3. For each workload that currently mounts a ServiceAccount token Secret, decide if you can migrate it to projected ServiceAccount tokens using the EKS-bound token volume instead of a Secret. For Pods you control, plan to:

   • Remove volumes[].secret.secretName and matching volumeMounts that reference the token Secret.
   • Use the automatically mounted ServiceAccount token file at /var/run/secrets/kubernetes.io/serviceaccount/token or add a projected serviceAccountToken volume with audience and expirationSeconds if you need custom settings.

4. Implement the change for one representative Deployment as a pattern:

   # Example: edit a Deployment to stop using a token Secret volume
   kubectl -n <namespace> edit deployment <name>
   

   In the editor:

   • Delete any volumes entry that uses secret: with the token Secret’s name.
   • Delete any volumeMounts that mount that volume.
   • (Optional) Add a projected volume, for example:
     volumes:
       - name: sa-token
         projected:
           sources:
             - serviceAccountToken:
                 path: token
                 audience: "<expected-audience>"
                 expirationSeconds: 3600
     
     and mount it via volumeMounts in the container.

5. After updating all dependent workloads and confirming they run correctly (check logs and readiness), safely remove unused ServiceAccount token Secrets:

   # Show candidate Secrets one more time
   kubectl get secrets -A --field-selector type=kubernetes.io/service-account-token
   
   # For each Secret confirmed unused:
   kubectl -n <namespace> delete secret <secret-name>
   

6. Verify the cluster no longer relies on long-lived ServiceAccount token Secrets, and that remaining ones (if any) are explicitly justified (for example, for legacy or third-party components you cannot yet change):

   kubectl get secrets -A --field-selector type=kubernetes.io/service-account-token -o wide
   

   Document any remaining Secrets, their owners, and a migration plan to projected tokens where feasible.

Using kubectl

# 1) List all long-lived ServiceAccount token Secrets (run on any machine with kubectl access)
kubectl get secrets -A --field-selector type=kubernetes.io/service-account-token

# Optional: show age to spot very old tokens
kubectl get secrets -A --field-selector type=kubernetes.io/service-account-token \
  -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,SA:.metadata.annotations.kubernetes\.io/service-account\.name,AGE:.metadata.creationTimestamp'

What indicates a problem

• Large numbers of kubernetes.io/service-account-token Secrets that are:
  • Very old (e.g., months/years), and
  • Still mounted into Pods or used by external systems.
• Workloads or external clients relying on these Secrets instead of using projected ServiceAccount tokens via the TokenRequest API (audience + short expiry).

---

# 2) For a candidate Secret, inspect details (replace with a real Secret name)
kubectl describe secret -n NAMESPACE SECRET_NAME

Problem indicators

• Type: kubernetes.io/service-account-token.
• No clear operational rotation process; token appears to be a long‑lived credential used by:
  • CI/CD systems,
  • External scripts/tools,
  • Third‑party integrations, etc.

---

# 3) Find which Pods mount a given ServiceAccount token Secret
# (use the ServiceAccount name from the Secret annotation)
SA_NAME="example-serviceaccount"
NAMESPACE="example-namespace"

kubectl get pods -n "$NAMESPACE" \
  -o jsonpath='{range .items[?(@.spec.serviceAccountName=="'"$SA_NAME"'")]}{.metadata.name}{"\n"}{end}'

If many Pods use a ServiceAccount that auto‑generates a long‑lived token Secret and those Pods (or sidecars/agents) read the Secret directly, you likely have a pattern that should be migrated to use projected tokens.

---

# 4) Discover external usage candidates (Secrets referenced in non-Pod objects)
kubectl get secrets -A --field-selector type=kubernetes.io/service-account-token \
  -o json | jq -r '.items[] | "\(.metadata.namespace) \(.metadata.name)"' | while read ns name; do
  echo "Checking references to $ns/$name"
  kubectl get all,ingress,cronjob,job,configmap,secret,serviceaccount,service -A -o yaml \
    | grep -q "$name" && echo "  Referenced somewhere in cluster" || true
done

Problem indicators

• ServiceAccount token Secrets referenced in ConfigMaps, other Secrets, or annotations as static credentials.
• Evidence the token value has been copied out of the cluster (you’ll usually confirm this via process/doc review, not kubectl).

---

# 5) Check if Pods are already using projected ServiceAccount tokens
kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{"\n"}{end}' \
  | while read ns pod; do
      echo "Pod: $ns/$pod"
      kubectl get pod -n "$ns" "$pod" -o jsonpath='{.spec.volumes[*].projected.sources[*].serviceAccountToken}{"\n"}' 2>/dev/null \
        | grep -q '{}' && echo "  Uses projected serviceAccountToken volume" || echo "  No projected SA token volume"
    done

Problem indicators

• Sensitive or externally‑integrated workloads not using projected.serviceAccountToken volumes at all, while still having long‑lived SA token Secrets auto‑created.

---

# 6) For a specific Pod, inspect projected token configuration
kubectl get pod -n NAMESPACE POD_NAME -o yaml

In the output, look under .spec.volumes:

• Healthy pattern (what you want to see for security‑sensitive workloads):

  volumes:
  - name: sa-token
    projected:
      sources:
      - serviceAccountToken:
          audience: "sts.amazonaws.com"
          expirationSeconds: 3600
          path: "token"
  

• Problem pattern:

  • Only a default SA token Secret volume (legacy behavior), no projected.serviceAccountToken.
  • Application docs/manifests instruct users to read /var/run/secrets/kubernetes.io/serviceaccount/token as a de‑facto long‑lived credential.

Automation

#!/usr/bin/env bash
# Purpose: Report use of legacy ServiceAccount token Secrets vs projected tokens in an EKS cluster
# Runs on: any machine with kubectl access and appropriate RBAC

set -euo pipefail

echo "=== Cluster-wide ServiceAccount token Secret inventory ==="
echo "NOTE: Presence of these Secrets is not automatically 'bad',"
echo "      but they should be reviewed and migrated to projected tokens where possible."
echo

# 1) List all service-account token Secrets with basic metadata
echo "1) All Secrets of type kubernetes.io/service-account-token:"
kubectl get secrets -A \
  --field-selector type=kubernetes.io/service-account-token \
  -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,SA:.metadata.annotations.kubernetes\.io/service-account\.name,AGE:.metadata.creationTimestamp' \
  | sed 's/T/ /; s/Z//' \
  | sort
echo

# 2) Highlight “old” token Secrets (likely long‑lived)
#    Adjust the threshold (in days) as appropriate for your environment.
THRESHOLD_DAYS=7
echo "2) Token Secrets older than ${THRESHOLD_DAYS} days (candidates for review):"
kubectl get secrets -A \
  --field-selector type=kubernetes.io/service-account-token \
  --sort-by=.metadata.creationTimestamp \
  -o json \
  | jq -r --arg days "${THRESHOLD_DAYS}" '
    .items[]
    | . as $s
    | (now - (.metadata.creationTimestamp | sub("Z$"; "") | strptime("%Y-%m-%dT%H:%M:%S") | mktime)) / 86400
      as $age
    | select($age >= ($days|tonumber))
    | [
        .metadata.namespace,
        .metadata.name,
        .metadata.annotations["kubernetes.io/service-account.name"],
        ( $age | floor | tostring )
      ]
      | @tsv' 2>/dev/null \
  | awk 'BEGIN { OFS="\t"; print "NAMESPACE","SECRET","SERVICEACCOUNT","AGE_DAYS" } { print }' \
  || echo "  (jq not available; skipping age-based listing)"
echo

# 3) For each ServiceAccount, show whether it is mounting a Secret token
#    vs using a projected serviceAccountToken volume.
echo "3) ServiceAccounts and how their tokens are consumed by Pods:"
echo "   - If a ServiceAccount has Pods mounting a Secret token volume, that is legacy usage."
echo "   - If Pods use projected serviceAccountToken volumes (audience/expiration), that is preferred."
echo

# Build a map of SA -> Pods & volume types
# This produces lines like:
# NAMESPACE  SERVICEACCOUNT  POD  VOLUME_NAME  VOLUME_TYPE  SECRET_NAME  TOKEN_AUDIENCE  TOKEN_EXPIRATIONSECONDS
kubectl get pods -A -o json \
  | jq -r '
    .items[]
    | .metadata as $meta
    | .spec as $spec
    | ($spec.serviceAccountName // "default") as $sa
    | ($meta.namespace) as $ns
    | ($meta.name) as $pod
    | ($spec.volumes // [])
    | map(
        if .projected and (.projected.sources[]? | has("serviceAccountToken")) then
          .projected.sources[]
          | select(has("serviceAccountToken"))
          | [
              $ns,
              $sa,
              $pod,
              .serviceAccountToken.path,
              "projected-token",
              "",
              (.serviceAccountToken.audience // ""),
              ((.serviceAccountToken.expirationSeconds // 0) | tostring)
            ]
        elif has("secret") and .secret.secretName then
          [
              $ns,
              $sa,
              $pod,
              .name,
              "secret-token",
              .secret.secretName,
              "",
              ""
          ]
        else
          empty
        end
      )
    | .[]
    | @tsv
  ' 2>/dev/null \
  | awk 'BEGIN {
      OFS="\t";
      print "NAMESPACE","SERVICEACCOUNT","POD","VOLUME","TOKEN_CONSUMPTION","SECRET_NAME","AUDIENCE","EXPIRATION_SECONDS";
    } { print }'
echo

cat <<'EOF'

How to interpret this output:

1) Section 1 (all token Secrets)
   - Any rows mean there are legacy ServiceAccount token Secrets in the cluster.
   - These are long-lived by default and should generally be phased out where possible.

2) Section 2 (old token Secrets)
   - Secrets with AGE_DAYS >= THRESHOLD_DAYS are more likely to be long-lived and should be prioritized for review.
   - Problem indication:
       * Many old Secrets that remain referenced by workloads.
       * Tokens for ServiceAccounts that are no longer used.

3) Section 3 (how Pods consume tokens)
   - TOKEN_CONSUMPTION = secret-token:
       * Pods are mounting a Secret of type kubernetes.io/service-account-token.
       * This indicates continued use of long-lived token Secrets (problematic pattern to review).
   - TOKEN_CONSUMPTION = projected-token:
       * Pods are using projected serviceAccountToken volumes (preferred pattern).
       * Check AUDIENCE and EXPIRATION_SECONDS are set appropriately for your use case.

Follow-up review guidance (manual):
   - For each ServiceAccount whose Pods show TOKEN_CONSUMPTION=secret-token:
       * Identify why the Secret is used instead of a projected token.
       * Where possible, update manifests to use projected serviceAccountToken volumes
         (with explicit audience and short expiration) instead of mounting the Secret.
   - For any old, unused token Secrets:
       * Confirm they are not referenced by any Pod or external process, then delete them.

EOF