Containers Should Use A Read-Only Root Filesystem

More Info:

Verifies readOnlyRootFilesystem is true. A writable root filesystem lets an attacker persist tools or modify binaries inside a running container.

Risk Level

Medium

Address

Security

Compliance Standards

• Cloudanix Best Practice

Triage and Remediation

Remediation

Remediation

Manual Steps

1. Identify noncompliant pods (run on any machine with kubectl access):

   kubectl get pods --all-namespaces -o json | jq -r '
     [ .items[]
     | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
     | .metadata as $m
     | (($m.ownerReferences // [])[] | select(.controller) | .kind + "/" + $m.namespace + "/" + .name) as $owner
     | ((.spec.containers // []) + (.spec.initContainers // []))[]
     | select(.securityContext.readOnlyRootFilesystem != true)
     | $owner // ("Pod/" + $m.namespace + "/" + $m.name)
     ] | unique[]'
   

2. For each owning workload (Deployment/StatefulSet/DaemonSet/Job/CronJob) or standalone Pod found in step 1, export its manifest (run on any machine with kubectl access). Example for a Deployment:

   kubectl -n NAMESPACE get deployment DEPLOYMENT_NAME -o yaml > /tmp/workload.yaml
   

   For a standalone Pod:

   kubectl -n NAMESPACE get pod POD_NAME -o yaml > /tmp/workload.yaml
   

3. Edit the manifest to set a read-only root filesystem (edit /tmp/workload.yaml on any machine with kubectl access):

   • Under each .spec.template.spec.containers[] and .spec.template.spec.initContainers[] (or .spec.containers[] / .spec.initContainers[] for a standalone Pod), ensure:
     securityContext:
       readOnlyRootFilesystem: true
     
     If securityContext exists, add readOnlyRootFilesystem: true inside it.

4. If the container needs writable paths, add emptyDir volumes and mount them (edit the same /tmp/workload.yaml):

   • Under spec.template.spec.volumes (or spec.volumes for a Pod) add, for each writable path:
     - name: writable-tmp
       emptyDir: {}
     
   • Under the corresponding container’s volumeMounts:
     volumeMounts:
       - name: writable-tmp
         mountPath: /path/that/must/be/writable
     
   • Remove any reliance on writing to / or other parts of the root filesystem.

5. Apply the updated manifest (run on any machine with kubectl access):

   kubectl apply -f /tmp/workload.yaml
   

   For controllers (Deployments, etc.), this will roll out new pods. For standalone Pods, you may need to delete and recreate if they are not controlled:

   kubectl -n NAMESPACE delete pod POD_NAME
   kubectl apply -f /tmp/workload.yaml
   

6. Verify compliance (run on any machine with kubectl access):

   kubectl get pods --all-namespaces -o json | jq -r '
     [ .items[]
     | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
     | .metadata as $m
     | (.spec.nodeName // "") as $node
     | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
     | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
     | ((.spec.containers // []) + (.spec.initContainers // []))[]
     | (.securityContext.readOnlyRootFilesystem == true) as $ok
     | "kind=Pod ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=v1"
       + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
       + (if $node   == ""   then "" else " node=\($node)" end)
       + (if $labels == ""   then "" else " labels=\($labels)" end)
       + (if $own    == null then "" else " owner=\($own.kind)/\($m.namespace)/\($own.name)/\($own.uid)" end)
       + " container=\(.name) image=\(.image)"
       + " readOnlyRootFilesystem=\(if .securityContext.readOnlyRootFilesystem == null then "unset" else .securityContext.readOnlyRootFilesystem end)"
       + " is_compliant=\(if $ok then "true" else "false" end)"
     ] as $rows
     | if ($rows | map(select(. | test("is_compliant=false"))) | length) == 0
       then "is_compliant=true"
       else $rows[]
       end'
   

Using kubectl

On any machine with kubectl access:

1. Identify the non‑compliant Pod and its owner

kubectl get pods -A
kubectl get pod <POD_NAME> -n <POD_NAMESPACE> -o yaml

If the Pod is controlled by a higher-level object (Deployment, StatefulSet, DaemonSet, Job, CronJob), you’ll see ownerReferences in the Pod spec. Always fix the controller, not the individual Pod.

2. Patch a Deployment to use a read‑only root filesystem

Example for a Deployment owning the Pod:

kubectl patch deployment <DEPLOYMENT_NAME> -n <POD_NAMESPACE> \
  --type merge \
  -p '{
    "spec": {
      "template": {
        "spec": {
          "containers": [
            {
              "name": "<CONTAINER_NAME>",
              "securityContext": {
                "readOnlyRootFilesystem": true
              }
            }
          ]
        }
      }
    }
  }'

If the container needs a writable path, also add an emptyDir volume and mount it:

kubectl patch deployment <DEPLOYMENT_NAME> -n <POD_NAMESPACE> \
  --type merge \
  -p '{
    "spec": {
      "template": {
        "spec": {
          "volumes": [
            {
              "name": "writable-tmp",
              "emptyDir": {}
            }
          ],
          "containers": [
            {
              "name": "<CONTAINER_NAME>",
              "securityContext": {
                "readOnlyRootFilesystem": true
              },
              "volumeMounts": [
                {
                  "name": "writable-tmp",
                  "mountPath": "/path/that/must/be/writable"
                }
              ]
            }
          ]
        }
      }
    }
  }'

3. Equivalent manifest edits (for GitOps or IaC)

Edit your Deployment manifest (in Git or your IaC system) to include:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  namespace: my-namespace
spec:
  template:
    spec:
      volumes:
        - name: writable-tmp
          emptyDir: {}
      containers:
        - name: my-container
          image: my-image:tag
          securityContext:
            readOnlyRootFilesystem: true
          volumeMounts:
            - name: writable-tmp
              mountPath: /path/that/must/be/writable

Apply it:

kubectl apply -f deployment.yaml

Use the corresponding kubectl patch/manifest changes for other controllers:

• StatefulSet: kubectl patch statefulset <NAME> -n <NS> ...
• DaemonSet: kubectl patch daemonset <NAME> -n <NS> ...
• Job: kubectl patch job <NAME> -n <NS> ...
• CronJob: kubectl patch cronjob <NAME> -n <NS> ...

4. Verification

After the new Pods are running, rerun the audit (on any machine with kubectl):

kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
  | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | .metadata as $m
  | (.spec.nodeName // "") as $node
  | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
  | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
  | ((.spec.containers // []) + (.spec.initContainers // []))[]
  | (.securityContext.readOnlyRootFilesystem == true) as $ok
  | "kind=Pod ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=v1"
    + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
    + (if $node   == ""   then "" else " node=\($node)" end)
    + (if $labels == ""   then "" else " labels=\($labels)" end)
    + (if $own    == null then "" else " owner=\($own.kind)/\($m.namespace)/\($own.name)/\($own.uid)" end)
    + " container=\(.name) image=\(.image)"
    + " readOnlyRootFilesystem=\(if .securityContext.readOnlyRootFilesystem == null then "unset" else .securityContext.readOnlyRootFilesystem end)"
    + " is_compliant=\(if $ok then "true" else "false" end)"
  ] as $rows
  | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end'

Confirm that all relevant containers now show readOnlyRootFilesystem=true and is_compliant=true.

Automation

#!/usr/bin/env bash
#
# Remediate: Ensure containers use a read-only root filesystem
# Scope: Any machine with kubectl access to the EKS cluster
#
# Strategy:
# - Discover all non-system namespaces
# - For each workload type (Deployment, StatefulSet, DaemonSet, Job, CronJob, Pod):
#   - Patch every container and initContainer with securityContext.readOnlyRootFilesystem=true
#   - Skip resources that already set it to true (patch is idempotent)
# - Print a final verification summary using the benchmark audit command
#
# Requirements:
# - kubectl configured for the target cluster
# - jq installed

set -euo pipefail

# ----------------------------
# Helper functions
# ----------------------------

have_cmd() {
  command -v "$1" >/dev/null 2>&1
}

if ! have_cmd kubectl; then
  echo "ERROR: kubectl is required but not found in PATH." >&2
  exit 1
fi

if ! have_cmd jq; then
  echo "ERROR: jq is required but not found in PATH." >&2
  exit 1
fi

# Ensure we can talk to the cluster
if ! kubectl version --short >/dev/null 2>&1; then
  echo "ERROR: kubectl cannot connect to the cluster. Check kubeconfig/context." >&2
  exit 1
fi

# ----------------------------
# Discovery
# ----------------------------

echo "Discovering non-system namespaces..."
NAMESPACES=$(kubectl get ns -o json | jq -r '
  .items[]
  | select(.metadata.name as $n
           | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  .metadata.name
')

if [ -z "${NAMESPACES}" ]; then
  echo "No non-system namespaces found; nothing to do."
  exit 0
fi

echo "Target namespaces:"
echo "${NAMESPACES}" | sed 's/^/  - /'

# ----------------------------
# Patching logic
# ----------------------------

# For workload controllers with pod templates (Deployment, StatefulSet, DaemonSet, Job, CronJob)
patch_pod_template() {
  local ns="$1"
  local kind="$2"
  local name="$3"

  # Build a JSON patch that forces readOnlyRootFilesystem=true on all containers and initContainers.
  # This is idempotent: setting the same value repeatedly is safe.
  read -r -d '' PATCH <<'EOF' || true
{
  "spec": {
    "template": {
      "spec": {
        "containers": (
          .spec.template.spec.containers // [] |
          map(
            .securityContext = ((.securityContext // {}) + {"readOnlyRootFilesystem": true})
          )
        ),
        "initContainers": (
          .spec.template.spec.initContainers // [] |
          map(
            .securityContext = ((.securityContext // {}) + {"readOnlyRootFilesystem": true})
          )
        )
      }
    }
  }
}
EOF

  # Apply server-side patch by constructing JSON via jq against the live object
  echo "Patching ${kind}/${ns}/${name} pod template..."
  kubectl get "$kind" "$name" -n "$ns" -o json \
    | jq "${PATCH}" \
    | kubectl apply -f -
}

# For standalone Pods (not controlled by a higher-level object)
patch_pod() {
  local ns="$1"
  local name="$2"

  read -r -d '' PATCH <<'EOF' || true
{
  "spec": {
    "containers": (
      .spec.containers // [] |
      map(
        .securityContext = ((.securityContext // {}) + {"readOnlyRootFilesystem": true})
      )
    ),
    "initContainers": (
      .spec.initContainers // [] |
      map(
        .securityContext = ((.securityContext // {}) + {"readOnlyRootFilesystem": true})
      )
    )
  }
}
EOF

  echo "Patching Pod/${ns}/${name}..."
  kubectl get pod "$name" -n "$ns" -o json \
    | jq "${PATCH}" \
    | kubectl apply -f -
}

# ----------------------------
# Main loop: patch controllers
# ----------------------------

for ns in ${NAMESPACES}; do
  echo "Processing namespace: ${ns}"

  # Deployments
  for name in $(kubectl get deploy -n "${ns}" -o jsonpath='{.items[*].metadata.name}' 2>/dev/null || true); do
    [ -z "$name" ] && continue
    patch_pod_template "${ns}" "deployment" "${name}"
  done

  # StatefulSets
  for name in $(kubectl get statefulset -n "${ns}" -o jsonpath='{.items[*].metadata.name}' 2>/dev/null || true); do
    [ -z "$name" ] && continue
    patch_pod_template "${ns}" "statefulset" "${name}"
  done

  # DaemonSets
  for name in $(kubectl get daemonset -n "${ns}" -o jsonpath='{.items[*].metadata.name}' 2>/dev/null || true); do
    [ -z "$name" ] && continue
    patch_pod_template "${ns}" "daemonset" "${name}"
  done

  # Jobs
  for name in $(kubectl get job -n "${ns}" -o jsonpath='{.items[*].metadata.name}' 2>/dev/null || true); do
    [ -z "$name" ] && continue
    patch_pod_template "${ns}" "job" "${name}"
  done

  # CronJobs (v1)
  for name in $(kubectl get cronjob -n "${ns}" -o jsonpath='{.items[*].metadata.name}' 2>/dev/null || true); do
    [ -z "$name" ] && continue
    # For CronJobs, the pod template is under spec.jobTemplate.spec.template
    read -r -d '' PATCH_CJ <<'EOF' || true
{
  "spec": {
    "jobTemplate": {
      "spec": {
        "template": {
          "spec": {
            "containers": (
              .spec.jobTemplate.spec.template.spec.containers // [] |
              map(
                .securityContext = ((.securityContext // {}) + {"readOnlyRootFilesystem": true})
              )
            ),
            "initContainers": (
              .spec.jobTemplate.spec.template.spec.initContainers // [] |
              map(
                .securityContext = ((.securityContext // {}) + {"readOnlyRootFilesystem": true})
              )
            )
          }
        }
      }
    }
  }
}
EOF
    echo "Patching cronjob/${ns}/${name} pod template..."
    kubectl get cronjob "$name" -n "$ns" -o json \
      | jq "${PATCH_CJ}" \
      | kubectl apply -f -
  done

  # Standalone Pods (those without a controller ownerReference)
  PODS=$(kubectl get pods -n "${ns}" -o json | jq -r '
    .items[]
    | select((.metadata.ownerReferences // []) | map(select(.controller == true)) | length == 0)
    | .metadata.name
  ')
  for name in ${PODS}; do
    [ -z "$name" ] && continue
    patch_pod "${ns}" "${name}"
  done
done

# ----------------------------
# Verification
# ----------------------------

echo
echo "Verification: running the benchmark audit command..."
kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
  | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | .metadata as $m
  | (.spec.nodeName // "") as $node
  | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
  | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
  | ((.spec.containers // []) + (.spec.initContainers // []))[]
  | (.securityContext.readOnlyRootFilesystem == true) as $ok
  | "kind=Pod ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=v1"
    + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
    + (if $node   == ""   then "" else " node=\($node)" end)
    + (if $labels == ""   then "" else " labels=\($labels)" end)
    + (if $own    == null then "" else " owner=\($own.kind)/\($m.namespace)/\($own.name)/\($own.uid)" end)
    + " container=\(.name) image=\(.image)"
    + " readOnlyRootFilesystem=\(if .securityContext.readOnlyRootFilesystem == null then "unset" else .securityContext.readOnlyRootFilesystem end)"
    + " is_compliant=\(if $ok then "true" else "false" end)"
  ] as $rows
  | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end'