Secrets in Container Environment Variables

More Info:

Ensure ECS task definition log configuration is enabled. This rule checks if secrets are passed as container environment variables in Amazon ECS task definitions. It marks the rule as non-compliant if one or more environment variable keys match a key listed in the secretKeys parameter .

Risk Level

Low

Address

Reliability, Security

Compliance Standards

• APRA CPS 234 (Australia)
• AWS Startup Security Baseline
• BSI C5 (Germany)
• Brazil LGPD
• CCPA / CPRA (California)
• CIS Critical Security Controls v8
• CMMC 2.0
• CSA Cloud Controls Matrix v4
• Cloudanix Best Practice
• DPDPA
• Digital Operational Resilience Act (EU)
• Essential 8
• ISO/IEC 27017
• ISO/IEC 27018
• ISO/IEC 27701
• KSA PDPL
• MAS Technology Risk Management (Singapore)
• MITRE ATT&CK (Cloud)
• NIST SP 800-171
• NYDFS 23 NYCRR 500
• SWIFT Customer Security Controls Framework
• Sarbanes-Oxley IT General Controls
• UK NCSC Cyber Assessment Framework

Triage and Remediation

Remediation

Remediation

Using Console

To remediate secrets being exposed in container environment variables in AWS Kubernetes using the AWS console, follow these steps:

1. Identify the exposed secrets: First, identify which secrets are being exposed in the container environment variables. You can do this by examining the Kubernetes deployment configuration or inspecting the running pods.

2. Store secrets securely: AWS provides a service called AWS Secrets Manager that allows you to store, retrieve, and manage sensitive data such as passwords, API keys, and other secrets. Store the secrets in AWS Secrets Manager instead of hardcoding them in the container environment variables.

3. Update Kubernetes deployment configuration:

   • Open the AWS Management Console and navigate to the Amazon EKS console.
   • Select your EKS cluster and navigate to the “Workloads” section.
   • Find the deployment that contains the exposed secrets and click on it to view the details.
   • Update the deployment configuration to fetch the secrets from AWS Secrets Manager instead of using them directly as environment variables.

4. Use AWS IAM roles for service accounts: Instead of directly accessing AWS Secrets Manager from your application code, you can use IAM roles for service accounts (IRSA) to securely provide AWS permissions to your Kubernetes pods. This way, your pods can access AWS services securely without needing to store AWS credentials or secrets in the container environment variables.

5. Monitor and audit: Regularly monitor and audit your Kubernetes clusters for any exposed secrets or misconfigurations. Set up alerts and notifications to detect any unauthorized access or changes to your secrets.

By following these steps, you can remediate secrets being exposed in container environment variables in AWS Kubernetes and ensure that your sensitive data is stored and accessed securely.

Using CLI

To remediate the issue of storing secrets in container environment variables in AWS Kubernetes using AWS CLI, follow these steps:

1. Create a Secret in AWS Secrets Manager:

   • Use the AWS CLI to create a secret in AWS Secrets Manager that will store the sensitive information securely. For example, you can create a secret named my-secret with key-value pairs for your sensitive data.
   aws secretsmanager create-secret --name my-secret --secret-string '{"username":"my_username", "password":"my_password"}'
   

2. Update Kubernetes Deployment YAML:

   • Modify your Kubernetes deployment YAML file to reference the secret stored in AWS Secrets Manager. Replace the environment variables containing sensitive data with references to the secret.
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: my-deployment
   spec:
     template:
       spec:
         containers:
         - name: my-container
           image: my-image
           env:
           - name: USERNAME
             valueFrom:
               secretKeyRef:
                 name: my-secret
                 key: username
           - name: PASSWORD
             valueFrom:
               secretKeyRef:
                 name: my-secret
                 key: password
   

3. Apply the Changes:

   • Apply the updated deployment YAML file to your Kubernetes cluster using kubectl.
   kubectl apply -f deployment.yaml
   

4. Verify the Deployment:

   • Verify that the deployment has been updated successfully and the sensitive data is now being sourced from the AWS Secrets Manager secret.
   kubectl get pods
   kubectl describe pod <pod_name>
   

By following these steps, you can remediate the issue of storing secrets in container environment variables in AWS Kubernetes by securely storing sensitive information in AWS Secrets Manager and referencing them in your Kubernetes deployment configuration.

Using Python

To remediate the issue of storing secrets in container environment variables in AWS Kubernetes using Python, you can follow these steps:

1. Store Secrets in AWS Secrets Manager:

   • Store your sensitive information such as API keys, passwords, etc., securely in AWS Secrets Manager.

2. Grant Access to Secrets:

   • Ensure that the Kubernetes service account has the necessary permissions to access the secrets stored in AWS Secrets Manager.

3. Install AWS SDK for Python (Boto3):

   • Install the Boto3 library in your Python environment by running the following command:
     pip install boto3
     

4. Retrieve Secrets in Python:

   • Write a Python script to retrieve the secrets from AWS Secrets Manager. Here is an example script:
     import boto3
     
     def get_secret(secret_name):
         client = boto3.client('secretsmanager')
         response = client.get_secret_value(SecretId=secret_name)
         secret = response['SecretString']
         return secret
     
     # Example usage
     secret_value = get_secret('your_secret_name')
     

5. Inject Secrets into Kubernetes Pods:

   • Use Kubernetes Secrets to inject the retrieved secrets into your application pods. You can create a Kubernetes Secret object using the retrieved secret value in your Python script.

6. Update Kubernetes Deployment:

   • Update your Kubernetes Deployment manifest to mount the secret as a volume or set it as an environment variable in your application container.

7. Deploy the Updated Application:

   • Deploy the updated application to your AWS EKS cluster using kubectl or any other deployment tool.

By following these steps, you can remediate the issue of storing secrets in container environment variables in AWS Kubernetes using Python and securely manage your application’s sensitive information.

Using Terraform

# Store the sensitive value in a Kubernetes Secret instead of a plain env var
resource "kubernetes_secret" "app_secrets" {
  metadata {
    name      = "APP_SECRETS_NAME"          # replace with your Secret name
    namespace = "APP_NAMESPACE"             # replace with your namespace
  }

  data = {
    DB_PASSWORD = "BASE64_ENCODED_PASSWORD" # replace with base64-encoded value
    # add more keys that match your secretKeys threshold list as needed
  }

  type = "Opaque"
}

# Use the Secret in the Pod spec instead of passing the secret directly in env
resource "kubernetes_deployment" "app" {
  metadata {
    name      = "APP_DEPLOYMENT_NAME"       # replace with your Deployment name
    namespace = "APP_NAMESPACE"             # must match namespace above
    labels = {
      app = "APP_LABEL"                     # replace with your app label
    }
  }

  spec {
    replicas = 1

    selector {
      match_labels = {
        app = "APP_LABEL"
      }
    }

    template {
      metadata {
        labels = {
          app = "APP_LABEL"
        }
      }

      spec {
        container {
          name  = "APP_CONTAINER_NAME"       # replace with your container name
          image = "APP_IMAGE"               # replace with your image

          # DO NOT use: env { name = "DB_PASSWORD" value = "plaintext" }
          # Instead, source from the Secret for any key in your secretKeys list
          env {
            name = "DB_PASSWORD"
            value_from {
              secret_key_ref {
                name = kubernetes_secret.app_secrets.metadata[0].name
                key  = "DB_PASSWORD"
              }
            }
          }

          # add more env blocks with value_from.secret_key_ref for each secret key
        }
      }
    }
  }
}

This moves sensitive values (those whose env var names match your secretKeys list) out of plain environment variables and into Kubernetes Secrets, which the containers consume via env.value_from.secret_key_ref.

This change is an in-place update of the Deployment; the Pods will be recreated but the Deployment resource itself is not replaced.

To verify, terraform plan should show:

• A kubernetes_secret being created (or updated if it already exists).
• A change to kubernetes_deployment.app.spec.template.spec.container[*].env replacing value with value_from.secret_key_ref for the secret keys, and no remaining plaintext value fields for those keys.