Cloud CDN Global User Created Backend Services Failover Policy Should Be Enabled

More Info:

Ensure Cloud CDN global user created backend services have failover policy enabled.

Risk Level

High

Address

Operational Maturity, Performance Efficiency, Reliability, Security

Compliance Standards

• CIS GCP

Triage and Remediation

Remediation

Remediation

Using Console

Below are concise, GCP‑Console–only steps to enable a failover policy on a Cloud CDN backend service (for a global external HTTP(S) load balancer).

1. Open the backend service for your CDN

1. In the Google Cloud Console, go to:
   Navigation menu → Network services → Load balancing
2. Make sure you’re in the right project and on the HTTP(S) Load Balancing tab.
3. Find and click the global external HTTP(S) load balancer that uses Cloud CDN.
4. On the load balancer detail page, go to the Backends tab.
5. Under Backend services, click the backend service that has Cloud CDN enabled (name will be something like backend-service-...).

2. Edit the backend service

1. On the backend service details page, click Edit at the top.
2. Confirm Cloud CDN is enabled (toggle on). If not, enable it if required by your policy.

3. Configure failover on the backend(s)

Failover is configured at the backend entry inside the backend service:

1. In the Backends section of the edit page, locate your primary backend (e.g., a MIG, NEG, or bucket backend).
2. If you already have a backup backend listed:
   • Click the pencil (edit) icon next to the backup backend.
   • Check Enable failover (or Use as failover / similar wording depending on UI version).
   • Set the Failover ratio (e.g., 0.8), which defines when traffic starts failing over based on healthy capacity.
   • Click Save for that backend entry.
3. If you do not have a backup backend configured yet:
   • Click Add backend.
   • Choose the backend type (e.g., instance group, NEG, or Cloud Storage bucket) you want to use as the failover target.
   • Set Capacity as appropriate (often 1.0 if it’s a full backup).
   • Check Use for failover only (or Enable failover depending on UI wording).
   • Optionally set Failover ratio.
   • Click Done or Save for that backend row.

4. Save the backend service

1. After configuring the primary and failover backends, scroll to the bottom and click Save on the backend service.
2. Wait for the configuration to propagate (usually a few minutes).

5. Verify the failover policy

1. Go back to the Backends tab of the load balancer.
2. Open the backend service again and verify:
   • You see both primary and failover backends.
   • The failover flag is turned on for the backup backend.
   • The Failover ratio is set as required by your policy.

This enables a failover policy for your global Cloud CDN–fronted backend service using the GCP Console.

Using CLI

Below is how to enable / configure the failover policy on a global backend service used by Cloud CDN using gcloud.

---

1. Identify the affected backend service(s)

# List all global backend services
gcloud compute backend-services list --global

# Inspect one backend service to confirm it's used by Cloud CDN and see existing config
gcloud compute backend-services describe BACKEND_SERVICE_NAME --global

Look for:

• cdnPolicy → confirms Cloud CDN is enabled
• backends → ensure you have at least one failover backend (failover: true) defined, or plan to add one
• failoverPolicy → will be missing or incorrectly set today

---

2. Ensure primary and failover backends exist

If you don’t yet have a failover backend configured, you must add one first (same backend service, different backend entry with failover: true):

# Example: add a failover backend instance group or NEG
gcloud compute backend-services add-backend BACKEND_SERVICE_NAME \
  --global \
  --instance-group FAILOVER_INSTANCE_GROUP_NAME \
  --instance-group-zone FAILOVER_INSTANCE_GROUP_ZONE \
  --balancing-mode UTILIZATION \
  --max-utilization 0.8 \
  --failover

Adjust flags for your resource type (e.g. --network-endpoint-group / --global-network-endpoint-group instead of --instance-group if using NEGs).

---

3. Enable / set the failover policy

Use --failover-policy on the backend service:

gcloud compute backend-services update BACKEND_SERVICE_NAME \
  --global \
  --failover-policy=disableConnectionDrainOnFailover=true,dropTrafficIfUnhealthy=true

• disableConnectionDrainOnFailover=true
  New connections switch immediately to the failover backend (no draining delay).
• dropTrafficIfUnhealthy=true
  If both primary and failover backends are unhealthy, traffic is dropped instead of being sent to unhealthy backends.

If you want different behavior, adjust the values:

# Example: keep connection draining but still drop traffic when both sides unhealthy
gcloud compute backend-services update BACKEND_SERVICE_NAME \
  --global \
  --failover-policy=disableConnectionDrainOnFailover=false,dropTrafficIfUnhealthy=true

---

4. Verify the configuration

gcloud compute backend-services describe BACKEND_SERVICE_NAME --global \
  --format="flattened(failoverPolicy,backends)"

You should see something like:

failoverPolicy.disableConnectionDrainOnFailover: true
failoverPolicy.dropTrafficIfUnhealthy: true
backends:
- ...
- failover: true
  ...

That completes enabling the failover policy for a global Cloud CDN backend service via gcloud.

Using Python

Below is a concise, step‑by‑step way to enable failover policy for global Cloud CDN backend services in GCP using Python.

---

1. Prerequisites

1. Install the Python client:
   pip install google-cloud-compute
   
2. Authenticate (one of):
   gcloud auth application-default login
   
3. Ensure your account has roles such as:
   • roles/compute.admin (or equivalent permissions on backend services).

---

2. What You Need to Change

For global HTTP(S)/SSL/TCP proxy backend services, the BackendService resource supports:

"failoverPolicy": {
  "disableConnectionDrainOnFailover": false,
  "dropTrafficIfUnhealthy": true,
  "failoverRatio": 0.5
}

Your goal is to set failoverPolicy for each Cloud CDN backend service that doesn’t have it.

Note: This only applies to global backend services (not regional), typically used by global HTTP(S) Load Balancers.

---

3. Python Code – Enable Failover Policy for a Single Backend Service

from google.cloud import compute_v1

PROJECT_ID = "YOUR_PROJECT_ID"
BACKEND_SERVICE_NAME = "YOUR_GLOBAL_BACKEND_SERVICE_NAME"

def enable_failover_policy_on_backend_service(
    project_id: str,
    backend_service_name: str,
    failover_ratio: float = 0.5,
    drop_traffic_if_unhealthy: bool = True,
    disable_connection_drain_on_failover: bool = False,
):
    client = compute_v1.BackendServicesClient()

    # Get existing backend service (global)
    backend_service = client.get(
        project=project_id,
        backend_service=backend_service_name,
    )

    # If already has failoverPolicy, you can choose to skip or update
    if backend_service.failover_policy is None:
        backend_service.failover_policy = compute_v1.BackendServiceFailoverPolicy(
            failover_ratio=failover_ratio,
            drop_traffic_if_unhealthy=drop_traffic_if_unhealthy,
            disable_connection_drain_on_failover=disable_connection_drain_on_failover,
        )
    else:
        # Update existing policy as needed
        backend_service.failover_policy.failover_ratio = failover_ratio
        backend_service.failover_policy.drop_traffic_if_unhealthy = drop_traffic_if_unhealthy
        backend_service.failover_policy.disable_connection_drain_on_failover = (
            disable_connection_drain_on_failover
        )

    # Perform PATCH (global backend service)
    operation = client.patch(
        project=project_id,
        backend_service=backend_service_name,
        backend_service_resource=backend_service,
    )

    # Wait for operation to complete (optional but recommended)
    op_client = compute_v1.GlobalOperationsClient()
    op_client.wait(
        operation=operation.name,
        project=project_id,
    )
    print(f"Failover policy enabled/updated on backend service: {backend_service_name}")


if __name__ == "__main__":
    enable_failover_policy_on_backend_service(PROJECT_ID, BACKEND_SERVICE_NAME)

---

4. Optional: Automatically Fix All Global CDN Backend Services

To automatically remediate all global backend services with Cloud CDN enabled and no failoverPolicy, you can extend the script:

from google.cloud import compute_v1

PROJECT_ID = "YOUR_PROJECT_ID"

def remediate_all_global_cdn_backend_services(project_id: str):
    bs_client = compute_v1.BackendServicesClient()
    op_client = compute_v1.GlobalOperationsClient()

    # List all global backend services
    request = compute_v1.ListBackendServicesRequest(project=project_id)
    for bs in bs_client.list(request=request):
        # Only target global, CDN-enabled services without failoverPolicy
        if not getattr(bs, "cdn_policy", None):
            continue  # not a CDN backend
        if bs.failover_policy is not None:
            continue  # already has failoverPolicy

        print(f"Updating failoverPolicy for backend service: {bs.name}")

        bs.failover_policy = compute_v1.BackendServiceFailoverPolicy(
            failover_ratio=0.5,
            drop_traffic_if_unhealthy=True,
            disable_connection_drain_on_failover=False,
        )

        op = bs_client.patch(
            project=project_id,
            backend_service=bs.name,
            backend_service_resource=bs,
        )
        op_client.wait(operation=op.name, project=project_id)

    print("Remediation complete.")

if __name__ == "__main__":
    remediate_all_global_cdn_backend_services(PROJECT_ID)

---

5. Validate

1. In the Cloud Console:
   • Network services → Cloud CDN → Backend services
   • Open each backend service and check the Failover policy section.
2. Or via gcloud:
   gcloud compute backend-services describe YOUR_GLOBAL_BACKEND_SERVICE_NAME \
       --global --project=YOUR_PROJECT_ID \
       --format="get(failoverPolicy)"
   

If you share the exact policy your security tool expects (e.g., required failoverRatio), I can adjust the Python example to match it precisely.

Using Terraform

resource "google_compute_backend_service" "CDN_BACKEND_SERVICE" {
  name                  = "CDN_BACKEND_SERVICE_NAME"     # replace with your backend service name
  protocol              = "HTTP"
  load_balancing_scheme = "EXTERNAL_MANAGED"
  enable_cdn            = true

  # Primary backend (e.g., primary NEG or instance group)
  backend {
    group    = GOOGLE_COMPUTE_BACKEND_GROUP_SELF_LINK_PRIMARY   # replace with self_link of primary backend
    failover = false
  }

  # Failover backend (e.g., secondary NEG or instance group)
  backend {
    group    = GOOGLE_COMPUTE_BACKEND_GROUP_SELF_LINK_FAILOVER  # replace with self_link of failover backend
    failover = true
  }

  # Enable and configure the failover policy
  failover_policy {
    disable_connection_drain_on_failover = false
    drop_traffic_if_unhealthy            = true
    failover_ratio                       = 0.5   # adjust this threshold if your policy requires a different ratio
  }
}

This change does not normally force replacement of the backend service; it is applied in-place by updating the existing google_compute_backend_service.

After you add or update this block, terraform plan should show an in-place update (~) to google_compute_backend_service.CDN_BACKEND_SERVICE with a new or changed failover_policy block and (if you added the secondary backend) the additional backend entry.

Additional Reading:

• https://cloud.google.com/cdn/docs/overview