Incident Report

Summary

On 7 February 2024, multiple Google Cloud services experienced a partial zonal service outage in the europe-west8-b zone for a duration of 1 hour, 8 minutes. To our customers whose services were impacted during this service outage, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.

Root Cause

The europe-west8 region recently received a networking upgrade to increase capacity and improve resilience. In order to minimize the risk, Google executes these as "make-before-break," adding new capacity before decommissioning old capacity. The final step of this upgrade was to remove now-unused fiber connections between zones in the region. Deployment automation creates work orders for onsite technicians to remove unused fiber cabling from network devices and fiber patch panels.

On 7 February 2024, between 02:21 and 02:46 US/Pacific, onsite technicians performing this planned network maintenance inadvertently unplugged several fibers that were adjacent to those in the work order, but still in use for production traffic. As a result, a portion of the europe-west8-b zone unintentionally became isolated from a portion of the backbone network at 02:46 US/Pacific.

Remediation and Prevention

Google engineers were alerted to the partial outage via internal monitoring on 7 February 2024, 02:56 US/Pacific and immediately started an investigation. Once the nature and scope of the issue became clear, Google engineers began reverting the fiber changes and restoring network capacity at 03:52 US/Pacific.

Sufficient capacity to serve customer traffic was restored by 03:54 US/Pacific, mitigating impact to the affected products. Full capacity was restored by 04:07 US/Pacific.

Google is committed to preventing recurrence of this incident. The following actions have been identified:

• Google engineers have paused all work of this kind globally, starting on 8 February 2024. This pause will remain in effect until the actions below have been implemented to reduce the risk of recurrence.
• Complete the rollout of an enhanced physical work safety program, which includes updates to the current process for execution of planned work related to interfaces or devices serving customer traffic. The following action items within that program are relevant to this incident:
  • Creation of an automated notification system for the start / end of planned work related to interfaces or devices serving customer traffic.
  • Division of planned work into execution batches.
  • Require operational team supervision and monitoring for planned work.
  • Add multi-step verification of traffic status before fibers are disconnected.
  • Include additional controls as part of working procedures for the execution of critical tasks to ensure compliance with documented processes.

Detailed Description of Impact

On 7 February 2024, from 02:46 to 03:54 US/Pacific, multiple Google Cloud services experienced a partial zonal service outage for europe-west8-b. Affected services included:

Google Kubernetes Engine

• GKE clusters in europe-west8-b were unavailable.
• Customers may also have experienced failures when attempting to create, delete, or modify VMs in the affected zone.

Cloud Key Management Service (KMS)

• Cloud KMS experienced a partial zonal outage for services in europe-west8-b, including Hardware Security Module (HSM), External Key Manager (EKM), Secret Manager, and Private CA.

Google Cloud Bigtable

• Customers experienced a service outage with non-high availability instances in europe-west8-b for the duration of the outage.

Virtual Private Cloud (VPC)

• VPC customers may have experienced increased packet loss in the affected zone.

Google Cloud Deploy

• Customers in europe-west8-b would have experienced errors creating rollouts and releases.

Google Compute Engine

• VMs in a subset of europe-west8-b were unreachable for the duration of the outage.
• VM creations and deletions in europe-west8-c started failing as services failed over to that zone.

Persistent Disk (PD)

• PD devices in a subset of europe-west8-b would have been unavailable for the duration of the outage.
• PD services related to snapshots, creation of new disks, and image creation for the affected zone would have experienced failures.
• A small number of VMs with Persistent Disks in a different zone within the region saw guest errors.

Google Cloud Networking

• Cloud NAT, Cloud Interconnect, Cloud VPN, Cloud VR were unavailable for europe-west8-b.
• Cloud Network programming was delayed for all customers in the europe-west8-b zone.

Service Directory

• Customers in europe-west8-b experienced read and write errors for the duration of the outage.

Traffic Director

• Stale configurations and load balancing assignments for all customers in europe-west8-b. This would have appeared as configuration updates not propagating and load balancing assignments not reacting to changes in load.
• All newly restarted clients would have been unable to load configuration and receive load balancing assignments.

Cloud Load Balancing

• Approximately 50% of load balancers/target pools in europe-west8-b were unreachable.
• Customers with Load Balancers configured in this zone would find them inconsistently available. If no Load Balancers were configured and available in another zone or region, requests to the customer's project would result in 500 errors.

Google Cloud DNS

• Customers in europe-west8-b would have been unable to write new DNS records for the duration of the outage.
• Cloud DNS public name servers were unreachable from europe-west8-b during the outage, and intermittently unreachable from other zones in the region as a result of throttling as traffic moved from europe-west8-b to other zones in region

Memorystore for Redis

• Cloud Redis Standalone instances in europe-west8-b were unavailable for the duration of the outage.

Cloud BigQuery

• A small number of projects (one customer) may have experienced errors for API calls for a period of approximately 15 minutes for the tabledata.insertAll API. A very low error rate might have been present for jobs.insert and jobs.query operations as well, but these were mitigated much quicker through automatic recovery mechanisms.

Cloud Dataflow

• A small number of customer projects may have experienced stuck streaming jobs in europe-west8-b during the duration of the incident.

Cloud Build

• Builds would have terminated with status “INTERNAL_ERROR” for approximately 20% of builds in europe-west8 for the first approximately 20 minutes of the outage. Intra-region failover healed the user impact thereafter.

Cloud SQL

• A small number of instance create operations failed in the europe-west8 region.
• Existing HA instances were moved to a healthy zone to restore connectivity.
• Existing Zonal instances in europe-west8-b would have remained unavailable for the duration of the outage.

Cloud Armor

• Propagation of updates to Cloud Armor Security policies in Cloud Console stalled globally.

Cloud Filestore

• Up to 100% error rates for CreateBackup, CreateInstance, CreateSnapshot, and DeleteInstance operations for europe-west8-b zone.
• Zonal instances in the affected zone were unavailable.

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support.

(All Times US/Pacific)

Incident Start: 7 Feb 2024 02:46

Incident End: 7 Feb 2024 03:54

Duration: 1 hour, 8 minutes

Affected Services and Features:

Google Kubernetes Engine Cloud Key Management Service Google BigQuery Google Cloud Bigtable Virtual Private Cloud (VPC) Google Cloud Deploy Google Compute Engine Persistent Disk Google Cloud Networking Service Directory Traffic Director Cloud Load Balancing Google Cloud DNS Memorystore for Redis Cloud Dataflow Cloud Build Cloud SQL Cloud Armor Cloud Filestore

Regions/Zones: Europe-west8-b

Description:

Multiple Google Cloud products experienced service unavailability for 1 hour and 8 minutes in europe-west8-b. The preliminary root cause appears to be a network decommissioning maintenance activity that was not executed as planned. Google will complete a full Incident Report in the following days that will provide a detailed root cause.

Customer Impact:

During the impact timeframe: Most customer services using this zone were unavailable.

Summary: Multiple product outage in europe-west8-b

The multi product outage in europe-west8-b has been resolved for all affected users as of Wednesday, 2024-02-07 05:02 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

Summary: Multiple product outage in europe-west8-b

Mitigation work is currently underway by our engineering team.. The mitigation is expected to complete by Wednesday, 2024-02-07 05:15 US/Pacific.

We will provide an update by Wednesday, 2024-02-07 05:15 US/Pacific with current details.

Diagnosis: Persistent Disk Customers are unable to their services in europe-west8-b.

Workaround: None at this time

Summary: Multiple product outage in europe-west8-b

Mitigation work is currently underway by our engineering team.. The mitigation is expected to complete by Wednesday, 2024-02-07 05:15 US/Pacific.

We will provide an update by Wednesday, 2024-02-07 05:00 US/Pacific with current details.

Diagnosis: Persistent Disk Customers are unable to their services in europe-west8-b.

Workaround: None at this time

Summary: Multiple product outage in europe-west8-b

Description: We are experiencing an issue with Persistent Disk, Google Cloud Networking, Service Directory, Traffic Director, Cloud Load Balancing, Google Cloud DNS, Cloud Memorystore, Cloud Dataflow, Cloud Build, Cloud SQL, Cloud Filestore beginning on Wednesday, 2024-02-07 02:46 US/Pacific.

Other products are likely impacted. Mitigation work is currently underway by our engineering team.. The mitigation is expected to complete by Wednesday, 2024-02-07 05:00 US/Pacific.

We will provide an update by Wednesday, 2024-02-07 05:00 US/Pacific with current details.

Diagnosis: Customers are unable to reach any of the impacted Google Cloud products in europe-west8-b.

Workaround: None at this time

Summary: Multiple product outage in europe-west8-b

Description: We are experiencing an issue with Google Kubernetes Engine, Cloud Key Management Service, Google BigQuery, Google Cloud Bigtable, Virtual Private Cloud (VPC), Google Cloud Deploy, Google Compute Engine, Persistent Disk, Google Cloud Networking, Service Directory, Traffic Director, Cloud Load Balancing, Google Cloud DNS, Cloud Memorystore, Cloud Dataflow, Cloud Build, Cloud SQL, Cloud Filestore, Identity and Access Management beginning on Wednesday, 2024-02-07 02:46 US/Pacific.

Other products are likely impacted. Mitigation work is currently underway by our engineering team. The mitigation is expected to complete by Wednesday, 2024-02-07 04:45 US/Pacific.

We will provide an update by Wednesday, 2024-02-07 04:45 US/Pacific with current details.

Diagnosis: Customers are unable to reach any of the impacted Google Cloud products in europe-west8-b.

Workaround: None at this time

Summary: Multiple product outage in europe-west8-b

Description: We are experiencing an issue with Google Kubernetes Engine, Cloud Key Management Service, Google BigQuery, Google Cloud Bigtable, Virtual Private Cloud (VPC), Google Cloud Deploy, Google Compute Engine, Persistent Disk, Google Cloud Networking, Service Directory, Traffic Director, Cloud Load Balancing, Google Cloud DNS, Cloud Memorystore, Cloud Dataflow, Cloud Build, Cloud SQL, Cloud Filestore, Identity and Access Management beginning on Wednesday, 2024-02-07 02:46 US/Pacific.

Other products are likely impacted. Mitigation work is currently underway by our engineering team. The mitigation is expected to complete by Wednesday, 2024-02-07 04:45 US/Pacific.

We will provide an update by Wednesday, 2024-02-07 04:45 US/Pacific with current details.

Diagnosis: Customers are unable to reach any of the impacted Google Cloud products in europe-west8-b.

Workaround: None at this time.

Summary: Multiple product outage in europe-west8-b

Description: We are experiencing an issue with Google Kubernetes Engine, Cloud Key Management Service, Google BigQuery, Google Cloud Bigtable, Virtual Private Cloud (VPC), Google Cloud Deploy, Google Compute Engine, Persistent Disk, Google Cloud Networking, Service Directory, Traffic Director, Cloud Load Balancing, Google Cloud DNS, Cloud Memorystore, Cloud Dataflow, Cloud Build, Cloud SQL, Cloud Filestore, Identity and Access Management beginning on Wednesday, 2024-02-07 02:46 US/Pacific.

Other products are likely impacted. Mitigation work is currently underway by our engineering team. The mitigation is expected to complete by Wednesday, 2024-02-07 04:30 US/Pacific.

We will provide an update by Wednesday, 2024-02-07 04:30 US/Pacific with current details.

Diagnosis: Customers are unable to reach any of the impacted Google Cloud products in europe-west8-b.

Workaround: None at this time.

Summary: Multiple product outage in europe-west8-b

Description: We are experiencing an issue with Google Kubernetes Engine, Cloud Key Management Service, Google BigQuery, Google Cloud Bigtable, Virtual Private Cloud (VPC), Google Cloud Deploy, Google Compute Engine, Persistent Disk, Google Cloud Networking, Service Directory, Traffic Director, Cloud Load Balancing, Google Cloud DNS, Cloud Memorystore, Cloud Dataflow, Cloud Build, Cloud SQL, Cloud Filestore beginning on Wednesday, 2024-02-07 02:46 US/Pacific.

Other products are likely impacted. Mitigation work is currently underway by our engineering team. The mitigation is expected to complete by Wednesday, 2024-02-07 04:30 US/Pacific.

We will provide an update by Wednesday, 2024-02-07 04:30 US/Pacific with current details.

Diagnosis: Customers are unable to reach any of the impacted Google Cloud products in europe-west8-b.

Workaround: None at this time.

Summary: Multiple product outage in europe-west8-b

Description: We are experiencing an issue with Google Kubernetes Engine, Cloud Key Management Service, Google BigQuery, Google Cloud Bigtable, Virtual Private Cloud (VPC), Google Cloud Deploy, Google Compute Engine, Persistent Disk, Google Cloud Networking, Service Directory, Traffic Director, Cloud Load Balancing, Google Cloud DNS, Cloud Memorystore, Cloud Dataflow, Cloud Build, Cloud SQLbeginning on Wednesday, 2024-02-07 02:46 US/Pacific.

Other products are likely impacted. Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point.

We will provide an update by Wednesday, 2024-02-07 04:30 US/Pacific with current details.

Diagnosis: Customers are unable to reach any of the impacted Google Cloud products in europe-west8-b.

Workaround: None at this time.

Summary: Multiple product outage in europe-west8-b

Description: We are experiencing an issue with Cloud Key Management Service, Google BigQuery, Google Cloud Bigtable, Virtual Private Cloud (VPC), Google Cloud Deploy, Google Compute Engine, Persistent Disk, Google Cloud Networking, Service Directory, Traffic Director, Cloud Load Balancing, Google Cloud DNS, Cloud Memorystore, Cloud Dataflow, Cloud Build beginning on Wednesday, 2024-02-07 02:46 US/Pacific.

Other products are likely impacted. Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point.

We will provide an update by Wednesday, 2024-02-07 04:15 US/Pacific with current details.

Diagnosis: Customers are unable to reach any of the impacted Google Cloud products in europe-west8-b.

Workaround: None at this time.

Summary: Multiple product outage in europe-west8-b

Description: We are experiencing an issue with Cloud Key Management Service, Google BigQuery, Google Cloud Bigtable, Virtual Private Cloud (VPC), Google Cloud Deploy, Google Compute Engine, Persistent Disk, Google Cloud Networking, Service Directory, Traffic Director, Cloud Load Balancing, Google Cloud DNS, Cloud Memorystore, Cloud Dataflow beginning on Wednesday, 2024-02-07 02:46 US/Pacific.

Other products are likely impacted. Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point.

We will provide an update by Wednesday, 2024-02-07 04:15 US/Pacific with current details.

Diagnosis: Customers are unable to reach any of the impacted Google Cloud products in europe-west8-b.

Workaround: None at this time.

Summary: Multiple product outage in europe-west8-b

Description: We are experiencing an issue with Cloud Key Management Service, Google BigQuery, Google Cloud Bigtable, Virtual Private Cloud (VPC), Google Cloud Deploy, Google Compute Engine, Persistent Disk, Google Cloud Networking, Service Directory, Traffic Director, Cloud Load Balancing, Google Cloud DNS, Cloud Memorystore beginning on Wednesday, 2024-02-07 02:46 US/Pacific.

Other products are likely impacted. Our engineering team continues to investigate the issue and identify affected products and services.

We will provide an update by Wednesday, 2024-02-07 04:15 US/Pacific with current details.

Diagnosis: Customers are unable to reach any of the impacted Google Cloud products in europe-west8-b.

Workaround: None at this time.

Summary: Multiple product outage in europe-west8-b

Description: We are experiencing an issue with Cloud Key Management Service, Google BigQuery, Google Cloud Bigtable, Virtual Private Cloud (VPC), Google Cloud Deploy, Google Compute Engine, Persistent Disk, Google Cloud Networking, Service Directory, Traffic Director, Cloud Load Balancing, Google Cloud DNS, Cloud Memorystore beginning on Wednesday, 2024-02-07 02:46 US/Pacific.

Other products are likely impacted. Our engineering team continues to investigate the issue and identify affected products and services.

We will provide an update by Wednesday, 2024-02-07 04:15 US/Pacific with current details.

Diagnosis: Most customers will find all VMs in europe-west8-b will be unreachable.

Workaround: None at this time.

Summary: Multiple product outage in europe-west8-b

Description: We are experiencing an issue with Virtual Private Cloud (VPC) beginning on Wednesday, 2024-02-07 02:46 US/Pacific.

Other products are likely impacted. Our engineering team continues to investigate the issue and identify affected products and services.

We will provide an update by Wednesday, 2024-02-07 04:00 US/Pacific with current details.

Diagnosis: Most customers will find all VMs in europe-west8-b will be unreachable.

Workaround: None at this time.