A faulty network configuration update disrupted inter-zone connectivity across three regions, temporarily disrupting VMware Engine stretched clusters.
A faulty configuration update on Google Cloud VMware Engine (GCVE) caused a multi-region disruption on Tuesday, disrupting inter-zone connectivity across three regions.
The incident, which lasted for over ten hours, began at 5:00 PM UTC on July 14 and was resolved by 04:46 AM UTC on July 15. It affected VMware Engine stretched clusters in Sydney (australia-southeast1), Melbourne (australia-southeast2), and Frankfurt (europe-west3).
Google later identified a recent network configuration update as the cause of the inter-zone network disruption and mitigated the issue by rolling back the faulty configuration to its last-known configuration.
Google traces the fault
The first status update, posted at 08:24 PM UTC on July 14, described the incident as a network connectivity issue affecting stretched clusters, while compute and storage services remained unaffected. At that time, GCVE VMs were running as expected, but the company acknowledged that customers may experience connectivity issues with the VMs.
But soon after, the preliminary investigation indicated that the issue could be stemming from an underlying network connectivity issue affecting the infrastructure that links the zones within a stretch cluster.
“This disruption is causing synchronization issues between the affected zones, and some GCVE customers using Stretched Cluster may experience inter-site communication failures to their GCVE environments within the affected zones,” Google Cloud said in a notification.
While the company was working on restoring full connectivity, Google advised moving workloads to the healthy side of the stretched cluster, where feasible, and only after consulting Google Support.
Less than two hours after the first update, Google Cloud identified underlying inter-zone communication failures and Border Gateway Protocol (BGP) session flapping between cluster zones. “Specifically, network connectivity has been lost between the affected zones and the witness appliance. Because the witness appliance is currently unreachable, the cluster zones are unable to safely synchronize state. As a result, VMs on the affected sites are becoming isolated and may be left without writable data,” noted the company.
And at 11:05 PM UTC, it posted that the investigation has identified a recent configuration update that is the likely cause of the inter-zone network disruption, and at 04:46 AM UTC on July 15, the engineering team mitigated the issue by rolling back the faulty configuration to its last-known good value.
“Google made a network setting change that accidentally broke the connection between the two data center zones in VMware Engine. The virtual machines themselves kept running fine, but nobody could reach them, and there was a risk that some machines might lose the ability to save data properly. This indicates that even managed cloud infrastructure can experience failures in critical shared network components,” said Pareekh Jain, CEO at EIIRTrend & Pareekh Consulting.
Neil Shah, vice president at Counterpoint Research, said the real culprit here is the SDN orchestration control plane, where a routine internal network update or configuration tweak introduced routing failure across multiple zones. “While most of the physical nodes are distributed for exactly this redundancy purpose, they are still tightly coupled to a singular shared orchestration fabric, so if that control plane crashes, then everything comes crashing down, and the physical distributed nodes become irrelevant.”
Stretched clusters fall short
Although the outage did not bring down virtual machines, the incident undermined the primary reason enterprises deploy stretched clusters.
“Stretched clusters are designed to keep applications running if one site fails. When the network connecting the two sites is disrupted, that resilience breaks down, leaving workloads inaccessible despite healthy compute and storage. The incident shows that network infrastructure can become a single point of failure,” highlighted Jain.
Jain noted companies use this setup specifically for their most important systems, the ones that can’t afford to go offline, like hospital records, banking systems, or company databases. A 12-hour outage on systems like that can mean lost money, missed deadlines, angry customers, and in some industries, legal or regulatory trouble.
Rethinking resilience
The incident also highlights that deploying stretched clusters alone does not eliminate dependency on the cloud provider’s underlying networking and control plane.
“If CIOs are looking to achieve absolute digital sovereignty, mission-critical production data must be decoupled from the automation layer. The asynchronous geo-separation with multi-cloud deployment could be a more viable strategy to avoid a single systematic point of failure,” added Shah.
Jain added that leaders should ask their cloud provider exactly what parts are shared versus separate, keep a true backup plan outside that same provider for their most critical systems, regularly test what happens if the provider’s systems fail, and make sure contracts account for compensation if this happens again.
