A series of lightning strikes in Europe caused a few headaches for Google this week, as the powerful jolts resulted in Google Cloud Platform errors and data loss.
On Aug. 13, "four successive lightning strikes on the local utilities grid that powers our European datacenter caused a brief loss of power to storage systems" that host disk capacity for Google Compute Engine (GCE) in western Europe, Google said this week.
"Although automatic auxiliary systems restored power quickly, and the storage systems are designed with battery backup, some recently written data was located on storage systems which were more susceptible to power failure from extended or repeated battery drain," Google said. "In almost all cases the data was successfully committed to stable storage, although manual intervention was required in order to restore the systems to their normal serving state. However, in a very few cases, recent writes were unrecoverable, leading to permanent data loss on the Persistent Disk."
Data loss was minimal (less than 0.000001 percent), Google said. In most cases, the "affected disks sporadically returned I/O errors to their attached GCE instances, and also typically returned errors for management operations such as snapshot creation."
Google apologized for the glitches and said it is "wholly" responsible for the outage.
"However, we would like to take this opportunity to highlight an important reminder for our customers: GCE instances and Persistent Disks within a zone exist in a single Google datacenter and are therefore unavoidably vulnerable to datacenter-scale disasters," the search giant warned. "Customers who need maximum availability should be prepared to switch their operations to another GCE zone. For maximum durability we recommend GCE snapshots and Google Cloud Storage as resilient, geographically replicated repositories for your data."
Google had already been working to upgrade storage hardware so that it is less susceptible to problems associated with power failures. "Most Persistent Disk storage is already running on this hardware," it said, but Google engineers did another review of its datacenter post-lightning strike and identified several "opportunities...to increase physical and procedural resilience."