Google has published its account of the power failure that took out one of its London availability zones last month, and the central finding is that the site's emergency generators were never asked to start.
The outage hit europe-west-2-b on 20 October, with service interruptions reported at Lloyds, HSBC, Royal Mail and Revolut. The company set out the sequence in an update to its Google Cloud status page on 4 November, which was reported by Simon Sharwood of The Register.
The building is not short of power routes. "The site relies on a 2N utility power architecture," Google said: two independent utility feeders, A and B, either of which can carry the load on its own, with a further standby utility connection available for a manual campus-wide transfer.
Behind the grid supply sits a central high-voltage generator bank, controlled by a dual-redundant programmable logic controller. The PLC is the industrial computer that decides, among other things, when the generators should run.
The sequence began six days before the outage. A quarterly generator load test on 14 October surfaced issues that needed attention, and "Circuit breaker feeder B was physically opened to allow for corrective repairs," Google said. The site was then running on feeder A alone.
On 20 October an external underground cable belonging to the power supplier, connected to primary feeder A, failed. With feeder B open for repairs, the zone lost its grid connection entirely.
The uninterruptible power supplies did what they are rated to do. They are specified to carry the zone for five minutes after the loss of the grid connection, which is intended to be long enough for the generators to come up to load. They worked. Then they ran out.
The generators did not come up. Google attributes this to a "voltage-sensing design flaw and PLC misconfiguration" that "prevented the facility's generator plant" from starting.
The specific failure is set out plainly in the update: "PLC logic indicated that a healthy utility source was available." Both feeds were de-energised. The controller's reading of them was not. No automatic start command was issued, because on the logic as configured there was nothing to start for.
Google says it reprogrammed the PLC once the utility connection was restored, has made electrical changes at europe-west-2-b and has tested them. It listed two follow-up actions: to "bolster notification speed by developing an automated alert-to-communication mechanism for location-impacting incidents", and to "expand platform-wide testing outcomes for infrastructure-level events".
It is worth being precise about what failed here, because on one reading nothing did. The generators were present and serviceable. The controller was redundant twice over. The start command was never issued because, by the only measure that governs the issuing of start commands, the site was powered.
The generator plant therefore performed to specification throughout. A standby generator exists to start when utility power is absent. Utility power was not absent. It was merely not there.
This is a distinction the equipment is entitled to draw. A PLC is not asked whether the lights are on. It is asked to read a voltage-sensing input and act on what the input says. It read the input. It acted on what the input said. At no point in the sequence did it do anything other than what it was configured to do.
Dual redundancy, our analysts note, is the practice of installing two controllers so that one can take over if the other stops working. Neither controller stopped working. They agreed. Redundancy of this kind doubles the number of devices reaching a conclusion and leaves the conclusion exactly where it was.
The five-minute battery rating has attracted some comment on the grounds that five minutes was not enough. This is unfair to the battery. Five minutes is the time allowed for generators to start. The generators did not need five minutes. They needed to be asked.
The Department of Unnecessary Research has reviewed the architecture and finds it internally sound. The site had no power. The site also had power, on the single instrument wired to decide the question. Only one of those two conditions was connected to the starter motor.