This April, most 911 services went out in seven states for six hours; the entire state of Washington was affected by the near total blackout. And per a new Federal Communications Commission report, the root cause of the crisis—which is not believed to have led to any deaths—was very simple. From the Washington Post:
At the center of the disruption was a system maintained by a third-party contractor, a Colorado-based company called Intrado. Intrado owns and operates a routing service, taking in 911 calls and directing them to the most appropriate public safety answering point, or PSAP, in industry parlance. Ordinarily, Intrado’s automated system assigns a unique identifying code to each incoming call before passing it on — a method of keeping track of phone calls as they move through the system.
But on April 9, the software responsible for assigning the codes maxed out at a pre-set limit; the counter literally stopped counting at 40 million calls. As a result, the routing system stopped accepting new calls, leading to a bottleneck and a series of cascading failures elsewhere in the 911 infrastructure.
The Post’s account might be slightly off—based on the FCC report, it doesn’t appear that the Intrado system assigned a code of the type that created the problem to every call, just certain kinds—but the gist remains: A computer database in Colorado hit a size limit, and 911 went out for millions of people across the country.