What do an emergency operations center in Cochise County, Ariz., Colorado hospitals, the Idaho Bureau of Corrections, and many 911 call centers across Massachusetts have in common? They were all brought down by a two-day nationwide CenturyLink outage from Dec. 27-28 that also resulted in internet outages for tens of thousands of CenturyLink’s residential customers. The cause? CenturyLink blamed a single, faulty third-party network management card in Denver for disrupting services for CenturyLink and other phone companies, notably Verizon, from Alaska to Florida.

Hours after outage began, two days after Christmas, CenturyLink issued a general statement:

“CenturyLink experienced a network event on one of our six transport networks beginning on December 27 that impacted voice, IP, and transport services for some of our customers. The event also impacted CenturyLink’s visibility into our network management system, impairing our ability to troubleshoot and prolonging the duration of the outage.”

That “network event” caused serious disruptions to critical services in 37 states, including 911, according to Brian Kyes, president of the Massachusetts Major City Chiefs of Police Association.

“This is affecting 911 (wireline & wireless) delivery to most of Massachusetts,” Kyes said in a statement during the outage to the Boston Herald. “We have heard from MEMA that this issue may also affect some landlines but I have not heard of any specific situations or communities that have been impacted. We are advising all police and fire chiefs to test their local 911 systems and notify their residents of potential issues by reverse 911, social media or any other means that they have at their disposal. The interruption in service may depend on a particular phone carrier and the information that we have is that it may be intermittent.”

CenturyLink outages on Dec. 27, 2018. (Image: Downdetector.com)

The disruptions affected much of Massachusetts — a state served primarily by Verizon Communications, because CenturyLink is a major commercial services vendor inside and outside of its local landline service areas and supplies some connectivity services to Verizon, mostly for wireless customers.

ATM networks also went down in certain parts of the country. CenturyLink is one of many vendors providing data connectivity between the cashpoint machines and several banking institutions.

Also impacted, the Idaho Department of Corrections, including inmate phone systems, and the Idaho Department of Education, which lost the ability to make or receive calls.

Consumers also noticed their internet connections were often down or sporadic in some locations, primarily because CenturyLink’s backbone network became saturated with rogue packets.

The Denver Post presented a more detailed technical explanation about the outage:

CenturyLink said the [defective] card was propagating “invalid frame packets” that were sent out over its secondary network, which controlled the flow of data traffic.

“Once on the secondary communication channel, the invalid frame packets multiplied, forming loops and replicating high volumes of traffic across the network, which congested controller card CPUs (central processing unit) network-wide, causing functionality issues and rendering many nodes unreachable,” the company said in a statement.

Once the syndrome gets going, it can be difficult to trace back to its original source and to stop, a big reason networks are designed to isolate failures early and contain them.

“We have learned through experience about these different types of failure modes. We build our systems to try and localize those failures,” said Craig Partridge, chair of the computer science department at Colorado State University in Fort Collins and a member of the Internet Hall of Fame. “I would hope that what is going on is that CenturyLink is trying to understand why a relatively well-known failure mode has bit them.”

The Federal Communications Commission also expects answers to some questions, opening another investigation of the phone company. In 2015, CenturyLink agreed to pay a $16 million settlement to the federal agency after a seven-state outage in April 2014.

Pai

FCC Chairman Ajit Pai said the agency would once again take a look at CenturyLink, focusing on disruptions to emergency services.

“When an emergency strikes, it’s critical that Americans are able to use 911 to reach those who can help,” Pai said in a statement. “This inquiry will include an examination of the effect that CenturyLink’s outage appears to have had on other providers’ 911 services.”

A retired manager at Qwest, a former Baby Bell now owned by CenturyLink, strongly criticized CenturyLink’s lack of communications with customers and an apparent lack of network redundancy.

“For a company in the communication business, they sure failed on this,” said Albuquerque resident Sam Martin. “I participated on the Qwest Disaster Recovery teams, and I do not recall ever having the network down for this kind of time and certainly never the 911 network. The 911 network should never have been down. The lack of this network can contribute to delays in rescue and fire saving lives.”

Martin is dubious about CenturyLink’s explanation for the network outage, suggesting a defective network card may be only a part of the problem.

“The explanations given so far are not valid,” Martin said. “The public may not be aware of it, but the communication network has redundancy and for essential services like inter-office trunking and 911 calls, there are duplicate fiber optic feeds – “rings” that duplicate the main circuit in another path – and switching equipment to these locations so that they may be switched electronically and automatically upon failure to a back-up network ring. When these systems are operating properly, the customer is unaware a failure occurred. If the automatic switching does not take place, employees involved with disaster recovery can intervene and manually switch the affected network to another fiber ring or electronic hub and service is restored until the actual damage is fixed.”

None of those things appeared to happen in this case, and the outage persisted for 48 hours before all services were restored.

“CenturyLink has to have a disaster recovery plan with redundancies in place for electrical, inbound and outbound local and toll-free carriers, as well as network and hardware component redundancies. CenturyLink should be able to switch between multiple fiber optic rings or central offices in case entire networks of phones go down. They would then locate and repair, or replace, defective telecommunication components without the customer ever knowing. The fact that this did not happen is discouraging and scary for the consumer. The fact that it happened nationwide is even more surprising and disturbing. Hopefully the truth will come out soon.”

A critical editorial in the Albuquerque Journal added:

We need answers from CenturyLink beyond the cryptic “a network element” caused the outage. We need to know how many CenturyLink and Verizon customers were affected. And we need to know what they – and other internet and phone providers – are doing to prevent similar outages or worse from happening in the future. Because if the outage showed nothing else, it’s that like an old-time string of Christmas lights, we are living in an interconnected world.

And when one light goes out, they can all go out.

KTVB in Boise, Idaho reported on CenturyLink’s massive outage on Dec. 27-28 and how it impacted local businesses and government services. (3:09)

WASHINGTON (Reuters) – AT&T will pay $5.25 million to settle a U.S. investigation after two outages in 2017 prevented about 15,000 callers from making emergency “911” calls, the company and a federal regulator said last week.

The Federal Communications Commission said Thursday AT&T had agreed to make changes to reduce the likelihood and impact of future 911 outages and improve notifying 911 call centers of outages.

AT&T said it has “taken steps to prevent this from happening again.”

The FCC said the 911 service outages were the result of planned network changes implemented by AT&T inadvertently interfering with the company’s routing of 911 calls.

The FCC said the March 2017 outage lasted about five hours, resulting in the failure of 911 calls from some 12,600 unique users, while the May 2017 outage lasted 47 minutes, resulting in 2,600 failed 911 calls.

The FCC said during the March outage the company failed to “quickly, clearly, and fully notify all affected 911 call centers.”

AT&T said it had cooperated with the review and agreed that “providing access to emergency 911 services is critically important.”

Several other carriers agreed to settlements after an April 2014 outage affected 11 million telephone users.

Verizon Communications agreed to a $3.4 million fine after a six-hour 911 outage in April 2014 that affected about 750,000 wireless consumers in nine California counties.

CenturyLink agreed to a $16 million settlement in the April 2014 outage.

The FCC said the outages at the carriers in April 2014 resulted in 6,600 missed 911 calls about domestic violence, assault, motor vehicle accidents, a heart attack, an overdose, and an intruder breaking into a residence.

The April 2014 outage was the result of a preventable software coding error at a call management center in Colorado, the FCC said.

In 2015, T Mobile US agreed to a $17.5 million settlement after two 911 service outages nationwide in August 2014. The separate but related outages lasted approximately three hours and affected almost all of T-Mobile’s then 50 million customers.

Reporting by David Shepardson; Editing by Lisa Shumaker and David Gregorio

A January 911 outage that crippled the emergency response system across the Tampa area came under scrutiny this week at a Hillsborough County government hearing on the matter.

“As a consumer and as a business owner, I’ve not been satisfied with the transition nor do I trust anyone from the company standing up here at the podium and saying ‘trust me.’ I’d like to see something that is guaranteed,” said Commissioner Victor Crist. “I would like to see something in writing that is guaranteed to my voters, my constituents and this board. Can I be clearer than that?”

Taking much of the heat from the clearly exasperated county commission was David Frezza, Frontier’s vice president of network operations.

“We deeply regret that the event on January 31st impacted the emergency services,” Frezza told the county commissioners.

In January, emergency 911 lines suddenly went out of service in several Florida counties around the Tampa Bay area for several hours. Frontier’s explanation initially blamed contractors and an accidental fiber cut.

But at this week’s hearing, Frezza blamed the outage squarely on CenturyLink, which he said removed both main and backup fiber communications cables for a road widening project underway near Clewiston, a small south-central Florida town on the shores of Lake Okeechobee. CenturyLink is the local phone company serving that area. That alone was apparently enough to interrupt 911 service in Pinellas, Hillsborough, Sarasota, Manatee, and Polk counties.

Frontier’s service area in the Tampa region.

The outage took Frontier several hours to track down, which all the more irritated county commissioners because CenturyLink sent advance notice about the work project, although Frezza denied CenturyLink gave the company enough details to recognize its potential danger to the 911 system.

“I assure you that had CenturyLink alerted us to the intent to work on both the primary and secondary paths simultaneously, we would have taken action to prevent such actions,” Frezza said. “Frontier recognizes that regardless of these circumstances, we are ultimately responsible for the quality and resiliency of the services we provide.”

But Freeza also admitted the company had room to improve line mapping and marking to help other telecom companies identify critical Frontier infrastructure. Before the outage, Frontier tracked maintenance notifications via e-mail. But now Frezza said Frontier will do it over the phone.

After several problems dealing with Frontier, including a widely criticized transition from Verizon’s billing systems to Frontier’s own system, county commissioners seemed reluctant to give Frontier just one more chance to explain.

“You have to give us peace of mind,” Commissioner Stacy White said. “We have to be able to tell the citizens of Hillsborough County with a straight face that we and Frontier have everything in place to reduce the likelihood that our 911 systems aren’t going to be knocked down.”

Frontier spokesman Bob Elek said the company had already implemented an improved backup system with two additional network paths for 911 calls and a third on the way.

“We have created enough redundancy in the network to ensure any future events should have backup to make sure it flows smoothly,” said Elek. The county commission curtly told Frontier to “put it in writing and come back.”

One county is not taking a chance with Frontier again. Manatee County officials report they are permanently cutting the cord on Frontier and moving to an internet-based call routing system that will be managed by Motorola. The county made the move after it gave up trying to get their questions and concerns resolved.

“What happened should never have happened. However, just trying to get answers out of them at this point has been hard to do,” said Jake Saur, the county’s chief of emergency communications. “It is set up in two geographically diverse locations, so if one side is knocked down or taken out, the other side takes it up. We don’t believe there will be outages like Frontier.”

WTVT in Tampa covered the Hillsborough County, Fla. hearing regarding Frontier’s 911 failures in January, 2018. (2:01)

Several Bay Area counties in western Florida were without 911 service for several hours Wednesday after two failures at Frontier Communications left residents without any way to call for help.

There first outage began around midnight and the second started at around 10am Wednesday morning. Both outages took about two hours each to troubleshoot and repair.

Local officials criticized Frontier for service disruptions that had the potential for disaster for residents across the area.

“It’s definitely frustrating, yes,” said Jacob Saur, Manatee County Emergency Communications Center chief. “The main concern is if someone is needing help from first responders and they can’t get that help, then, we have a big problem.”

Frontier’s 911 system is supposed to run over two different networks to provide redundancy in case of an outage. But that didn’t happen. Instead, the company mistakenly connected its two networks together, with no backup workaround.

“For some reason, those two pieces were combined to one network,” said Frontier spokesperson Bob Elek. “So. when that one network was touched or impacted, it took both of them down. It took the service. The problem is a strange one. Honestly, I don’t think I’ve ever seen something like this before unless there was massive network damage. So, it should be an easy one to repair and fix so it doesn’t happen again and we will definitely do it.”

The first outage was traced to routine network maintenance of CenturyLink/Level 3-owned equipment affected by a Florida road project near Clewiston. The second outage occurred because of a fiber cable cut.

Tampa Police Chief Brian Dugan can’t believe Frontier did not have a proper contingency plan in place to deal with critical 911 service.

“There will still have to be some answers on why there was no redundancy, what type of disaster recovery program there was,” Dugan told WTVT in Tampa.

Frontier claims it is moving away from copper wire routing of 911 calls and claims it will route future 911 calls through the internet.

“Using that old technology of copper wire to route 911 calls is going to go away,” said Saur. “So, we are preparing for the future by routing 911 calls through the internet. However, it takes time to get that in place.”

WTVT in Tampa reports Frontier had no backup plan for Wednesday’s 911 outage that interrupted service in multiple counties. (2:55)

At least 7,597,945 Floridians are without cable, broadband and phone service after Hurricane Irma left large sections of Florida without power or wired telecommunications service.

According to the Federal Communications Commission, service providers have self-reported multiple large outages including 911 service, cell towers, cable service, and radio and television services. In many cases, the outages are directly tied to power interruptions, and many expect service will resume soon after Florida utilities get electricity service back up and running.

Telephone

Providers report 390 central switching offices are non functioning across Florida at this time. These offices handle landline calls and wired broadband from AT&T and Frontier Communications, among other telephone companies. Where physical damage from wind or water has occurred, it could take weeks or months before those offices are fully restored, but temporary provisions for restoring basic phone service are likely to be in place within days.

There are also significant 911 outages across Florida:

No 911 Service, Calls Going Unanswered: Big Cypress Indian Reservation, Collier County EOC, Ft. Myers Police Department, Glades County Sheriff, Glades County Sheriff Back Up, Hardee County Back Up, Hendry County Sheriff, Highlands County Sheriff, Lee County Emergency Dispatch Center EOC, Naples PD, Okeechobee County Sheriff’s Department, Homestead Air Force, Marathon SO, and Ocean Reef.

911 Service With No Automatic Caller Location Information (ALI): Hardee County Sheriff and Lee County Sheriff

Re-routed with ALI: Atlantic Beach PD, Belle Glade PD, Indian River SO, Manalapan PD, Miami Beach PD, Neptune Beach PD, and St. Augustine PD

Re-routed without ALI: 4 Clewiston Police Department, Desoto County Sheriff, Lee County Backup, and Sanibel Police Department

Potential Outage: The Broward County South Region 911 Center lost power as of 9/10/17 at 2:28PM EDT. We do not know if 911 calls are being rerouted.

Wireless Service

Overall, 27.4% of cell sites in the disaster area of Florida are out of service. The worst outages are in:

• Collier County: 160 of 212 cell sites are down. (75.5%)
• Hendry County: 31 out of 46 cell sites are down. (67.4%)
• Highlands County: 42 out of 80 cell sites are down. (52.5%)
• Lee County: 186 out of 343 cell sites are down. (54.2%)
• Miami-Dade County: 739 out of 1,435 cell sites down. (51.5%)
• Monroe County: 87 out of 108 cell sites down. (80.6%)

Broadcasting

There are two television stations out of service in Florida: WVFW and WZVN. There are 26 radio stations off the air: W227CP, W300CL, W222AW, WROK, WAOA-FM, WHKR, WLZR, WIOD, WOLZ, WINZ, WBTT, WCKT, WRUM, WQOL, WZZR, WSWN, WOTW, WMFQ, WXUS, WYCG, WEAT, WTIR, WMYR, WCNZ, and W294AN.