What do an emergency operations center in Cochise County, Ariz., Colorado hospitals, the Idaho Bureau of Corrections, and many 911 call centers across Massachusetts have in common? They were all brought down by a two-day nationwide CenturyLink outage from Dec. 27-28 that also resulted in internet outages for tens of thousands of CenturyLink’s residential customers. The cause? CenturyLink blamed a single, faulty third-party network management card in Denver for disrupting services for CenturyLink and other phone companies, notably Verizon, from Alaska to Florida.

Hours after outage began, two days after Christmas, CenturyLink issued a general statement:

“CenturyLink experienced a network event on one of our six transport networks beginning on December 27 that impacted voice, IP, and transport services for some of our customers. The event also impacted CenturyLink’s visibility into our network management system, impairing our ability to troubleshoot and prolonging the duration of the outage.”

That “network event” caused serious disruptions to critical services in 37 states, including 911, according to Brian Kyes, president of the Massachusetts Major City Chiefs of Police Association.

“This is affecting 911 (wireline & wireless) delivery to most of Massachusetts,” Kyes said in a statement during the outage to the Boston Herald. “We have heard from MEMA that this issue may also affect some landlines but I have not heard of any specific situations or communities that have been impacted. We are advising all police and fire chiefs to test their local 911 systems and notify their residents of potential issues by reverse 911, social media or any other means that they have at their disposal. The interruption in service may depend on a particular phone carrier and the information that we have is that it may be intermittent.”

CenturyLink outages on Dec. 27, 2018. (Image: Downdetector.com)

The disruptions affected much of Massachusetts — a state served primarily by Verizon Communications, because CenturyLink is a major commercial services vendor inside and outside of its local landline service areas and supplies some connectivity services to Verizon, mostly for wireless customers.

ATM networks also went down in certain parts of the country. CenturyLink is one of many vendors providing data connectivity between the cashpoint machines and several banking institutions.

Also impacted, the Idaho Department of Corrections, including inmate phone systems, and the Idaho Department of Education, which lost the ability to make or receive calls.

Consumers also noticed their internet connections were often down or sporadic in some locations, primarily because CenturyLink’s backbone network became saturated with rogue packets.

The Denver Post presented a more detailed technical explanation about the outage:

CenturyLink said the [defective] card was propagating “invalid frame packets” that were sent out over its secondary network, which controlled the flow of data traffic.

“Once on the secondary communication channel, the invalid frame packets multiplied, forming loops and replicating high volumes of traffic across the network, which congested controller card CPUs (central processing unit) network-wide, causing functionality issues and rendering many nodes unreachable,” the company said in a statement.

Once the syndrome gets going, it can be difficult to trace back to its original source and to stop, a big reason networks are designed to isolate failures early and contain them.

“We have learned through experience about these different types of failure modes. We build our systems to try and localize those failures,” said Craig Partridge, chair of the computer science department at Colorado State University in Fort Collins and a member of the Internet Hall of Fame. “I would hope that what is going on is that CenturyLink is trying to understand why a relatively well-known failure mode has bit them.”

The Federal Communications Commission also expects answers to some questions, opening another investigation of the phone company. In 2015, CenturyLink agreed to pay a $16 million settlement to the federal agency after a seven-state outage in April 2014.

Pai

FCC Chairman Ajit Pai said the agency would once again take a look at CenturyLink, focusing on disruptions to emergency services.

“When an emergency strikes, it’s critical that Americans are able to use 911 to reach those who can help,” Pai said in a statement. “This inquiry will include an examination of the effect that CenturyLink’s outage appears to have had on other providers’ 911 services.”

A retired manager at Qwest, a former Baby Bell now owned by CenturyLink, strongly criticized CenturyLink’s lack of communications with customers and an apparent lack of network redundancy.

“For a company in the communication business, they sure failed on this,” said Albuquerque resident Sam Martin. “I participated on the Qwest Disaster Recovery teams, and I do not recall ever having the network down for this kind of time and certainly never the 911 network. The 911 network should never have been down. The lack of this network can contribute to delays in rescue and fire saving lives.”

Martin is dubious about CenturyLink’s explanation for the network outage, suggesting a defective network card may be only a part of the problem.

“The explanations given so far are not valid,” Martin said. “The public may not be aware of it, but the communication network has redundancy and for essential services like inter-office trunking and 911 calls, there are duplicate fiber optic feeds – “rings” that duplicate the main circuit in another path – and switching equipment to these locations so that they may be switched electronically and automatically upon failure to a back-up network ring. When these systems are operating properly, the customer is unaware a failure occurred. If the automatic switching does not take place, employees involved with disaster recovery can intervene and manually switch the affected network to another fiber ring or electronic hub and service is restored until the actual damage is fixed.”

None of those things appeared to happen in this case, and the outage persisted for 48 hours before all services were restored.

“CenturyLink has to have a disaster recovery plan with redundancies in place for electrical, inbound and outbound local and toll-free carriers, as well as network and hardware component redundancies. CenturyLink should be able to switch between multiple fiber optic rings or central offices in case entire networks of phones go down. They would then locate and repair, or replace, defective telecommunication components without the customer ever knowing. The fact that this did not happen is discouraging and scary for the consumer. The fact that it happened nationwide is even more surprising and disturbing. Hopefully the truth will come out soon.”

A critical editorial in the Albuquerque Journal added:

We need answers from CenturyLink beyond the cryptic “a network element” caused the outage. We need to know how many CenturyLink and Verizon customers were affected. And we need to know what they – and other internet and phone providers – are doing to prevent similar outages or worse from happening in the future. Because if the outage showed nothing else, it’s that like an old-time string of Christmas lights, we are living in an interconnected world.

And when one light goes out, they can all go out.

KTVB in Boise, Idaho reported on CenturyLink’s massive outage on Dec. 27-28 and how it impacted local businesses and government services. (3:09)

Downdetector shows a major service outage for Spectrum customers in Central Florida.

A widespread service outage affecting Spectrum customers in central Florida that began Sunday caused a crowd of 100-150 customers to turn out at a Spectrum office on Semoran Boulevard in Orlando this morning to switch cable boxes or cancel service.

Customers lost television service Sunday evening and the outage continues in many areas, leaving thousands without service for more than 14 hours. Orange County Public Schools spent this morning without internet service, also provided by Spectrum. The school district e-mailed parents:

“OCPS families, we want to make you aware that throughout our community the internet and networks are down throughout due to issues with Spectrum. This outage is impacting many of our schools. At this time, Spectrum cannot provide a timeframe for restoration of service. We want to make you aware that contacting schools may be limited due to the outages. We also want to assure you our digital classrooms are always prepared to adjust to such circumstances as they can use blended methods for learning so instruction can continue regardless of problems with the internet. Thank you for your support. We will update you as we get new information.”

Many customers are angry about what they perceive as a deterioration in service after Charter Communications acquired Bright House Networks.

“It seems like since Spectrum has taken over from Bright House, every time the wind blows, the cable and the boxes go out and you have to come down here and stand in a line to change a box. It’s a waste,” Spectrum customer George Roberts told WFTV.

Communities affected include: Orlando and surrounding suburbs, Cape Canaveral, Sanford, Daytona Beach, Sky Lake, Palm Coast, and beyond.

“The storms last night caused damage to operations impacting customers in the Central Florida region,” said Spectrum spokesperson Joe Durkin. “I won’t speculate on completion but as Spectrum engineers are working to restore full video services to our customers and as time goes on – some areas are coming back. We confirm there’s no internet outage at all that could still be affecting Orange County Public Schools.”

Spectrum, like most cable operators, will not issue a service outage credit unless customers specifically request one. The best way to do that is to login to Spectrum’s website and use online chat or call your local cable office and ask for a service outage credit.

WFTV in Orlando reports angry crowds gathered at a Orlando Spectrum cable store to switch boxes or cancel service because of a service outage impacting Central Florida. (2:08)

Frontier Communications spent more time working on ways to keep Minnesota customers from turning up at upcoming public hearings to discuss their poor service than actually resolving those customers’ service troubles.

Minnesota has a big problem with Frontier. The company has been the subject of an unprecedented number of customer complaints and negative comments — 439 in just a five-week period from Feb. 12 – March 19, 2018 about poor service, repair crews that don’t show up, woefully inadequate internet service, poor billing and customer service practices, and false advertising. As a result, the Minnesota Public Utilities Commission (PUC) launched an investigation into Frontier’s service performance in the state (Note: most links in this article will require a free account at the Minnesota Department of Commerce to read. Register here.), which is about the same time Frontier’s top executives in the state began a campaign of damage control focused primarily on keeping internet complaints out of the public record.

The complaints, summarized below by the Minnesota PUC, are familiar to many Frontier Communications customers around the country:

Some parties allege being without telephone service for about a week’s time on multiple occasions. Such instances resulted in customers being unable to access 911 or connect medical devices dependent on land telephone lines. Missed incoming calls, noise on phone lines and other phone quality complaints are not infrequent. Nearly all comments mention that they are being charged for service product(s) not being provided as promised, often with related billing and cancellation disputes as a consequence.

Nearly all parties complain that Frontier’s customer service representatives provide inconsistent information on available service in the customer’s area and its price. Many report routinely being sold higher level (more costly) service or hardware as a remedy for service problems that remain or return after the recommended solution is in place. Customers often note being told later that the upgraded service they were sold is not available at their location.

Many complaints concern home service visits that require subsequent visits to correct or augment earlier actions, often with charges but no resulting remedy. Often customers say they experience long delays in getting repairs scheduled, must take lengthy time from work to await for service representatives to arrive only to find problems cannot be remedied. Missed service appointments, mistaken disconnections, unrequested service additions, installation and wiring errors are common complaints.

Customers frequently report discovering they are allegedly on a contract with penalties for ending service early even if they had explicitly refused to accept long term contracts. Apparently such contracts automatically renew without customer notice upon payment of the first month of the new period. Customers indicate being warned of damaging credit reports in addition to accumulating penalties if they do not pay disputed bills. Billing disputes also include promised discounts not being provided, penalties accumulating on disputed amounts, and checks being sent but not being credited to accounts.

Based on decades of experience, the PUC staff knew trouble when they saw it, and found the complaints about Frontier credible and serious.

“The total number of comments and complaints, often with detailed documentation, appears to indicate that widespread problems with service quality, customer service and billing exist,” PUC staffers wrote. “Customers express the very highest levels of frustration over service quality and over their interactions with Frontier representatives. Customers express despair over their billing and lack of alternatives. Finally, they express outright ‘gratitude for the hope that someone might come to their aid.”

Customers hoping for rescue discovered Frontier’s legal team instead, on a mission to do everything possible to limit the scope of the state’s investigation and discourage public participation by suggesting customers with internet complaints would not be welcome at the hearings.

Frontier, joined by fellow independent phone company CenturyLink, immediately realized the implications of holding public hearings about the performance of their DSL service in Minnesota. Both companies likely receive an even larger number of service complaints than regulators do, and here is just a sampling:

‘If you don’t like our service in the countryside, move to town!’

Graham Adams: “We have had Frontier for a little over 2 years and have had nothing but problems. Internet is constantly out for days sometimes weeks at a time. I think it’s preposterous they can charge me $42 a month for 5 Mbps service that is inconsistent at best. Because we live outside city limits Frontier is the only internet service available.”

Christopher Krolak:  “I have been a Frontier Communications customer for about 4.5 years. I live in an area where there isn’t a lot of competition for high speed internet. I pay $30 per month for “up to 6 Mbps” service but real world speeds are best case 2 Mbps and fall to 0.3 Mbps during peak times. When I’ve called about the large discrepancy between advertised speed and actual speed, Frontier has responded that the area I live in is only provisioned for about 2 Mbps speed and an infrastructure upgrade is required. Frontier is unwilling to give any timeline forecast for when such upgrade will be made.”

Sylvia Svihel: “We have been a customer of Frontier’s for 41 years as it has been the only land line in our area. Our phone, internet and Dish service are tied into the same package. The prices keep going up. We did upgrade our service for a faster speed but we see zero improvement on the speed…just an even higher bill. I have lost track of how many times we have contacted Frontier on lost service. They usually just say it’s the modem and to reboot it and everything will be OK. I reboot the modem, sometimes multiple times a day.”

Jay Johnson: “I have been a Frontier customer for internet for a long time. The service I pay for is “up to 6 Mbps” but I’d be lucky to get 1.2 Mbps. They have a monopoly in this part of Mille Lacs County. There are really no other options other than satellite or cellular and those are not really any better speed and certainly not price.”

Roger Wikstrom: “We have had Frontier service for 32 years. Beginning about 20 years ago we added internet service, which has always been unreliable. […]We complained many times and had dozens of service calls over the years. At one point, the technician told us we were out in the country, the brass at Frontier did not really care about our service, and that if we wanted good service from Frontier, we should move to town.”

Based on a growing record of complaints, the PUC sought to hold public hearings to gather more information from consumers and to better understand the problems being experienced by Frontier customers. Almost immediately, Frontier began to claim the complaints were few and far between, and most of the complaints seen on the record pertain to the company’s DSL internet service, which Frontier claims is not subject to oversight by the PUC and cannot be a subject on the agenda of the public hearings.

Frontier’s Lawyers: It would confuse customers and give them false hope if they believed the Commission can force Frontier to improve DSL service.

Frontier’s attorneys lecture the Minnesota Department of Commerce

Frontier’s attorneys have repeatedly objected to any investigation or hearings that cover anything beyond the performance of Frontier’s landline telephone service. Frontier was joined by CenturyLink, which also argues Minnesota no longer has any jurisdiction over broadband issues, noting a state court recently ruled telecommunications services are subject to state regulation and oversight, while “information services” like internet access are not.

Frontier was particularly irritated that the hearings could stray into an open mic session filled with consumers upset about Frontier’s DSL service. Unless customers were warned in advance the public meetings were not to include discussions about internet service, it “would create false expectations and confusion for customers.” In fact, if regulators permitted this, Frontier claims it would “violate federal law.”

“Holding public hearings directed to internet access service complaints would not be constructive because the Commission would be precluded from taking action concerning internet service rates or service quality using any information it may collect during the public hearings,” Frontier added.

Here is where the Republican-dominated FCC comes to the aid of Frontier and CenturyLink. At the insistence of FCC Chairman Ajit Pai, stripping away state oversight of poorly performing telecom companies was a key industry benefit gained with the implementation of Pai’s “Restoring Internet Freedom Order,” implemented on Jan. 8, 2018. That FCC Order swept away former FCC Chairman Thomas Wheeler’s favored classification of broadband as a “telecommunications service,” which is subject to oversight, and instead put it firmly back in unregulated territory as an “information service.” That proved helpful to CenturyLink’s argument:

In making its decision the FCC broadly preempted state regulation and decided that “regulation of broadband Internet access service should be governed principally by a uniform set of federal regulations, rather than by a patchwork that includes separate state and local requirements.” The FCC expressly preempted any ‘public utility-type’ regulations, . . . akin to those found in Title II of the Act and its implementing rules . . .”

Frontier’s lawyers made so much noise about the prospect of internet complaints being heard at public hearings, the Commission elected to allow Frontier to draft the public hearing notices that would be inserted into customer bills and published in newspapers around the state. The Commission also allowed Frontier to clarify the limits of the Commission’s jurisdiction over internet service — a decision it would soon regret.

Minnesota is unusual because it is served by dozens of smaller, typically independent telephone companies, which include Frontier and its subsidiary Citizens Telecommunications of Minnesota.

Give Frontier an inch, and they take a mile, according to some company critics who told Stop the Cap! were astonished on April 30th when Frontier shared its draft notice with the public. The Minnesota attorney general’s office politely characterized Frontier’s notice as a “very narrow reading of the Commission’s jurisdiction over internet service.”

Here it is, as originally proposed by Frontier in April:

The jurisdiction of the MPUC includes telephone services, but does not include Internet services or the speed or quality of access or connections to the Internet or the communications services, such as Voice Over IP, that are provided using only the Internet.

The attorney general’s office objected to Frontier’s characterization of VoIP phone service as completely unregulated. A subsequent proposed revision by Frontier was not welcomed by the attorney general’s office either:

The jurisdiction of the MPUC includes telephone services, but does not include Internet access services or the rates, speed, quality, or availability of Internet services.

After motions to reconsider, the Commission ultimately reversed its earlier decision allowing Frontier to write its own text:

While the Commission does not want to mislead the public into believing the Commission has jurisdiction over matters that are solely within the province of federal entities, neither does the Commission want to erroneously disavow any aspect of the jurisdiction it does have over the goods and services that Frontier provides to its Minnesota customers.

Given the tension between these two objectives—and the fact that this dispute is arising in the context of drafting the language of a public notice—the Commission will resolve this matter by simply eliminating the requirement that the notice address the topic of the Commission’s jurisdiction over aspects of internet services.

Frontier DSL in Watertown: “47 minutes to upload one small photo to Facebook.”

While Frontier argues about jurisdiction issues, customers like Dr. Kathleen McCann — a dentist serving rural Watertown Township in Carver County, share their stories about how inadequate internet access directly harms local communities, and in her case, her patients.

Dr. McCann

“Frontier Communications is my only option for internet,” McCann told regulators. “My internet service is worse than dial-up. I am charged for ‘DSL High Speed Broadband’ on my monthly bill, but my download speeds are only averaging 2 Mbps and the upload speeds average 0.28 Mbps. As a dentist, I am not able to email dental X-rays. It took me 47 minutes to upload one small photo to Facebook recently.”

McCann added what is even worse than her DSL speed is Frontier’s service. She claims there are “frequent drops” every day, and a technician from Frontier measured an average of 20 small service outages a day. One day her service dropped 400 times. Outages can last days.

“The most recent Frontier internet outage began March 3 and as of March 7, there are at least 27 homes in my neighborhood still without internet service,” McCann added. “This is unacceptable, especially since many of these 27 Frontier customers are running their businesses entirely from home. Calls to Frontier, when finally answered after sometimes 40 minutes on hold, are ineffective.”

Public meetings to discuss Frontier service are scheduled in these areas of Minnesota (exact locations to be determined):

• Ely: September 4, 2018, at 6:00 p.m.
• McGregor: September 5, 2018, at 6:00 p.m.
• Wyoming: September 12, 2018, at 6:00 p.m.
• Slayton: September 25, 2018, at 6:00 p.m.
• Lakeville: September 26, 2018, at 2:00 p.m. and 6:00 p.m.

Comcast’s Business division has introduced the first automatic 4G backup internet connection service for commercial customers who experience an internet outage or network problem.

Comcast’s Connection Pro ($29.95/mo) offers automatic switching to a backup 4G LTE wireless internet service that will keep business customers connected to the internet until Comcast’s wired broadband connection is repaired and goes back online.

“Internet connectivity is critical for any business. Losing their connection – even shortly – can be disruptive,” said Jeff Lewis, vice president, data product management, Comcast Business. “Comcast Business understands that businesses need a redundant back-up solution to help stay connected and provide greater peace of mind in the event of a power or internet outage.”

The service targets small businesses and retailers and is marketed as a backup for cash registers/credit card point of sale terminals, email, and cloud services, and includes battery backup to maintain connectivity for up to eight hours in the event of a power outage.

Business customers can also access an online control panel to remotely monitor outages at individual business locations.

The culprit?

A major internet outage is affecting internet users nationwide since this morning, particularly those subscribed to Comcast’s internet service.

“Some customers are having issues with their XFINITY Internet service. We apologize & appreciate your patience while we work to fix,” Comcast tweeted to its customers this morning.

The outages are not just affecting Comcast customers, however, with sporadic slowdowns and problems also reported by Charter/Spectrum, AT&T, and Verizon customers.

One factor that may explain the outage affecting customers beyond Comcast is CenturyLink’s Level 3, which provides backbone services for several ISPs.

Outages affecting Level 3 just happen to be in major Comcast service areas.

Level 3 eventually did take responsibility for the outage in a statement to the media:

On Monday, Nov. 6, our network experienced a service disruption affecting some customers with IP-based services. The disruption was caused by a configuration error. We know how important these services are to our customers. Our technicians were able to restore service within approximately 90 minutes.

On Nov. 1, Level 3 officially became part of CenturyLink, as part of a $30 billion acquisition. Hopefully CenturyLink will spend a bit more and build additional redundancy into Level 3’s network.

As of 30 minutes ago, Comcast claims the internet outage has eased.