For two months, Hagerstown, Md.-based Antietam Broadband, an independent cable company serving Washington County, has been facing an organized effort to bring its broadband business to its knees.

A series of prolonged service outages that the cable company characterized as a “multi-pronged concerted attack” affected both internet and voice service for 40-50% of its customers. A “malicious and targeted attack outside of our network” caused connections to slow, and in many cases stop, the company admitted. The attack flooded the company’s service delivery network with “more than one million malicious hits against our network every 5 minutes,” according to Antietam Broadband president Brian Lynch.

As a result, some customers have been without stable service for over a week at a time, and with the attacks continuing for at least two months, many are growing increasingly frustrated by the repeated outages.

“It has come on for an hour and then gone down again,” said Barbara Ewald of Hagerstown, telling the Herald-Media it was her 11th day of problems with her internet service. “I can’t keep crawling around to reset it. I’m not going to try to set it. It’s not my responsibility. We deserve better than this.”

Other customers have experienced outages up to a dozen times a day, depending on how sustained the attacks are on Antietam’s network.

Letter from Brian Lynch, President of Antietam Broadband.

“I’ve never seen anything like it,” Lynch said. “There’s an entity trying to interrupt communication. The way they’re doing it is very unique; it’s multi-pronged.”

Evidently not a traditional denial-of-service-disruption, Lynch won’t go into much detail about the exact nature of the attack, except to say it targets the company’s cable modems which manage voice and data service.

A recent public meeting at the Washington County Free Library attracted at least 80 local residents to complain about the state of Antietam’s service.

Penny Wolters of Hagerstown told the audience her son was unable to complete a college assignment in time and some of her bills were paid late because of the ongoing service disruptions. Other subscribers who telecommute or operate in-home businesses had to leave Antietam’s service area to find a reliable connection. Calls to customer service were met “with rudeness” according to several complaining customers. A representative told one customer she should be grateful the company offered her any service credit for an outage that had been a problem for over six weeks.

Even with the help of Cable Labs, CommScope/ARRIS and Zcorum, and security experts from around the country, the attacks continued, although they seem to be ebbing. But after two months dealing with the problem, a local grassroots group, “More Cable Options for Hagerstown” launched to find alternatives for Antietam.

“My goal is to give the citizens of Washington County a platform to discuss their frustration with Antietam Broadband,” organizer Cherish Marriott told the newspaper. “I’d also like to give them the tools and information on how to bring other service providers to the area.”

For Lynch, the priority has become repairing the damage to the company’s reputation.

“We want to personally apologize to our customers for the disruption and frustration these past weeks have caused in your life,” Lynch wrote on the company’s Facebook page. “We recognize the internet and phone services we provide are of the utmost importance to you. We would like to thank you for the support and patience you have shown us during this challenging time.”

WDVM in Hagerstown reports on the service outage issues that began in April. Hagerstown’s mayor wonders why the city apologized for the cable company’s problems. (1:31)

To address customer concerns, Antietam provided this recent update to customers:

• “We have gone two weeks since the last event. We believe that the measures we’ve taken have been effective, and we are closely watching our network in case the attackers try again.”
• “We are increasing the credit to residential customers from two to three weeks. Credits will be automatically placed on your bill. If you previously received two weeks of credit, on your next bill you will see the additional week of credit. These additional credits will include leased modems and/or routers provided by Antietam.”
• “We have seen no evidence that the attackers had access to any customer data stored on our systems.”
• “We will continue to stress test our networks to provide every safeguard available against further cyberattacks.”

Local organizers meet to discuss ongoing problems with Antietam Broadband and what alternatives exist. WDVM reports. (2:00)

What do an emergency operations center in Cochise County, Ariz., Colorado hospitals, the Idaho Bureau of Corrections, and many 911 call centers across Massachusetts have in common? They were all brought down by a two-day nationwide CenturyLink outage from Dec. 27-28 that also resulted in internet outages for tens of thousands of CenturyLink’s residential customers. The cause? CenturyLink blamed a single, faulty third-party network management card in Denver for disrupting services for CenturyLink and other phone companies, notably Verizon, from Alaska to Florida.

Hours after outage began, two days after Christmas, CenturyLink issued a general statement:

“CenturyLink experienced a network event on one of our six transport networks beginning on December 27 that impacted voice, IP, and transport services for some of our customers. The event also impacted CenturyLink’s visibility into our network management system, impairing our ability to troubleshoot and prolonging the duration of the outage.”

That “network event” caused serious disruptions to critical services in 37 states, including 911, according to Brian Kyes, president of the Massachusetts Major City Chiefs of Police Association.

“This is affecting 911 (wireline & wireless) delivery to most of Massachusetts,” Kyes said in a statement during the outage to the Boston Herald. “We have heard from MEMA that this issue may also affect some landlines but I have not heard of any specific situations or communities that have been impacted. We are advising all police and fire chiefs to test their local 911 systems and notify their residents of potential issues by reverse 911, social media or any other means that they have at their disposal. The interruption in service may depend on a particular phone carrier and the information that we have is that it may be intermittent.”

CenturyLink outages on Dec. 27, 2018. (Image: Downdetector.com)

The disruptions affected much of Massachusetts — a state served primarily by Verizon Communications, because CenturyLink is a major commercial services vendor inside and outside of its local landline service areas and supplies some connectivity services to Verizon, mostly for wireless customers.

ATM networks also went down in certain parts of the country. CenturyLink is one of many vendors providing data connectivity between the cashpoint machines and several banking institutions.

Also impacted, the Idaho Department of Corrections, including inmate phone systems, and the Idaho Department of Education, which lost the ability to make or receive calls.

Consumers also noticed their internet connections were often down or sporadic in some locations, primarily because CenturyLink’s backbone network became saturated with rogue packets.

The Denver Post presented a more detailed technical explanation about the outage:

CenturyLink said the [defective] card was propagating “invalid frame packets” that were sent out over its secondary network, which controlled the flow of data traffic.

“Once on the secondary communication channel, the invalid frame packets multiplied, forming loops and replicating high volumes of traffic across the network, which congested controller card CPUs (central processing unit) network-wide, causing functionality issues and rendering many nodes unreachable,” the company said in a statement.

Once the syndrome gets going, it can be difficult to trace back to its original source and to stop, a big reason networks are designed to isolate failures early and contain them.

“We have learned through experience about these different types of failure modes. We build our systems to try and localize those failures,” said Craig Partridge, chair of the computer science department at Colorado State University in Fort Collins and a member of the Internet Hall of Fame. “I would hope that what is going on is that CenturyLink is trying to understand why a relatively well-known failure mode has bit them.”

The Federal Communications Commission also expects answers to some questions, opening another investigation of the phone company. In 2015, CenturyLink agreed to pay a $16 million settlement to the federal agency after a seven-state outage in April 2014.

Pai

FCC Chairman Ajit Pai said the agency would once again take a look at CenturyLink, focusing on disruptions to emergency services.

“When an emergency strikes, it’s critical that Americans are able to use 911 to reach those who can help,” Pai said in a statement. “This inquiry will include an examination of the effect that CenturyLink’s outage appears to have had on other providers’ 911 services.”

A retired manager at Qwest, a former Baby Bell now owned by CenturyLink, strongly criticized CenturyLink’s lack of communications with customers and an apparent lack of network redundancy.

“For a company in the communication business, they sure failed on this,” said Albuquerque resident Sam Martin. “I participated on the Qwest Disaster Recovery teams, and I do not recall ever having the network down for this kind of time and certainly never the 911 network. The 911 network should never have been down. The lack of this network can contribute to delays in rescue and fire saving lives.”

Martin is dubious about CenturyLink’s explanation for the network outage, suggesting a defective network card may be only a part of the problem.

“The explanations given so far are not valid,” Martin said. “The public may not be aware of it, but the communication network has redundancy and for essential services like inter-office trunking and 911 calls, there are duplicate fiber optic feeds – “rings” that duplicate the main circuit in another path – and switching equipment to these locations so that they may be switched electronically and automatically upon failure to a back-up network ring. When these systems are operating properly, the customer is unaware a failure occurred. If the automatic switching does not take place, employees involved with disaster recovery can intervene and manually switch the affected network to another fiber ring or electronic hub and service is restored until the actual damage is fixed.”

None of those things appeared to happen in this case, and the outage persisted for 48 hours before all services were restored.

“CenturyLink has to have a disaster recovery plan with redundancies in place for electrical, inbound and outbound local and toll-free carriers, as well as network and hardware component redundancies. CenturyLink should be able to switch between multiple fiber optic rings or central offices in case entire networks of phones go down. They would then locate and repair, or replace, defective telecommunication components without the customer ever knowing. The fact that this did not happen is discouraging and scary for the consumer. The fact that it happened nationwide is even more surprising and disturbing. Hopefully the truth will come out soon.”

A critical editorial in the Albuquerque Journal added:

We need answers from CenturyLink beyond the cryptic “a network element” caused the outage. We need to know how many CenturyLink and Verizon customers were affected. And we need to know what they – and other internet and phone providers – are doing to prevent similar outages or worse from happening in the future. Because if the outage showed nothing else, it’s that like an old-time string of Christmas lights, we are living in an interconnected world.

And when one light goes out, they can all go out.

KTVB in Boise, Idaho reported on CenturyLink’s massive outage on Dec. 27-28 and how it impacted local businesses and government services. (3:09)

Parts of Montana were left without phone, internet, or cable television service for the third time in the last few weeks after the latest outage from Spectrum caused widespread interruptions.

“Here we go again,” complained Spectrum customer Greg Dugdale. “Charter/Spectrum phones and internet down in Havre and then they are back up, then they are down again.”

The worst affected area is Great Falls, which is coping with its third major service outage.

KRTV-TV reported Spectrum officials are blaming repairs being performed on a third-party carrier’s network for the latest service interruption. Although Spectrum was aware of the problem, they were completely reliant on the other provider to correct it.

Customers are increasingly frustrated about Spectrum’s repeated service problems. An outage in early October wiped out service for almost 48 hours. Last Friday, another outage took out service for several hours. In both cases, the company blamed damage to a fiber optic line, for which it apparently has no backup.

Spectrum officials apologized for the interruption, but will not issue automatic service credits for affected customers. Those looking for a credit will need to reach out to Spectrum customer service to request one.

At least 7,597,945 Floridians are without cable, broadband and phone service after Hurricane Irma left large sections of Florida without power or wired telecommunications service.

According to the Federal Communications Commission, service providers have self-reported multiple large outages including 911 service, cell towers, cable service, and radio and television services. In many cases, the outages are directly tied to power interruptions, and many expect service will resume soon after Florida utilities get electricity service back up and running.

Telephone

Providers report 390 central switching offices are non functioning across Florida at this time. These offices handle landline calls and wired broadband from AT&T and Frontier Communications, among other telephone companies. Where physical damage from wind or water has occurred, it could take weeks or months before those offices are fully restored, but temporary provisions for restoring basic phone service are likely to be in place within days.

There are also significant 911 outages across Florida:

No 911 Service, Calls Going Unanswered: Big Cypress Indian Reservation, Collier County EOC, Ft. Myers Police Department, Glades County Sheriff, Glades County Sheriff Back Up, Hardee County Back Up, Hendry County Sheriff, Highlands County Sheriff, Lee County Emergency Dispatch Center EOC, Naples PD, Okeechobee County Sheriff’s Department, Homestead Air Force, Marathon SO, and Ocean Reef.

911 Service With No Automatic Caller Location Information (ALI): Hardee County Sheriff and Lee County Sheriff

Re-routed with ALI: Atlantic Beach PD, Belle Glade PD, Indian River SO, Manalapan PD, Miami Beach PD, Neptune Beach PD, and St. Augustine PD

Re-routed without ALI: 4 Clewiston Police Department, Desoto County Sheriff, Lee County Backup, and Sanibel Police Department

Potential Outage: The Broward County South Region 911 Center lost power as of 9/10/17 at 2:28PM EDT. We do not know if 911 calls are being rerouted.

Wireless Service

Overall, 27.4% of cell sites in the disaster area of Florida are out of service. The worst outages are in:

• Collier County: 160 of 212 cell sites are down. (75.5%)
• Hendry County: 31 out of 46 cell sites are down. (67.4%)
• Highlands County: 42 out of 80 cell sites are down. (52.5%)
• Lee County: 186 out of 343 cell sites are down. (54.2%)
• Miami-Dade County: 739 out of 1,435 cell sites down. (51.5%)
• Monroe County: 87 out of 108 cell sites down. (80.6%)

Broadcasting

There are two television stations out of service in Florida: WVFW and WZVN. There are 26 radio stations off the air: W227CP, W300CL, W222AW, WROK, WAOA-FM, WHKR, WLZR, WIOD, WOLZ, WINZ, WBTT, WCKT, WRUM, WQOL, WZZR, WSWN, WOTW, WMFQ, WXUS, WYCG, WEAT, WTIR, WMYR, WCNZ, and W294AN.

“I think that in general, there is probably too much commercial interruption in television,” Disney CEO Bob Iger told investors on the company’s first-quarter earnings call.

As some commercial networks cut “hour-long” TV shows to as little as 39 minutes, viewers have taken notice and are tuning out.

TV advertising loads — how many ad breaks and for how long — have been increasing since the 1950s, leaving viewers with less and less of their favorite shows. Networks have gotten creative over the years finding new spaces for advertising content by compressing opening titles and closing credits, telling producers to plan for shorter shows, using product placement to showcase brands and products inside shows, and editing or compressing older programs that can no longer fit in their allotted time slots.

The idea of reducing advertising is a recent phenomena, but one being taken seriously by networks that have watched ad revenue and ratings slip, as viewers learn to bypass or avoid advertising, or watch something else.

Iger promised an unspecified reduction in advertising on Disney-owned ABC and ESPN. It is uncertain if that will result in a slight decline of a minute per hour or something more dramatic. Time Warner, Inc., began opting for the latter in 2015, cutting ad time on certain shows on its TruTV network by half. Viacom and 21st Century Fox followed with experimental ad reductions of their own that same year.

Reducing advertising does not necessarily mean revenue reduction. Earlier experiments raised ad rates to compensate for the reduced number of commercials. Advertisers were reportedly pleased by the experiments because shorter ad breaks encouraged viewers to avoid channel surfing, giving higher exposure to advertising messages.