What do an emergency operations center in Cochise County, Ariz., Colorado hospitals, the Idaho Bureau of Corrections, and many 911 call centers across Massachusetts have in common? They were all brought down by a two-day nationwide CenturyLink outage from Dec. 27-28 that also resulted in internet outages for tens of thousands of CenturyLink’s residential customers. The cause? CenturyLink blamed a single, faulty third-party network management card in Denver for disrupting services for CenturyLink and other phone companies, notably Verizon, from Alaska to Florida.

Hours after outage began, two days after Christmas, CenturyLink issued a general statement:

“CenturyLink experienced a network event on one of our six transport networks beginning on December 27 that impacted voice, IP, and transport services for some of our customers. The event also impacted CenturyLink’s visibility into our network management system, impairing our ability to troubleshoot and prolonging the duration of the outage.”

That “network event” caused serious disruptions to critical services in 37 states, including 911, according to Brian Kyes, president of the Massachusetts Major City Chiefs of Police Association.

“This is affecting 911 (wireline & wireless) delivery to most of Massachusetts,” Kyes said in a statement during the outage to the Boston Herald. “We have heard from MEMA that this issue may also affect some landlines but I have not heard of any specific situations or communities that have been impacted. We are advising all police and fire chiefs to test their local 911 systems and notify their residents of potential issues by reverse 911, social media or any other means that they have at their disposal. The interruption in service may depend on a particular phone carrier and the information that we have is that it may be intermittent.”

CenturyLink outages on Dec. 27, 2018. (Image: Downdetector.com)

The disruptions affected much of Massachusetts — a state served primarily by Verizon Communications, because CenturyLink is a major commercial services vendor inside and outside of its local landline service areas and supplies some connectivity services to Verizon, mostly for wireless customers.

ATM networks also went down in certain parts of the country. CenturyLink is one of many vendors providing data connectivity between the cashpoint machines and several banking institutions.

Also impacted, the Idaho Department of Corrections, including inmate phone systems, and the Idaho Department of Education, which lost the ability to make or receive calls.

Consumers also noticed their internet connections were often down or sporadic in some locations, primarily because CenturyLink’s backbone network became saturated with rogue packets.

The Denver Post presented a more detailed technical explanation about the outage:

CenturyLink said the [defective] card was propagating “invalid frame packets” that were sent out over its secondary network, which controlled the flow of data traffic.

“Once on the secondary communication channel, the invalid frame packets multiplied, forming loops and replicating high volumes of traffic across the network, which congested controller card CPUs (central processing unit) network-wide, causing functionality issues and rendering many nodes unreachable,” the company said in a statement.

Once the syndrome gets going, it can be difficult to trace back to its original source and to stop, a big reason networks are designed to isolate failures early and contain them.

“We have learned through experience about these different types of failure modes. We build our systems to try and localize those failures,” said Craig Partridge, chair of the computer science department at Colorado State University in Fort Collins and a member of the Internet Hall of Fame. “I would hope that what is going on is that CenturyLink is trying to understand why a relatively well-known failure mode has bit them.”

The Federal Communications Commission also expects answers to some questions, opening another investigation of the phone company. In 2015, CenturyLink agreed to pay a $16 million settlement to the federal agency after a seven-state outage in April 2014.

Pai

FCC Chairman Ajit Pai said the agency would once again take a look at CenturyLink, focusing on disruptions to emergency services.

“When an emergency strikes, it’s critical that Americans are able to use 911 to reach those who can help,” Pai said in a statement. “This inquiry will include an examination of the effect that CenturyLink’s outage appears to have had on other providers’ 911 services.”

A retired manager at Qwest, a former Baby Bell now owned by CenturyLink, strongly criticized CenturyLink’s lack of communications with customers and an apparent lack of network redundancy.

“For a company in the communication business, they sure failed on this,” said Albuquerque resident Sam Martin. “I participated on the Qwest Disaster Recovery teams, and I do not recall ever having the network down for this kind of time and certainly never the 911 network. The 911 network should never have been down. The lack of this network can contribute to delays in rescue and fire saving lives.”

Martin is dubious about CenturyLink’s explanation for the network outage, suggesting a defective network card may be only a part of the problem.

“The explanations given so far are not valid,” Martin said. “The public may not be aware of it, but the communication network has redundancy and for essential services like inter-office trunking and 911 calls, there are duplicate fiber optic feeds – “rings” that duplicate the main circuit in another path – and switching equipment to these locations so that they may be switched electronically and automatically upon failure to a back-up network ring. When these systems are operating properly, the customer is unaware a failure occurred. If the automatic switching does not take place, employees involved with disaster recovery can intervene and manually switch the affected network to another fiber ring or electronic hub and service is restored until the actual damage is fixed.”

None of those things appeared to happen in this case, and the outage persisted for 48 hours before all services were restored.

“CenturyLink has to have a disaster recovery plan with redundancies in place for electrical, inbound and outbound local and toll-free carriers, as well as network and hardware component redundancies. CenturyLink should be able to switch between multiple fiber optic rings or central offices in case entire networks of phones go down. They would then locate and repair, or replace, defective telecommunication components without the customer ever knowing. The fact that this did not happen is discouraging and scary for the consumer. The fact that it happened nationwide is even more surprising and disturbing. Hopefully the truth will come out soon.”

A critical editorial in the Albuquerque Journal added:

We need answers from CenturyLink beyond the cryptic “a network element” caused the outage. We need to know how many CenturyLink and Verizon customers were affected. And we need to know what they – and other internet and phone providers – are doing to prevent similar outages or worse from happening in the future. Because if the outage showed nothing else, it’s that like an old-time string of Christmas lights, we are living in an interconnected world.

And when one light goes out, they can all go out.

KTVB in Boise, Idaho reported on CenturyLink’s massive outage on Dec. 27-28 and how it impacted local businesses and government services. (3:09)

Vermont residents want better internet service and protection for universal access to phone service, even if customers have to pay a surcharge on their bill to make sure the traditional landline is still available in 100% of the state.

In contrast, some residents complain, Vermont regulators want to make life easier for a telecom industry that wants to abandon universal service, fails to connect rural customers to the internet, and has left major cell phone signal gaps around the state.

In 2017, Vermont commissioned two surveys that asked 400 business and 418 residential customers about their telecommunications services. It was quickly clear from the results that most wanted some changes.

Vermont is a largely rural, small state that presents a difficult business case for many for-profit telecom companies. Verizon Communications sold its landline business in northern New England, including Vermont, to FairPoint Communications in 2007. FairPoint limped along for several years until it declared bankruptcy and was eventually acquired by Consolidated Communications, which still provides telephone service to most of the state.

Many rural areas are not furnished with anything close to the FCC’s definition of broadband (25/3 Mbps). DSL service at speeds hovering around 4 Mbps is still common, especially in areas relying on Verizon’s old copper wire infrastructure.

Comcast dominates cable service in the state, except for a portion of east-central Vermont, which is served by Charter Communications (Spectrum). Getting cable broadband service in rural Vermont remains challenging, as those areas usually fail the Return On Investment test. Still, 85% of respondents said they had internet service available in their home. Vermont’s Department of Public Service (DPS) estimates about 7% of homes in Vermont have no internet provider offering service, with a larger percentage served only by the incumbent phone company.

The majority of residents own cell phones, with Verizon (47.5%) and AT&T (31.8%) dominating market share. Sprint has 0.6% of the market; T-Mobile has a 1.2% share, both largely due to coverage issues. But even with Verizon and AT&T, 40% of respondents said cell phone companies cannot deliver a signal in their homes. As a result, many residents are hanging on to landline service, which is considered more reliable than cell service. Some 40% of those relying on cell phones said they would consider going back to a landline for various reasons.

With the FCC ready to cut support funding for rural landline service, residents were asked if the state should maintain funding to assure continued universal access to landline service. Among respondents, 87.8% thought it was either “very important” or “somewhat important.” The study also found 51% would accept a general statewide rate increase to cover those costs, while 29% preferred rate increases be targeted to rural ratepayers only. About 16% did not like either idea, and 4% liked both.

When asked whether residents would be interested in having a fiber connection in their home, 80% of respondents said “yes,” but 30% were unwilling to pay a higher bill to get it.

Ironically, the state’s most aggressive residential fiber buildouts are run by some of the smallest community-owned internet providers, rural electric or telecom co-ops, and small independent phone companies. What makes these smaller providers different is that they answer to their customers, not shareholders.

Comcast and Charter are the two largest cable companies in Vermont.

Recently, the DPS released a 100+ page final draft of its 2018 Vermont Telecommunications Plan, setting out a vision of where Vermont should be a decade from now, especially regarding rural broadband expansion, pole attachment issues, and cell tower/small cell zoning.

The report acknowledges the state’s own existing rural broadband expansion fund is woefully inadequate. In 2017, the Vermont Universal Service Fund paid out $220,000 to assist ISPs with building out networks to rural areas.

“The amount of money available to the fund pales in comparison to the amount of funding requests that the Department receives, which is generally in the millions of dollars,” the draft report states. “With approximately 20,000 unserved and underserved addresses in Vermont, the Connectivity Initiative cannot make a meaningful dent in the number of underserved locations.”

Much of the report focuses on ideas to lure incumbent providers into volunteering to expand their networks, either through deregulation, pole attachment reform, direct subsidies, or cost sharing arrangements that split the expenses of network extensions between providers, the state, and residents.

A significant weakness in the report covered the forthcoming challenge of upgrading a state like Vermont with 5G wireless service:

Small cell deployment has been attempted along rural routes with very limited success and the national efforts to expand small-cell, distributed-antenna systems, and 5G upgrades have focused on urban areas. The common refrain on 5G is that ‘it’s not coming to rural America.’ 5G should come to rural Vermont and the state should take efforts to improve its reach into rural areas.

A small cell attached to a light pole.

To accomplish this, the report only recommends streamlining the permitting process for new cell towers and small cells. The report says nothing about how the state can make a compelling case to convince providers to spend millions to deploy small cells in rural areas.

Where for-profit providers refuse to provide service, local communities and co-op phone or electric providers often step into the void. But Vermont prohibits municipalities from using tax dollars to fund telecommunications projects, which the law claims was designed to protect communities from investing in ‘unprofitable broadband networks.’

The draft proposal offers a mild recommendation to change the law to allow towns to bond for some capital expenditures on existing or starting networks:

Vermont could use a similar program to help start Communications Union Districts as well as allow towns to invest in existing networks of incumbent providers. Limitations on the authority to bond would need to be put in place. Such limitations should include focusing capital to underserved locations only, limiting the amount (or percentage) of tax payer dollars allowed to be collateralized, and setting technical requirements for the service.

Such restrictions often leave community broadband projects untenable and lacking a sufficient service area or customer base to cover construction and operating expenses. Instead, only the most difficult and costly-to-serve areas would be served. The current law also prevents local communities from making decisions on the local level to meet local needs.

At a public hearing last Tuesday in Montpelier, the report faced immediate criticism from a state legislator and some consumers for being vague and lacking measurable, attainable goals.

“This plan does not appear to move Vermont ahead in a way that enables it to compete effectively,” said state Sen. Mark MacDonald (D-Orange). “It doesn’t seem to be moving forward at the pace we’re capable of.”

Clay Purvis, director of telecommunications and connectivity for the DPS, defended the plan by alluding to how the state intends to establish a cooperative environment for internet service providers and help cut through red tape. But much of those efforts are focused on resolving controversial pole attachment fees and disputes, assuming there are providers fighting to place their competing infrastructure on those utility poles.

Montpelier resident Stephen Whitaker was profoundly underwhelmed by the report.

“There is no plan at all here,” Whitaker said at Tuesday’s public hearing. “There is some new background information from the 2014 version, but there’s no plan there. There is no objective to reach the statutory goals.”

Despite a strong economy, Verizon Communications will shed 10,400 employees and cut $10 billion in costs as part of a transformation initiative promoted by the company’s newest top executive.

“These changes are well-planned and anticipated, and they will be seamless to our customers,” said Verizon CEO Hans Vestberg. “This is a moment in time, given our financial and operational strength, to begin to better serve customers with more agility, speed and flexibility.”

For Andrew Challenger, vice president of Challenger, Gray & Christmas, an outplacement firm that closely monitors corporate layoffs, Verizon’s willingness to let go of 7% of their workforce is an ominous sign of possible additional job reductions in the future.

Vestberg has advocated reorienting Verizon towards a potentially lucrative 5G wireless future. The estimated $10 billion in cost savings could placate investors on Wall Street alarmed about increased spending Verizon is likely to undertake to deploy 5G infrastructure over the next five years.

In October, Verizon offered more than 44,000 employees a voluntary buyout package and announced it would transfer thousands of current employees to Infosys, an outsourcing company headquartered in India. The voluntary separation package included up to 60 weeks’ salary, bonus and benefits, depending on length of service. This morning, accepted participants received word of their last day of employment, which will be the last day of this year or at the end of March or June, 2019. Verizon currently has 152,300 employees.

Challenger believes Verizon is likely to continue letting employees go as the company faces ongoing pressures on its landline and business service units and endures cord-cutting for its FiOS fiber to the home service. Challenger told CNBC the 44,000 workers who took Verizon’s offer likely made a smart decision. Companies that offer voluntary buyouts in good times can be a sign of likely layoffs when the economy slows down. With record low unemployment, the Verizon workers leaving the telecom company are likely to find new jobs much easier than those forced to look during a recession.

Verizon employees transferred to Infosys may be among the next to be targeted in future layoffs, according to Challenger. Verizon workers will be working closely with low-paid Indian staffers who may eventually replace them.

Workers who are assigned to train cheaper workers should keep their eyes open and resumes ready, Challenger warned.

Verizon Communications is on track to gradually cut up to a quarter of its workforce, and has now offered 44,000 employees “voluntary severance” packages, while also outsourcing many information technology jobs to India’s Infosys, Ltd., in a deal worth an estimated $700 million.

The Wall Street Journal received confirmation from Verizon this afternoon the company is seeking to reduce its workforce to cut $10 billion in costs and invest in its forthcoming 5G wireless network.

Analysts claim Verizon’s new CEO Hans Vestberg chose the voluntary layoff route in part to send a message to investors and Wall Street before the company embarks on a costly upgrade to its wireless network. Wall Street generally dislikes and downgrades companies starting large, long-term spending projects. By cutting its workforce and other expenses, Verizon hopes to offset some of the sting of that spending to appease investors.

To entice employees to retire, Verizon is offering three weeks’ pay for each year of service up to 60 weeks. But not every employee will qualify for the offer. About 2,500 Verizon workers employed in the company’s IT department have been notified their jobs are being transferred to Bengaluru, India-based Infosys, Ltd., where they will continue work for Verizon as outsourced contractors. Verizon has notified affected workers they do not qualify for the current voluntary severance offer and will lose their 2018 bonus if they refuse to accept a position at Infosys.

In all, about 30 percent of Verizon’s 153,100 employees have either been offered early retirement deals or are facing an involuntary transfer to Infosys. Affected employees are being told Verizon is investing “more in transforming the business versus running the business,” which may suggest additional outsourcing and third-party contracting arrangements may be forthcoming.

Infosys has a controversial record in the United States. The company has been accused of forcing high-paid American workers to train their low wage foreign replacements, often arriving from India on the H-1B visa. Those workers typically remain in the U.S. for about a year and temporarily manage job functions that eventually are transitioned to workers back in India or other developing countries.

Verizon claims it expects up to 1,000 workers to accept the voluntary severance offer, but has not indicated whether additional forced layoffs may follow.

Verizon has dramatically downsized since 2011, when it employed 195,900 workers — shedding more than 42,000 employees in the last seven years.

WASHINGTON (Reuters) – Four industry groups representing major internet providers and cable companies filed suit on Wednesday seeking to block California’s new law to mandate net neutrality rules.

The groups represent companies including AT&T Inc, Verizon Communications Inc, Comcast Corp and Charter Communications Inc. The lawsuit came after the U.S. Justice Department on Sunday filed its own lawsuit to block the new law.

The lawsuit filed by the American Cable Association, CTIA – The Wireless Association, NCTA – The Internet & Television Association and USTelecom – The Broadband Association, called California’s law a “classic example of unconstitutional state regulation” and urged the court to block it before it is set to take effect Jan. 1.

U.S. Attorney General Jeff Sessions said on Sunday in a statement that the “the California legislature has enacted an extreme and illegal state law attempting to frustrate federal policy.”

This marked the latest clash between the Trump administration and California, which have sparred over environmental, immigration and other hot-button issues.

In December, the Federal Communications Commission said in repealing the Obama-era rules that it was preempting states from setting their own rules governing internet access.

California Attorney General Xavier Becerra said on Sunday the Trump Administration was ignoring “millions of Americans who voiced strong support for net neutrality rules.”

The Trump administration rules were a win for internet providers but opposed by companies like Facebook Inc, Amazon.com Inc and Alphabet Inc.

Under President Donald Trump, the FCC voted 3-2 in December along party lines to reverse rules that barred internet service providers from blocking or throttling traffic or offering paid fast lanes, also known as paid prioritization.

In August, 22 states and a coalition of trade groups representing major tech companies urged a federal appeals court to reinstate the rules. The states argue that the FCC cannot preempt state rule because it is not setting any limits on conduct by internet providers.

A federal judge on Monday set a Nov. 14 hearing in Sacramento on the Justice Department lawsuit.