What do an emergency operations center in Cochise County, Ariz., Colorado hospitals, the Idaho Bureau of Corrections, and many 911 call centers across Massachusetts have in common? They were all brought down by a two-day nationwide CenturyLink outage from Dec. 27-28 that also resulted in internet outages for tens of thousands of CenturyLink’s residential customers. The cause? CenturyLink blamed a single, faulty third-party network management card in Denver for disrupting services for CenturyLink and other phone companies, notably Verizon, from Alaska to Florida.

Hours after outage began, two days after Christmas, CenturyLink issued a general statement:

“CenturyLink experienced a network event on one of our six transport networks beginning on December 27 that impacted voice, IP, and transport services for some of our customers. The event also impacted CenturyLink’s visibility into our network management system, impairing our ability to troubleshoot and prolonging the duration of the outage.”

That “network event” caused serious disruptions to critical services in 37 states, including 911, according to Brian Kyes, president of the Massachusetts Major City Chiefs of Police Association.

“This is affecting 911 (wireline & wireless) delivery to most of Massachusetts,” Kyes said in a statement during the outage to the Boston Herald. “We have heard from MEMA that this issue may also affect some landlines but I have not heard of any specific situations or communities that have been impacted. We are advising all police and fire chiefs to test their local 911 systems and notify their residents of potential issues by reverse 911, social media or any other means that they have at their disposal. The interruption in service may depend on a particular phone carrier and the information that we have is that it may be intermittent.”

CenturyLink outages on Dec. 27, 2018. (Image: Downdetector.com)

The disruptions affected much of Massachusetts — a state served primarily by Verizon Communications, because CenturyLink is a major commercial services vendor inside and outside of its local landline service areas and supplies some connectivity services to Verizon, mostly for wireless customers.

ATM networks also went down in certain parts of the country. CenturyLink is one of many vendors providing data connectivity between the cashpoint machines and several banking institutions.

Also impacted, the Idaho Department of Corrections, including inmate phone systems, and the Idaho Department of Education, which lost the ability to make or receive calls.

Consumers also noticed their internet connections were often down or sporadic in some locations, primarily because CenturyLink’s backbone network became saturated with rogue packets.

The Denver Post presented a more detailed technical explanation about the outage:

CenturyLink said the [defective] card was propagating “invalid frame packets” that were sent out over its secondary network, which controlled the flow of data traffic.

“Once on the secondary communication channel, the invalid frame packets multiplied, forming loops and replicating high volumes of traffic across the network, which congested controller card CPUs (central processing unit) network-wide, causing functionality issues and rendering many nodes unreachable,” the company said in a statement.

Once the syndrome gets going, it can be difficult to trace back to its original source and to stop, a big reason networks are designed to isolate failures early and contain them.

“We have learned through experience about these different types of failure modes. We build our systems to try and localize those failures,” said Craig Partridge, chair of the computer science department at Colorado State University in Fort Collins and a member of the Internet Hall of Fame. “I would hope that what is going on is that CenturyLink is trying to understand why a relatively well-known failure mode has bit them.”

The Federal Communications Commission also expects answers to some questions, opening another investigation of the phone company. In 2015, CenturyLink agreed to pay a $16 million settlement to the federal agency after a seven-state outage in April 2014.

Pai

FCC Chairman Ajit Pai said the agency would once again take a look at CenturyLink, focusing on disruptions to emergency services.

“When an emergency strikes, it’s critical that Americans are able to use 911 to reach those who can help,” Pai said in a statement. “This inquiry will include an examination of the effect that CenturyLink’s outage appears to have had on other providers’ 911 services.”

A retired manager at Qwest, a former Baby Bell now owned by CenturyLink, strongly criticized CenturyLink’s lack of communications with customers and an apparent lack of network redundancy.

“For a company in the communication business, they sure failed on this,” said Albuquerque resident Sam Martin. “I participated on the Qwest Disaster Recovery teams, and I do not recall ever having the network down for this kind of time and certainly never the 911 network. The 911 network should never have been down. The lack of this network can contribute to delays in rescue and fire saving lives.”

Martin is dubious about CenturyLink’s explanation for the network outage, suggesting a defective network card may be only a part of the problem.

“The explanations given so far are not valid,” Martin said. “The public may not be aware of it, but the communication network has redundancy and for essential services like inter-office trunking and 911 calls, there are duplicate fiber optic feeds – “rings” that duplicate the main circuit in another path – and switching equipment to these locations so that they may be switched electronically and automatically upon failure to a back-up network ring. When these systems are operating properly, the customer is unaware a failure occurred. If the automatic switching does not take place, employees involved with disaster recovery can intervene and manually switch the affected network to another fiber ring or electronic hub and service is restored until the actual damage is fixed.”

None of those things appeared to happen in this case, and the outage persisted for 48 hours before all services were restored.

“CenturyLink has to have a disaster recovery plan with redundancies in place for electrical, inbound and outbound local and toll-free carriers, as well as network and hardware component redundancies. CenturyLink should be able to switch between multiple fiber optic rings or central offices in case entire networks of phones go down. They would then locate and repair, or replace, defective telecommunication components without the customer ever knowing. The fact that this did not happen is discouraging and scary for the consumer. The fact that it happened nationwide is even more surprising and disturbing. Hopefully the truth will come out soon.”

A critical editorial in the Albuquerque Journal added:

We need answers from CenturyLink beyond the cryptic “a network element” caused the outage. We need to know how many CenturyLink and Verizon customers were affected. And we need to know what they – and other internet and phone providers – are doing to prevent similar outages or worse from happening in the future. Because if the outage showed nothing else, it’s that like an old-time string of Christmas lights, we are living in an interconnected world.

And when one light goes out, they can all go out.

KTVB in Boise, Idaho reported on CenturyLink’s massive outage on Dec. 27-28 and how it impacted local businesses and government services. (3:09)

AT&T customers with Samsung Galaxy 8 Active or LG’s V30 or V40 smartphones began noticing a new icon on their phones starting last weekend: an italicized 5GE, leading some to believe 5G wireless service has now reached AT&T’s network.

Not so fast, AT&T.

AT&T’s use of 5GE, which stands for “5G Evolution” in AT&T’s techie parlance, is another example of how wireless carriers exploit up and coming technology upgrades that are unprotected from overzealous marketing misuse. The actual 5G standard is different from 5GE, and customers using 5G on millimeter wave frequencies can expect very different performance in comparison to today’s 4G LTE experience. But with 5G being hyped in the media, AT&T is attempting to capture some of that excitement for itself.

The company’s marketing division managed to accomplish a speed and technology upgrade without spending millions of dollars on actual 5G network upgrades — just by changing an icon on customers’ phones and making them believe they are getting a 5G experience. In fact, 5GE is actually just the latest evolution of 4G LTE already known to Verizon customers as LTE-Advanced or LTE Plus on Sprint’s network — technology including carrier aggregation, 256 QAM, and 4×4 MIMO that has been in use on competing cellular networks in the U.S. since at least 2016. But just as Verizon customers saw significant speed improvements from Verizon’s updates to the 4G LTE standard, as AT&T deploys similar upgrades in each of its markets, customers should notice similar performance improvements.

AT&T claims 5GE is already live in 400+ markets with more to come. In the short term, the “upgrade” that was pushed to AT&T network devices last weekend only switched on the 5GE icon, which will mean little to AT&T customers already reached by 5GE and never knew it until this past weekend, and nothing to those still waiting for the upgrade to arrive.

Walter Piecyk, an analyst at BTIG Research, says AT&T’s latest spectrum deployments will matter more than whatever the company brands its latest upgrade, and could eventually allow AT&T to surpass Verizon Wireless in network performance.

AT&T’s recent effort to improve its network by deploying more wireless spectrum — up to 60 MHz in many areas, is not the 5G upgrade customers might expect, but it will deliver faster speeds and more performance on today’s smartphones.

AT&T calls its forthcoming actual 5G network 5G+, and the company is launching a modest but authentic 5G experience in limited “innovation zones” in Jacksonville, Fla., Atlanta, Ga., Indianapolis, Ind., Louisville, Ky., New Orleans, La., Charlotte and Raleigh, N.C., Oklahoma City, Okla., as well as Dallas, Houston, San Antonio, and Waco, Tex.

In a money-saving maneuver, AT&T’s combined spectrum upgrades include 20 MHz of FirstNet first responder spectrum (prime 700 MHz spectrum shared with AT&T customers except during emergencies) it received in 2017, 20 MHz of AWS-3 spectrum (1755-1780 MHz for uplink operations and 2155-2180 MHz for downlink) it acquired for $18 billion in 2015, and 20 MHz of WCS spectrum (2300 MHz) it acquired from NextWave for $650 million back in 2012. All of this spectrum is expected to be activated at the same time as technicians work to upgrade each AT&T cell tower. This dramatically cuts AT&T’s costs and truck rolls for incremental upgrades.

AT&T calls its improved 4G LTE network “5G Evolution”

“We’re turning up not only the FirstNet spectrum that we got, but all of this other spectrum that we’ve acquired over the last few years,” AT&T CEO Randall Stephenson told investors at a December conference. “So as we climb these cell towers, we turn up the spectrum. By the time we get to end of 2019, we will have increased the capacity on AT&T’s network by 50%. I mean, you just have to pause and think about this. The entire AT&T wireless network capacity is going to increase over the next 14 months by 50%. I mean, that’s huge.”

Some areas have already received partial upgrades, others may find newly improved rural coverage as AT&T meets its commitments to the government’s FirstNet platform, which calls for more robust rural coverage. Some areas that never had AT&T coverage before may get it for the first time.

AT&T’s biggest competitor, Verizon, has commanded a lead in 4G LTE coverage from 2010 forward after utilizing a considerable amount of its available spectrum for the faster standard. But Verizon has not been a robust bidder for new spectrum recently, except for the millimeter wave frequencies it bought for its emerging 5G network. It has some additional unused AWS-3 spectrum it can use for expansion, but Piecyk believes Verizon may already be using those frequencies in many markets where it is likely facing a spectrum crunch.

While AT&T lights up 60 MHz of additional spectrum, Verizon is primarily depending on the ongoing conversion of 10-15 MHz of existing spectrum it now uses for 3G service to LTE each year. But the company is reportedly running out of frequencies in areas where data demand requires that extra spectrum the most.

The only short term solution for Verizon, which is not participating in marketing hoopla like 5GE, is to make its current spectrum more efficient. That means more cell towers sharing the same frequencies to reduce the load on each tower, improved antenna technology, and using newly available spectrum in the CBRS and millimeter wave bands to manage network traffic. Verizon may even use unlicensed shared spectrum to handle some of the load. Unfortunately, smartphones equipped to take advantage of these new bands are not yet available and may not be until 2020.

For AT&T, improved network performance is seen as a key to resume robust growth in new subscribers.  After Verizon dramatically improved its LTE network in 2014, AT&T stopped growing its lucrative post-paid phone subscriber base, according to Piecyk. Now it may be AT&T’s chance to turn the tables on Verizon.

This AT&T produced video helps consumers understand what 5G, beam forming, small cells, and coverage differences between 4G and 5G are all about. Notice the 5G trial speed test showed download speeds topping out at around 137 Mbps. (4:26)

Vermont residents want better internet service and protection for universal access to phone service, even if customers have to pay a surcharge on their bill to make sure the traditional landline is still available in 100% of the state.

In contrast, some residents complain, Vermont regulators want to make life easier for a telecom industry that wants to abandon universal service, fails to connect rural customers to the internet, and has left major cell phone signal gaps around the state.

In 2017, Vermont commissioned two surveys that asked 400 business and 418 residential customers about their telecommunications services. It was quickly clear from the results that most wanted some changes.

Vermont is a largely rural, small state that presents a difficult business case for many for-profit telecom companies. Verizon Communications sold its landline business in northern New England, including Vermont, to FairPoint Communications in 2007. FairPoint limped along for several years until it declared bankruptcy and was eventually acquired by Consolidated Communications, which still provides telephone service to most of the state.

Many rural areas are not furnished with anything close to the FCC’s definition of broadband (25/3 Mbps). DSL service at speeds hovering around 4 Mbps is still common, especially in areas relying on Verizon’s old copper wire infrastructure.

Comcast dominates cable service in the state, except for a portion of east-central Vermont, which is served by Charter Communications (Spectrum). Getting cable broadband service in rural Vermont remains challenging, as those areas usually fail the Return On Investment test. Still, 85% of respondents said they had internet service available in their home. Vermont’s Department of Public Service (DPS) estimates about 7% of homes in Vermont have no internet provider offering service, with a larger percentage served only by the incumbent phone company.

The majority of residents own cell phones, with Verizon (47.5%) and AT&T (31.8%) dominating market share. Sprint has 0.6% of the market; T-Mobile has a 1.2% share, both largely due to coverage issues. But even with Verizon and AT&T, 40% of respondents said cell phone companies cannot deliver a signal in their homes. As a result, many residents are hanging on to landline service, which is considered more reliable than cell service. Some 40% of those relying on cell phones said they would consider going back to a landline for various reasons.

With the FCC ready to cut support funding for rural landline service, residents were asked if the state should maintain funding to assure continued universal access to landline service. Among respondents, 87.8% thought it was either “very important” or “somewhat important.” The study also found 51% would accept a general statewide rate increase to cover those costs, while 29% preferred rate increases be targeted to rural ratepayers only. About 16% did not like either idea, and 4% liked both.

When asked whether residents would be interested in having a fiber connection in their home, 80% of respondents said “yes,” but 30% were unwilling to pay a higher bill to get it.

Ironically, the state’s most aggressive residential fiber buildouts are run by some of the smallest community-owned internet providers, rural electric or telecom co-ops, and small independent phone companies. What makes these smaller providers different is that they answer to their customers, not shareholders.

Comcast and Charter are the two largest cable companies in Vermont.

Recently, the DPS released a 100+ page final draft of its 2018 Vermont Telecommunications Plan, setting out a vision of where Vermont should be a decade from now, especially regarding rural broadband expansion, pole attachment issues, and cell tower/small cell zoning.

The report acknowledges the state’s own existing rural broadband expansion fund is woefully inadequate. In 2017, the Vermont Universal Service Fund paid out $220,000 to assist ISPs with building out networks to rural areas.

“The amount of money available to the fund pales in comparison to the amount of funding requests that the Department receives, which is generally in the millions of dollars,” the draft report states. “With approximately 20,000 unserved and underserved addresses in Vermont, the Connectivity Initiative cannot make a meaningful dent in the number of underserved locations.”

Much of the report focuses on ideas to lure incumbent providers into volunteering to expand their networks, either through deregulation, pole attachment reform, direct subsidies, or cost sharing arrangements that split the expenses of network extensions between providers, the state, and residents.

A significant weakness in the report covered the forthcoming challenge of upgrading a state like Vermont with 5G wireless service:

Small cell deployment has been attempted along rural routes with very limited success and the national efforts to expand small-cell, distributed-antenna systems, and 5G upgrades have focused on urban areas. The common refrain on 5G is that ‘it’s not coming to rural America.’ 5G should come to rural Vermont and the state should take efforts to improve its reach into rural areas.

A small cell attached to a light pole.

To accomplish this, the report only recommends streamlining the permitting process for new cell towers and small cells. The report says nothing about how the state can make a compelling case to convince providers to spend millions to deploy small cells in rural areas.

Where for-profit providers refuse to provide service, local communities and co-op phone or electric providers often step into the void. But Vermont prohibits municipalities from using tax dollars to fund telecommunications projects, which the law claims was designed to protect communities from investing in ‘unprofitable broadband networks.’

The draft proposal offers a mild recommendation to change the law to allow towns to bond for some capital expenditures on existing or starting networks:

Vermont could use a similar program to help start Communications Union Districts as well as allow towns to invest in existing networks of incumbent providers. Limitations on the authority to bond would need to be put in place. Such limitations should include focusing capital to underserved locations only, limiting the amount (or percentage) of tax payer dollars allowed to be collateralized, and setting technical requirements for the service.

Such restrictions often leave community broadband projects untenable and lacking a sufficient service area or customer base to cover construction and operating expenses. Instead, only the most difficult and costly-to-serve areas would be served. The current law also prevents local communities from making decisions on the local level to meet local needs.

At a public hearing last Tuesday in Montpelier, the report faced immediate criticism from a state legislator and some consumers for being vague and lacking measurable, attainable goals.

“This plan does not appear to move Vermont ahead in a way that enables it to compete effectively,” said state Sen. Mark MacDonald (D-Orange). “It doesn’t seem to be moving forward at the pace we’re capable of.”

Clay Purvis, director of telecommunications and connectivity for the DPS, defended the plan by alluding to how the state intends to establish a cooperative environment for internet service providers and help cut through red tape. But much of those efforts are focused on resolving controversial pole attachment fees and disputes, assuming there are providers fighting to place their competing infrastructure on those utility poles.

Montpelier resident Stephen Whitaker was profoundly underwhelmed by the report.

“There is no plan at all here,” Whitaker said at Tuesday’s public hearing. “There is some new background information from the 2014 version, but there’s no plan there. There is no objective to reach the statutory goals.”

Despite a strong economy, Verizon Communications will shed 10,400 employees and cut $10 billion in costs as part of a transformation initiative promoted by the company’s newest top executive.

“These changes are well-planned and anticipated, and they will be seamless to our customers,” said Verizon CEO Hans Vestberg. “This is a moment in time, given our financial and operational strength, to begin to better serve customers with more agility, speed and flexibility.”

For Andrew Challenger, vice president of Challenger, Gray & Christmas, an outplacement firm that closely monitors corporate layoffs, Verizon’s willingness to let go of 7% of their workforce is an ominous sign of possible additional job reductions in the future.

Vestberg has advocated reorienting Verizon towards a potentially lucrative 5G wireless future. The estimated $10 billion in cost savings could placate investors on Wall Street alarmed about increased spending Verizon is likely to undertake to deploy 5G infrastructure over the next five years.

In October, Verizon offered more than 44,000 employees a voluntary buyout package and announced it would transfer thousands of current employees to Infosys, an outsourcing company headquartered in India. The voluntary separation package included up to 60 weeks’ salary, bonus and benefits, depending on length of service. This morning, accepted participants received word of their last day of employment, which will be the last day of this year or at the end of March or June, 2019. Verizon currently has 152,300 employees.

Challenger believes Verizon is likely to continue letting employees go as the company faces ongoing pressures on its landline and business service units and endures cord-cutting for its FiOS fiber to the home service. Challenger told CNBC the 44,000 workers who took Verizon’s offer likely made a smart decision. Companies that offer voluntary buyouts in good times can be a sign of likely layoffs when the economy slows down. With record low unemployment, the Verizon workers leaving the telecom company are likely to find new jobs much easier than those forced to look during a recession.

Verizon employees transferred to Infosys may be among the next to be targeted in future layoffs, according to Challenger. Verizon workers will be working closely with low-paid Indian staffers who may eventually replace them.

Workers who are assigned to train cheaper workers should keep their eyes open and resumes ready, Challenger warned.