OpenSignal’s State of American Wireless Networks – Aug. 2017

Verizon Wireless customers are seeing declining wireless internet speeds and the greater potential for congestion because Verizon Wireless is experiencing the impact of some overburdened cell sites in some of its largest markets.

Walter Piecyk from BTIG Research reports over the last few weeks, Verizon has begun using the last 10MHz of PCS spectrum left in its inventory in New York City, nine months earlier than expected.

Verizon’s reserve spectrum in PCS Band 2 near 1900MHz is not as ideal as lower frequency spectrum better able to manage inside buildings in a city as densely packed as New York, but if that is all the company has left for immediate use, that is what it will use. The newly activated frequencies, first uncovered by Milan Milanovic, are not yet operational across all of Verizon’s extensive cell network in the Big Apple. Verizon’s need to activate its last remaining PCS frequencies suggests former chief financial officer Fran Shammo may have been overly optimistic when he claimed Verizon was only using 40% of its spectrum inventory. That may be true in smaller cities, but is no longer the case in large metropolitan areas.

“This latest action also means that the only spectrum Verizon has left to convert to LTE in NYC is the 25MHz of 800MHz spectrum that the FCC gave it for free in 1984,” wrote Piecyk. “Unfortunately, that 800MHz spectrum is being used to support CDMA voice traffic and legacy 3G data for enterprise/IoT applications. Meanwhile, Dish sits on 125MHz of vacant spectrum in NYC.”

BTIG Research has been carefully tracking Verizon’s deployment of its spectrum for years. In New York, LTE expansion has depended heavily on spectrum acquisitions and enabling LTE+, which bonds frequencies together to increase speed and capacity.

BTIG Research Tracks Verizon Wireless’ LTE Deployment in NYC

• 20 MHz: December 2010 – launched LTE on the 20MHz of 700MHz spectrum it bought in the 2008 700MHz auction for $0.46/MHz/POP for the Northeast regional license and $0.77/MHz/POP nationwide.
• 40 MHz: December of 2013 – XLTE-branded rollout of AWS spectrum, which mainly included the spectrum it bought from Cable in 2011 for $0.69/MHz/POP, but also the spectrum it acquired in the 2006 AWS-1 auction, where it spent $1.33/MHz/POP for the Northeast regional license and $0.73/MHz/POP overall.
• 20 MHz: December of 2014 – LTE conversion begins on PCS spectrum. Verizon purchased 10MHz from Northcoast as part of a larger transaction valued at $1.58/MHz/POP in 2003, 10MHz covering NYC from NextWave for $4.63/MHz/POP in 2004, and 20MHz from NextWave in 2005 as part of a larger transaction valued at $2.85/MHz/POP. (Link)
• 10 MHz: Q1 of 2016 – This enabled Verizon to deliver 15MHz x 15MHz connections on Band 2, thereby improving speeds. When this happened we predicted the remaining PCS spectrum would be used in early 2018. (Link)
• 10 MHz: Q3 of 2017 – Once again, this was spotted by Milanovic (Link), who notes that it has not been deployed on all sites. This effectively expands the Band 2 deployment to a 20MHz x 20MHz deployment.

The company has also attempted to increase capacity with network densification, which adds more cell sites to divide up the traffic load. But activating a new cell site can take years, especially if Verizon encounters zoning and permitting problems or public opposition. Small cells can ease congestion in particularly dense traffic areas, but are not enough alone to deal with increasing network traffic.

Verizon’s own business practices have also complicated things for the wireless company. Ditching two-year contracts and subsidized phones in favor of customers acquiring devices at retail prices financed through wireless carriers like Verizon have led to a slowdown in subscriber upgrades as consumers hold on to their devices for longer.

Most phones acquired in the last year or two now support Voice over LTE (VoLTE), which means phone calls travel over Verizon’s LTE network, not the legacy CDMA network Verizon has used for well over a decade. Verizon has to dedicate a significant amount of prime spectrum in the 850MHz band for its CDMA network. Although Verizon claims it has migrated “more than 50%” of its voice traffic to the newer, more efficient VoLTE standard, that is below analysts’ expectations.

Piecyk thinks it may be possible Verizon has been slow to convert because of the record low phone upgrade rate of its customers. As a result, it cannot repurpose its CDMA spectrum for LTE use. Discussions with Verizon engineers suggest the company may eventually cut back CDMA spectrum, but will likely still keep 5 x 5MHz reserved for CDMA voice calling for at least the next four years to support its customers with older devices.

As part of its network densification effort, Verizon is once again relying on fiber optic buildouts, some of which it may take on itself in areas where it does not provide landline service. Verizon will be placing cables with 1,700 strands of fiber, so it is obviously thinking about future network demands.

Before it can deploy additional upgrades or acquire more spectrum, customers can anticipate more “network management” techniques, suspects Piecyk, especially now that unlimited data plans are for sale again. Verizon already limits its “unlimited” plan to 22GB of usage per month, before wireless data speeds are throttled. OpenSignal believes Verizon’s recent speed drops are a result of its unlimited plans putting more pressure on its network.

“We suspect management will now follow T-Mobile’s lead and suppress video quality like BingeOn to help with the rise in network traffic,” Piecyk wrote. “They might also discuss control of overall peak data speeds. However, if no mobile applications require more than 10Mbps service, would it make any sense to suppress the speeds on your customers’ phone? What’s the benefit other than offering a convenient excuse on why your speed tests are slower than the competition?”

South Jersey: The worst broadband problems are in the southernmost counties closest to Delaware.

Customers hoping New Jersey’s telecom regulator would compel Verizon to expand fiber to the home service across southern New Jersey are out of luck.

The New Jersey Board of Public Utilities (BPU) approved a settlement between Verizon New Jersey, Inc., Cumberland County, and 18 southern New Jersey towns that alleged Verizon failed to properly maintain its wireline network in areas where it has chosen not to deploy FiOS — its fiber to the home service. But the settlement will only compel Verizon to maintain its existing copper network and offer token DSL and FiOS expansion in some unserved rural communities.

“We have heard our customers’ concerns in South Jersey and are pleased to have reached an agreement with the approval of all 17 towns on a maintenance plan going forward,” said Ray McConville, a Verizon spokesman. “We look forward to staying in regular communication with the towns to ensure our customers continue to receive the level of service they expect and deserve.”

“While the Board was fully prepared to proceed on this matter, the parties were able to reach a negotiated settlement which takes into consideration the needs of each community,” said Richard S. Mroz, president, N.J. Board of Public Utilities.

But some residents of those communities beg to differ.

“It’s another example of Chris Christie’s hand-picked regulators letting Verizon off the hook and sticking us in a digital divide,” complained Jeff Franklin, a Verizon DSL customer in Cumberland County. “Verizon should not be allowed to offer one half of the state modern broadband while sticking the rest of us with its slow DSL service.”

Franklin is upset that communities bypassed by Verizon’s FiOS network appear to have little chance of getting it in the future, now that regulators have agreed to allow Verizon to fix its own copper network.

“All the Board did was force Verizon to do what it should have been doing all along, taking care of its own network,” Franklin complained to Stop the Cap! 

Verizon did agree to expand its fiber network into the communities of Estell Manor, Weymouth Township, Corbin City, and Lower Alloways Creek Township, but only because of a 2014 agreement with Verizon compelling them to offer broadband to residents who read and complete a “Bona Fide Retail Request” (BFRR) form which stipulates homes and businesses in Verizon’s New Jersey territory can get broadband if they don’t have it now as long as these criteria are met:

• Have no access to broadband service from a cable provider or Verizon;
• Have no access to 4G-based wireless service; and
• Sign a contract for at least one (1) year of broadband service and pay a $100 deposit.

“BFRR is a joke because it requires potential customers have no access to 4G wireless service,” claimed Franklin. “You have to go to the government’s National Broadband Map to determine eligibility, which is very tough because — surprise, surprise — Verizon itself contributed its 4G wireless coverage information for that map and as far as Verizon is concerned, their 4G coverage in New Jersey is beautiful, even though it really isn’t.”

If a single provider submits map data that shows a home address is already covered by 4G wireless service, even if that isn’t accurate on the ground, that customer is ineligible under the terms of BFRR. Even if they were able to subscribe to 4G broadband, most plans are strictly data capped or throttled.

Under the settlement, Verizon gets to choose what technology to deploy. Outside of the four communities getting FiOS, the rest of South Jersey will have to continue relying on Verizon’s DSL service. Verizon has agreed to extend DSL to 2,000 new residences and businesses in Upper Pittsgrove, Downe, Commercial, Mannington, Pilesgrove, and South Harrison. It will also fix some of its DSL speed congestion problems and monitor for future ones as part of the settlement.

But DSL won’t work if Verizon’s wireline network stays in poor shape. The company has agreed to deploy its “Proactive Preventative Maintenance Tool” (PPMT) to scan its copper network to identify and repair or replace defective cables. Verizon has also agreed to daily inspections of outside facilities and fix any detected problems within 30 days, as well as regularly reporting back on the condition of its infrastructure inside the towns affected under the settlement.

This agreement took a year and a half to reach and will keep the two parties out of court, but many are not satisfied being left with Verizon’s DSL service.

“Unfortunately, the BPU continues to allow Verizon to pick and choose which residents will receive modern telecommunications at an affordable cost,” Greg Facemyer, a Hopewell Township committeeman in Cumberland County, told NewsWorks. “The state legislature needs to recognize these inequities and step in and level the playing field for South Jersey. Otherwise, our region will continue to fall even farther behind and be less competitive.”

The average Charter/Spectrum customer shares their internet connection with up to 499 of their neighbors, according to an admission made today by Charter Communications CEO Thomas Rutledge.

“Our average node size is around 500 homes,” Rutledge told investors on a morning conference call.

According to a lawsuit filed by the New York State Attorney General Eric Schneiderman, from about 2012, Spectrum-TWC’s network across New York typically provided about 304Mbps (8 x 38Mbps channels) of bandwidth to be shared among all the subscribers in a service group. In some areas, this would mean that 300 customers in a node would have around 1Mbps of bandwidth to use if all 300 subscribers used the internet at the same time. Time Warner Cable had begun expanding bandwidth on DOCSIS nodes to 16 channels at the time Charter Communications acquired the company, giving customers shared bandwidth of about 608Mbps.

Remarkably, Rutledge’s admission suggests some Charter customers may be serviced by DOCSIS nodes even more populated than the ones in New York State that regularly failed to deliver advertised internet speeds and prompted the Attorney General to file a lawsuit against Charter.

New York’s lawsuit claimed as of February 2016, the average Time Warner Cable customer in the state shared their connection with about 340 other customers. Information obtained from Time Warner Cable found some nodes with as few as 32 subscribers while the most overcongested had as many as 621 subscribers.

Rutledge’s comments this morning suggest Charter/Spectrum customers may be sharing their connection with up to 499 of their neighbors, making them more likely to experience congestion potentially worse than experienced with Time Warner Cable. Standard internet service from Charter is also much faster than Time Warner Cable’s corresponding Standard plan — 60Mbps vs. 15Mbps, which has the potential to lead to even worse slowdowns if customers use their internet connections at the same time.

Rutledge defended the average node size by claiming Charter has a lot of fiber in its network.

“And we have the ability to take that fiber deeper,” Rutledge said. “We have the ability incrementally to take the network to a passive network and to do that at reasonably efficient capital cost through time and to do that in very targeted ways where we need the capacity. So we’re very comfortable with the extensibility of our network and the ability to put high capacity anywhere in our network.”

Rutledge said node expansions take place through a “market demand driven sort of process.”

“There are bunch of ways you can manage capacity on our network,” Rutledge explained. “We can do what are called virtual node splits. If you clear analog spectrum and go all-digital, [that can create] excess capacity in your network, and [if] you have demand to put more capacity in a node, there [are] two ways of doing it. One way is to physically split a node into a smaller node, which requires the placing of an electronic device in the field, and maybe the extension of some fiber. It depends on how the architecture of that is structured, but it’s relatively inexpensive on a grand scale capital perspective, but a lot more expensive than a digital or virtual node split. And you can do those if you have channel capacity by just recreating additional DOCSIS paths to create a virtual node essentially. And so we manage our network for the future based on the actual load on the network as opposed to some theoretical issue.”

Unlimited data is back.

AT&T has followed Verizon Wireless back the era of unlimited data plans, starting tomorrow.

The AT&T Unlimited plan will be available to all customers, not just those signed up with DirecTV, and will be expensive. A single line unlimited voice, text, and data plan will reportedly cost $100 a month. Customers switching four lines to unlimited data will pay $180 after a $40 bill credit kicks in 60 days after signing up. This means for the first two months, customers will pay $220 for the privilege of unlimited data.

The new plan is open to residential and business/corporate accounts and business customers will also get the benefit of any corporate discounts.

AT&T’s definition of “unlimited” actually means 22GB. If you exceed that amount, AT&T reserves the right to slow your data connection “during periods of network congestion.”

The plan includes:

• unlimited calls from the U.S. to Canada and Mexico
• unlimited texts to over 120 countries
• talk, text and use data in Canada and Mexico with no roaming charges when adding the free Roam North America feature
• the ability to switch off AT&T’s “Stream Saver” which limits online video playback to 480p

“We’re always listening to our customers and will continue to evolve to provide more choice, more convenience, and more value,” claims AT&T in a press release. But observers believe AT&T listens to the competitive realities of the marketplace more than its customers who never wanted to lose the option of unlimited data in the first place.

This is part two of a multi-part series examining Time Warner Cable’s internal documents, made partly public as a result of a lawsuit filed by the New York Attorney General. You can read part one here.

In the summer of 2014, Time Warner Cable had a problem. For three years, the Federal Communications Commission had been issuing reports about the quality of broadband service from the nation’s largest internet service providers. The raw data collected by about 800 subscribers of Time Warner Cable who volunteered to participate in the project began to worry executives because it showed their broadband service was oversold in certain large cities and was no longer capable of consistently achieving advertised speeds. Even worse, the company’s congestion problems threatened to lower Time Warner Cable’s internet performance score at the FCC.

Sam Knows (but so does Time Warner Cable): The Not-So-Independent, Not-So-Confidential FCC Speed Test Program

The FCC commissioned a private company – Sam Knows – to distribute modified internet routers to gather data about the internet connections of thousands of volunteers and shared the results with the FCC to incorporate into its Measuring Broadband America program. After the FCC issued its first report in 2011, providers quickly learned the consequences of overpromising and underdelivering when Cablevision was called out for dramatically overselling its broadband service and not delivering the speeds customers paid to receive. While Cablevision executives publicly attacked the FCC speed test program as unreliable and wrong, they also quietly opened the company’s checkbook and spent millions quickly upgrading their facilities. The metric they failed to achieve was the FCC’s 80/80 test: “speed that at least 80% of the subscribers experience at least 80% of the time over peak periods.”

Here is what broadband performance on an oversold broadband service looks like. Notice Cablevision’s 2011 speed ranking plummets during peak usage periods when too many customers are sharing too little available bandwidth.

The incident embarrassed and damaged Cablevision’s reputation, and no cable operator wanted to be the next highlighted company for a public spanking by the FCC.

Time Warner Cable’s “Slow-Motion Train Wreck”

In 2013, a Time Warner Cable executive recognized the company’s practice of limiting company-financed expansion of their upstream connections with the rest of the internet would have serious implications for their own speed test scores, because customers were encountering nightly slowdowns on popular websites like YouTube caused by overcongested connections. Company executives feared customers participating in the Sam Knows/FCC program would soon reveal Time Warner Cable’s internet speeds were beginning to suffer some of the same peak usage problems Cablevision was encountering in 2011. The executive’s solution? Temporarily expand upstream connections just long enough to protect Time Warner Cable’s broadband speed scores:

“Our Sam Knows scores are like watching a slow-motion train wreck. We need to get in front of this. One thing I think we may need to be prepared to do is just give more ports to Cogent during sweeps month [when FCC results are measured for purposes of the MBA report]. We don’t have to make any promises, we just have to make it work temporarily.”

But even tricks like that failed to help Time Warner Cable’s speed scores in New York City, where serious congestion problems were obvious, even as late as last year:

The lawsuit filed by New York’s Attorney General revealed that FCC panelists in New York were getting speeds consistently well below the speeds they paid for, especially those paying for premium speeds:

FCC/Sam Knows Time Warner Cable Maxx Panelists in New York Speed Test Reports:

100Mbps subscribers received 73-87% of advertised speed (<80% advertised speed over six month period)
200Mbps subscribers received 49-58% of advertised speed (<60% advertised speed over six month period)
300Mbps subscribers received 33-52% of advertised speed (<38-74% advertised speed over six month period)

Speed test results showed consistent speed deficiencies between 2013-2016 occuring for many reasons, according to the lawsuit, including customers using outdated, company-supplied cable modems insufficient to support the customer’s speed plan, chronically oversold neighborhood groups that Time Warner Cable did not split or upgrade with additional capacity, and inadequate upstream/backbone connections to properly deliver content originating outside of Time Warner Cable’s own broadband network.

Overprovisioning Your Broadband Speed = “Putting Lipstick on a Pig”

We subscribe to 50/5Mbps service but receive closer to 62/6Mbps because Spectrum/Time Warner Cable overprovisions our service.

Instead of investing adequately in network upgrades and node splits, a July 7, 2014 internal email from Time Warner Cable’s former head of corporate strategy told senior colleagues the best way out of this dilemma was to cheat on the FCC broadband tests:

“We recommend increasing over-provisioning our modem speeds to around 20% to drive our Sam Knows scores >100% and then to market that we deliver more than promised speeds.”

In plain English, Time Warner Cable boosted the maximum allowed speed of each customer by about 20%. As a result, during non-peak usage times customers would find, for example, a plan advertising 50/5Mbps speed now delivered around 60/6Mbps. Although some customers considered overprovisioning a hidden free upgrade, Time Warner Cable’s motives were not altruistic. Because the Sam Knows testing program averages scores received from periodic testing, Time Warner Cable padded the results with higher-than-advertised speeds when their network was not congested, which compensated for the slower speeds and worse performance customers were getting during peak usage times. The lawsuit also alleges the practice helped to hide the abundance of obsolete rented cable modems still in use across Time Warner Cable’s broadband network.

The strategy worked to boost Time Warner Cable’s scores, but only as far as the FCC was concerned. Some customers were still finding their visits to YouTube, Netflix, and other websites littered with buffering problems and degraded resolution videos just about every evening. In 2013, before Time Warner Cable went ahead with its overprovisioning plan, the company’s own network engineers called the practice putting “lipstick on a pig.”

The Attorney General had its own analogy:

Using the highway analogy, Spectrum-TWC’s overprovisioning strategy amounts to allowing cars to go faster than the posted speed limit at certain times to compensate for the fact that often the highway slowed to a crawl. Boosting the average results with outlier results masked the enormous frustration for most subscribers stuck in traffic.

Breaking the FCC’s Rules

The lawsuit also alleges Time Warner Cable broke its own commitment to the FCC in the Code of Conduct it signed as a participant in the FCC’s testing program.

The FCC’s Code of Conduct required Spectrum-TWC to “at all times act in good faith” and not do anything “if the intended consequence of such act or omission is to enhance, degrade or tamper with the results of any test.” Specifically, the Code of Conduct prohibited the company from “modifying or improving services delivered to any class of subscribers” that was not “consistent with normal business practices.”

Stop the Cap! has also learned Time Warner Cable was able to identify each participant of the FCC/Sam Knows Time Warner Cable panel. This allowed the cable company to secretly verify the line quality and equipment in use by each participant, and give extra attention to those customers/volunteers to make sure service was performing as well as possible. In fact, executives instructed customer service representatives to assign FCC panelists “VIP treatment” and “best in class devices” when swapping modems, even as the company continued to supply deficient equipment to other customers who were not FCC panelists.

Still to Come: Playing games with online gamers, company officials tell the truth about bandwidth costs and Net Neutrality, and more….