The Federal Communications Commission has announced a new updated version of its FCC Speed Test app, helping consumers evaluate their internet connection while also sharing performance data with the Commission.

The new version is designed with more accurate measurements of users’ mobile internet connections in mind, including emerging 5G services.

“This new and improved app is an important tool that will empower consumers to collect information about the services they are receiving,” said Monisha Ghosh, the FCC’s chief technology officer. “These improvements will build on the success of this effort over the years and help the FCC bridge the digital divide.”

Versions are available for iOS in the Apple App Store and Android in the Google Play Store.

Users running the app will be able to check upload and download speed, network latency, packet loss, and jitter on both wired and wireless networks. Results are shared anonymously with the FCC, which compiles network performance data as part of an agency mandate, the Measuring Broadband America program. That program reports whether the nation’s service providers are delivering internet speeds that match their advertising claims.

AT&T was unhappy with the low internet speed score the FCC was about to give the telecom giant, so it made a few phone calls and got the government regulator to effectively rig the results in its favor.

“Regulatory capture” is a term becoming more common in administrations that enable regulators that favor friendly relations with large companies over consumer protection, and under the Trump Administration, a very business-friendly FCC has demonstrated it is prepared to go the distance for some of the country’s largest telecom companies.

Today, the Wall Street Journal reported AT&T successfully got the FCC to omit DSL speed test results from the agency’s annual “Measuring Broadband America” report. Introduced during the Obama Administration, the internet speed analysis was designed to test whether cable and phone companies are being honest about delivering the broadband speed they advertise. Using a small army of test volunteers that host a free speed testing router in their home (full disclosure: Stop the Cap! is a volunteer host), automated testing of broadband performance is done silently by the equipment on an ongoing basis, with results sent to SamKnows, an independent company contracted to manage the data for the FCC’s project.

In 2011, the first full year of the program, results identified an early offender — Cablevision/Optimum, which advertised speed it couldn’t deliver to many of its customers because its network was oversold and congested. Within months, the company invested millions to dramatically expand internet capacity and speeds quickly rose, sometimes beyond the advertised level. In general, fiber and cable internet providers traditionally deliver the fastest and most reliable internet speed. Phone companies selling DSL service usually lag far behind in the results. One of those providers happened to be AT&T.

In the last year, the Journal reports AT&T successfully appealed to the FCC to keep its DSL service’s speed performance out of the report and withheld important information from the FCC required to validate some of the agency’s results.

The newspaper also found multiple potential conflicts of interest in both the program and SamKnows, its contracted partner:

• Providers get the full names of customers using speed test equipment, and some (notably Cablevision/Optimum) regularly give speed test customers white glove treatment, including prioritized service, performance upgrades and extremely fast response times during outages that could affect the provider’s speed test score. Jack Burton, a former Cablevision engineer said “there was an effort to make sure known [users] had up-to-date equipment” like modems and routers. Cablevision also marked as “high priority” the neighborhoods that contained speed-testing users, ensuring that those neighborhoods got upgraded ahead of others, said other former Cablevision engineers close to the effort.
• Providers can tinker with the raw data, including the right to exclude results from speed test volunteers subscribed to an “unpopular” speed tier (usually above 100 Mbps), those using outdated or troublesome equipment, or are signed up to an “obsolete” speed plan, like low-speed internet. Over 25% of speed test results (presumably unfavorable to the provider) were not included in the last annual report because cable and phone companies objected to their inclusion.
• SamKnows sells providers immediate access to speed test data and the other data volunteers measure for a fee, ostensibly to allow providers to identify problems on their networks before they end up published in the FCC’s report. Critics claim this gives providers an incentive to give preferential treatment to customers with speed testing equipment.

Some have claimed internet companies have gained almost total leverage over the FCC speed testing project.

The Journal:

Internet experts and former FCC officials said the setup gives the internet companies enormous leverage. “How can you go to the party who controls the information and say, ‘please give me information that may implicate you?’ ” said Tom Wheeler, a former FCC chairman who stepped down in January 2017. Jim Warner, a retired network engineer who has helped advise the agency on the test for years, told the FCC in 2015 that the rules for providers were too lax. “It’s not much of a code of conduct,” Mr. Warner said.

An FCC spokesman told the Journal the program has a transparent process and that the agency will continue to enable it “to improve, evolve, and provide meaningful results as we move forward.”

The stakes of the FCC’s speed tests are enormous for providers, now more reliant than ever on the highly profitable broadband segment of their businesses. They also allow providers to weaponize  favorable performance results to fight off consumer protection efforts that attempt to hold providers accountable for selling internet speeds undelivered. In some high stakes court cases, the FCC’s speed test reports have been used to defend providers, such as the lawsuit filed by New York’s Attorney General against Charter Communications over the poor performance of Time Warner Cable. The parties eventually settled that case.

In 2018, the key takeaway from the report celebrated by providers in testimony, marketing, and lobbying, was that “for most of the major broadband providers that were tested, measured download speeds were 100% or better of advertised speeds during the peak hours.”

Comcast often refers to the FCC’s results in claims about XFINITY internet service: “Recent testing performed by the FCC confirms that Comcast’s broadband internet access service is one of the fastest, most reliable broadband services in the United States.” But in 2018, Comcast also successfully petitioned to FCC to exclude speed test results from 214 of its testing customers, the highest number surveyed among individual providers. In contrast, Charter got the FCC to ignore results from 148 of its customers, Mediacom asked the FCC to ignore results from 46 of its internet customers.

Among the most remarkable findings uncovered by the Journal was the revelation AT&T successfully got the FCC to exclude all of its DSL customers’ speed test results, claiming that it would not be proper to include data for a service no longer being marketed to customers. AT&T deems its DSL service “obsolete” and no longer worthy of being covered by the FCC. But the company still actively markets DSL to prospective customers. This year, AT&T also announced it was no longer cooperating with SamKnows and its speed test project, claiming AT&T has devised a far more accurate speed testing project itself that it intends to use to self-report customer speed testing data.

Cox also managed to find an innovative way out of its poor score for internet speed consistency, which the FCC initially rated a rock bottom 37% of what Cox advertises. Cox claimed its speed test results were faulty because SamKnows’ tests sent traffic through an overcongested internet link yet to be upgraded. That ‘unfairly lowered Cox’s ratings’ for many of its Arizona customers, the company successfully argued, and the FCC put Cox’s poor speed consistency rating in a fine print footnote, which included both the 37% rating and a predicted/estimated reliability rating of 85%, assuming Cox properly routed its internet traffic.

The FCC report also downplays or doesn’t include data about internet slowdowns on specific websites, like Netflix or YouTube. Complaints about buffering on both popular streaming sites have been regularly cited by angry customers, but the FCC’s annual report signals there is literally nothing wrong with most providers.

Providers still fear their own network slowdowns or problems during known testing periods. The Journal reports many have a solution for that problem as well — temporarily boosting speeds and targeting better performance of popular websites and services during testing periods and returning service to normal after tests are finished.

James Cannon, a longtime cable and telecom engineering executive who left Charter in February admitted that is standard practice at Spectrum.

“I know that goes on,” he told the Journal. “If they have a scheduled test with a government agency, they will be very careful about how that traffic is routed on the network.”

As a result, the FCC’s “independent” annual speed test report is now compromised by large telecom companies, admits Maurice Dean, a telecom and media consultant with 22 years’ experience working on streaming, cable and telecom projects.

“It is problematic,” Dean said. “This attempt to ‘enhance’ performance for these measurements is a well-known practice in the industry,’ and makes the FCC results “almost meaningless for describing actual user experience.”

Tim Wu, a longtime internet advocate, likened the speed test program as more theoretical than actual, suggesting it was like measuring the speed of a car after getting rid of traffic.

Sen. Joe Manchin (D-W.V.) wants every West Virginian to test their internet speed and send his office the results to ferret out deceptive service maps and uncover more information about the state’s ongoing broadband problems.

“We’re urging everyone to do these speed tests,” Manchin told residents in Lewis County last Sunday. “We need to know, and people need to be involved in West Virginia, if they ever want to have broadband, high-speed internet and cell service. This is what we’re fighting for.”

Manchin is on a mission to debunk FCC Chairman Ajit Pai’s claims that rural broadband has grown under Pai’s leadership. Manchin believes the FCC’s broadband coverage maps are wildly inaccurate, advertised speeds are not met, and many rural residents in the state are left without internet access.

The senator intends to send the speed test results to Pai’s office, and he wants consumers to use the FCC’s own free speed test app (for Android and iOS) to “cover [Pai]” with piles of speed test results shining a light on the problem.

“There’s an FCC app that you can download on your phone,” Wes Kungel, legislative director for Manchin’s office told WVNews. “If you download that, you can hit a little button and it will tell you your speeds. If you email or mail that to our office, we will put it in a letter and send it personally to Chairmen Pai.”

Sen. Manchin

The ongoing problem with faulty broadband service maps have allowed a select group of telecom companies (many responsible for the data used by those maps) to receive federal funding to expand their own broadband businesses while preventing others from getting funding claiming the new providers would receive government funding to overlap their existing service areas.

“This is really where it all started,” he said. “[People] contacted us a few years back and basically they weren’t getting the coverage. They could not get coverage because [the FCC] said the maps showed that there was already coverage here. So we came out and did the speed tests.”

Manchin argues that West Virginia is among the most broadband-challenged states and inaccurate maps will result in the state not getting its fair share of the estimated $20 billion the FCC plans to distribute in rural America to improve broadband service.

“There’s no state that needs it more than rural West Virginia,” Manchin added. “A ‘urban’ community is 50,000 (people) or more. We don’t have one city in West Virginia with 50,000 so we have nothing to compare it to. We are all rural and we’re going to have to fight for every dollar that we can to get connected.”

West Virginians can submit their speed test results to Sen. Manchin’s office by following this link.

This is part two of a multi-part series examining Time Warner Cable’s internal documents, made partly public as a result of a lawsuit filed by the New York Attorney General. You can read part one here.

In the summer of 2014, Time Warner Cable had a problem. For three years, the Federal Communications Commission had been issuing reports about the quality of broadband service from the nation’s largest internet service providers. The raw data collected by about 800 subscribers of Time Warner Cable who volunteered to participate in the project began to worry executives because it showed their broadband service was oversold in certain large cities and was no longer capable of consistently achieving advertised speeds. Even worse, the company’s congestion problems threatened to lower Time Warner Cable’s internet performance score at the FCC.

Sam Knows (but so does Time Warner Cable): The Not-So-Independent, Not-So-Confidential FCC Speed Test Program

The FCC commissioned a private company – Sam Knows – to distribute modified internet routers to gather data about the internet connections of thousands of volunteers and shared the results with the FCC to incorporate into its Measuring Broadband America program. After the FCC issued its first report in 2011, providers quickly learned the consequences of overpromising and underdelivering when Cablevision was called out for dramatically overselling its broadband service and not delivering the speeds customers paid to receive. While Cablevision executives publicly attacked the FCC speed test program as unreliable and wrong, they also quietly opened the company’s checkbook and spent millions quickly upgrading their facilities. The metric they failed to achieve was the FCC’s 80/80 test: “speed that at least 80% of the subscribers experience at least 80% of the time over peak periods.”

Here is what broadband performance on an oversold broadband service looks like. Notice Cablevision’s 2011 speed ranking plummets during peak usage periods when too many customers are sharing too little available bandwidth.

The incident embarrassed and damaged Cablevision’s reputation, and no cable operator wanted to be the next highlighted company for a public spanking by the FCC.

Time Warner Cable’s “Slow-Motion Train Wreck”

In 2013, a Time Warner Cable executive recognized the company’s practice of limiting company-financed expansion of their upstream connections with the rest of the internet would have serious implications for their own speed test scores, because customers were encountering nightly slowdowns on popular websites like YouTube caused by overcongested connections. Company executives feared customers participating in the Sam Knows/FCC program would soon reveal Time Warner Cable’s internet speeds were beginning to suffer some of the same peak usage problems Cablevision was encountering in 2011. The executive’s solution? Temporarily expand upstream connections just long enough to protect Time Warner Cable’s broadband speed scores:

“Our Sam Knows scores are like watching a slow-motion train wreck. We need to get in front of this. One thing I think we may need to be prepared to do is just give more ports to Cogent during sweeps month [when FCC results are measured for purposes of the MBA report]. We don’t have to make any promises, we just have to make it work temporarily.”

But even tricks like that failed to help Time Warner Cable’s speed scores in New York City, where serious congestion problems were obvious, even as late as last year:

The lawsuit filed by New York’s Attorney General revealed that FCC panelists in New York were getting speeds consistently well below the speeds they paid for, especially those paying for premium speeds:

FCC/Sam Knows Time Warner Cable Maxx Panelists in New York Speed Test Reports:

100Mbps subscribers received 73-87% of advertised speed (<80% advertised speed over six month period)
200Mbps subscribers received 49-58% of advertised speed (<60% advertised speed over six month period)
300Mbps subscribers received 33-52% of advertised speed (<38-74% advertised speed over six month period)

Speed test results showed consistent speed deficiencies between 2013-2016 occuring for many reasons, according to the lawsuit, including customers using outdated, company-supplied cable modems insufficient to support the customer’s speed plan, chronically oversold neighborhood groups that Time Warner Cable did not split or upgrade with additional capacity, and inadequate upstream/backbone connections to properly deliver content originating outside of Time Warner Cable’s own broadband network.

Overprovisioning Your Broadband Speed = “Putting Lipstick on a Pig”

We subscribe to 50/5Mbps service but receive closer to 62/6Mbps because Spectrum/Time Warner Cable overprovisions our service.

Instead of investing adequately in network upgrades and node splits, a July 7, 2014 internal email from Time Warner Cable’s former head of corporate strategy told senior colleagues the best way out of this dilemma was to cheat on the FCC broadband tests:

“We recommend increasing over-provisioning our modem speeds to around 20% to drive our Sam Knows scores >100% and then to market that we deliver more than promised speeds.”

In plain English, Time Warner Cable boosted the maximum allowed speed of each customer by about 20%. As a result, during non-peak usage times customers would find, for example, a plan advertising 50/5Mbps speed now delivered around 60/6Mbps. Although some customers considered overprovisioning a hidden free upgrade, Time Warner Cable’s motives were not altruistic. Because the Sam Knows testing program averages scores received from periodic testing, Time Warner Cable padded the results with higher-than-advertised speeds when their network was not congested, which compensated for the slower speeds and worse performance customers were getting during peak usage times. The lawsuit also alleges the practice helped to hide the abundance of obsolete rented cable modems still in use across Time Warner Cable’s broadband network.

The strategy worked to boost Time Warner Cable’s scores, but only as far as the FCC was concerned. Some customers were still finding their visits to YouTube, Netflix, and other websites littered with buffering problems and degraded resolution videos just about every evening. In 2013, before Time Warner Cable went ahead with its overprovisioning plan, the company’s own network engineers called the practice putting “lipstick on a pig.”

The Attorney General had its own analogy:

Using the highway analogy, Spectrum-TWC’s overprovisioning strategy amounts to allowing cars to go faster than the posted speed limit at certain times to compensate for the fact that often the highway slowed to a crawl. Boosting the average results with outlier results masked the enormous frustration for most subscribers stuck in traffic.

Breaking the FCC’s Rules

The lawsuit also alleges Time Warner Cable broke its own commitment to the FCC in the Code of Conduct it signed as a participant in the FCC’s testing program.

The FCC’s Code of Conduct required Spectrum-TWC to “at all times act in good faith” and not do anything “if the intended consequence of such act or omission is to enhance, degrade or tamper with the results of any test.” Specifically, the Code of Conduct prohibited the company from “modifying or improving services delivered to any class of subscribers” that was not “consistent with normal business practices.”

Stop the Cap! has also learned Time Warner Cable was able to identify each participant of the FCC/Sam Knows Time Warner Cable panel. This allowed the cable company to secretly verify the line quality and equipment in use by each participant, and give extra attention to those customers/volunteers to make sure service was performing as well as possible. In fact, executives instructed customer service representatives to assign FCC panelists “VIP treatment” and “best in class devices” when swapping modems, even as the company continued to supply deficient equipment to other customers who were not FCC panelists.

Still to Come: Playing games with online gamers, company officials tell the truth about bandwidth costs and Net Neutrality, and more….

Any regular reader here knows the drill. ISPs sell you Internet service offering speeds you may or may not actually get. Giant equipment manufacturers like Cisco issue endless dramatic warnings about Internet brownouts and traffic jams. Industry shills suggest the Internet requires massive investments to keep up with broadband traffic that require usage caps and consumption billing so heavy users “pay their fair share.”

Are these claims correct or just an excuse to charge you even more for less service? While astroturf sock puppet groups claim to have authoritative facts to prove their claims, an independent, not-for-profit organization in the Netherlands is taking the temperature of the Internet with the largest Internet measurement network ever assembled.

The RIPE Atlas probe

They need your help.

The Réseaux IP Européens Network Coordination Centre (RIPE NCC) in Amsterdam coordinates an international network of volunteers who agree to host network probes on their home or work Internet connection that accurately measures Internet connectivity, reliability, and speeds in real-time. The RIPE Atlas project has been up and running for more than three years, mostly across Europe. But now the group needs a bigger network of volunteers in the United States and Canada.

The project is interactive, meaning participants can perform their own connectivity tests by utilizing the global network of RIPE Atlas probes. By hosting a probe you will be able to specify your own measurements using all other probes in the measurement network. So by hosting just one probe, you can get access to a very large number of vantage points from which to do your own measurements. For instance if you operate a web site and would like to monitor its availability from -say- Germany, you could specify a measurement from 10 probes in Germany every 10 minutes for a week and have hard figures available to you.

Volunteer participants will be sent a modified TP-Link portable router with customized firmware at no charge. The device is smaller than a pack of playing cards and is yours to keep. No special configurations are needed for NATs or firewalls; no incoming connections need to be directed to the probe. Once connected, the probe will contact RIPE’s measurement infrastructure, obtain a list of measurements it needs to perform and pass the results back. Because the probe does not observe any local traffic, it is perfectly OK, and in fact recommended, to install it behind a local switch port to isolate it from other traffic. A home router will do fine as well.  The probe is powered via its USB connector. That connector is not used to exchange any data; it is just there to obtain power. You can provide this by connecting it to a nearby USB port or you can use one of the ubiquitous USB power supplies.

We host a RIPE Atlas probe here at Stop the Cap! along with a SamKnows FCC National Speed Test router. Neither interferes in any way with our Internet connection and both contribute useful information about the true state of the Internet and our provider.

We highly recommend participation in the RIPE Atlas program. You can apply to take part in the project on the RIPE Atlas website.

Be patient. The group typically sends out probes to new applicants once every quarter, so it could be up to three months before a probe reaches you from Amsterdam. The group will contact you in e-mail just before mailing your probe to re-confirm your mailing address. The device is then sent in a padded envelope via Air Mail from the Netherlands.