The country’s largest cable internet service provider needed help from an app developer in Portland, Ore. to let it know its broadband pipes were full and to do something about it.

Comcast customers were complaining about slow downloads from the Panic website and the company’s own workers were saying largely the same thing when attempting to remotely connect to the company’s servers from home.

Because Panic’s web servers have just a single connection to the internet via Cogent, it would be a simple matter to track down where the traffic bottleneck was occurring, assuming there was one. The company asked for volunteers to run a test transferring 20MB of data first from Panic’s server and then again from a control server hosted with Linode, a popular and well-respected hosting company.

The results were pretty stunning.

With speeds often around only 356.3kbps for Comcast customers connecting to Panic, something was definitely up. It also explained why employees had a rough time connecting to the company’s server as well — Panic’s workers are based in Portland, Ore., where Comcast is used by almost every employee.

The slowdowns were not related to the time of day and because the problem persisted for weeks, it wasn’t a temporary technical fault. Panic’s blog picks up the story about what is behind all this:

Peering.

Major internet pipes, like Cogent, have peering agreements with network providers, like Comcast. These companies need each other — Cogent can’t exist if their network doesn’t go all the way to the end user, and Comcast can’t exist if they can’t send their customer’s data all over the world. One core tenet of peering is that it is “settlement-free” — neither party pays the other party to exchange their traffic. Instead, each party generates revenue from their customers. Cogent generates revenue from us. Comcast generates revenue from us at home. Everyone wins, right?

After a quick Google session, I learned that Cogent and Comcast have quite a storied history. This history started when Cogent started delivering a great deal of video content to Comcast customers… content from Netflix. and suddenly, the “peering pipe” that connects Cogent and Comcast filled up and slowed dramatically down.

Normally when these peering pipes “fill up”, more capacity is added between the two companies. But, if you believe Cogent’s side of the story, Comcast simply decided not to play ball — and refused to add any additional bandwidth unless Cogent paid them. In other words, Comcast didn’t like being paid nothing to deliver Netflix traffic, which competes with its own TV and streaming offerings. This Ars Technica article covers it well. (How did Netflix solve this problem in 2014? Netflix entered into a business agreement to pay Comcast directly. And suddenly, more peering bandwidth opened up between Comcast and Cogent, like magic.)

We felt certain history was repeating itself: the peering connection between Comcast and Cogent was once again saturated. Cogent said their hands were tied. What now?

In addition to giving the internet public policy community new evidence that peering fights leaving customers stuck in the middle might be heating up once again. It also suggests if Comcast was unaware of the problem, it does not reflect well on the cable company to wait weeks until a customer reports such a serious slowdown before fixing it.

The folks at Panic took a chance and reported the problem to Comcast, bypassing the usual customer support route in favor of a corporate contact who listed a direct email address on the company’s website. Comcast took the request seriously and eventually responded, “give us one to two weeks, and if you re-run your test I think you’ll be happy with the results.”

Indeed, the problem was fixed. The folks at Panic say according to Comcast, two primary changes were made:

1. Comcast added more capacity for Cogent traffic. (As suspected, the pipe was full.)
2. Cogent made some unspecified changes to their traffic engineering.

The folks at Panic and their users are happy that the problem is fixed, but some questions remain:

1. Is Comcast intentionally throttling web traffic in an attempt to extract a more favorable peering agreement with Cogent?
2. How could Comcast not know this particular connection was hopelessly over-capacity for several weeks, leaving customers to deal with heavily throttled traffic.

“While this story amazingly had a happy ending, I’m not looking forward to the next time we’re stuck in the middle of a peering dispute between two companies,” wrote Cabel. “It feels absolutely inevitable, all the more so now that net neutrality is gone. Here’s hoping the next time it happens, the responsible party is as responsive as Comcast was this time.”

Panic explains internet slowdowns resulting from peering disputes in this (3:30) video.

Unclear and unenforced Net Neutrality rules in India give a cautionary tale to U.S. internet users who could soon find Net Neutrality guarantees replaced in the U.S. with industry-written rules filled with loopholes or no Net Neutrality protections at all.

As India considers stronger enforcement of Net Neutrality protection, broadband providers have been merrily violating current Net Neutrality guidelines with fast lanes, sometimes advertised openly. Many of those ISPs are depending on obfuscation and grey areas to effectively give their preferred partners a leg up on the competition while claiming they are not giving them preferential treatment.

Medianama notes Ortel advertises two different internet speeds for its customers – one for regular internet traffic and the other for preferred partner websites cached by Ortel inside its network. The result is that preferred websites load 10-40 times faster than regular internet traffic.

Ortel’s vice president of broadband business, Jiji John, said Ortel is not violating Net Neutrality.

“Cache concept is totally based on the Internet user’s browsing. ISP does not control the contents and it has nothing to do with Net Neutrality,” John said in a statement.

Critics contend ISPs like Ortel may not control the contents of websites, but they do control which websites are cached and which are not.

Alliance Broadband, a West Bengal-based Internet provider, goes a step further and advertises higher speeds for Hotstar — a legal streaming platform, Google and popular movie, TV and software torrents, which arrive at speeds of 3-12Mbps faster than the rest of the internet. Alliance takes this further by establishing a reserved lane for each service, meaning regardless of what else one does with their internet connection, Hotstar content will arrive at 8Mbps, torrents at 12Mbps and the rest of the internet at 5Mbps concurrently. This means customers can get up to 25Mbps when combining traffic from the three sources, even if they are only subscribed to a much slower tier.

Alliance Broadband’s rate card. Could your ISP be next?

Which services are deemed “preferred” is up to the ISP. While Alliance may favor Google, Wishnet in West Bengal offers up preferential speeds for YouTube videos.

The ISPs claim these faster speeds are a result of “peering” those websites on its own internal network, reducing traffic slowdowns and delays. In some cases, the ISPs store the most popular content on its own servers, where it can be delivered to customers more rapidly. This alone does not violate Net Neutrality, but when an ISP reserves bandwidth for a preferred partner’s website or application, that can come at the expense of those websites that do not have this arrangement. Some ISPs have sought to devote extra bandwidth to those reserved lanes so it does not appear to impact on other traffic, but it still gives preferential treatment to some over others.

Remarkably, Indian ISPs frequently give preferential treatment to peer-to-peer services that routinely flout copyright laws while leaving legal streaming services other than Hotstar on the slow lane, encouraging copyright theft.

American ISPs have already volunteered not to block of directly impede the traffic of websites, but this may not go far enough to prevent the kinds of clever preferential runarounds ISPs can engineer where Net Neutrality is already in place, but isn’t well defined or enforced.

Phillip “Speaking as a Customer” Dampier

Two defenders of large Internet Service Providers are coming to the defense of the broadband industry by questioning a Guardian article that reported major Internet Service Providers were intentionally allowing a degradation in performance of Content Delivery Networks and other high volume Internet traffic in a dispute over money.

Richard Bennett and Dan Rayburn today both published articles attempting to discredit Battle for the Net’s effort highlighting the impact interconnection disputes can have on consumers.

Rayburn:

On Monday The Guardian ran a story with a headline stating that major Internet providers are slowing traffic speeds for thousands of consumers in North America. While that’s a title that’s going to get a lot of people’s attention, it’s not accurate. Even worse, other news outlets like Network World picked up on the story, re-hashed everything The Guardian said, but then mentioned they could not find the “study” that The Guardian is talking about. The reason they can’t find the report is because it does not exist.

[…] Even if The Guardian article was trying to use data collected via the BattlefortheNet website, they don’t understand what data is actually being collected. That data is specific to problems at interconnection points, not inside the last mile networks. So if there isn’t enough capacity at an interconnection point, saying ISPs are “slowing traffic speeds” is not accurate. No ISP is slowing down the speed of the consumers’ connection to the Internet as that all takes place inside the last mile, which is outside of the interconnection points. Even the Free Press isn’t quoted as saying ISPs are “slowing” down access speed, but rather access to enough capacity at connection points.

Bennett:

In summary, it appears that Battle for the Net may have cooked up some dubious tests to support their predetermined conclusion that ISPs are engaging in evil, extortionate behavior.

It may well be the case that they want to, but AT&T, Verizon, Charter Cable, Time Warner Cable, Brighthouse, and several others have merger business and spectrum auction business pending before the FCC. If they were manipulating customer experience in such a malicious way during the pendency of the their critical business, that would constitute executive ineptitude on an enormous scale. The alleged behavior doesn’t make customers stick around either.

I doubt the ISPs are stupid enough to do what the Guardian says they’re doing, and a careful examination of the available test data says that Battle for the Net is actually cooking the books. There is no way a long haul bandwidth and latency test says a thing about CDN performance. Now it could be that Battle for the Net has as a secret test that actually measures CDNs, but if so it’s certainly a well-kept one. Stay tuned.

The higher line measures speeds received by Comcast customers connecting to websites handled by GTT in Atlanta. The lower line represents speeds endured by AT&T customers, as measured by MLab.

Stop the Cap! was peripherally mentioned in Rayburn’s piece because we originally referenced one of the affected providers as a Content Delivery Network (CDN). In fact, GTT is a Tier 1 IP Network, providing service to CDNs, among others — a point we made in a correction prompted by one of our readers yesterday.

Both Rayburn and Bennett scoff at Battle for the Net’s methodology, results, and conclusion your Internet Service Provider might care more about money than keeping customers satisfied with decent Internet speeds. Bennett alludes to the five groups backing the Battle for the Net campaign as “comrades” and Rayburn comes close to suggesting the Guardian piece represented journalistic malpractice.

Much was made of the missing “study” that the Guardian referenced in its original piece. Stop the Cap! told readers in our original story we did not have a copy to share either, but would update the story once it became available.

We published our own story because we were able to find, without much difficulty, plenty of raw data collected by MLab from consumers conducting voluntary Internet Health Tests, on which Battle for the Net drew its conclusions about network performance. A review of that data independently confirmed all the performance assertions made in the Guardian story, with or without a report. There are obvious and undeniable significant differences in performance between certain Internet Service Providers and traffic distribution networks like GTT.

So let’s take a closer look at the issues Rayburn and Bennett either dispute or attempt to explain away:

1. MLab today confirmed there is a measurable and clear problem with ISPs serving around 75% of Americans that apparently involves under-provisioned interconnection capacity. That means the connection your ISP has with some content distributors is inadequate to handle the amount of traffic requested by customers. Some very large content distributors like Netflix increasingly use their own Content Delivery Networks, while others rely on third-party distributors to move that content for them. But the problem affects more than just high traffic video websites. If Stop the Cap! happens to reach you through one of these congested traffic networks and your ISP won’t upgrade that connection without compensation, not only will video traffic suffer slowdowns and buffering, but so will traffic from every other website, including ours, that happens to be sent through that same connection.

MLab: “Customers of Comcast, Time Warner Cable, and Verizon all saw degraded performance [in NYC] during peak use hours when connecting across transit ISPs GTT and Tata. These patterns were most dramatic for customers of Comcast and Verizon when connecting to GTT, with a low-speed of near 1 Mbps during peak hours in May. None of the three experienced similar problems when connecting with other transit providers, such as Internap and Zayo, and Cablevision did not experience the same extent of problems.”

Our initial findings show persistent performance degradation experienced by customers of a number of major access ISPs across the United States during the first half of 2015. While the ISPs involved differ, the symptoms and patterns of degradation are similar to those detailed in last year’s Interconnections study: decreased download throughput, increased latency and increased packet loss compared to the performance through different access ISPs in the same region. In nearly all cases degradation was worse during peak use hours. In last year’s technical report, we found that peak-hour degradation was an indicator of under-provisioned interconnection capacity whose shortcomings are only felt when traffic grows beyond a certain threshold.

Patterns of degraded performance occurred across the United States, impacting customers of various access ISPs when connecting to measurement points hosted within a number of transit ISPs in Atlanta, Chicago, Los Angeles, New York, Seattle, and Washington, D.C. Many of these access-transit ISP pairs have not previously been available for study using M-Lab data. In September, 2014, several measurement points were added in transit networks across the United States, making it possible to measure more access-transit ISP interconnection points. It is important to note that while we are able to observe and record these episodes of performance degradation, nothing in the data allows us to draw conclusions about who is responsible for the performance degradation. We leave determining the underlying cause of the degradation to others, and focus solely on the data, which tells us about consumer conditions irrespective of cause.

Rayburn attempts to go to town highlighting MLab’s statement that the data does not allow it to draw conclusions about who is responsible for the traffic jam. But any effort to extend that to a broader conclusion the Guardian article is “bogus” is folly. MLab’s findings clearly state there is a problem affecting the consumer’s Internet experience. To be fair, Rayburn’s view generally accepts there are disputes involving interconnection agreements, but he defends the current system that requires IP networks sending more traffic than they return to pay the ISP for a better connection.

Rayburn’s website refers to him as “the voice of industry.”

1. Rayburn comes to the debate with a different perspective than ours. Rayburn’s website highlights the fact he is the “voice of the industry.” He also helped launch the industry trade group Streaming Video Alliance, which counts Comcast as one of its members. Anyone able to afford the dues for sponsor/founding member ($25,000 annually); full member ($12,500); or supporting member ($5,500) can join.

Stop the Cap! unreservedly speaks only for consumers. In these disputes, paying customers are the undeniable collateral damage when Internet slowdowns occur and more than a few are frequently inconvenienced by congestion-related slowdowns.

It is our view that allowing paying customers to be caught in the middle of these disputes is a symptom of the monopoly/duopoly marketplace broadband providers enjoy. In any industry where competition demands a provider deliver an excellent customer experience, few would ever allow these kinds of disputes to alienate customers. In Atlanta, Los Angeles, and Chicago, for example, AT&T has evidently made a business decision to allow its connections with GTT to degrade to just a fraction of the performance achieved by other providers. Nothing else explains consistent slowdowns that have affected AT&T U-verse and DSL customers for months on end that involve GTT while Comcast customers experience none of those problems.

We also know why this is happening because AT&T and GTT have both confirmed it to Ars Technica, which covered this specific slowdown back in March. As is always the case about these disputes, it’s all about the money:

AT&T is seeking money from network operators and won’t upgrade capacity until it gets paid. Under its peering policy, AT&T demands payment when a network sends more than twice as much traffic as it receives.

“Some providers are sending significantly more than twice as much traffic as they are receiving at specific interconnection points, which violates our peering policy that has been in place for years,” AT&T told Ars. “We are engaged in commercial-agreement discussions, as is typical in such situations, with several ISPs and Internet providers regarding this imbalanced traffic and possible solutions for augmenting capacity.”

Missing from this discussion are AT&T customers directly affected by slowdowns. AT&T’s attitude seems uninterested in the customer experience and the company feels safe stonewalling GTT until it gets a check in the mail. It matters less that AT&T customers have paid $40, 50, even 70 a month for high quality Internet service they are not getting.

In a more competitive marketplace, we believe no ISP would ever allow these disputes to impact paying subscribers, because a dissatisfied customer can cancel service and switch providers. That is much less likely if you are an AT&T DSL customer with no cable competition or if your only other choice cannot offer the Internet speed you need.

1. Consolidating the telecommunications industry will only guarantee these problems will get worse. If AT&T is allowed to merge with DirecTV and expand Internet service to more customers in rural areas where cable broadband does not reach, does that not strengthen AT&T’s ability to further stonewall content providers? Of course it does. In fact, even a company the size of Netflix eventually relented and wrote a check to Comcast to clear up major congestion problems experienced by Comcast customers in 2014. Comcast could have solved the problem itself for the benefit of its paying customers, but refused. The day Netflix’s check arrived, problems with Netflix magically disappeared.

More mergers and more consolidation does not enhance competition. It entrenches big ISPs to play more aggressive hardball with content providers at the expense of consumers.

Even Rayburn concedes these disputes are “not about ‘fairness,’ it’s business,” he writes. “Some pay based on various business terms, others might not. There is no law against it, no rule that prohibits it.”

Battle for the Net’s point may be that there should be.

The nation’s first Net Neutrality complaint filed with the Federal Communications Commission accuses Time Warner Cable of refusing to provide the best possible path for its broadband customers to watch a series of high-definition webcams covering San Diego Bay.

Commercial Network Services’ CEO Barry Bahrami wrote the FCC that Time Warner Cable is degrading its ability to exercise free expression by choosing which Internet traffic providers it directly peers with and which it does not:

I am writing to initiate an informal complaint against Time Warner Cable (TWC) for violating the “No Paid Prioritization” and “No Throttling” sections of the new Net Neutrality rules for failure to fulfill their obligations to their BIAS consumers by opting to exchange Internet traffic over higher latency (and often more congested) transit routes instead of directly to the edge provider over lower latency peering routes freely available to them through their presence on public Internet exchanges, unless a payment is made to TWC by the edge provider. These violations are occurring on industry recognized public Internet peering exchanges where both autonomous systems maintain a presence to exchange Internet traffic, but are unable to due to the management policy of TWC. As you know, there is no management policy exception to the No Paid Prioritization rule.

By refusing to accept the freely available direct route to the edge-provider of the consumers’ choosing, TWC is unnecessarily increasing latency and congestion between the consumer and the edge provider by instead sending traffic through higher latency and routinely congested transit routes. This is a default on their promise to the BIAS consumer to deliver to the edge and make arrangements as necessary to do that.

The website responsible for initiating the complaint shows live webcam footage of the San Diego Bay.

Bahrami’s complaint deals with interconnection issues, which are not explicitly covered by the FCC’s Net Neutrality rules that prohibit intentional degradation or paid prioritization of network traffic. For years, ISPs have agreed to “settlement-free peering” arrangements with bandwidth providers that exchange traffic in roughly equal amounts with one another. To qualify for this kind of free interconnection arrangement, CNS’ webcams must be hosted by a company that receives about as much traffic from Time Warner Cable customers as it sends back to them — an unlikely prospect.

As bandwidth intensive content knocks traffic figures out of balance, ISPs have started demanding financial compensation from content producers if they want performance guarantees. This is what led Comcast, Verizon and AT&T to insist on paid interconnection agreements with the traffic monster Netflix.

Time Warner Cable is calling on the FCC to dismiss Bahrami’s letter on the grounds it is not a valid Net Neutrality complaint.

“[The FCC should] reject any complaint that is premised on the notion that every edge provider around the globe is entitled to enter into a settlement-free peering arrangement,” Time Warner Cable responds. That is a nice way of telling CNS it doesn’t get a premium pathway to Time Warner Cable customers for free just because of Net Neutrality rules.

Bahrami responds Time Warner’s attitude is based on a distinction without much difference because he is effectively being told CNS must pay extra for a suitable connection with Time Warner to guarantee his web visitors will have a good experience.

“This is not a valid complaint, and there is no way the FCC is going to side with them,” Dan Rayburn, a telecom analyst at Frost & Sullivan and the founding member of the Streaming Video Alliance told Motherboard. “The rules say you can’t block or throttle, but there’s no rule that says Time Warner Cable has to give CNS settlement-free peering. I don’t see how the FCC could possibly say there’s a violation here.”

The FCC made it clear in its Net Neutrality policy it intends “to watch, learn, and act as required, but not intervene now, especially not with prescriptive rules” with respect to interconnection matters.

That makes it likely Bahrami’s complaint will either be tossed out on grounds it is not a Net Neutrality violation or more likely dismissed but kept in what will likely be a growing file of future cases of interconnection disputes between ISPs and content producers. If that file grows too large too quickly, the FCC may be compelled to act.

If your YouTube, Netflix, or Amazon Video experience isn’t what it should be, your Internet Service Provider is likely to blame.

A consumer group today implicated several major Internet providers including Comcast, AT&T, Time Warner Cable and Verizon in an Internet slowdown scheme that prevented customers from getting the broadband performance they are paying for.

A study* of 300,000 Internet users conducted by Battleforthenet found evidence some of America’s largest providers are not adequately providing connectivity for Content Delivery Networks (CDNs) that supply high-capacity traffic coming from the Internet’s most popular websites.

Significant performance degradation was measured on the networks of the five largest American ISPs, which provide Internet connectivity for 75% of U.S. households.

“For too long, Internet access providers and their lobbyists have characterized Net Neutrality protections as a solution in search of a problem,” Tim Karr from Free Press told the Guardian newspaper, which had advance notice of the study. “Data compiled using the Internet Health Test show us otherwise – that there is widespread and systemic abuse across the network. The irony is that this trove of evidence is becoming public just as many in Congress are trying to strip away the open Internet protections that would prevent such bad behavior.”

The study revealed network performance issues that would typically be invisible to most broadband customers performing generic speed tests to measure their Internet speed. The Open Technology Institute’s M-Lab devised a more advanced speed test that would compare the performance of high traffic CDNs across several providers. CDNs were created to reduce the distance between a customer and the content provider and balance high traffic loads more evenly to reduce congestion. The shorter the distance a Netflix movie has to cross, for example, the less of a chance network problems will disrupt a customer’s viewing.

If technicians controlled the Internet, the story would end there. But it turns out money has gotten between Internet engineers with intentions of moving traffic as efficiently as possible and the executives who want to be paid something extra to carry the traffic their customers want.

That may explain why Comcast can deliver 21.4Mbps median download speeds for traffic distributed by a CDN Tier1 IP network called GTT to customers in Atlanta, while AT&T only managed to squeeze through around 200kbps — one-fifth of 1Mbps. It turns out AT&T’s connection with GTT may be maxed out and AT&T will not upgrade capacity to a network that sends AT&T customers more than twice the traffic it receives from them without direct compensation from GTT.

Internet traffic jam, at least for AT&T customers in Atlanta trying to reach content delivered by GTT.

An AT&T U-verse customer in Atlanta would probably not attribute the poor performance depicted in M-Lab’s performance test directly to AT&T because Internet responsiveness for other websites would likely appear normal. Customers might blame the originating website instead. But M-Lab’s performance results shows the trouble is limited to AT&T, not other providers like Comcast.

AT&T: Slow down, you move too fast.

The issues of performance and peering agreements that provide enough capacity to meet demand are close cousins of Net Neutrality, which is supposed to prevent content producers from being forced to pay for assurances their traffic will reach end users. But that seems to be exactly what AT&T is asking for from GTT.

“It would be unprecedented and unjustified to force AT&T to provide free backbone services to other backbone carriers and edge providers, as Cogent et al seek,” AT&T wrote in response to a request from several CDNs to disallow AT&T’s merger with DirecTV. “Nor is there any basis for requiring AT&T to augment network capacity for free and without any limits. Opponents’ proposals would shift the costs of their services onto all AT&T subscribers, many of whom do not use Opponents’ services, and would harm consumers.”

* – When a copy of the study becomes publicly available, we will supply a link to it.

Correction: It is more accurate to describe GTT as a “Tier1 IP network” which supplies services to CDN’s, among others. More detail on what GTT does can be found here.