The country’s largest cable internet service provider needed help from an app developer in Portland, Ore. to let it know its broadband pipes were full and to do something about it.

Comcast customers were complaining about slow downloads from the Panic website and the company’s own workers were saying largely the same thing when attempting to remotely connect to the company’s servers from home.

Because Panic’s web servers have just a single connection to the internet via Cogent, it would be a simple matter to track down where the traffic bottleneck was occurring, assuming there was one. The company asked for volunteers to run a test transferring 20MB of data first from Panic’s server and then again from a control server hosted with Linode, a popular and well-respected hosting company.

The results were pretty stunning.

With speeds often around only 356.3kbps for Comcast customers connecting to Panic, something was definitely up. It also explained why employees had a rough time connecting to the company’s server as well — Panic’s workers are based in Portland, Ore., where Comcast is used by almost every employee.

The slowdowns were not related to the time of day and because the problem persisted for weeks, it wasn’t a temporary technical fault. Panic’s blog picks up the story about what is behind all this:

Peering.

Major internet pipes, like Cogent, have peering agreements with network providers, like Comcast. These companies need each other — Cogent can’t exist if their network doesn’t go all the way to the end user, and Comcast can’t exist if they can’t send their customer’s data all over the world. One core tenet of peering is that it is “settlement-free” — neither party pays the other party to exchange their traffic. Instead, each party generates revenue from their customers. Cogent generates revenue from us. Comcast generates revenue from us at home. Everyone wins, right?

After a quick Google session, I learned that Cogent and Comcast have quite a storied history. This history started when Cogent started delivering a great deal of video content to Comcast customers… content from Netflix. and suddenly, the “peering pipe” that connects Cogent and Comcast filled up and slowed dramatically down.

Normally when these peering pipes “fill up”, more capacity is added between the two companies. But, if you believe Cogent’s side of the story, Comcast simply decided not to play ball — and refused to add any additional bandwidth unless Cogent paid them. In other words, Comcast didn’t like being paid nothing to deliver Netflix traffic, which competes with its own TV and streaming offerings. This Ars Technica article covers it well. (How did Netflix solve this problem in 2014? Netflix entered into a business agreement to pay Comcast directly. And suddenly, more peering bandwidth opened up between Comcast and Cogent, like magic.)

We felt certain history was repeating itself: the peering connection between Comcast and Cogent was once again saturated. Cogent said their hands were tied. What now?

In addition to giving the internet public policy community new evidence that peering fights leaving customers stuck in the middle might be heating up once again. It also suggests if Comcast was unaware of the problem, it does not reflect well on the cable company to wait weeks until a customer reports such a serious slowdown before fixing it.

The folks at Panic took a chance and reported the problem to Comcast, bypassing the usual customer support route in favor of a corporate contact who listed a direct email address on the company’s website. Comcast took the request seriously and eventually responded, “give us one to two weeks, and if you re-run your test I think you’ll be happy with the results.”

Indeed, the problem was fixed. The folks at Panic say according to Comcast, two primary changes were made:

1. Comcast added more capacity for Cogent traffic. (As suspected, the pipe was full.)
2. Cogent made some unspecified changes to their traffic engineering.

The folks at Panic and their users are happy that the problem is fixed, but some questions remain:

1. Is Comcast intentionally throttling web traffic in an attempt to extract a more favorable peering agreement with Cogent?
2. How could Comcast not know this particular connection was hopelessly over-capacity for several weeks, leaving customers to deal with heavily throttled traffic.

“While this story amazingly had a happy ending, I’m not looking forward to the next time we’re stuck in the middle of a peering dispute between two companies,” wrote Cabel. “It feels absolutely inevitable, all the more so now that net neutrality is gone. Here’s hoping the next time it happens, the responsible party is as responsive as Comcast was this time.”

Panic explains internet slowdowns resulting from peering disputes in this (3:30) video.

If your YouTube, Netflix, or Amazon Video experience isn’t what it should be, your Internet Service Provider is likely to blame.

A consumer group today implicated several major Internet providers including Comcast, AT&T, Time Warner Cable and Verizon in an Internet slowdown scheme that prevented customers from getting the broadband performance they are paying for.

A study* of 300,000 Internet users conducted by Battleforthenet found evidence some of America’s largest providers are not adequately providing connectivity for Content Delivery Networks (CDNs) that supply high-capacity traffic coming from the Internet’s most popular websites.

Significant performance degradation was measured on the networks of the five largest American ISPs, which provide Internet connectivity for 75% of U.S. households.

“For too long, Internet access providers and their lobbyists have characterized Net Neutrality protections as a solution in search of a problem,” Tim Karr from Free Press told the Guardian newspaper, which had advance notice of the study. “Data compiled using the Internet Health Test show us otherwise – that there is widespread and systemic abuse across the network. The irony is that this trove of evidence is becoming public just as many in Congress are trying to strip away the open Internet protections that would prevent such bad behavior.”

The study revealed network performance issues that would typically be invisible to most broadband customers performing generic speed tests to measure their Internet speed. The Open Technology Institute’s M-Lab devised a more advanced speed test that would compare the performance of high traffic CDNs across several providers. CDNs were created to reduce the distance between a customer and the content provider and balance high traffic loads more evenly to reduce congestion. The shorter the distance a Netflix movie has to cross, for example, the less of a chance network problems will disrupt a customer’s viewing.

If technicians controlled the Internet, the story would end there. But it turns out money has gotten between Internet engineers with intentions of moving traffic as efficiently as possible and the executives who want to be paid something extra to carry the traffic their customers want.

That may explain why Comcast can deliver 21.4Mbps median download speeds for traffic distributed by a CDN Tier1 IP network called GTT to customers in Atlanta, while AT&T only managed to squeeze through around 200kbps — one-fifth of 1Mbps. It turns out AT&T’s connection with GTT may be maxed out and AT&T will not upgrade capacity to a network that sends AT&T customers more than twice the traffic it receives from them without direct compensation from GTT.

Internet traffic jam, at least for AT&T customers in Atlanta trying to reach content delivered by GTT.

An AT&T U-verse customer in Atlanta would probably not attribute the poor performance depicted in M-Lab’s performance test directly to AT&T because Internet responsiveness for other websites would likely appear normal. Customers might blame the originating website instead. But M-Lab’s performance results shows the trouble is limited to AT&T, not other providers like Comcast.

AT&T: Slow down, you move too fast.

The issues of performance and peering agreements that provide enough capacity to meet demand are close cousins of Net Neutrality, which is supposed to prevent content producers from being forced to pay for assurances their traffic will reach end users. But that seems to be exactly what AT&T is asking for from GTT.

“It would be unprecedented and unjustified to force AT&T to provide free backbone services to other backbone carriers and edge providers, as Cogent et al seek,” AT&T wrote in response to a request from several CDNs to disallow AT&T’s merger with DirecTV. “Nor is there any basis for requiring AT&T to augment network capacity for free and without any limits. Opponents’ proposals would shift the costs of their services onto all AT&T subscribers, many of whom do not use Opponents’ services, and would harm consumers.”

* – When a copy of the study becomes publicly available, we will supply a link to it.

Correction: It is more accurate to describe GTT as a “Tier1 IP network” which supplies services to CDN’s, among others. More detail on what GTT does can be found here.

Netflix has agreed to compensate Comcast in return for assurances that the cable company’s subscribers would no longer be caught in the middle of a dispute between Comcast and one of Netflix’s content distributors.

The multi-year agreement between the two companies will bring Netflix direct access to Comcast’s broadband network with a Service Level Agreement that will guarantee streaming stability for customers who have loudly complained about Netflix’s deteriorating performance.

The controversial arrangement has probably established a precedent for other large Internet Service Providers likely to seek compensation to handle Netflix traffic. As of this evening, both AT&T and Verizon have already acknowledged they are negotiating with Netflix for similar arrangements.

Caught in the middle of the dispute are Comcast customers paying for a reliable Internet connection and getting slowing connections and re-buffering problems while attempting to watch Netflix content during peak usage times.

One side accuses Comcast of violating Net Neutrality while the other blames Netflix for dumping enormous Internet traffic on Internet Service Providers without compensation for network upgrades. Also in the crossfire is Cogent, a third-party company delivering Netflix content to Comcast’s front door.

How Netflix Distributes Its Streaming Movies and TV Shows

Netflix has traditionally avoided owning the “pipes” that distribute movies and TV shows to paying customers. Instead, it usually contracts with “transit providers” to send content from Netflix headquarters on to “content distribution networks (CDN)” that manage video streaming. A Netflix video may pass through a number of connections on a variety of independently owned networks before it arrives at the front door of your Internet Service Provider. Companies like Comcast handle “the last mile” of the journey that began at Netflix and ends at your computer or television set.

Netflix does not rely on just one transit provider to handle its traffic. Level 3, Cogent, and XO Communications all reportedly serve in that capacity, depending on where traffic is headed. The same is true for the CDN’s Netflix contracts with to regionally stream content to each subscriber.

Netflix determines how to handle your streaming movie request behind the scenes, selecting a CDN that is close to you and capable of delivering the most stable streaming experience at that moment. If you are a Comcast or Verizon customer, Netflix often selects Cogent to handle its content. Cogent is also well known for its relatively low cost.

If you are served by Cablevision, Frontier, or certain other providers like Google Fiber, Netflix will instead direct your streaming request to a CDN located within your provider’s own network. These “Open Connect” boxes store Netflix content in a type of cache and can stream it to customers directly without sending video packets across multiple third-party networks. Theoretically, Open Connect offers an efficient and stable way of distributing Netflix content to customers. It also saves Netflix money and in return, it costs the ISP nothing — Netflix pays for the equipment and service.

Cogent vs. Big Telecom

Netflix and YouTube together are now estimated to cover 50 percent of all video traffic on the Internet, and that traffic is growing. Cogent dutifully passes that video content along to Internet Service Providers like Verizon and Comcast that have customers waiting to watch. But it is a two-way street. Any outbound traffic from customers could also be forwarded to Cogent to send on. Traditionally, both sides have managed the traffic by gradually increasing the bandwidth and speed of their connections to one-another. But as Netflix traffic grows and grows, companies like Comcast and Verizon believe they are being saddled with the costs to upgrade their networks in ways that are out of proportion to the traffic they send in the other direction. ISPs often grumble about the cost but keep on upgrading to keep paying customers happy. Verizon and Comcast are suspected of dragging their feet on those upgrades in an effort to win compensation.

Verizon and Comcast argue they should be paid by content producers responsible for generating tons of Internet traffic to help cover the cost of upgrades. Instead, Netflix offered its Open Connect boxes, which keep Netflix traffic within an ISPs own network, reducing the necessity of constantly upgrading connections with other transit providers. Verizon and Comcast don’t want Netflix’s solution — they want cold hard cash.

Conflict of Interest

Some network engineers cannot understand all the controversy about Comcast’s arrangement with Netflix. Some believe Netflix is simply shifting traffic away from third-party Cogent to Comcast directly, presumably at a cost savings. They suggest customers will be happy that streaming quality is restored and Netflix also wins a guaranteed level of performance they never had with Cogent.

But that argument does not explain why Netflix was compelled to make a financial arrangement with Comcast. The two companies have been in negotiations on the subject of traffic compensation for months. Many industry observers believe those talks went nowhere until Netflix customers began complaining about the increasing network slowdowns. Some even dropped their Netflix subscriptions over the issue.

Netflix CEO Reed Hastings admitted he made a deal with Comcast to restore customer confidence in Netflix and end subscriber frustration. It was also increasingly clear Comcast was in no hurry to improve things on its own, despite the fact its own customers were the ones most directly affected.

So why wouldn’t Comcast (or Verizon or Time Warner Cable) take Netflix up on its offer of free Open Connect boxes that would reasonably solve streaming problems without forcing anyone to spend a fortune on upgrades? Simply put, all three companies are direct competitors of Netflix. Helping Netflix offer a top quality streaming experience is not in the best interests of Comcast (or others) that are facing potential cord-cutting customer losses in their subscription video businesses. Verizon has partnered with Redbox to deliver streamed video, Comcast operates Streampix, its own online streaming service, and Time Warner Cable offers a variety of on-demand and streamed video content for its cable TV subscribers. None of these services have suffered from traffic congestion issues.

ISP Payday

What About Net Neutrality? What About Paying Customers?

With Net Neutrality tossed out by the courts, there is little any regulator can do to resolve disputes until Net Neutrality can be properly enforced under a stronger regulatory framework. Some argue the congestion issues creating the problems with Netflix are not a true violation of Net Neutrality in any event because providers are not artificially prioritizing traffic.

They are simply not keeping up with upgrades that just so happen to directly impact a competitor while leaving their own services unscathed.

Providers also seem characteristically unconcerned about complaining customers, passing blame for the problem on to Netflix. Besides, they remind you, paying for an Internet connection alone does not entitle you to any guarantee of performance.

The Dam Breaks

With this week’s agreement between Comcast and Netflix, both AT&T and Verizon wasted no time admitting they are both seeking compensation from Netflix as well. Other providers are likely to follow.

Netflix warned investors that paid agreements with ISPs could adversely affect its earnings due to increased costs. Although stopping short of suggesting price increases for Netflix customers could come as a result, Wall Street wasted no time worrying about the financial impact of deals like the one between Netflix and Comcast.

The Wall Street Journal reported the momentum appears to be shifting in favor of large Internet providers like Comcast and AT&T and away from content producers.

Janney Capital analyst Tony Wible suggested Comcast’s toll booth could create a barrier for other content producers if the cable company asks for significant compensation.

“Although there is no prioritization benefit [from the deal], we suspect that the exchange of money for resolution/performance could (if large) effectively limit competition,” said Wible. “In essence, Netflix could be trading [profit] margins for subscribers. Few others can match Netflix’s [spending budget to acquire content] without incurring massive losses. The competition may now have to cope with additional fees that sway their willingness to compete if they do not already have a large subscriber base.”

In other words, a new Internet startup could face hard questions from investors about how it intends to cover ISP demands for compensation in return for a suitable connection to reach customers. A large venture like Netflix has enough resources to handle those costs and negotiate for a better deal while a smaller startup may not.

[flv]http://www.phillipdampier.com/video/WSJ Netflix Comcast Agreement 2-24-14.flv[/flv]

Netflix has signed a deal with Comcast to ensure smooth streaming, in what is being called a landmark agreement. Wall Street Journal reporter Shalini Ramachandran explains the agreement. (3:39)

Netflix kicks off the summer rating the streaming video performance of some of America’s largest ISPs, and the results deliver only a few surprises.

Google Fiber is the runaway leader, but Verizon FiOS — also a fiber-based network — is lagging behind several cable operators, notably Charter Cable and Suddenlink.

In mid-June, GigaOM reported Verizon was engaged in a battle with Cogent — a bandwidth provider Netflix depends on to help it reach Verizon customers.

Cogent promptly blamed Verizon for the slowdown:

Cogent and Verizon peer to each other at about ten locations and they exchange traffic through several ports. These ports typically send and receive data at speeds of around 10 gigabit per second. When the ports start to fill up (usually at 50 percent of their capacity), the internet companies add more ports. In this case, through, Verizon is allowing the ports that connect to Cogent to get crammed. ”They are allowing the peer connections to degrade,” said Dave Schaeffer, chief executive officer of Cogent said in an interview. “Today some of the ports are at 100 percent capacity.”

“Think of it as the on-ramp to the freeway being log-jammed,” Schaeffer said. And that means your Netflix content, especially content sent by Netflix’s content delivery network, slows down, and you get pixelated pictures and buffering.

Verizon just as quickly shot back at GigaOm and Cogent:

Cogent is not compliant with one of the basic and long-standing requirements for most settlement-free peering arrangements: that traffic between the providers be roughly in balance. When the traffic loads are not symmetric, the provider with the heavier load typically pays the other for transit (see our ex parte filing [PDF] from the 2010 Comcast/Level 3 spat for more info on peering and transit agreements). This isn’t a story about Netflix, or about Verizon “letting” anybody’s traffic deteriorate. This is a fairly boring story about a bandwidth provider that is unhappy that they are out of balance and will have to make alternative arrangements for capacity enhancements, just like any other interconnecting ISP.

Customers don’t care. They just know their efforts to watch Arrested Development are being stymied, and Netflix’s June ISP results illustrate the degraded performance customers are getting.

Cablevision, the top performing cable operator, can likely thank its recent investments in network upgrades for improved performance, not its participation in Netflix’s OpenConnect Content Delivery Network, designed to improve streaming performance for participating ISPs. Cablevision is a member, but so are Frontier Communications (#14) and Clearwire (#17 and dead last).

OpenConnect couldn’t help Frontier DSL or Clearwire wireless customers achieve good results — the technology in use and the upstream connections both companies maintain with the Internet backbone mattered much more.

In general, fiber performs best when everyone is getting along, cable comes in second, DSL third, and wireless last.

But if you want the best performance possible, and Google Fiber is not in your neighborhood, your best bet is to move to Sweden, where the top-six providers all outperformed every American cable, DSL, and wireless provider. In Finland, the top-four beat everyone but Google Fiber. The nine best-rated ISPs in Denmark also outclassed their American counterparts, while in Norway a half-dozen providers did better.

But many ISPs in the United States can still be proud: the top eight beat Mexico. Mediacom, AT&T U-verse & DSL, Bright House, CenturyLink, Windstream, Frontier, Verizon DSL and Clearwire have some work to do… if they want to keep up with those speed mavens in Guadalajara.