The country’s largest cable internet service provider needed help from an app developer in Portland, Ore. to let it know its broadband pipes were full and to do something about it.

Comcast customers were complaining about slow downloads from the Panic website and the company’s own workers were saying largely the same thing when attempting to remotely connect to the company’s servers from home.

Because Panic’s web servers have just a single connection to the internet via Cogent, it would be a simple matter to track down where the traffic bottleneck was occurring, assuming there was one. The company asked for volunteers to run a test transferring 20MB of data first from Panic’s server and then again from a control server hosted with Linode, a popular and well-respected hosting company.

The results were pretty stunning.

With speeds often around only 356.3kbps for Comcast customers connecting to Panic, something was definitely up. It also explained why employees had a rough time connecting to the company’s server as well — Panic’s workers are based in Portland, Ore., where Comcast is used by almost every employee.

The slowdowns were not related to the time of day and because the problem persisted for weeks, it wasn’t a temporary technical fault. Panic’s blog picks up the story about what is behind all this:

Peering.

Major internet pipes, like Cogent, have peering agreements with network providers, like Comcast. These companies need each other — Cogent can’t exist if their network doesn’t go all the way to the end user, and Comcast can’t exist if they can’t send their customer’s data all over the world. One core tenet of peering is that it is “settlement-free” — neither party pays the other party to exchange their traffic. Instead, each party generates revenue from their customers. Cogent generates revenue from us. Comcast generates revenue from us at home. Everyone wins, right?

After a quick Google session, I learned that Cogent and Comcast have quite a storied history. This history started when Cogent started delivering a great deal of video content to Comcast customers… content from Netflix. and suddenly, the “peering pipe” that connects Cogent and Comcast filled up and slowed dramatically down.

Normally when these peering pipes “fill up”, more capacity is added between the two companies. But, if you believe Cogent’s side of the story, Comcast simply decided not to play ball — and refused to add any additional bandwidth unless Cogent paid them. In other words, Comcast didn’t like being paid nothing to deliver Netflix traffic, which competes with its own TV and streaming offerings. This Ars Technica article covers it well. (How did Netflix solve this problem in 2014? Netflix entered into a business agreement to pay Comcast directly. And suddenly, more peering bandwidth opened up between Comcast and Cogent, like magic.)

We felt certain history was repeating itself: the peering connection between Comcast and Cogent was once again saturated. Cogent said their hands were tied. What now?

In addition to giving the internet public policy community new evidence that peering fights leaving customers stuck in the middle might be heating up once again. It also suggests if Comcast was unaware of the problem, it does not reflect well on the cable company to wait weeks until a customer reports such a serious slowdown before fixing it.

The folks at Panic took a chance and reported the problem to Comcast, bypassing the usual customer support route in favor of a corporate contact who listed a direct email address on the company’s website. Comcast took the request seriously and eventually responded, “give us one to two weeks, and if you re-run your test I think you’ll be happy with the results.”

Indeed, the problem was fixed. The folks at Panic say according to Comcast, two primary changes were made:

1. Comcast added more capacity for Cogent traffic. (As suspected, the pipe was full.)
2. Cogent made some unspecified changes to their traffic engineering.

The folks at Panic and their users are happy that the problem is fixed, but some questions remain:

1. Is Comcast intentionally throttling web traffic in an attempt to extract a more favorable peering agreement with Cogent?
2. How could Comcast not know this particular connection was hopelessly over-capacity for several weeks, leaving customers to deal with heavily throttled traffic.

“While this story amazingly had a happy ending, I’m not looking forward to the next time we’re stuck in the middle of a peering dispute between two companies,” wrote Cabel. “It feels absolutely inevitable, all the more so now that net neutrality is gone. Here’s hoping the next time it happens, the responsible party is as responsive as Comcast was this time.”

Panic explains internet slowdowns resulting from peering disputes in this (3:30) video.