Level 3’s global network: Orange lines represent Level 3-owned infrastructure, yellow lines show leased or co-owned connections.

Five of the largest Internet Service Providers in the country are intentionally sabotaging your use of the Internet by allowing their network connections to degrade unless they receive extra compensation from content companies they often directly compete with.

Mark Taylor, vice president of content and media for Level 3, wrote a lengthy primer on how Internet providers exchange traffic with each other across a vast global network. While clients of Level 3 are likely to have few problems exchanging traffic back and forth across Level 3’s global network, vital interconnections with other providers that make sure everyone can communicate with everyone else on the Internet are occasional trouble spots.

Every provider has different options to reach other providers, but favor those offering the most direct route possible to minimize “hops” between networks, which slow down the connection and increase the risk of service interruptions. These connections are often arranged through peering agreements. Level 3 has 51 peers, minimized in number to keep traffic moving as efficiently as possible.

This oversaturated port in Dallas cannot handle all the traffic trying to pass through it, so Internet packets are often dropped and traffic speeds are slowed.

Taylor writes most peering arrangements were informal agreements between engineers and did not involve any money changing hands. Today, 48 of the 51 Level 3 peering agreements don’t involve compensation. In fact, Level 3 refuses to pay “arbitrary charges to add interconnection capacity.” Taylor feels such upgrades are a matter of routine and are not costly for either party.

Peering agreements have been a very successful part of the Internet experience, even if end users remain completely in the dark about how Internet traffic moves around the world. In the view of many, customers don’t need to know and shouldn’t care, because their monthly Internet bill more than covers the cost of transporting data back and forth.

Because of ongoing upgrades the average utilization of Level 3’s connections is around 36 percent of capacity — busy enough to justify keeping the connection and providing spare capacity for days when Internet traffic explodes during breaking news or over the holidays.

However, Taylor says more than a year ago, something suddenly changed at five U.S. Internet Service Providers. They stopped periodic upgrades and allowed some of their connections to become increasingly busy with traffic. Today, six of Level 3’s 51 peer connections are now 90 percent saturated with traffic for several hours a day, which causes traffic to degrade or get lost.

“[The] congestion [has become] permanent, has been in place for well over a year and […] our peer refuses to augment capacity,” Taylor wrote. “They are deliberately harming the service they deliver to their paying customers. They are not allowing us to fulfill the requests their customers make for content.”

Taylor adds all but one of the affected connections are U.S. consumer broadband networks with a dominant or exclusive market share. Where competition exists, no provider allows their Internet connections to degrade, said Taylor.

Taylor won’t directly name the offenders, but he left an easy-to-follow trail:

“The companies with the congested peering interconnects also happen to rank dead last in customer satisfaction across all industries in the U.S.,” Taylor wrote. “Not only dead last, but by a massive statistical margin of almost three standard deviations.”

Taylor footnotes the source for his rankings, the American Consumer Satisfaction Index. The five worse providers listed for consumer satisfaction:

• Comcast
• Time Warner Cable
• Charter Communications
• Cox Communications
• Verizon

AT&T has also made noises about insisting on compensation for its own network upgrades, blaming Netflix traffic.

In fact, Netflix traffic seems to be a common point of contention among Internet Service Providers that also sell their own television packages. They now insist the streaming video provider establish direct, paid connections with their networks. Level 3 is affected because it carries a substantial amount of traffic on behalf of Netflix.

Ultimately, the debate is about who pays for network upgrades to keep up with traffic growth. Taylor says Level 3’s cost to add an extra 10Gbps port would be between $10-20 thousand dollars, spare change for multi-billion dollar Americans cable and phone companies. Normally, competition would never allow a traffic dispute like this interfere with a customer’s usage experience. Angry customers would simply switch providers. But the lack of competition prevents this from happening in the United States, leaving customers in the middle.

This leaves Taylor with a question: “Shouldn’t a broadband consumer network with near monopoly control over their customers be expected, if not obligated, to deliver a better experience than this?”

Neville Chamberlain, British Prime Minister, 1937-1940

“If you don’t succeed, try, try, try again.” — Neville Chamberlain, 1938

Another day, another damage control effort from FCC chairman Thomas Wheeler, still reeling from days of criticism in response to his plan to revisit the issue of Net Neutrality next month.

In a lengthy blog post, Wheeler still believes it’s all a big misunderstanding:

“Some recent commentary has had a misinformed interpretation of the Open Internet Notice of Proposed Rulemaking (NPRM) currently before the Commission,” writes Wheeler. “There are two things that are important to understand.  First, this is not a final decision by the Commission but rather a formal request for input on a proposal as well as a set of related questions.  Second, as the Notice makes clear, all options for protecting and promoting an Open Internet are on the table.”

Except they are not.

Wheeler channels former British Prime Minister Neville Chamberlain by declaring a deep desire for “peace in our time” with half-measures instead of direct confrontation with Big Telecom interests.

“I believe this process will put us on track to have tough, enforceable Open Internet rules on the books in an expeditious manner, ending a decade of uncertainty and litigation,” Wheeler declares. “The idea of Net Neutrality (or the Open Internet) has been discussed for a decade with no lasting results. Today Internet Openness is being decided on an ad hoc basis by big companies. Further delay will only exacerbate this problem.”

The troubles with Net Neutrality are a problem of the agency’s own making and its leadership’s utter failure to show courage in the face of Verizon, Comcast, and AT&T’s power and influence. Former FCC chairman Michael Powell (now top cable industry lobbyist) created the problem when he invented a classification for broadband as an “information service” out of thin air without any clear authority. At the heart of Powell’s “policy statement” were four basic Internet principles:

1. Consumers are entitled to access the lawful Internet content of their choice.
2. Consumers are entitled to run applications and use services of their choice, subject to the needs of law enforcement.
3. Consumers are entitled to connect their choice of legal devices that do not harm the network.
4. Consumers are entitled to competition among network providers, application and service providers, and content providers.

Powell’s principles stood as long as the FCC’s policies moved in lock-step with the telecommunications industry. When the FCC strayed from industry talking points and started showing some enforcement teeth, some of the same telecom companies that send the FCC cupcakes took them to court.

Former FCC chairman Julius Genachowski who insisted the FCC had authority over broadband because he said so believed the best way forward was to involve the industry in the development of Net Neutrality policies they could live with. After multiple private phone conversations and closed-door meetings, companies like Verizon helped write the guidelines for protecting the Open Internet and then, after they were implemented, sued the FCC in federal court.

“We are deeply concerned by the FCC’s assertion of broad authority for sweeping new regulation of broadband networks and the Internet itself,” said Michael E. Glover, Verizon’s senior vice president and deputy general counsel. “We believe this assertion of authority goes well beyond any authority provided by Congress, and creates uncertainty for the communications industry, innovators, investors and consumers.”

That’s gratitude for you, and it wasn’t the first time.

Phillip “Your Wallet = Czechoslovakia” Dampier

In 2010, an exasperated D.C. Circuit Court of Appeals didn’t exactly encounter Perry Mason when the FCC legal team showed up to defend its order demanding Comcast cease throttling broadband traffic. When the FCC threatened to fine Comcast, the cable company sued claiming the FCC had no authority over how they run their broadband business. Commission lawyer Austin C. Schlick delivered a less-than-robust defense of the FCC’s scheme.

“If I’m going to lose I would like to lose more narrowly,” Schlick confided. “But above all, we want guidance from this Court so that when we do this rule-making, if we decide rules are appropriate we’d like to know what we need to do to establish jurisdiction.”

Justice A. Raymond Randolph had none of it.

“We don’t give guidance,” Randolph grumbled, “we decide cases.” The FCC lost.

Legal experts already knew the FCC was on thin ice.  First, the Powell’s statement was never codified by the Commission’s own rulemaking procedure.  Second, the Commission framed the broadband policy as a set of “guidelines,” a term considered legally vague.  Third, the FCC relied on the concept of “ancillary” authority — borrowing regulatory authority from so-called “policy statements” coming from Congress, to claim jurisdiction.

DC Circuit Court

So it should come as no surprise that the same framework declared invalid when the FCC tried to spank Comcast was just as useless in shoring up the FCC’s authority to enforce Net Neutrality.

U.S. Circuit Judge David Tatel, writing for a three-judge panel, said that while the FCC has the power to regulate Verizon and other broadband companies, it chose the wrong legal framework for its open-Internet regulations.

“Given that the commission has chosen to classify broadband providers in a manner that exempts them from treatment as common carriers, the Communications Act expressly prohibits the commission from nonetheless regulating them as such,” Tatel wrote.

Judge Tatel could not have been more clear. In his second ruling, he noted the FCC’s ongoing resistance to reclassify broadband service under the well-grounded definition of a “telecommunications service” is at the heart of the problem.

But Wheeler, like his immediate predecessor Julias Genachowski, still stubbornly grips Powell’s flawed framework like a life-preserver off the Titanic:

The FCC promises Verizon they won’t have to sue again.

I am concerned that acting in a manner that ignores the Verizon court’s guidance, or opening an entirely new approach, invites delay that could tack on multiple more years before there are Open Internet rules in place.  We are asking for comment on a proposed a course of action that could result in an enforceable rule rather than continuing the debate over our legal authority that has so far produced nothing of permanence for the Internet.

I do not believe we should leave the market unprotected for multiple more years while lawyers for the biggest corporate players tie the FCC’s protections up in court.  Notwithstanding this, all regulatory options remain on the table. If the proposal before us now turns out to be insufficient or if we observe anyone taking advantage of the rule, I won’t hesitate to use Title II. However, unlike with Title II, we can use the court’s roadmap to implement Open Internet regulation now rather than endure additional years of litigation and delay.

Here is some news Wheeler can use: No matter what policies the FCC enacts or how, if they run contrary to the interests of Big Telecom companies, they will sue anyway. Net Neutrality appeasement by collaboration did not stop Verizon from promptly suing the FCC to overturn in court the rules the company helped write.

Wheeler needs to deal his reclassification card or get out of the game. It is increasingly clear it is the only legal basis under which the Court of Appeals will readily accept the FCC’s authority to oversee broadband.

Wheeler has his own set of Powell-like principles – the Four No-No’s of the Net:

Let me be clear, however, as to what I believe is not “commercially reasonable” on the Internet:

• Something that harms consumers is not commercially reasonable. For instance, degrading service in order to create a new “fast lane” would be shut down.

• Something that harms competition is not commercially reasonable. For instance, degrading overall service so as to force consumers and content companies to a higher priced tier would be shut down.

• Providing exclusive, prioritized service to an affiliate is not commercially reasonable. For instance, a broadband provider that also owns a sports network should not be able to give a commercial advantage to that network over another competitive sports network wishing to reach viewers over the Internet.

• Something that curbs the free exercise of speech and civic engagement is not commercially reasonable. For instance, if the creators of new Internet content or services had to seek permission from ISPs or pay special fees to be seen online such action should be shut down.

But there are plenty of loopholes in Wheeler’s proposals. First, “degrading service” goes undefined. As we’ve seen recently, there is a difference between purposely throttling a broadband connection and not maintaining and upgrading it to handle growing traffic. Second, Wheeler’s idea of what is “commercially reasonable” is not defined either. A provider could make all of its owned sports networks exempt from usage caps. That is neither “exclusive” or “prioritized.” It just doesn’t count against your usage allowance. Third, you might have open access to all of this content but won’t want it because your provider’s preferred partners get faster and more responsive service and less waiting for pages or videos to load.

Wheeler’s apparent naiveté about this industry and its behavior is beyond belief considering the decades he worked on behalf of the cable and wireless industry. Netflix foreshadows an Internet future without robust Net Neutrality. Verizon, Comcast and others ignore complaints about the degrading performance of Netflix, refusing to upgrade their connections of behalf of paying customers, until Netflix also agrees to pay them. When Netflix drops a check in the mail, the problem disappears. It doesn’t seem to matter that customers paying a very high price for Internet service cannot get the service they deserve unless someone else also pays.

If we can see this problem, it is extraordinarily curious why Wheeler cannot (or will not). Wheeler’s tough talk is cheap, but American broadband is not. Without direct action that reclassifies broadband as a telecommunications service, nothing Wheeler proposes or gets enacted is likely to survive the next inevitable court challenge.

Wheeler

Federal Communications Commission chairman Thomas Wheeler this morning defended his forthcoming Net Neutrality policies in front of an audience of cable executives attending the Cable Show in Los Angeles.

“If you read some of the press accounts about what we propose to do, those of you who oppose Net Neutrality might feel like a celebration was in order,” Wheeler told the cable industry audience. “Reports that we are gutting the Open Internet rules are incorrect. I am here to say wait a minute. Put away the party hats. The Open Internet rules will be tough, enforceable and, with the concurrence of my colleagues, in place with dispatch.”

“Let me be clear,” Wheeler continued. “If someone acts to divide the Internet between ‘haves’ and ‘have-nots,’ we will use every power at our disposal to stop it. I consider that to include Title II. Just because it is my strong belief that following the court’s roadmap will produce similar protections more quickly, does not mean I will hesitate to use Title II if warranted. And, in our Notice, we are asking for input as to whether this approach should be used.”

Wheeler also used the forum to acknowledge that cable companies are now the “principal provider of broadband” in the United States, a slap at telephone company DSL service that continues to lose market share.

Wheeler’s comments primarily addressed intentional interference with Internet traffic and remained silent about whether the FCC would allow providers to delay network upgrades that gradually allow service to degrade while selling improved “Quality of Service” contracts to content providers like Netflix.

“Prioritizing some traffic by forcing the rest of the traffic into a congested lane won’t be permitted under any proposed Open Internet rule,” Wheeler insisted. “We will not allow some companies to force Internet users into a slow lane so that others with special privileges can have superior service.”

[flv]http://www.phillipdampier.com/video/FCC Chairman Tom Wheeler’s Speech and Chat with Michael Powell 4-30-14.mp4[/flv]

FCC chairman Thomas Wheeler spoke before the 2014 NCTA Cable Show this morning to speak about Net Neutrality and chat with NCTA president Michael Powell. (39:15)

Wheeler’s remarks in full can be found below the jump:

… Continue Reading

Netflix has agreed to compensate Comcast in return for assurances that the cable company’s subscribers would no longer be caught in the middle of a dispute between Comcast and one of Netflix’s content distributors.

The multi-year agreement between the two companies will bring Netflix direct access to Comcast’s broadband network with a Service Level Agreement that will guarantee streaming stability for customers who have loudly complained about Netflix’s deteriorating performance.

The controversial arrangement has probably established a precedent for other large Internet Service Providers likely to seek compensation to handle Netflix traffic. As of this evening, both AT&T and Verizon have already acknowledged they are negotiating with Netflix for similar arrangements.

Caught in the middle of the dispute are Comcast customers paying for a reliable Internet connection and getting slowing connections and re-buffering problems while attempting to watch Netflix content during peak usage times.

One side accuses Comcast of violating Net Neutrality while the other blames Netflix for dumping enormous Internet traffic on Internet Service Providers without compensation for network upgrades. Also in the crossfire is Cogent, a third-party company delivering Netflix content to Comcast’s front door.

How Netflix Distributes Its Streaming Movies and TV Shows

Netflix has traditionally avoided owning the “pipes” that distribute movies and TV shows to paying customers. Instead, it usually contracts with “transit providers” to send content from Netflix headquarters on to “content distribution networks (CDN)” that manage video streaming. A Netflix video may pass through a number of connections on a variety of independently owned networks before it arrives at the front door of your Internet Service Provider. Companies like Comcast handle “the last mile” of the journey that began at Netflix and ends at your computer or television set.

Netflix does not rely on just one transit provider to handle its traffic. Level 3, Cogent, and XO Communications all reportedly serve in that capacity, depending on where traffic is headed. The same is true for the CDN’s Netflix contracts with to regionally stream content to each subscriber.

Netflix determines how to handle your streaming movie request behind the scenes, selecting a CDN that is close to you and capable of delivering the most stable streaming experience at that moment. If you are a Comcast or Verizon customer, Netflix often selects Cogent to handle its content. Cogent is also well known for its relatively low cost.

If you are served by Cablevision, Frontier, or certain other providers like Google Fiber, Netflix will instead direct your streaming request to a CDN located within your provider’s own network. These “Open Connect” boxes store Netflix content in a type of cache and can stream it to customers directly without sending video packets across multiple third-party networks. Theoretically, Open Connect offers an efficient and stable way of distributing Netflix content to customers. It also saves Netflix money and in return, it costs the ISP nothing — Netflix pays for the equipment and service.

Cogent vs. Big Telecom

Netflix and YouTube together are now estimated to cover 50 percent of all video traffic on the Internet, and that traffic is growing. Cogent dutifully passes that video content along to Internet Service Providers like Verizon and Comcast that have customers waiting to watch. But it is a two-way street. Any outbound traffic from customers could also be forwarded to Cogent to send on. Traditionally, both sides have managed the traffic by gradually increasing the bandwidth and speed of their connections to one-another. But as Netflix traffic grows and grows, companies like Comcast and Verizon believe they are being saddled with the costs to upgrade their networks in ways that are out of proportion to the traffic they send in the other direction. ISPs often grumble about the cost but keep on upgrading to keep paying customers happy. Verizon and Comcast are suspected of dragging their feet on those upgrades in an effort to win compensation.

Verizon and Comcast argue they should be paid by content producers responsible for generating tons of Internet traffic to help cover the cost of upgrades. Instead, Netflix offered its Open Connect boxes, which keep Netflix traffic within an ISPs own network, reducing the necessity of constantly upgrading connections with other transit providers. Verizon and Comcast don’t want Netflix’s solution — they want cold hard cash.

Conflict of Interest

Some network engineers cannot understand all the controversy about Comcast’s arrangement with Netflix. Some believe Netflix is simply shifting traffic away from third-party Cogent to Comcast directly, presumably at a cost savings. They suggest customers will be happy that streaming quality is restored and Netflix also wins a guaranteed level of performance they never had with Cogent.

But that argument does not explain why Netflix was compelled to make a financial arrangement with Comcast. The two companies have been in negotiations on the subject of traffic compensation for months. Many industry observers believe those talks went nowhere until Netflix customers began complaining about the increasing network slowdowns. Some even dropped their Netflix subscriptions over the issue.

Netflix CEO Reed Hastings admitted he made a deal with Comcast to restore customer confidence in Netflix and end subscriber frustration. It was also increasingly clear Comcast was in no hurry to improve things on its own, despite the fact its own customers were the ones most directly affected.

So why wouldn’t Comcast (or Verizon or Time Warner Cable) take Netflix up on its offer of free Open Connect boxes that would reasonably solve streaming problems without forcing anyone to spend a fortune on upgrades? Simply put, all three companies are direct competitors of Netflix. Helping Netflix offer a top quality streaming experience is not in the best interests of Comcast (or others) that are facing potential cord-cutting customer losses in their subscription video businesses. Verizon has partnered with Redbox to deliver streamed video, Comcast operates Streampix, its own online streaming service, and Time Warner Cable offers a variety of on-demand and streamed video content for its cable TV subscribers. None of these services have suffered from traffic congestion issues.

ISP Payday

What About Net Neutrality? What About Paying Customers?

With Net Neutrality tossed out by the courts, there is little any regulator can do to resolve disputes until Net Neutrality can be properly enforced under a stronger regulatory framework. Some argue the congestion issues creating the problems with Netflix are not a true violation of Net Neutrality in any event because providers are not artificially prioritizing traffic.

They are simply not keeping up with upgrades that just so happen to directly impact a competitor while leaving their own services unscathed.

Providers also seem characteristically unconcerned about complaining customers, passing blame for the problem on to Netflix. Besides, they remind you, paying for an Internet connection alone does not entitle you to any guarantee of performance.

The Dam Breaks

With this week’s agreement between Comcast and Netflix, both AT&T and Verizon wasted no time admitting they are both seeking compensation from Netflix as well. Other providers are likely to follow.

Netflix warned investors that paid agreements with ISPs could adversely affect its earnings due to increased costs. Although stopping short of suggesting price increases for Netflix customers could come as a result, Wall Street wasted no time worrying about the financial impact of deals like the one between Netflix and Comcast.

The Wall Street Journal reported the momentum appears to be shifting in favor of large Internet providers like Comcast and AT&T and away from content producers.

Janney Capital analyst Tony Wible suggested Comcast’s toll booth could create a barrier for other content producers if the cable company asks for significant compensation.

“Although there is no prioritization benefit [from the deal], we suspect that the exchange of money for resolution/performance could (if large) effectively limit competition,” said Wible. “In essence, Netflix could be trading [profit] margins for subscribers. Few others can match Netflix’s [spending budget to acquire content] without incurring massive losses. The competition may now have to cope with additional fees that sway their willingness to compete if they do not already have a large subscriber base.”

In other words, a new Internet startup could face hard questions from investors about how it intends to cover ISP demands for compensation in return for a suitable connection to reach customers. A large venture like Netflix has enough resources to handle those costs and negotiate for a better deal while a smaller startup may not.

[flv]http://www.phillipdampier.com/video/WSJ Netflix Comcast Agreement 2-24-14.flv[/flv]

Netflix has signed a deal with Comcast to ensure smooth streaming, in what is being called a landmark agreement. Wall Street Journal reporter Shalini Ramachandran explains the agreement. (3:39)