Netflix has agreed to compensate Comcast in return for assurances that the cable company’s subscribers would no longer be caught in the middle of a dispute between Comcast and one of Netflix’s content distributors.

The multi-year agreement between the two companies will bring Netflix direct access to Comcast’s broadband network with a Service Level Agreement that will guarantee streaming stability for customers who have loudly complained about Netflix’s deteriorating performance.

The controversial arrangement has probably established a precedent for other large Internet Service Providers likely to seek compensation to handle Netflix traffic. As of this evening, both AT&T and Verizon have already acknowledged they are negotiating with Netflix for similar arrangements.

Caught in the middle of the dispute are Comcast customers paying for a reliable Internet connection and getting slowing connections and re-buffering problems while attempting to watch Netflix content during peak usage times.

One side accuses Comcast of violating Net Neutrality while the other blames Netflix for dumping enormous Internet traffic on Internet Service Providers without compensation for network upgrades. Also in the crossfire is Cogent, a third-party company delivering Netflix content to Comcast’s front door.

How Netflix Distributes Its Streaming Movies and TV Shows

Netflix has traditionally avoided owning the “pipes” that distribute movies and TV shows to paying customers. Instead, it usually contracts with “transit providers” to send content from Netflix headquarters on to “content distribution networks (CDN)” that manage video streaming. A Netflix video may pass through a number of connections on a variety of independently owned networks before it arrives at the front door of your Internet Service Provider. Companies like Comcast handle “the last mile” of the journey that began at Netflix and ends at your computer or television set.

Netflix does not rely on just one transit provider to handle its traffic. Level 3, Cogent, and XO Communications all reportedly serve in that capacity, depending on where traffic is headed. The same is true for the CDN’s Netflix contracts with to regionally stream content to each subscriber.

Netflix determines how to handle your streaming movie request behind the scenes, selecting a CDN that is close to you and capable of delivering the most stable streaming experience at that moment. If you are a Comcast or Verizon customer, Netflix often selects Cogent to handle its content. Cogent is also well known for its relatively low cost.

If you are served by Cablevision, Frontier, or certain other providers like Google Fiber, Netflix will instead direct your streaming request to a CDN located within your provider’s own network. These “Open Connect” boxes store Netflix content in a type of cache and can stream it to customers directly without sending video packets across multiple third-party networks. Theoretically, Open Connect offers an efficient and stable way of distributing Netflix content to customers. It also saves Netflix money and in return, it costs the ISP nothing — Netflix pays for the equipment and service.

Cogent vs. Big Telecom

Netflix and YouTube together are now estimated to cover 50 percent of all video traffic on the Internet, and that traffic is growing. Cogent dutifully passes that video content along to Internet Service Providers like Verizon and Comcast that have customers waiting to watch. But it is a two-way street. Any outbound traffic from customers could also be forwarded to Cogent to send on. Traditionally, both sides have managed the traffic by gradually increasing the bandwidth and speed of their connections to one-another. But as Netflix traffic grows and grows, companies like Comcast and Verizon believe they are being saddled with the costs to upgrade their networks in ways that are out of proportion to the traffic they send in the other direction. ISPs often grumble about the cost but keep on upgrading to keep paying customers happy. Verizon and Comcast are suspected of dragging their feet on those upgrades in an effort to win compensation.

Verizon and Comcast argue they should be paid by content producers responsible for generating tons of Internet traffic to help cover the cost of upgrades. Instead, Netflix offered its Open Connect boxes, which keep Netflix traffic within an ISPs own network, reducing the necessity of constantly upgrading connections with other transit providers. Verizon and Comcast don’t want Netflix’s solution — they want cold hard cash.

Conflict of Interest

Some network engineers cannot understand all the controversy about Comcast’s arrangement with Netflix. Some believe Netflix is simply shifting traffic away from third-party Cogent to Comcast directly, presumably at a cost savings. They suggest customers will be happy that streaming quality is restored and Netflix also wins a guaranteed level of performance they never had with Cogent.

But that argument does not explain why Netflix was compelled to make a financial arrangement with Comcast. The two companies have been in negotiations on the subject of traffic compensation for months. Many industry observers believe those talks went nowhere until Netflix customers began complaining about the increasing network slowdowns. Some even dropped their Netflix subscriptions over the issue.

Netflix CEO Reed Hastings admitted he made a deal with Comcast to restore customer confidence in Netflix and end subscriber frustration. It was also increasingly clear Comcast was in no hurry to improve things on its own, despite the fact its own customers were the ones most directly affected.

So why wouldn’t Comcast (or Verizon or Time Warner Cable) take Netflix up on its offer of free Open Connect boxes that would reasonably solve streaming problems without forcing anyone to spend a fortune on upgrades? Simply put, all three companies are direct competitors of Netflix. Helping Netflix offer a top quality streaming experience is not in the best interests of Comcast (or others) that are facing potential cord-cutting customer losses in their subscription video businesses. Verizon has partnered with Redbox to deliver streamed video, Comcast operates Streampix, its own online streaming service, and Time Warner Cable offers a variety of on-demand and streamed video content for its cable TV subscribers. None of these services have suffered from traffic congestion issues.

ISP Payday

What About Net Neutrality? What About Paying Customers?

With Net Neutrality tossed out by the courts, there is little any regulator can do to resolve disputes until Net Neutrality can be properly enforced under a stronger regulatory framework. Some argue the congestion issues creating the problems with Netflix are not a true violation of Net Neutrality in any event because providers are not artificially prioritizing traffic.

They are simply not keeping up with upgrades that just so happen to directly impact a competitor while leaving their own services unscathed.

Providers also seem characteristically unconcerned about complaining customers, passing blame for the problem on to Netflix. Besides, they remind you, paying for an Internet connection alone does not entitle you to any guarantee of performance.

The Dam Breaks

With this week’s agreement between Comcast and Netflix, both AT&T and Verizon wasted no time admitting they are both seeking compensation from Netflix as well. Other providers are likely to follow.

Netflix warned investors that paid agreements with ISPs could adversely affect its earnings due to increased costs. Although stopping short of suggesting price increases for Netflix customers could come as a result, Wall Street wasted no time worrying about the financial impact of deals like the one between Netflix and Comcast.

The Wall Street Journal reported the momentum appears to be shifting in favor of large Internet providers like Comcast and AT&T and away from content producers.

Janney Capital analyst Tony Wible suggested Comcast’s toll booth could create a barrier for other content producers if the cable company asks for significant compensation.

“Although there is no prioritization benefit [from the deal], we suspect that the exchange of money for resolution/performance could (if large) effectively limit competition,” said Wible. “In essence, Netflix could be trading [profit] margins for subscribers. Few others can match Netflix’s [spending budget to acquire content] without incurring massive losses. The competition may now have to cope with additional fees that sway their willingness to compete if they do not already have a large subscriber base.”

In other words, a new Internet startup could face hard questions from investors about how it intends to cover ISP demands for compensation in return for a suitable connection to reach customers. A large venture like Netflix has enough resources to handle those costs and negotiate for a better deal while a smaller startup may not.

[flv]http://www.phillipdampier.com/video/WSJ Netflix Comcast Agreement 2-24-14.flv[/flv]

Netflix has signed a deal with Comcast to ensure smooth streaming, in what is being called a landmark agreement. Wall Street Journal reporter Shalini Ramachandran explains the agreement. (3:39)

The largest drops in Netflix streaming speeds are coming from ISPs that may be stalling necessary upgrades at the cost of their paying customers’ online experience.

Netflix performance for Verizon customers is deteriorating because Verizon may be delaying bandwidth upgrades until it receives compensation for handling the growing amount of traffic coming from the online video provider.

Verizon customers have increasingly complained about Netflix slowdowns during prime-time, especially in the northeast, and Netflix’s latest statistics confirm FiOS customers have seen average performance drop by as much as 14% in the last month alone.

Verizon told Stop the Cap! a few weeks ago the company was not interfering with Netflix traffic or degrading its performance, but there is growing evidence that may not be the whole story. The Wall Street Journal reports Netflix and at least one bandwidth provider suspect phone and cable companies are purposely stalling on upgrading connections to handle traffic growth from Netflix until they are compensated for carrying its video traffic.

The dispute involves the plumbing behind parts of the Internet that are invisible to consumers. As more people stream movies and television, that infrastructure is getting strained, intensifying the debate over who should pay for upgrades needed to satisfy America’s online-video habit.

Netflix wants broadband companies to hook up to its new video-distribution network without paying them fees for carrying its traffic. But the biggest U.S. providers—Verizon, Comcast, Time Warner Cable and AT&T Inc. —have resisted, insisting on compensation.

The bottleneck has made Netflix unwatchable for Jen Zellinger, an information-technology manager from Carney, Md., who signed up for the service last month. She couldn’t play an episode of “Breaking Bad” without it stopping, she said, even after her family upgraded their FiOS Internet service to a faster, more expensive package. “We tried a couple other shows, and it didn’t seem to make any difference,” she said. Mrs. Zellinger said she plans to drop her Netflix service soon if the picture doesn’t improve, though she will likely hold on to her upgraded FiOS subscription.

She and her husband thought about watching “House of Cards,” but she said they probably will skip it. “We’d be interested in getting to that if we could actually pull up the show,” she said.

Netflix relies on third-party traffic distributors to deliver much of its streamed programming to customers around the country. Cogent Communications Group is a Netflix favorite. Cogent maintains two-way connections with many Internet Service Providers. When incoming and outgoing traffic are generally balanced, providers don’t complain. But when Cogent started delivering far more traffic to Verizon customers than what it receives from them, Verizon sought compensation for the disparity.

“When one party’s getting all the benefit and the other’s carrying all the cost, issues will arise,” Craig Silliman, Verizon’s head of public policy and government affairs told the newspaper. The imbalance is primarily coming from the growth of online video, and as higher definition video grows more popular, traffic imbalances can grow dramatically worse.

A spat last summer between Cogent and some ISPs is nearly identical to the current slowdown. Ars Technica reported the traditional warning signs providers used to start upgrades are increasingly being ignored:

“Typically what happened is when the connections reached about 50 percent utilization, the two parties agreed to upgrade them and they would be upgraded in a timely manner,” Cogent CEO Dave Schaeffer told Ars. “Over the past year or so, as we have continued to pick up Netflix traffic, Verizon has continuously slowed down the rate of upgrading those connections, allowing the interconnections to become totally saturated and therefore degrading the quality of throughput.”

Schaeffer said this is true of all the big players to varying degrees, naming Comcast, Time Warner, CenturyLink, and AT&T. Out of those, he said that “AT&T is the best behaved of the bunch.”

Letting ports fill up can be a negotiating tactic. Verizon and Cogent each have to spend about $10,000 for equipment when a port is added, Schaeffer said—pocket change for companies of this size. But instead of the companies sharing equal costs, Verizon wants Cogent to pay because more traffic is flowing from Cogent to Verizon than vice versa.

Cablevision, which participates in Netflix’s Open Connect program, experiences no significant speed degradation during prime time. The same cannot be said of Time Warner Cable, which refuses to take part.

Netflix offered a solution to help Internet Service Providers manage its video traffic. Netflix’s Open Connect offers free peering at common Internet exchanges as well as free storage appliances that ISPs can connect directly to their network to distribute video to customers. Free is always good, and Netflix claims many ISPs around the world have already taken them up on the offer, slashing their transit costs along the way.

A few major North American ISPs have also agreed to take part in Open Connect, including Frontier Communications, Clearwire, Telus, Bell, Cablevision and Google Fiber. Open Connect participating ISPs also got an initial bonus for participating they could offer customers – exclusive access to SuperHD streaming.

But most Americans would not get super high-resolution streaming because the largest ISP’s refused to participate, seeking direct compensation from content providers to carry traffic across their digital pipes instead.

On Sep. 26, 2013 Netflix decided to offer SuperHD streaming to all customers, regardless of their ISP. As a result, one major ISP told the newspaper Netflix traffic from Cogent at least quadrupled. ISPs taking Netflix up on Open Connect saw almost no degradation from the increased traffic, but not so for Verizon, AT&T, Time Warner Cable, and Comcast customers.

Net Neutrality advocates fear the country’s largest phone and cable companies are making an end-run around the concept of an Open Internet. Providers can honestly guarantee not to interfere with certain web traffic, but also refuse to keep up with needed upgrades to accommodate it unless they receive payment. The slowdowns and unsatisfactory performance are the same in the end for those caught in the middle – paying customers.

“Customers are already paying for it,” said industry observer Benoît Felten. “You sell a service to the end-user which is you can access the Internet. You make a huge margin on that. Why should they get extra revenue for something that’s already being paid for?”

Some of the web’s biggest players including Microsoft, Google and Facebook may have already capitulated — agreeing to pay major providers for direct connections that guarantee a smoother browsing experience. Netflix has, thus far, held out against paying ISPs to properly manage the video content their subscribers want to watch but in some cases no longer can.

“…every kitten grows up to be a cat. They seem so harmless, at first, small, quiet, lapping up their saucer of milk. But once their claws get long enough, they draw blood, sometimes from the hand that feeds them. For those of us climbing to the top of the food chain, there can be no mercy. There is but one rule: Hunt or be hunted.” — Francis Underwood

Addicts of Netflix’s hit series House of Cards may need to grab a card of a different kind to cover overlimit fees charged by your Internet Service Provider for blowing past your usage allowance.

As online video streaming moves into the realm of 4K — the next generation of high-definition video — watching television shows and movies online could get very expensive because of the massive file sizes involved. It’s all just in time for ISP’s increasing enforcement of usage caps.

Gizmodo just did the math for those intending to spend a weekend watching the entire second season of the made-for-Netflix series in high-definition:

Streaming in 1080p on Netflix takes up 4.7GB/hour. So a regular one-hour episode of something debiting less than 5GB from your allotment is no big deal. However, with 4K, you’ve got quadruple the pixel count, so you’re burning through 18.8GB/hour. Even if you’re streaming with the new h.265 codec—which cuts the bit rate by about half, but still hasn’t found its way into many consumer products—you’re still looking at 7GB/hour.

But you’re not watching just one episode, are you? Of course not! You’re binging on House of Cards, watching the whole series if not in one weekend then certainly in one month. That’s 639 minutes of top-quality TV, which in 4K tallies up to 75GB if you’re using the latest and greatest codec, and nearly 200GB if not. That means, best case scenario, a quarter of your cap—a third, if you’re a U-Verse customer with a 250GB cap—spent on one television show. Throw in a normal month’s internet usage, and you’re toast.

Sure you can send 900+ emails, download hundreds of songs, upload hundreds of pictures, and play online games 24 hours a day, but you can’t also watch one standard and one HD movie a day at the same time without blowing well past your AT&T DSL limit.

What is worse is that h.265 is still more theoretical than actually available to most consumers, so customers will either have to settle with degraded video or prepare to eat close to 19GB an hour at the highest resolution. No wonder Netflix has introduced video degradation settings to save you from your ISP’s arbitrary cap. Of course, your video quality will suffer, especially on a big screen television.

Comcast customers (and presumably Time Warner Cable customers also eventually subjected to Comcast’s cap) will still have a generous 100GB left over to watch, browse, and send that avalanche of e-mails usage cappers love to boast about. If you live in the reality-based community and have a family active online, that 100GB isn’t going to go too far. Video game addicts regularly face downloading huge updates, many ranging from 8-12GB apiece. Call of Duty: Ghosts? That’s 39.5GB. Madden NFL 25? Another 12.51GB, says Gizmodo. Using a file backup cloud storage service can also eat your allowance for breakfast.

Gizmodo also mentions Sony’s Unlimited Video service has 70 titles (and growing) available in 4K. A Sony representative admits a single two-hour movie will burn up 40GB. Watch a few of those and you are well on your way to blowing your allowance Vegas-style.

AT&T cooked up the arbitrary de facto standard overlimit fee now adopted by many American ISPs, and granular it isn’t. Exceed your allowance by even 1 kilobyte and you will be charged an extra $10 for 50 extra gigabytes. Because AT&T, Comcast, Suddenlink, and others are not already paid enough for broadband service and their modem rental.

Online video is the online application most likely to put you over your limit. Most ISPs don’t like to talk about that, however. They prefer to explain caps in terms of activities no online user is likely to ever exceed, including sending thousands of e-mails, viewing hundreds of thousands of web pages, transferring boatloads of songs and images, and watching YouTube videos at low resolution.

If you don’t watch online video, your cable or phone company thanks you for paying for cable television instead. If you haven’t used a peer-to-peer network in years, chances are you won’t exceed any limits either. But as Internet usage continues to evolve, anything that appears to be a competitive threat delivered over your ISP’s broadband pipe can be effectively controlled with the elimination of flat rate Internet service and imposing overlimit fees that deter usage.

George Washington always knew the value of a dollar, and so does AT&T. It wants more of yours going into their bank account.

The Teaser Before the Sting:

Thank you for choosing AT&T. Throughout the year we’ve worked hard to improve the U-verse experience. For example, you can now enjoy live TV channels on your computer, smartphone or tablet at no extra cost within the U-verse app or at Uverse.com. We’re looking forward to delivering even more value in 2014.

Here it comes:

The monthly price of some U-verse products are increasing and may impact the cost of your services. These changes reflect increased business costs, including costs associated with higher programming fees.

The monthly price for the following U-verse TV service plans will increase $3 beginning with bills received on January 26, 2014: U-family to $62; U200 to $77; U200 Latino to $87; U300 to $92; U300 Latino to $102; U450 to $124; and U450 Latino to $134. (Grandfathered plans also will increase $3: U100 to $64 or $69, depending on when first ordered; and U400 to $119.)

The monthly price of each non-DVR TV receiver will increase from $7 to $8 and the monthly price of the Wi-Fi enabled Residential Gateway will increase from either $2 to $4 or from $6 to $7, depending on when it was first ordered. If you have a package that includes an HD-ready DVR, that will continue to be included with your TV service.

The Broadcast TV Surcharge will be $2.99 per month beginning on February 1, 2014, in all markets except for Flint, MI, Indianapolis, IN, Huntsville, AL, Topeka, KS and Lansing, MI, where it will be $1.99. This charge is to recover a portion of the amount local broadcasters charge AT&T to carry their channels.

For customers with AT&T U-verse High Speed Internet service, the monthly price when not combined with any other qualifying AT&T service (AT&T U-verse IPTV service, AT&T U-verse Voice, combined/unified billed AT&T DIRECTV or AT&T wireless) will increase by $5.00 effective March 1, 2014, as described below. By bundling your internet services with other AT&T U-verse products this price increase will not affect you. AT&T wireless or AT&T DirecTV customers with AT&T U-verse Broadband can avoid this price increase by combining their services onto a single bill. To speak to an AT&T representative, please contact our Complete Customer Care Center at 1-800-288-2020.

The monthly price for Basic will become $33, High Speed Internet Express will become $41, High Speed Internet Pro will become $46, High Speed Internet Elite will become $51, High Speed Internet Max will become $56, High Speed Internet Max Plus will become $61, High Speed Internet Max Turbo will become $71, and High Speed Internet Power will become $81.

If you are on a current U-verse TV, U-verse Voice, and/or U-verse High Speed Internet pricing promotion, the promotional benefit will continue until the applicable promotion ends or expires. For questions regarding any of the U-verse services, visit att.com/uversesupport.

AT&T wants customers to pay attention to their broadband account’s monthly usage limits: 150GB for DSL or 250GB for U-verse. Customers who exceed their allowance are more likely than ever to get a warning letter from AT&T threatening overlimit fees if they continue to ‘use too much’ Internet.

AT&T customers in Texas, Ohio, Oklahoma, and Florida have shared identical warnings with Stop the Cap! received during the last 10 days — in each case it was the first warning notice received about exceeding AT&T’s arbitrary allowance:

Dear AT&T High Speed Internet Service Customer,

We want to remind you that your AT&T High Speed Internet service includes 150 gigabytes (GB) of data for each billing period.

You have exceeded 150 GB this billing period.

We’ll waive the charges for additional data this month and notify you as your usage approaches 150 GB in future months.

The next time you exceed 150 GB you’ll be notified, but not billed. However if you go over your data plan in any subsequent billing period, we’ll provide you with an additional 50 GB of data for $10. You’ll be charged $10 for every incremental 50 GB of usage beyond your plan.

AT&T imposed usage caps a few years ago but has generally not enforced them, even when usage meters show an excess of 500GB in Internet traffic. Some AT&T customers still have no access to a working usage meter, making compliance even more difficult. Stop the Cap! has yet to receive a verified copy of a billing statement actually showing overages billed to customers, but the increasing number of warning letters may indicate overlimit fees are forthcoming for persistent ‘violators.’

We recommend that customers receiving these warning letters send a warning of their own by calling AT&T and threatening to cancel service over the issue of unacceptable usage caps. Let AT&T know that you consider usage-based billing a deal-breaker and you will begin exploring your options with other providers. Remind AT&T that they already earn a lot of money from you and that any overlimit fees that appear on your bill will mean the immediate termination of your account.