Netflix has agreed to compensate Comcast in return for assurances that the cable company’s subscribers would no longer be caught in the middle of a dispute between Comcast and one of Netflix’s content distributors.

The multi-year agreement between the two companies will bring Netflix direct access to Comcast’s broadband network with a Service Level Agreement that will guarantee streaming stability for customers who have loudly complained about Netflix’s deteriorating performance.

The controversial arrangement has probably established a precedent for other large Internet Service Providers likely to seek compensation to handle Netflix traffic. As of this evening, both AT&T and Verizon have already acknowledged they are negotiating with Netflix for similar arrangements.

Caught in the middle of the dispute are Comcast customers paying for a reliable Internet connection and getting slowing connections and re-buffering problems while attempting to watch Netflix content during peak usage times.

One side accuses Comcast of violating Net Neutrality while the other blames Netflix for dumping enormous Internet traffic on Internet Service Providers without compensation for network upgrades. Also in the crossfire is Cogent, a third-party company delivering Netflix content to Comcast’s front door.

How Netflix Distributes Its Streaming Movies and TV Shows

Netflix has traditionally avoided owning the “pipes” that distribute movies and TV shows to paying customers. Instead, it usually contracts with “transit providers” to send content from Netflix headquarters on to “content distribution networks (CDN)” that manage video streaming. A Netflix video may pass through a number of connections on a variety of independently owned networks before it arrives at the front door of your Internet Service Provider. Companies like Comcast handle “the last mile” of the journey that began at Netflix and ends at your computer or television set.

Netflix does not rely on just one transit provider to handle its traffic. Level 3, Cogent, and XO Communications all reportedly serve in that capacity, depending on where traffic is headed. The same is true for the CDN’s Netflix contracts with to regionally stream content to each subscriber.

Netflix determines how to handle your streaming movie request behind the scenes, selecting a CDN that is close to you and capable of delivering the most stable streaming experience at that moment. If you are a Comcast or Verizon customer, Netflix often selects Cogent to handle its content. Cogent is also well known for its relatively low cost.

If you are served by Cablevision, Frontier, or certain other providers like Google Fiber, Netflix will instead direct your streaming request to a CDN located within your provider’s own network. These “Open Connect” boxes store Netflix content in a type of cache and can stream it to customers directly without sending video packets across multiple third-party networks. Theoretically, Open Connect offers an efficient and stable way of distributing Netflix content to customers. It also saves Netflix money and in return, it costs the ISP nothing — Netflix pays for the equipment and service.

Cogent vs. Big Telecom

Netflix and YouTube together are now estimated to cover 50 percent of all video traffic on the Internet, and that traffic is growing. Cogent dutifully passes that video content along to Internet Service Providers like Verizon and Comcast that have customers waiting to watch. But it is a two-way street. Any outbound traffic from customers could also be forwarded to Cogent to send on. Traditionally, both sides have managed the traffic by gradually increasing the bandwidth and speed of their connections to one-another. But as Netflix traffic grows and grows, companies like Comcast and Verizon believe they are being saddled with the costs to upgrade their networks in ways that are out of proportion to the traffic they send in the other direction. ISPs often grumble about the cost but keep on upgrading to keep paying customers happy. Verizon and Comcast are suspected of dragging their feet on those upgrades in an effort to win compensation.

Verizon and Comcast argue they should be paid by content producers responsible for generating tons of Internet traffic to help cover the cost of upgrades. Instead, Netflix offered its Open Connect boxes, which keep Netflix traffic within an ISPs own network, reducing the necessity of constantly upgrading connections with other transit providers. Verizon and Comcast don’t want Netflix’s solution — they want cold hard cash.

Conflict of Interest

Some network engineers cannot understand all the controversy about Comcast’s arrangement with Netflix. Some believe Netflix is simply shifting traffic away from third-party Cogent to Comcast directly, presumably at a cost savings. They suggest customers will be happy that streaming quality is restored and Netflix also wins a guaranteed level of performance they never had with Cogent.

But that argument does not explain why Netflix was compelled to make a financial arrangement with Comcast. The two companies have been in negotiations on the subject of traffic compensation for months. Many industry observers believe those talks went nowhere until Netflix customers began complaining about the increasing network slowdowns. Some even dropped their Netflix subscriptions over the issue.

Netflix CEO Reed Hastings admitted he made a deal with Comcast to restore customer confidence in Netflix and end subscriber frustration. It was also increasingly clear Comcast was in no hurry to improve things on its own, despite the fact its own customers were the ones most directly affected.

So why wouldn’t Comcast (or Verizon or Time Warner Cable) take Netflix up on its offer of free Open Connect boxes that would reasonably solve streaming problems without forcing anyone to spend a fortune on upgrades? Simply put, all three companies are direct competitors of Netflix. Helping Netflix offer a top quality streaming experience is not in the best interests of Comcast (or others) that are facing potential cord-cutting customer losses in their subscription video businesses. Verizon has partnered with Redbox to deliver streamed video, Comcast operates Streampix, its own online streaming service, and Time Warner Cable offers a variety of on-demand and streamed video content for its cable TV subscribers. None of these services have suffered from traffic congestion issues.

ISP Payday

What About Net Neutrality? What About Paying Customers?

With Net Neutrality tossed out by the courts, there is little any regulator can do to resolve disputes until Net Neutrality can be properly enforced under a stronger regulatory framework. Some argue the congestion issues creating the problems with Netflix are not a true violation of Net Neutrality in any event because providers are not artificially prioritizing traffic.

They are simply not keeping up with upgrades that just so happen to directly impact a competitor while leaving their own services unscathed.

Providers also seem characteristically unconcerned about complaining customers, passing blame for the problem on to Netflix. Besides, they remind you, paying for an Internet connection alone does not entitle you to any guarantee of performance.

The Dam Breaks

With this week’s agreement between Comcast and Netflix, both AT&T and Verizon wasted no time admitting they are both seeking compensation from Netflix as well. Other providers are likely to follow.

Netflix warned investors that paid agreements with ISPs could adversely affect its earnings due to increased costs. Although stopping short of suggesting price increases for Netflix customers could come as a result, Wall Street wasted no time worrying about the financial impact of deals like the one between Netflix and Comcast.

The Wall Street Journal reported the momentum appears to be shifting in favor of large Internet providers like Comcast and AT&T and away from content producers.

Janney Capital analyst Tony Wible suggested Comcast’s toll booth could create a barrier for other content producers if the cable company asks for significant compensation.

“Although there is no prioritization benefit [from the deal], we suspect that the exchange of money for resolution/performance could (if large) effectively limit competition,” said Wible. “In essence, Netflix could be trading [profit] margins for subscribers. Few others can match Netflix’s [spending budget to acquire content] without incurring massive losses. The competition may now have to cope with additional fees that sway their willingness to compete if they do not already have a large subscriber base.”

In other words, a new Internet startup could face hard questions from investors about how it intends to cover ISP demands for compensation in return for a suitable connection to reach customers. A large venture like Netflix has enough resources to handle those costs and negotiate for a better deal while a smaller startup may not.

[flv]http://www.phillipdampier.com/video/WSJ Netflix Comcast Agreement 2-24-14.flv[/flv]

Netflix has signed a deal with Comcast to ensure smooth streaming, in what is being called a landmark agreement. Wall Street Journal reporter Shalini Ramachandran explains the agreement. (3:39)

Google has proposed expanding its gigabit fiber network to nine metropolitan areas around the United States, but none of them include cities in the Mid-Atlantic and Northeast dominated by Time Warner Cable, Comcast, and Verizon FiOS.

Altogether, the expansion project could bring fiber to the home Internet service to 34 new cities:

• Arizona: Phoenix, Scottsdale, Tempe
• California: San Jose, Santa Clara, Sunnyvale, Mountain View, Palo Alto
• Georgia: Atlanta, Avondale Estates, Brookhaven, College Park, Decatur, East Point, Hapeville, Sandy Springs, Smyrna
• North Carolina: Charlotte, Carrboro, Cary, Chapel Hill, Durham, Garner, Morrisville, Raleigh
• Oregon: Portland, Beaverton, Hillsboro, Gresham, Lake Oswego, Tigard
• Tennessee: Nashville-Davidson
• Texas: San Antonio
• Utah: Salt Lake City

Google’s Fiber Blog:

Now that we’ve learned a lot from our Google Fiber projects in Kansas City, Austin and Provo, we want to help build more ultra-fast networks. So we’ve invited cities in nine metro areas around the U.S.—34 cities altogether—to work with us to explore what it would take to bring them Google Fiber.

We aim to provide updates by the end of the year about which cities will be getting Google Fiber. Between now and then, we’ll work closely with each city’s leaders on a joint planning process that will not only map out a Google Fiber network in detail, but also assess what unique local challenges we might face. These are such big jobs that advance planning goes a long way toward helping us stick to schedules and minimize disruption for residents.

We’re going to work on a detailed study of local factors that could affect construction, like topography (e.g., hills, flood zones), housing density and the condition of local infrastructure. Meanwhile, cities will complete a checklist of items that will help them get ready for a project of this scale and speed. For example, they’ll provide us with maps of existing conduit, water, gas and electricity lines so that we can plan where to place fiber. They’ll also help us find ways to access existing infrastructure—like utility poles—so we don’t unnecessarily dig up streets or have to put up a new pole next to an existing one.

While we do want to bring Fiber to every one of these cities, it might not work out for everyone. But cities who go through this process with us will be more prepared for us or any provider who wants to build a fiber network. In fact, we want to give everyone a boost in their thinking about how to bring fiber to their communities; we plan to share what we learn in these 34 cities, and in the meantime you can check out some tips in a recent guest post on the Google Fiber blog by industry expert Joanne Hovis. Stay tuned for updates, and we hope this news inspires more communities across America to take steps to get to a gig.

Google does not guarantee every community will actually get the service, and a read between the lines makes it clear that a close working relationship between Google and city officials and utilities will be essential for projects to move forward. Bureaucratic red tape could be a fiber-killer in some of these communities, as could an intransigent utility fighting to keep Google fiber off utility-owned poles.

Google continues to completely ignore the northeastern United States for fiber expansion. Analysts suggest Google will not enter areas where fiber broadband service already exists, and this region of the country is home to the largest deployment of Verizon’s FiOS. Despite the fact Verizon has canceled further expansion, and large sections of the region have little chance of seeing a fiber upgrade anytime soon, Google seems more interested in serving the middle of the country and fast growing areas including North Carolina, Georgia, Phoenix and Texas. Its choice of San Jose obviously reflects the presence of Silicon Valley.

The largest drops in Netflix streaming speeds are coming from ISPs that may be stalling necessary upgrades at the cost of their paying customers’ online experience.

Netflix performance for Verizon customers is deteriorating because Verizon may be delaying bandwidth upgrades until it receives compensation for handling the growing amount of traffic coming from the online video provider.

Verizon customers have increasingly complained about Netflix slowdowns during prime-time, especially in the northeast, and Netflix’s latest statistics confirm FiOS customers have seen average performance drop by as much as 14% in the last month alone.

Verizon told Stop the Cap! a few weeks ago the company was not interfering with Netflix traffic or degrading its performance, but there is growing evidence that may not be the whole story. The Wall Street Journal reports Netflix and at least one bandwidth provider suspect phone and cable companies are purposely stalling on upgrading connections to handle traffic growth from Netflix until they are compensated for carrying its video traffic.

The dispute involves the plumbing behind parts of the Internet that are invisible to consumers. As more people stream movies and television, that infrastructure is getting strained, intensifying the debate over who should pay for upgrades needed to satisfy America’s online-video habit.

Netflix wants broadband companies to hook up to its new video-distribution network without paying them fees for carrying its traffic. But the biggest U.S. providers—Verizon, Comcast, Time Warner Cable and AT&T Inc. —have resisted, insisting on compensation.

The bottleneck has made Netflix unwatchable for Jen Zellinger, an information-technology manager from Carney, Md., who signed up for the service last month. She couldn’t play an episode of “Breaking Bad” without it stopping, she said, even after her family upgraded their FiOS Internet service to a faster, more expensive package. “We tried a couple other shows, and it didn’t seem to make any difference,” she said. Mrs. Zellinger said she plans to drop her Netflix service soon if the picture doesn’t improve, though she will likely hold on to her upgraded FiOS subscription.

She and her husband thought about watching “House of Cards,” but she said they probably will skip it. “We’d be interested in getting to that if we could actually pull up the show,” she said.

Netflix relies on third-party traffic distributors to deliver much of its streamed programming to customers around the country. Cogent Communications Group is a Netflix favorite. Cogent maintains two-way connections with many Internet Service Providers. When incoming and outgoing traffic are generally balanced, providers don’t complain. But when Cogent started delivering far more traffic to Verizon customers than what it receives from them, Verizon sought compensation for the disparity.

“When one party’s getting all the benefit and the other’s carrying all the cost, issues will arise,” Craig Silliman, Verizon’s head of public policy and government affairs told the newspaper. The imbalance is primarily coming from the growth of online video, and as higher definition video grows more popular, traffic imbalances can grow dramatically worse.

A spat last summer between Cogent and some ISPs is nearly identical to the current slowdown. Ars Technica reported the traditional warning signs providers used to start upgrades are increasingly being ignored:

“Typically what happened is when the connections reached about 50 percent utilization, the two parties agreed to upgrade them and they would be upgraded in a timely manner,” Cogent CEO Dave Schaeffer told Ars. “Over the past year or so, as we have continued to pick up Netflix traffic, Verizon has continuously slowed down the rate of upgrading those connections, allowing the interconnections to become totally saturated and therefore degrading the quality of throughput.”

Schaeffer said this is true of all the big players to varying degrees, naming Comcast, Time Warner, CenturyLink, and AT&T. Out of those, he said that “AT&T is the best behaved of the bunch.”

Letting ports fill up can be a negotiating tactic. Verizon and Cogent each have to spend about $10,000 for equipment when a port is added, Schaeffer said—pocket change for companies of this size. But instead of the companies sharing equal costs, Verizon wants Cogent to pay because more traffic is flowing from Cogent to Verizon than vice versa.

Cablevision, which participates in Netflix’s Open Connect program, experiences no significant speed degradation during prime time. The same cannot be said of Time Warner Cable, which refuses to take part.

Netflix offered a solution to help Internet Service Providers manage its video traffic. Netflix’s Open Connect offers free peering at common Internet exchanges as well as free storage appliances that ISPs can connect directly to their network to distribute video to customers. Free is always good, and Netflix claims many ISPs around the world have already taken them up on the offer, slashing their transit costs along the way.

A few major North American ISPs have also agreed to take part in Open Connect, including Frontier Communications, Clearwire, Telus, Bell, Cablevision and Google Fiber. Open Connect participating ISPs also got an initial bonus for participating they could offer customers – exclusive access to SuperHD streaming.

But most Americans would not get super high-resolution streaming because the largest ISP’s refused to participate, seeking direct compensation from content providers to carry traffic across their digital pipes instead.

On Sep. 26, 2013 Netflix decided to offer SuperHD streaming to all customers, regardless of their ISP. As a result, one major ISP told the newspaper Netflix traffic from Cogent at least quadrupled. ISPs taking Netflix up on Open Connect saw almost no degradation from the increased traffic, but not so for Verizon, AT&T, Time Warner Cable, and Comcast customers.

Net Neutrality advocates fear the country’s largest phone and cable companies are making an end-run around the concept of an Open Internet. Providers can honestly guarantee not to interfere with certain web traffic, but also refuse to keep up with needed upgrades to accommodate it unless they receive payment. The slowdowns and unsatisfactory performance are the same in the end for those caught in the middle – paying customers.

“Customers are already paying for it,” said industry observer Benoît Felten. “You sell a service to the end-user which is you can access the Internet. You make a huge margin on that. Why should they get extra revenue for something that’s already being paid for?”

Some of the web’s biggest players including Microsoft, Google and Facebook may have already capitulated — agreeing to pay major providers for direct connections that guarantee a smoother browsing experience. Netflix has, thus far, held out against paying ISPs to properly manage the video content their subscribers want to watch but in some cases no longer can.

David Raphael ran into trouble using his Verizon Internet connection last month, discovering major slowdowns when accessing Amazon’s cloud-server ‘AWS,’ which in addition to serving his employer also feeds Netflix video content to customers.

“One evening I also noticed a slowdown while using our service from my house,” Raphael writes on his blog. “I realized that the one thing in common between me and [my employer] was that we both had FiOS internet service from Verizon. Since we host all of our infrastructure on Amazon’s AWS – I decided to do a little test – I grabbed a URL from AWS S3 and loaded it. 40kB/s.”

Internet slowdowns while accessing different websites is nothing new. Just ask anyone trying to watch YouTube in the early evening.

But what was different this time is that a Verizon representative seemed to openly admit the company is purposefully throttling certain web traffic, as this chat screen capture suggests:

“Frankly, I was surprised he admitted to this,” Raphael writes. “I’ve since tested this almost every day for the last couple of weeks. During the day – the bandwidth is normal to AWS. However, after 4pm or so – things get slow. In my personal opinion, this is Verizon waging war against Netflix. Unfortunately, a lot of infrastructure is hosted on AWS. That means a lot of services are going to be impacted by this.”

That would certainly be the case as many large content distributors increasingly rely on cloud-based delivery services to reach subscribers over the shortest and fastest possible route. But broad-based interference with web traffic would also throw a major wrench in Verizon’s core marketing message for FiOS — its fiber-fast speed when compared against the cable competition. If subscribers notice their Netflix experience degraded to speeds that resemble dial-up, cable companies are going to get a lot of returning customers.

We reached out to Verizon for comment and it turns out the company has not declared war on Netflix after all.

“We treat all traffic equally, and that has not changed,” says Verizon spokesman Jarryd Gonzales. “Many factors can affect the speed a customer’s experiences for a specific site, including, that site’s servers, the way the traffic is routed over the Internet, and other considerations.  We are looking into this specific matter, but the company representative was mistaken. We we’re going to redouble our representative education efforts on this topic.”

Montana is among the bottom three states for Internet broadband performance and the state can partly blame Charter Communications for its poor service.

Net Index rates Montana so low because the state relies on slow speed DSL and cable broadband service provided by smaller players who either lack the will or resources to invest in improved service.

Among the worst providers: Charter Cable, which often suffers from capacity and connectivity problems in the state.

“Right now with Charter we are experiencing significant packet loss going out to major networks in the country,” Joshua Reynolds, president of JTech Communications in Bozeman told NBC Montana. “Its gotten so bad recently that he can’t connect to our file server and download files,” said Reynolds.

Reynolds said Charter’s slow service is now affecting his company by preventing an out-of-state employee from doing his job.

Brit Fontenot, director of economic development for the city of Bozeman is surprised Montana didn’t rank dead last. Fontenot told the television station local cable and phone providers are not investing in more reliable fiber optics to solve capacity slowdowns. The city is exploring taking matters into its own hands.

“The future is a ring, a community ring connecting around the community that allows data to be transmitted both internally and externally,” said Fontenot.

The city is now engaged in dialogue with local business leaders to get comments on the quality of local Internet service.

Charter Cable is the second worst-rated cable company in the nation, according to Consumer Reports.

Speed ratings in Montana range from serviceable to painful. The fastest average speeds are around 15Mbps and the worst are just above 3Mbps.

[flv]http://www.phillipdampier.com/video/NBC Montana State Broadband Third Worst 1-27-14.flv[/flv]

NBC Montana surveys the broadband situation in Montana and the results are not good. (1:39)