The country’s largest cable internet service provider needed help from an app developer in Portland, Ore. to let it know its broadband pipes were full and to do something about it.

Comcast customers were complaining about slow downloads from the Panic website and the company’s own workers were saying largely the same thing when attempting to remotely connect to the company’s servers from home.

Because Panic’s web servers have just a single connection to the internet via Cogent, it would be a simple matter to track down where the traffic bottleneck was occurring, assuming there was one. The company asked for volunteers to run a test transferring 20MB of data first from Panic’s server and then again from a control server hosted with Linode, a popular and well-respected hosting company.

The results were pretty stunning.

With speeds often around only 356.3kbps for Comcast customers connecting to Panic, something was definitely up. It also explained why employees had a rough time connecting to the company’s server as well — Panic’s workers are based in Portland, Ore., where Comcast is used by almost every employee.

The slowdowns were not related to the time of day and because the problem persisted for weeks, it wasn’t a temporary technical fault. Panic’s blog picks up the story about what is behind all this:

Peering.

Major internet pipes, like Cogent, have peering agreements with network providers, like Comcast. These companies need each other — Cogent can’t exist if their network doesn’t go all the way to the end user, and Comcast can’t exist if they can’t send their customer’s data all over the world. One core tenet of peering is that it is “settlement-free” — neither party pays the other party to exchange their traffic. Instead, each party generates revenue from their customers. Cogent generates revenue from us. Comcast generates revenue from us at home. Everyone wins, right?

After a quick Google session, I learned that Cogent and Comcast have quite a storied history. This history started when Cogent started delivering a great deal of video content to Comcast customers… content from Netflix. and suddenly, the “peering pipe” that connects Cogent and Comcast filled up and slowed dramatically down.

Normally when these peering pipes “fill up”, more capacity is added between the two companies. But, if you believe Cogent’s side of the story, Comcast simply decided not to play ball — and refused to add any additional bandwidth unless Cogent paid them. In other words, Comcast didn’t like being paid nothing to deliver Netflix traffic, which competes with its own TV and streaming offerings. This Ars Technica article covers it well. (How did Netflix solve this problem in 2014? Netflix entered into a business agreement to pay Comcast directly. And suddenly, more peering bandwidth opened up between Comcast and Cogent, like magic.)

We felt certain history was repeating itself: the peering connection between Comcast and Cogent was once again saturated. Cogent said their hands were tied. What now?

In addition to giving the internet public policy community new evidence that peering fights leaving customers stuck in the middle might be heating up once again. It also suggests if Comcast was unaware of the problem, it does not reflect well on the cable company to wait weeks until a customer reports such a serious slowdown before fixing it.

The folks at Panic took a chance and reported the problem to Comcast, bypassing the usual customer support route in favor of a corporate contact who listed a direct email address on the company’s website. Comcast took the request seriously and eventually responded, “give us one to two weeks, and if you re-run your test I think you’ll be happy with the results.”

Indeed, the problem was fixed. The folks at Panic say according to Comcast, two primary changes were made:

1. Comcast added more capacity for Cogent traffic. (As suspected, the pipe was full.)
2. Cogent made some unspecified changes to their traffic engineering.

The folks at Panic and their users are happy that the problem is fixed, but some questions remain:

1. Is Comcast intentionally throttling web traffic in an attempt to extract a more favorable peering agreement with Cogent?
2. How could Comcast not know this particular connection was hopelessly over-capacity for several weeks, leaving customers to deal with heavily throttled traffic.

“While this story amazingly had a happy ending, I’m not looking forward to the next time we’re stuck in the middle of a peering dispute between two companies,” wrote Cabel. “It feels absolutely inevitable, all the more so now that net neutrality is gone. Here’s hoping the next time it happens, the responsible party is as responsive as Comcast was this time.”

Panic explains internet slowdowns resulting from peering disputes in this (3:30) video.

Cablevision customers get very attractive promotions in the highly competitive northeastern United States, while Suddenlink customers in more rural areas pay more.

The majority of Cablevision and Suddenlink broadband customers want speeds of 100 Mbps or greater from the Altice-owned cable operators, and average monthly data usage by those customers is now reaching 200 GB per month.

Those statistics were part of a quarterly financial results presentation by Altice USA executives about how the company is doing in the United States.

Altice’s cable holdings include Cablevision, serving a generally affluent customer base in and around the New York City area where Verizon FiOS is its biggest competitor, and Suddenlink, which serves in less competitive markets where local economies are often challenged and phone company DSL still has a significant presence.

Regardless of whether customers receive broadband from Cablevision or Suddenlink, Altice USA CEO Dexter Goei made it clear consumers want faster internet service and are consuming exponentially more data than ever before.

Goei said Altice will continue to increase internet speeds over its existing hybrid fiber-coax network (HFC) even as it builds out its fiber to the home replacement network in some areas. At least 95% of Cablevision customers can now subscribe to 400 Mbps broadband on the company’s legacy HFC network. Around 72% of Suddenlink customers can get similar speeds today. Gigabit speed is available to 29% of Altice USA customers.

Goei said 90% of new Cablevision and Suddenlink customers now choose internet plans featuring 100 Mbps or faster broadband. The average data use of those customers “is now reaching about 200 GB” per month, Goei reported. For customers on HFC systems, Goei said the maximum speed Altice’s implementation of DOCSIS 3 can support is around 600 Mbps, depending on how many customers are sharing the connection. As customers transition to fiber service in the northeast, faster speeds are planned. In fact, Goei wants Cablevision to offer speeds even faster than Verizon FiOS, its chief competitor.

“In terms of the speed capabilities, we’ll have the ability to do higher speeds than the competition,” Goei said.

Altice USA’s fiber-to-the-home (FTTH) deployment is “well underway” in New York, New Jersey, and Connecticut, with plans to connect several hundred thousand customers to the new network starting later this year. Goei told investors Altice was accelerating the rollout this year with the hope of further reducing network and customer operation costs related to servicing the older coaxial network.

Cablevision and Suddenlink will gradually be rebranded as Altice, and the company has begun familiarizing customers with the new brand name in various ways, including the rollout of its new deluxe set-top box, called Altice One.

“This is our new entertainment platform with an all in one box, including TV, internet, Wi-Fi, integrated apps such as Netflix and a voice activated remote control,” said Goei. “The service includes an improved Wi-Fi experience […] as many TV boxes double up as Wi-Fi repeaters around the home. This is a key part of our strategy of enhancing the customer experience and we’ll have the capacity for ongoing upgrades and the addition of new apps as they become available.”

But that new platform comes at a cost. Currently, Cablevision customers can pay as much as $10 for each set-top box and $5 for a cable modem. Altice One is regularly priced at $25 a month — $10 more for a customer that has one television set-top box and cable modem. That makes Altice’s box among the most costly in the cable industry. The company is trying to hide the cost of its box by bundling it into promotions targeting price sensitive new customers.

In fact, the cost of service is increasingly becoming a factor, especially for Suddenlink customers. Over the last two years, Altice has been “harmonizing” Suddenlink’s rate plans, which used to be set based on the technical capabilities and performance of each cable system. Goei said Suddenlink comprised “five or six different customer bases” — each served by cable systems with different capabilities and rate plans. In the last two years, Suddenlink customers have been introduced to new rate plans, and some are paying considerably higher rates than before, especially for equipment and surcharges.

“All of that activity was probably more than we ever wanted to or anticipated as harmonizing all the different variables is not that easy,” Goei said. “And so we made a very concerted effort to not implement a usual or industry like price increase at the end of 2017, given all the various changes that happened over both customer bases as we harmonized them.” But Goei added the reprieve from rate hikes won’t last forever, promising a “rate event” strategy sometime this year, different from rate changes in past years.

Altice is emphasizing the progress it is making boosting internet speeds at its Cablevision and Suddenlink cable systems.

What Suddenlink and Cablevision charge for service is very dependent on what the competition is offering in Altice’s various markets. Goei paradoxically noted that some of the most attractive rates go to customers living in the most affluent areas of the New York Tri-State Area because of intense competition from Verizon FiOS. Prices have remained so low historically that, in Goei’s view, “it makes it very difficult for third parties to come into these markets” and compete with attractive offers that can match Cablevision. That also explains why Cablevision customers do not deal with data caps while Suddenlink customers often do.

Goei

Conversely, in Suddenlink service areas where less capable competitors exist, prices can be higher and service is considered less affordable. As a result, financial analysts have noted Suddenlink’s broadband growth has been anemic since Altice bought the company, presumably because would-be customers cannot afford the service or have chosen a more economic package sold by the phone company, even if it less capable.

Goei promised Altice would be more “nimble” in the future about targeting pricing in different service areas, taking current conditions on the ground into account when setting rates.

In more general terms, Altice is dealing with the same challenges most cable operators are facing these days. Cord-cutting continues to result in reduced numbers of video subscribers. The company also recently endured a multi-week programming dispute with Starz that cost the company video subscribers in the Cablevision service area. The dispute eventually ended with a new multi-year affiliation agreement that allows Altice systems to carry Starz and Starz Encore networks, on-demand services, and online access for several years.

But Altice clearly sees broadband as its key product going forward, which is why the company is upgrading its Cablevision and Suddenlink systems to support faster internet speeds.

AT&T is rolling out mobile 5G service for its wireless smartphone and tablet customers in a dozen U.S. cities by year’s end, starting in parts of Atlanta, Ga., and portions of Dallas and Waco, Tex.

“After significantly contributing to the first phase of 5G standards, conducting multi-city trials, and literally transforming our network for the future, we’re planning to be the first carrier to deliver standards-based mobile 5G – and do it much sooner than most people thought possible,” said Igal Elbaz, senior vice president, wireless network architecture and design. “Our mobile 5G firsts will put our customers in the middle of it all.”

AT&T’s mobile 5G will work differently from the fixed wireless home broadband service Verizon is launching this year using small small cell neighborhood antennas. But like Verizon, AT&T is taking a gradual, incremental approach to the next generation of wireless technology.

In 2017, AT&T announced what it calls “5G Evolution” service in almost two dozen cities, although this branding was derided as “fake 5G” in the tech press because, in reality, it is just an improvement of today’s widely deployed 4G LTE service. Similar technology is also in place at T-Mobile. In the fall of 2017, AT&T introduced 4G LTE-Licensed Assisted Access (LTE-LAA) technology in Indianapolis and parts of Chicago, Los Angeles and San Francisco. This network lays the foundation to offer gigabit speed wireless service, and is especially useful in areas where AT&T’s spectrum holdings are tight.

AT&T’s initial 5G rollout will serve parts of:
A – Atlanta, Ga.
B – Waco, Tex.
C – Dallas, Tex.

AT&T is preparing its existing wireless network to permit gradual migration to the completed 5G wireless standard over both existing and new spectrum.

This year, AT&T plans to launch some 5G service using millimeter wave spectrum, which is very line-of-sight and offers a more limited service area. But the technology will support very fast wireless speeds and offer plenty of bandwidth. AT&T could deploy this technology initially in dense population areas and places like stadiums, malls, and convention centers.

“Ultimately, we expect to reach theoretical peak speeds of multiple gigabits per second on devices through mobile 5G,” AT&T wrote in a press release. “While speed is important, we also expect to see much lower latency rates. With higher speeds and lower latency rates, our mobile 5G network will eventually unlock a number of new, exciting experiences for our customers.”

If past precedent means anything, AT&T will likely only initially offer 5G service in selected parts of each city. It needn’t hurry, because equipment designed to work with the new spectrum isn’t expected to become widely available until 2019. A gradual transition will also please shareholders by keeping network upgrade costs predictable over the next 3-5 years.

AT&T isn’t expected to use 5G technology anytime soon as part of its taxpayer-funded, rural wireless broadband deployment. AT&T currently uses its 4G LTE technology to power its fixed wireless rural broadband service. AT&T claims this service was designed to assure download speeds of at least 10 Mbps, although customers using it report speeds are often lower, although sometimes higher. AT&T does not offer and network performance guarantees, stating, “service performance may be affected by your proximity to a cell site, the capacity of the cell site, the number of other users connected to the same cell site, the surrounding terrain, radio frequency interference, applicable network management practices, and the applications you use.” That will also be true of AT&T’s forthcoming 5G network.

C Spire, an independent wireless company providing service in the southern United States is partnering with electric utility Entergy to jointly construct a new fiber optic network in remote sections of Mississippi to manage an electric smart grid and fiber broadband service.

C Spire will own and build the network, with Entergy contributing construction costs, according to C Spire vice president of government relations Ben Moncrief. The partnership grants Entergy leasing rights to use the fiber optic network to develop smart grid technology for rural Mississippi electric customers. Five individual fiber routes will be build, each with a capacity of 144 or more strands of fiber. Entergy will have exclusive use of its own fiber strands, but C Spire will get most of the capacity to power its backhaul facilities, including its network of cell towers, and eventually deploy the network for commercial and institutional users, with the possibility of expanding service to home and small businesses customers if there is adequate demand.

The fiber network will be uncharacteristically placed in some of the most rural parts of the state’s push to redevelop its rural economy to support digital businesses. C Spire itself has been in transition over the last five years, diversifying its core cellular business into fiber to the home broadband, phone, and television service targeting underserved, smaller communities across the state.

“A robust broadband infrastructure is critical to the success of our efforts to move Mississippi forward by growing the economy, fostering innovation, creating job opportunities and improving the quality of life for all our residents,” said Hu Meena, CEO of C Spire.

C Spire/Entergy Mississippi’s new fiber project

The construction project will involve placing fiber optic cable along five separate routes:

• Delta: 92 miles of fiber through Sunflower, Humphreys, Madison and Hinds counties and near the cities of Indianola, Inverness, Isola, Belzoni, Silver City, Yazoo City, Bentonia, Flora and Jackson.
• North: 51 miles in Attala, Leake and Madison counties, including the communities of McAdams, Kosciusko and Canton.
• Central: 33 miles in Madison, Rankin and Scott counties and near the towns of Canton, Sand Hill and Morton.
• South: 77 miles passing through Simpson, Jefferson Davis, Lawrence and Walthall counties and near the towns of Magee, Prentiss, Silver Creek, Monticello and Tylertown.
• Southwest: 49 miles in Franklin and Adams counties near Bude, Meadville, Roxie, Natchez and Eddiceton.

C Spire got the idea to collaborate with the electric utility after the Mississippi Public Service Commission inquired if Entergy’s plans to build a fiber optic smart grid network could also be used to develop improved broadband service for rural Mississippi. Entergy and C Spire decided to collaborate on the project to deliver both services over the same network.

Fort Collins residents saw their mailboxes filled with mailers last fall opposing community broadband, paid for by the state’s cable lobby.

The Loveland, Col. City Council approved Tuesday four measures that include a $2.5 million spending authorization to lay the groundwork to allow the city to develop a new public broadband network.

The city plans to move quickly, spending $300,000 to develop an in-depth business plan for the service, which the city may run itself. The money will also be spent on researching financing options and a general outreach campaign to explain the service to local residents. Another $2.2 million will cover the development of a detailed solicitation for proposals to build the fiber network an exploration of bonding options.

Some Council members were adamant they will not repeat the mistakes of other Colorado towns by taking muncipal broadband up for a public vote. Several Loveland City Council members commented on a campaign of demagoguery and distortion practiced by incumbent cable and phone companies in Fort Collins and Longmont, which financed expensive campaigns to try to block municipal broadband proposals from getting off the ground. Both industry-funded campaigns failed.

For one Council member, the extensive lobbying campaign in 2017 to smear Fort Collins’ proposal municipal network backfired.

Councilman John Fogle had previously supported requiring a public vote if Loveland decided to get into the broadband business. But then last November he witnessed Fort Collins endure a well-financed effort by the Colorado Cable Telecommunications Association and the Fort Collins Chamber of Commerce to defeat a similar broadband proposal. He changed his mind.

“It’s not an even playing field when incumbent industries will spend $900,000 at the drop of the hat to perpetuate … a monopoly,” Fogle said, noting that local governments cannot spend taxpayer dollars to fight lobbyists and defend their proposals.

Ball: We don’t need a public vote.

Councilman Rich Ball went even further, declaring unless he died or resigned, he would never support a public vote.

“We have the wonderful opportunity to collaborate or we can be the little city that I grew up in that always got beat … by Fort Collins and Longmont,” Ball said.

Many local residents supporting the Loveland public fiber network applauded the decision of local council members not to be tricked into an unfair fight with the well-financed telecom industry.

“I don’t want the Council to spend even five minutes entertaining Comcast’s circus of lies and distortions. I hope those TV ads run last fall in Fort Collins from that fake group sponsored by the Chamber of Commerce taught our state a lesson on what cable monopolies will do to protect their monopoly,” said Loveland resident Susan Collins. “They’ll do whatever it takes and you can lose if you play their game. We already had a vote when we elected our City Council. If people don’t like what they are doing, they can vote them out again.”

But Mayor Pro Tem Don Overcash expressed concern and requested the four measures be amended to require voter approval, believing the Council may be exceeding its authority.

“If citizens want to expand our powers to meet their needs, they have the right to do that,” Overcash said.

A handful of residents also worried they would be paying for a network they won’t use, choosing to stay with their local cable or phone company provider instead.

Loveland, Col.

Councilman Jeremy Jersvig complained that his fellow Council members were making “dictatorial” motions to move forward on the fiber network that, in his view, did not consider public opinion.

But Council members who support Loveland’s public fiber proposal noted:

• In a 2015 election, 82 percent of Loveland voters said “yes” to overriding a state law banning local governments from providing telecommunication services, such as high-speed internet. Other Colorado communities have gone through similar votes.
• The vote allowed the city to explore making high-speed internet available throughout the Loveland area, independently or in partnership, and without raising taxes. City Council will make the final decision on whether to provide this service, and what model to use if so.
• Ultra-fast internet service, with speeds greater than 1 gigabit per second, would be delivered through a citywide fiber-optic network, which is faster than what the local cable or phone company will provide.