Schneiderman

(Reuters) – New York state’s attorney general is probing whether three major Internet providers could be shortchanging consumers by charging them for faster broadband speeds and failing to deliver the speeds being advertised, according to documents seen by Reuters.

The letters, sent on Friday to executives at Verizon Communications, Cablevision Systems, and Time Warner Cable ask each company to provide copies of all disclosures they have made to customers, as well as copies of any testing they may have done of their Internet speeds.

“New Yorkers deserve the Internet speeds they pay for. But, it turns out, many of us may be paying for one thing, and getting another,” Attorney General Eric Schneiderman said in a statement.

In statements, spokesmen for the three companies expressed confidence in the speeds of their Internet services.

“We’re confident that we provide our customers the speeds and services we promise them and look forward to working with the AG to resolve this matter,” Time Warner Cable spokesman Bobby Amirshahi said.

Cablevision spokesman Charlie Schueler said the company’s Optimum Online service “consistently surpasses advertised broadband speeds, including in FCC (Federal Communications Commission) and internal tests. We are happy to provide any necessary performance information to the Attorney General as we do to our customers.”

A Verizon spokesman said the company would cooperate with Schneiderman’s office. “Verizon is confident in the robust and reliable Internet speeds it delivers to subscribers,” the spokesman said.

The attorney general’s investigation is particularly focused on so-called interconnection arrangements, or contractual deals that Internet service providers strike with other networks for the mutual exchange of data.

In the letters, Schneiderman’s office says it is concerned that customers paying a premium for higher speeds may be experiencing a disruption in their service due to technical problems and business disputes over interconnection agreements.

A 2014 study by the Measurement Lab Consortium, or M-Lab, found that customers’ Internet service tended to suffer at points where their broadband providers connected with long-haul Internet traffic carriers, including Cogent Communications Group.

“Internet service provider interconnection has a substantial impact on consumer Internet performance – sometimes a severely negative impact,” the study said, adding that business relationships rather than technical issues were often at the root of the problem.

A spokesman for the attorney general’s office said the 2014 study’s findings, coupled with consumer complaints and internal analysis, prompted the inquiry into Internet speeds.

Some of the letters also raise questions about speeds delivered by Time Warner Cable and Cablevision to consumers over “the last mile,” a term that refers to the point where a telecommunication chain reaches a retail consumer’s devices.

(Reporting by Sarah N. Lynch; Editing by Peter Cooney, Christian Plumb and Jonathan Oatis)

A Frontier Communications service outage in New York left more than 6,000 customers without Internet service for more than 24 hours, leaving businesses with no way to process credit card payments and idling home-based telecommuters.

The outage began early Sunday morning leaving customers near Buffalo, Rochester, and the Southern Tier with no broadband and no answers.

Daniel Virella of Irondequoit called Frontier about the outage and a representative spent 30 minutes troubleshooting his connection with no results.

“I [then] asked him if there was an outage and he says, ‘you know what you’re right,” Virella wrote. “I’m like ‘are you serious?'”

As calls poured into Frontier’s customer service center, nobody had any answers about what the problem was or when it would be fixed.

“There was a recording that said if you’re calling from Rochester, you’ve got a problem,” Stephen Lambert told WROC-TV. “I wish someone would tell me what the problem is.”

By late Sunday, customers took to social media to blast Frontier for its lack of response.

“[Frontier’s] Internet goes down constantly,” complained Rochester resident Mary Ellen Frye. “They are aware of the problem but have no idea when it will be fixed. [Their] service level [is] erratic and totally unacceptable!”

Sharon McCauley Barger was without Frontier Internet for two days in Wheatfield (near Niagara Falls).

“We had to add 2GB to our mobile plan because of this,” she complained.

For businesses affected by the outage, the costs were even higher.

A gas station on Winton Road in Rochester lost business as customers discovered their credit cards wouldn’t work because Frontier’s Internet was offline.

Manager Angel Perez told WROC there is every chance the damage done will last longer than the outage itself.

“The impact is definitely lost sales, customers. You don’t know, they just might not come back,” Perez said.

Eva McDaniel can commiserate. Her service has been out for weeks. She let Frontier know she was fed up with them for the last time.

“Very poor customer service and no resolution on an Internet outage for over a month,” she told the company on their Facebook page. “Good riddance Frontier! I am done!”

Frontier eventually issued a statement that a circuit board was responsible for the failure but it would take several more hours before service was restored. Although Frontier claimed they first received reports of the outage “late Sunday,” Stop the Cap! confirmed customers started calling Frontier about service problems early Sunday morning. Multiple customers were able to confirm the outage began around 7:30am Sunday and ended just before 10:30am Monday morning — more than 24 hours later.

Internet Service Providers are deregulated and are not required to report service outages except when they impact telephone service. The New York Public Service Commission does collect statistics about service outages, mostly as a result of customer complaints.

Customers have some recourse when an outage occurs:

1. Request a service credit for the outage. Providers typically do not give credit unless it is requested. For each day you experience a service outage, Frontier should credit you for one day of service. Multiple outages or extended service problems often call for even larger service credits, especially in response to a complaint filed with a state regulator;
2. File a complaint with a state regulator and/or the FCC. Providers with a poor service record could attract the attention of state or federal officials and provide useful ammunition when a company seeks to expand by buying up other providers and service areas.
3. If service problems are frequent, change providers if you can.

[flv]http://www.phillipdampier.com/video/WROC Rochester Frontier outage frustrates customers 8-10-15.mp4[/flv]

Stop the Cap! talks with WROC-TV about the major Internet outage affecting Frontier Communications DSL service in western New York. (2:36)

CRTC chairman Jean-Pierre Blais

Independent Internet Service Providers are hailing a decision by telecommunications regulators that will force big phone and cable companies to open their fiber optic networks to competitors, suggesting Canadian consumers will benefit from lower prices, fewer usage caps, and higher-speed Internet.

The Canadian Radio-television and Telecommunications Commission on Wednesday ordered companies like Bell/BCE, Telus, Rogers, Shaw, and others to sell wholesale access to their growing fiber optic networks, despite industry protests giving that access would harm future investment in fiber technology just as it is on the cusp of spreading across the country.

“We’re an evidence-based body, so we heard all of the positions of the various parties and we balanced those off through what we heard in our deliberations afterwards,” said CRTC chairman Jean-Pierre Blais. “In this particular case, we are concerned about the future of broadband in the country so we have to make sure we have a sustainable and competitive marketplace. It’s a wholesale decision that says Canadians can expect a better competitive marketplace because we are going to require incumbent cable and telephone companies to make their high-speed facilities available to competitors.”

[flv]http://www.phillipdampier.com/video/BNN Breaking News CRTC Decision Fiber 7-22-15.flv[/flv]

BNN broke into regular programming with this Special Report on the CRTC decision that will grant independent ISPs access to large telecom companies’ fiber optic networks. (3:13)

Large phone companies, including Bell, warned regulators in a hearing last fall that forcing them to open their networks to third parties would deter investment in fiber expansion. Canadian telecom companies now provide about three million homes with either fiber to the home or fiber to the neighborhood service. Blais, along with representatives of independent ISPs have rejected Bell’s arguments, arguing competition from cable operators was forcing telephone companies to upgrade their networks regardless of the wholesale access debate.

“Our view is the incumbent telcos have a market reason to invest in improving their plant through the investment in fiber,” Blais said. “That’s what Canadians expect and because of market conditions they have to do that investment. So we’re quite confident that’s going to happen.”

Canadian telecommunications companies have done well selling Internet and television services in a highly concentrated telecommunications and media marketplace. For example, BCE, the parent company of Bell Canada, Bell Media, and Bell TV owns a wireless carrier, a satellite TV provider, the CTV television network and many of its local affiliates, dozens of radio stations, more than two dozen cable networks, a landline telephone company, an Internet Service Provider, and ownership interests in sports teams like the Montreal Canadiens as well as a part interest in The Globe and Mail, Canada’s unofficial newspaper of record.

Companies like Rogers, Shaw, Vidéotron, Telus, and Bell have dominated the market for Internet access. But regulators began requiring these companies to sell access to their networks on a wholesale basis to smaller competitors to foster additional retail competition. Today, there are over 500 independent ISPs selling service in Canada, including well-known companies like TekSavvy, Primus, and Distributel. In the past few years, Internet enthusiasts have flocked to these alternative providers to escape a regime of usage caps and usage-based billing of Internet service common among most incumbent cable and phone companies. Competition from the independents, which offer more generous usage allowances or sell unlimited access, has forced some phone and cable companies to offer cap-free Internet service as well.

[flv]http://www.phillipdampier.com/video/BNN CRTC Decision Interview with Jean Pierre Blais 7-22-15.flv[/flv]

BNN interviewed CRTC chairman Jean-Pierre Blais about the commission’s decision to open up wholesale access to Canada’s fiber optic networks. (5:26)

Despite the competition, the majority of Canadians still do business with BCE, Rogers Communications, Quebecor (Vidéotron), Shaw Communications, or Telus, that collectively captured 75 percent of telecom revenue in 2013.

Although competitors have been able to purchase wholesale access to cable broadband and DSL service, nothing in the CRTC rules required big cable and phone companies to sell access to next generation fiber networks. That gap threatened the viability of independent ISPs, left with offering customers access to older cable/copper technology only. This week’s CRTC decision is the first step to grant access to fiber networks as well, although some ISPs are cautious about the impact of the decision until the CRTC provides pricing guidance.

“The commission took a great step today in favor of competition,” Matt Stein, CEO of Distributel Communications Ltd., told The Globe and Mail. “In giving us access to fiber to the premise, they have ensured that as speeds and demands increase, we’re going to continue to be able to provide service that customers want. It’s definitely going to be some time before these products make it to market. There’s going to be the costing and the implementation, and reasonably it could be a year or even longer before the products are actually out the door. But the heavy lifting? Today that was done.”

Bram Abramson, chief legal and regulatory officer for TekSavvy Solutions Inc., added some caution.

Distributel, an independent ISP, made a name for itself offering usage-cap free Internet access to Canadians.

“The devil really is in the details on this,” Abramson told the newspaper. “That’s why I say we like the direction, because there are a million ways in which this could become unworkable if implemented wrong. For example, what rates are we going to pay? We won’t know until those tariffs are done and settled.”

Other so-called “wireline incumbents” like Manitoba Telecom and SaskTel will also be required to make their fiber optic networks available to competitors.

Last fall, Bell warned the CRTC of the consequences of letting TekSavvy, Distributel, and others resell access to their fiber networks.

“We are not suggesting that mandated access will immediately grind investment to a halt in every location in Canada, but it is a question of balance and it will have an impact,” Mirko Bibic, chief legal and regulatory officer for BCE/Bell told CRTC commissioners at a hearing.

Bibic cautioned if the CRTC granted competitive access it could affect how the company allocated its capital investments and could lead it to shift spending to other areas instead.

“What we’re saying is a mandated access rule will affect the pace of deployment and the breadth of deployment,” Bibic said.

Bibic

Specifically, Bibic claimed Bell may call it quits on fiber expansion beyond the fiber-to-the-neighborhood service Bell sells under the Fibe brand in 80% of its service area in Ontario and Quebec. Bell had envisioned upgrading the network to straight fiber-to-the-home service, eliminating the rest of the legacy copper still in its network. But perhaps not anymore.

“If the commission forces the incumbent telephone operators to open access to fiber-to-the-home, BCE might not prioritize building that final leg in some communities,” Bibic warned. “The point is, with 80% of our territory covered […] we can hold and do really well with fiber-to-the-node for longer than we otherwise might.”

Nonsense, independent ISPs told the CRTC, pointing to the cable industry’s preparations to introduce DOCSIS 3.1 cable broadband and vastly increase broadband speeds well in excess of what a fiber-to-the-neighborhood network can offer.

“First of all, [telephone companies] have a natural incentive to build wherever there is a cable carrier, because otherwise the cable carrier will eat their lunch,” said Chris Tacit, counsel to the Canadian Network Operators Consortium, which represents the interests of independent ISPs. “There’s a reason that they’re sinking all that money into [fiber-to-the-home], it’s because they have to keep up. Now, I don’t believe for a minute that they are going to stop investing if they have to grant access.”

Regulators in the United States have traditionally sided with large telecommunications companies and have largely allowed phone and cable companies to keep access to their advanced broadband networks to themselves. Republicans have largely defended the industry position that regulation and forced open access would deter private investment and competitors should construct networks of their own. In some cases, they have. Google Fiber is now the most prominent overbuilder, but several dozen independent providers are also slowly wiring fiber optics in communities already served by cable and telephone company-provided broadband. Whether it is better to inspire new entrants to build their own networks or grant them access to existing ones is an ongoing political debate.

But the CRTC has not given independent ISPs a free ride. The commission announced it will begin moving towards “disaggregated” network availability for smaller ISPs, which will require them to invest in network equipment to connect with incumbent networks on a more local level, starting in Ontario and Quebec.

The CRTC under Blais’ leadership is gaining a reputation of being pro-consumer, a departure from the CRTC’s often-industry-friendly past. Blais has presided over rulings to regulate wholesale wireless roaming fees to lower consumer costs and forced pay television providers to unbundle their huge TV channel packages so consumers can get rid of scores of channels they don’t watch.

[flv]http://www.phillipdampier.com/video/The Globe and Mail Internet competitors welcome CRTC decision on broadband access 7-23-15.flv[/flv]

Canadian Press spoke with independent ISPs about their reaction to the wholesale access decision. (1:18)

Phillip “Speaking as a Customer” Dampier

Two defenders of large Internet Service Providers are coming to the defense of the broadband industry by questioning a Guardian article that reported major Internet Service Providers were intentionally allowing a degradation in performance of Content Delivery Networks and other high volume Internet traffic in a dispute over money.

Richard Bennett and Dan Rayburn today both published articles attempting to discredit Battle for the Net’s effort highlighting the impact interconnection disputes can have on consumers.

Rayburn:

On Monday The Guardian ran a story with a headline stating that major Internet providers are slowing traffic speeds for thousands of consumers in North America. While that’s a title that’s going to get a lot of people’s attention, it’s not accurate. Even worse, other news outlets like Network World picked up on the story, re-hashed everything The Guardian said, but then mentioned they could not find the “study” that The Guardian is talking about. The reason they can’t find the report is because it does not exist.

[…] Even if The Guardian article was trying to use data collected via the BattlefortheNet website, they don’t understand what data is actually being collected. That data is specific to problems at interconnection points, not inside the last mile networks. So if there isn’t enough capacity at an interconnection point, saying ISPs are “slowing traffic speeds” is not accurate. No ISP is slowing down the speed of the consumers’ connection to the Internet as that all takes place inside the last mile, which is outside of the interconnection points. Even the Free Press isn’t quoted as saying ISPs are “slowing” down access speed, but rather access to enough capacity at connection points.

Bennett:

In summary, it appears that Battle for the Net may have cooked up some dubious tests to support their predetermined conclusion that ISPs are engaging in evil, extortionate behavior.

It may well be the case that they want to, but AT&T, Verizon, Charter Cable, Time Warner Cable, Brighthouse, and several others have merger business and spectrum auction business pending before the FCC. If they were manipulating customer experience in such a malicious way during the pendency of the their critical business, that would constitute executive ineptitude on an enormous scale. The alleged behavior doesn’t make customers stick around either.

I doubt the ISPs are stupid enough to do what the Guardian says they’re doing, and a careful examination of the available test data says that Battle for the Net is actually cooking the books. There is no way a long haul bandwidth and latency test says a thing about CDN performance. Now it could be that Battle for the Net has as a secret test that actually measures CDNs, but if so it’s certainly a well-kept one. Stay tuned.

The higher line measures speeds received by Comcast customers connecting to websites handled by GTT in Atlanta. The lower line represents speeds endured by AT&T customers, as measured by MLab.

Stop the Cap! was peripherally mentioned in Rayburn’s piece because we originally referenced one of the affected providers as a Content Delivery Network (CDN). In fact, GTT is a Tier 1 IP Network, providing service to CDNs, among others — a point we made in a correction prompted by one of our readers yesterday.

Both Rayburn and Bennett scoff at Battle for the Net’s methodology, results, and conclusion your Internet Service Provider might care more about money than keeping customers satisfied with decent Internet speeds. Bennett alludes to the five groups backing the Battle for the Net campaign as “comrades” and Rayburn comes close to suggesting the Guardian piece represented journalistic malpractice.

Much was made of the missing “study” that the Guardian referenced in its original piece. Stop the Cap! told readers in our original story we did not have a copy to share either, but would update the story once it became available.

We published our own story because we were able to find, without much difficulty, plenty of raw data collected by MLab from consumers conducting voluntary Internet Health Tests, on which Battle for the Net drew its conclusions about network performance. A review of that data independently confirmed all the performance assertions made in the Guardian story, with or without a report. There are obvious and undeniable significant differences in performance between certain Internet Service Providers and traffic distribution networks like GTT.

So let’s take a closer look at the issues Rayburn and Bennett either dispute or attempt to explain away:

1. MLab today confirmed there is a measurable and clear problem with ISPs serving around 75% of Americans that apparently involves under-provisioned interconnection capacity. That means the connection your ISP has with some content distributors is inadequate to handle the amount of traffic requested by customers. Some very large content distributors like Netflix increasingly use their own Content Delivery Networks, while others rely on third-party distributors to move that content for them. But the problem affects more than just high traffic video websites. If Stop the Cap! happens to reach you through one of these congested traffic networks and your ISP won’t upgrade that connection without compensation, not only will video traffic suffer slowdowns and buffering, but so will traffic from every other website, including ours, that happens to be sent through that same connection.

MLab: “Customers of Comcast, Time Warner Cable, and Verizon all saw degraded performance [in NYC] during peak use hours when connecting across transit ISPs GTT and Tata. These patterns were most dramatic for customers of Comcast and Verizon when connecting to GTT, with a low-speed of near 1 Mbps during peak hours in May. None of the three experienced similar problems when connecting with other transit providers, such as Internap and Zayo, and Cablevision did not experience the same extent of problems.”

Our initial findings show persistent performance degradation experienced by customers of a number of major access ISPs across the United States during the first half of 2015. While the ISPs involved differ, the symptoms and patterns of degradation are similar to those detailed in last year’s Interconnections study: decreased download throughput, increased latency and increased packet loss compared to the performance through different access ISPs in the same region. In nearly all cases degradation was worse during peak use hours. In last year’s technical report, we found that peak-hour degradation was an indicator of under-provisioned interconnection capacity whose shortcomings are only felt when traffic grows beyond a certain threshold.

Patterns of degraded performance occurred across the United States, impacting customers of various access ISPs when connecting to measurement points hosted within a number of transit ISPs in Atlanta, Chicago, Los Angeles, New York, Seattle, and Washington, D.C. Many of these access-transit ISP pairs have not previously been available for study using M-Lab data. In September, 2014, several measurement points were added in transit networks across the United States, making it possible to measure more access-transit ISP interconnection points. It is important to note that while we are able to observe and record these episodes of performance degradation, nothing in the data allows us to draw conclusions about who is responsible for the performance degradation. We leave determining the underlying cause of the degradation to others, and focus solely on the data, which tells us about consumer conditions irrespective of cause.

Rayburn attempts to go to town highlighting MLab’s statement that the data does not allow it to draw conclusions about who is responsible for the traffic jam. But any effort to extend that to a broader conclusion the Guardian article is “bogus” is folly. MLab’s findings clearly state there is a problem affecting the consumer’s Internet experience. To be fair, Rayburn’s view generally accepts there are disputes involving interconnection agreements, but he defends the current system that requires IP networks sending more traffic than they return to pay the ISP for a better connection.

Rayburn’s website refers to him as “the voice of industry.”

1. Rayburn comes to the debate with a different perspective than ours. Rayburn’s website highlights the fact he is the “voice of the industry.” He also helped launch the industry trade group Streaming Video Alliance, which counts Comcast as one of its members. Anyone able to afford the dues for sponsor/founding member ($25,000 annually); full member ($12,500); or supporting member ($5,500) can join.

Stop the Cap! unreservedly speaks only for consumers. In these disputes, paying customers are the undeniable collateral damage when Internet slowdowns occur and more than a few are frequently inconvenienced by congestion-related slowdowns.

It is our view that allowing paying customers to be caught in the middle of these disputes is a symptom of the monopoly/duopoly marketplace broadband providers enjoy. In any industry where competition demands a provider deliver an excellent customer experience, few would ever allow these kinds of disputes to alienate customers. In Atlanta, Los Angeles, and Chicago, for example, AT&T has evidently made a business decision to allow its connections with GTT to degrade to just a fraction of the performance achieved by other providers. Nothing else explains consistent slowdowns that have affected AT&T U-verse and DSL customers for months on end that involve GTT while Comcast customers experience none of those problems.

We also know why this is happening because AT&T and GTT have both confirmed it to Ars Technica, which covered this specific slowdown back in March. As is always the case about these disputes, it’s all about the money:

AT&T is seeking money from network operators and won’t upgrade capacity until it gets paid. Under its peering policy, AT&T demands payment when a network sends more than twice as much traffic as it receives.

“Some providers are sending significantly more than twice as much traffic as they are receiving at specific interconnection points, which violates our peering policy that has been in place for years,” AT&T told Ars. “We are engaged in commercial-agreement discussions, as is typical in such situations, with several ISPs and Internet providers regarding this imbalanced traffic and possible solutions for augmenting capacity.”

Missing from this discussion are AT&T customers directly affected by slowdowns. AT&T’s attitude seems uninterested in the customer experience and the company feels safe stonewalling GTT until it gets a check in the mail. It matters less that AT&T customers have paid $40, 50, even 70 a month for high quality Internet service they are not getting.

In a more competitive marketplace, we believe no ISP would ever allow these disputes to impact paying subscribers, because a dissatisfied customer can cancel service and switch providers. That is much less likely if you are an AT&T DSL customer with no cable competition or if your only other choice cannot offer the Internet speed you need.

1. Consolidating the telecommunications industry will only guarantee these problems will get worse. If AT&T is allowed to merge with DirecTV and expand Internet service to more customers in rural areas where cable broadband does not reach, does that not strengthen AT&T’s ability to further stonewall content providers? Of course it does. In fact, even a company the size of Netflix eventually relented and wrote a check to Comcast to clear up major congestion problems experienced by Comcast customers in 2014. Comcast could have solved the problem itself for the benefit of its paying customers, but refused. The day Netflix’s check arrived, problems with Netflix magically disappeared.

More mergers and more consolidation does not enhance competition. It entrenches big ISPs to play more aggressive hardball with content providers at the expense of consumers.

Even Rayburn concedes these disputes are “not about ‘fairness,’ it’s business,” he writes. “Some pay based on various business terms, others might not. There is no law against it, no rule that prohibits it.”

Battle for the Net’s point may be that there should be.

Customers of Lithuania’s Teo are getting a free speed upgrade — from 500Mbps before to 600Mbps now — on the company’s fiber to the home network. They are also paying less than half the price of what you pay for 15Mbps.

“Internet bandwidth is constantly increasing and high-speed becomes a market norm,” said Teo’s Nerijus Ivanauskas. “Therefore, we see that the added value that customers receive from purchasing a basic service becomes an increasingly important factor when choosing a service provider.”

Lithuanians have several choices for broadband service and price competition has kept broadband speeds faster than what North Americans typically receive, at a fraction of the price. Fiber to the home service is increasingly common in populated areas and is very affordable. Budget-minded customers happy with 100/100Mbps Internet access can get it from Teo for less than $13.50 a month.

Teo’s fiber network passes 837,000 households as of the first quarter of this year. That represents almost 70% of Lithuania. Lithuania was already well ahead of the United States and Canada, with an average broadband speed of 45.11Mbps — 4th place in the European Union and 9th fastest country in the world. Teo also leads the world in fast Wi-Fi. More than 3,000 Teo hotspots serve up speeds averaging 15.4Mbps to every connected client.

As broadband speeds continue to soar in Lithuania, Internet Service Providers have been forced to offer extras to customers to compete. Teo offers 300GB of free cloud storage space, free anti-virus protection, and special parental controls to help protect children from adult content.

[flv]http://www.phillipdampier.com/video/Teo Internet Speed Lithuania 6-2015.mp4[/flv]

A Teo advertisement showing off its fiber broadband speeds, ubiquitous free Wi-Fi network, anti-virus and child protection features. (0:45)