The country’s largest cable internet service provider needed help from an app developer in Portland, Ore. to let it know its broadband pipes were full and to do something about it.

Comcast customers were complaining about slow downloads from the Panic website and the company’s own workers were saying largely the same thing when attempting to remotely connect to the company’s servers from home.

Because Panic’s web servers have just a single connection to the internet via Cogent, it would be a simple matter to track down where the traffic bottleneck was occurring, assuming there was one. The company asked for volunteers to run a test transferring 20MB of data first from Panic’s server and then again from a control server hosted with Linode, a popular and well-respected hosting company.

The results were pretty stunning.

With speeds often around only 356.3kbps for Comcast customers connecting to Panic, something was definitely up. It also explained why employees had a rough time connecting to the company’s server as well — Panic’s workers are based in Portland, Ore., where Comcast is used by almost every employee.

The slowdowns were not related to the time of day and because the problem persisted for weeks, it wasn’t a temporary technical fault. Panic’s blog picks up the story about what is behind all this:

Peering.

Major internet pipes, like Cogent, have peering agreements with network providers, like Comcast. These companies need each other — Cogent can’t exist if their network doesn’t go all the way to the end user, and Comcast can’t exist if they can’t send their customer’s data all over the world. One core tenet of peering is that it is “settlement-free” — neither party pays the other party to exchange their traffic. Instead, each party generates revenue from their customers. Cogent generates revenue from us. Comcast generates revenue from us at home. Everyone wins, right?

After a quick Google session, I learned that Cogent and Comcast have quite a storied history. This history started when Cogent started delivering a great deal of video content to Comcast customers… content from Netflix. and suddenly, the “peering pipe” that connects Cogent and Comcast filled up and slowed dramatically down.

Normally when these peering pipes “fill up”, more capacity is added between the two companies. But, if you believe Cogent’s side of the story, Comcast simply decided not to play ball — and refused to add any additional bandwidth unless Cogent paid them. In other words, Comcast didn’t like being paid nothing to deliver Netflix traffic, which competes with its own TV and streaming offerings. This Ars Technica article covers it well. (How did Netflix solve this problem in 2014? Netflix entered into a business agreement to pay Comcast directly. And suddenly, more peering bandwidth opened up between Comcast and Cogent, like magic.)

We felt certain history was repeating itself: the peering connection between Comcast and Cogent was once again saturated. Cogent said their hands were tied. What now?

In addition to giving the internet public policy community new evidence that peering fights leaving customers stuck in the middle might be heating up once again. It also suggests if Comcast was unaware of the problem, it does not reflect well on the cable company to wait weeks until a customer reports such a serious slowdown before fixing it.

The folks at Panic took a chance and reported the problem to Comcast, bypassing the usual customer support route in favor of a corporate contact who listed a direct email address on the company’s website. Comcast took the request seriously and eventually responded, “give us one to two weeks, and if you re-run your test I think you’ll be happy with the results.”

Indeed, the problem was fixed. The folks at Panic say according to Comcast, two primary changes were made:

1. Comcast added more capacity for Cogent traffic. (As suspected, the pipe was full.)
2. Cogent made some unspecified changes to their traffic engineering.

The folks at Panic and their users are happy that the problem is fixed, but some questions remain:

1. Is Comcast intentionally throttling web traffic in an attempt to extract a more favorable peering agreement with Cogent?
2. How could Comcast not know this particular connection was hopelessly over-capacity for several weeks, leaving customers to deal with heavily throttled traffic.

“While this story amazingly had a happy ending, I’m not looking forward to the next time we’re stuck in the middle of a peering dispute between two companies,” wrote Cabel. “It feels absolutely inevitable, all the more so now that net neutrality is gone. Here’s hoping the next time it happens, the responsible party is as responsive as Comcast was this time.”

Panic explains internet slowdowns resulting from peering disputes in this (3:30) video.

After 10 months of negotiations between Frontier Communications and the Communications Workers of America (CWA) over the phone company’s job cuts, 1,400 Frontier workers in West Virginia and Ashburn, Va., walked off the job Sunday.

The Communications Workers of America claims they have been unable to reach an agreement on a fair contract with Frontier despite three extensions. The original contract expired in August, 2017. The CWA claims their members have waited long enough and called a strike.

“We have been very clear throughout the bargaining process that our top priority is keeping good jobs in our communities,” said Ed Mooney, vice president of CWA District 2-13. “Going on strike is never easy. It’s a hardship for our members and the customers who we are proud to serve. But the job cuts at Frontier have gone too far — we know it and Frontier’s customers know it. It’s time for Frontier to start investing in maintaining and rebuilding its network in West Virginia.”

The CWA claims Frontier has let go of some of its most experienced technicians while outsourcing an increasing number of jobs to outside contractors. Frontier has also cut over 500 jobs in the area since 2012 and has announced a plan for additional layoffs this month. The union claims Frontier’s customers are suffering too.

“We’re taking a stand,” said Johnny Bailey, president of CWA Local 2226 in Bluefield. “Customers are waiting way too long to have their problems resolved, and too often we’re back fixing the same problems over and over again. Frontier is leaving West Virginia behind. The network has been neglected and there are just not enough experienced, well-trained workers left to handle the service requests.”

According to CWA, complaints filed with the West Virginia Public Service Commission have increased steadily over the past three years, rising 69% from 639 in 2014 to 1,072 complaints in 2017.

“The complaints at Frontier have risen so high in the last few years it is has gotten to the point [… where] we are embarrassed by the product that we have to serve,” said Jeff Anderson, president of CWA Local 2004, which covers large parts of north-central West Virginia, including Harrison, Marion, Monongalia, Taylor, and Doddridge counties. “In some areas we have good service but we beg for that and we ask the company and we will do anything we can to get our people better service cause ultimately that is what keeps our jobs.”

Frontier countered the company is already extremely generous with its workforce.

“Frontier is one of West Virginia’s best employers,” the company said in a statement. “Average annual wages for the Company’s union employees exceed $64,500, and more than half of all union employees earn more than $75,000 per year. For comprehensive family medical coverage, most employees pay less than $150 per month for family coverage, with no annual deductible and low co-pays. Including employee benefits, the Company’s average employee cost per CWA member is more than $100,000.”

Frontier said it has activated its strike contingency plan, which will require Frontier’s management, outside contractors and Frontier employees from other areas to handle service calls and other tasks formerly done by striking workers.

Customers can expect to encounter Frontier’s picket lines in several places:

FCC Chairman Ajit Pai is under investigation by the Inspector General of the Federal Communications Commission after being alleged of improperly taking actions to benefit Sinclair Broadcast Group, while one of his colleagues, Commissioner Michael O’Rielly, is the subject of an ethics complaint after allegedly violating the Hatch Act by openly advocating for the re-election of President Donald Trump.

Pai’s actions as head of the FCC under the Trump Administration have been under scrutiny by some members of Congress since last fall. Ranking Member of the House Energy & Commerce Committee, Rep. Frank Pallone, Jr. (D-N.J.) and Ranking Member of the House Committee on Oversight and Government Reform, Elijah Cummings (D-Md.) signed a joint letter addressed to FCC Inspector General James Hunt last November requesting an investigation after they claimed Chairman Pai “has repeatedly refused to adequately respond to Congressional inquiries” on the matter.

Pallone and Cummings noted press reports that Pai specifically timed certain FCC regulatory actions to directly benefit Sinclair, seen as politically friendly to the Trump Administration and Republicans. As evidence, they included multiple examples of suspiciously timed regulatory changes that seemed to coincide with Sinclair’s business deals and the company’s lobbying efforts in Washington:

Sinclair-Bonten License Transfer Application

Chairman Pai rescinded a guidance in February (2017), effectively loosening the scrutiny the FCC’s staff applied to deals that could skirt local TV ownership restrictions by using a sharing agreement (effectively allowing Sinclair to control stations owned by another company). The FCC approved a deal three months later where Sinclair used several of these sharing agreements, potentially to circumvent the rules.

Reinstatement of the UHF Discount Rule

Press reports indicated in March, 2017, Sinclair was in talks with Tribune Media Company about a potential merger, but analysts remarked the deal would likely require the FCC to reinstate an outdated rule called the “UHF discount.” This rule, left over from the days of analog television and finally rescinded in 2016, did not count UHF television stations above Channel 13 the same as VHF stations (Chs. 2-13) when defining how many TV stations a single company can own. The theory behind the discount was that analog UHF reception was more difficult and, as a result, such stations were less valuable than their lower channel counterparts. But digital television largely erased that distinction because UHF reception has improved, TV stations can be “mapped” by digital tuners to any channel number, and, in some areas, digital VHF stations suffer more reception problems than UHF stations do.

Chairman Pai suddenly announced his plan to reinstate the outdated UHF discount rule the same month Sinclair began talks with Tribune. Sinclair announced its proposed acquisition of Tribune’s TV stations just two weeks after the FCC reinstated the UHF discount. If approved, the transaction would solidify Sinclair as the country’s largest TV group owner with a potential to reach 70% of the country, which is far in excess of the current 39% limit.

Next Gen TV (ATSC 3.0)

Sinclair has been one of the main proponents of the ATSC 3,0 (also known as “Next Gen TV”) transition, and its subsidiary holds patents that reports indicate could provide billions of dollars in licensing fees to Sinclair. Chairman Pai announced during his first full month in office a proposal to allow the TV industry to accelerate a transition to the new standard.

Since that time, the FCC has pushed ATSC 3.0 forward and the new technology has begun to be tested in the United States. Some consumer groups worry the new technology will be costly if consumers cannot afford or find converter boxes for existing televisions, although ATSC 3.0 proponents promise stations will continue to broadcast a Standard Definition version of existing TV stations for at least five years after the transition begins.

New televisions supporting the standard have already gone on sale in South Korea at prices ranging from around $900-$1,500US. The government is subsidizing TV station owners a minimum of $1.75 billion as part of a TV station repack that will precede the introduction of ATSC 3.0. But no subsidies will be given to consumers. Those buying ATSC 3.0 tuners or televisions will do so out of their own pocket if they wish to continue watching over-the-air stations. Sinclair will also get a royalty payment for each new television or tuner sold.

Main Studio Rule

The FCC voted last October to eliminate rules requiring a local broadcast station to maintain a physical presence in the market in which it operates. This means a station could deliver programming to a station’s transmitter from another city, with no local programming or personnel. This move would make Sinclair’s potential merger even more profitable by eliminating many of the costs of maintaining local stations, particularly labor and news-gathering costs.

Broadcast Ownership Rules

Chairman Pai plans to significantly change the existing broadcast media ownership limits. This would clear away virtually all remaining obstacles to Sinclair increasing its reach beyond the Tribune merger proposal and acquire still more television stations. Sinclair has carefully prepared for this eventuality by contractually obligating the new owner(s) of stations Sinclair is required to sell to remain under whatever ownership cap still exists to sell those stations back to Sinclair if and when Sinclair requests it.

According to the two Democrats, “all of these actions — when taken in context with reported meetings between the Trump Administration, Sinclair, and Chairman Pai’s office — have raised serious concerns about whether Pai’s actions comply with the FCC’s mandate to be independent.”

Pai’s critics are also concerned about the increased partisanship of the chairman and another Republican FCC Commissioner Michael O’Rielly. Both turned up at the Conservative Political Action Conference (CPAC) in Maryland last week.

The NRA’s “Charlton Heston Courage Under Fire Award” for Ajit Pai’s Assassination of Net Neutrality Includes a Kentucky Long Gun

When Pai arrived on stage to deliver a short speech, Dan Schneider, executive director of the American Conservative Union, which sponsors CPAC, took the microphone to introduce the FCC chairman.

“Ajit Pai is the most courageous, heroic person that I know,” Schneider said. “He has received countless death threats. His property has been invaded by the George Soros crowd. He has a family, and his family has been abused in different ways. Chairman Pai, thank you for everything you’ve done.”

He then turned the podium over to Carolyn Meadows, second vice president of the National Rifle Association, who surprised Pai with the NRA’s “Charlton Heston Courage Under Fire Award,” a rare honor given only to firebrand conservatives willing to push through their political agenda regardless of criticism or voter backlash. Pai was being recognized for ignoring the comments of tens of millions of supporters of net neutrality and pushing through a complete repeal of the open internet rules, regardless of the possible political consequences.

Previous award winners include controversial former Milwaukee Sheriff David A. Clarke Jr., Undersecretary John Bolton, who once threw a tape dispenser at a female government contractor and chased her down a Moscow hotel hallway, conservative talk show host Rush Limbaugh, and Vice President Mike Pence.

The honor included a “Kentucky hand-made long gun,” said Meadows, who promised to store the gun for Pai at an NRA museum. That prompted a Tweet from the former director of the Office of Government Ethics, Walter Shaub, claiming Pai’s gun award likely violated federal ethics rules.

Anyone care to explain to me why the FCC thinks that the ethics rules allow Ajit Pai to accept the gift of an expensive handmade gun from the NRA, an entity whose interests he can affect (and has affected) by the performance of his official duties? Am I missing something? https://t.co/S6ocyWIV7H

— Walter Shaub (@waltshaub) February 24, 2018

As criticism of the FCC chairman grew, Pai’s office sent letters on Thursday to both the NRA and the American Conservative Union declining the handmade weapon. Pai indirectly blamed the NRA, claiming his staff has asked at the event that the gun not be given to him. But the NRA came up with its own compromise, storing the gun until Pai left office.

“As you know, once my staff became aware of what was happening, they asked backstage that the musket not be presented to me to ensure that this could be first discussed with and vetted by career ethics attorneys in the FCC’s Office of General Counsel,” Pai wrote, an FCC source told Politico. “Therefore, upon their counsel, I must respectfully decline the award. I have also been advised by the FCC’s career ethics attorneys that I would not be able to accept the award upon my departure from government service.”

FCC Commissioner Michael O’Rielly Calls for the Re-Election of President Trump and Violates the Hatch Act

At the same CPAC event, FCC Commissioner Michael O’Rielly also managed to find himself the subject of controversy in response to a question.

Q. What can the FCC do to stop the constant “ping pong” of issues, like net neutrality, every time the party in power changes in the nation’s capital?
A. “I think what we can do is make sure as conservatives that we elect good people to both the House, Senate and make sure that President Trump gets re-elected,” O’Rielly answered.

Experts claim O’Rielly violated the Hatch Act, a law that prohibits employees in the executive branch of the federal government, except the president, vice-president, and certain designated high-level officials from engaging in certain forms of political activity. Telling the public to re-elect President Trump counts as a violation.

The Office of Special Counsel (OSC) already warned government officials so steer clear of President Trump’s already announced 2020 re-election campaign. In short, the Hatch Act “prohibits federal employees, while on duty or in the workplace, from expressly advocating for or against his reelection in 2020,” the OSC wrote in a guidance memo distributed to all federal agencies.

American Oversight, a group that monitors ethics issues in Washington, filed a formal complaint with the OSC against O’Rielly on Feb. 23:

“American Oversight respectfully requests that the Office of Special Counsel (“OSC”) immediately open an investigation into whether Michael O’Rielly, Commissioner on the Federal Communications Commission (FCC), violated the Hatch Act during an appearance at the Conservative Political Action Conference today, February 23, 2018. We do not believe this presents a hard question.

“Appearing in his capacity as a commissioner of the FCC, Commissioner O’Rielly improperly engaged in partisan political activity by expressly advocating for the re-election of Donald Trump
and exhorting the crowd to “elect good people to the House [and] the Senate.” Specifically, during the panel discussion, Commissioner O’Reilly delivered the following remarks:

“‘I think what we can do is make sure as conservatives that we elect good people to both the House, the Senate, and make sure that President Trump gets re-elected. But there’s another thing you can do. We’re going to have a fight over the Obama internet rules in the next couple months in the U.S. Senate. And that’s going to matter and that vote matters, and so making sure people take the right course on that, really does affect what policies we’re able to keep in place moving forward. So we can certainly use everyone’s help along those lines.’

“These remarks, made in Commissioner O’Rielly’s capacity as a commissioner at the FCC, constitute prohibited partisan political activity under the Hatch Act. As you know, the Hatch Act generally prohibits federal officials from engaging in partisan political activity while on duty. Advocating for the election of a candidate in a partisan election is the classic example of such prohibited activity.”

“The FCC controls our airwaves, the internet, and so many of the things we interact with every single day,” said Austin Evers, the executive director of American Oversight. “It should be independent, it should not be partisan, and bottom line, it should obey the law.”

Another group, Citizens for Responsibility and Ethics in Washington (CREW) is also reviewing the event.

“This certainly raises Hatch Act issues,” spokesman Jordan Libowitz told the Chicago Tribune. “[O’Rielly] is prohibited from taking part in partisan political activity using his official title or position.”

“The Young Turks” explain Ajit Pai’s attack on net neutrality and the award the NRA gave him for killing it. (7:16)

The Minnesota Public Utilities Commission (MPUC) has opened an inquiry into whether Frontier Communications is meeting its service obligations to customers after receiving a major spike in complaints about the phone company.

The MPUC acknowledged it has been “receiving a large volume of complaints related to the service quality, customer service, and billing practices of Frontier Communications.” The regulator is concerned that “after attempts to mediate these complaints, many of them remain unresolved.”

The investigation will include the Minnesota Department of Commerce and Minnesota’s Attorney General, both tasked with determining if Frontier is complying with MPUC rules and Minnesota state law.

Frontier provides service to more than 98,000 landlines in Minnesota, doing business as Frontier Communications and Citizens Telecommunications. Most Frontier customers are located in northeastern and southern Minnesota, as well as communities like Apple Valley, Burnsville, Farmington, and Rosemount.

A survey of filed complaints found many involved Frontier’s DSL internet service, which customers complained was slow and prone to frequent outages. Other complaints involved inaccurate billing and missed service calls, which sometimes led to delays of days or weeks before service could be restored.

“I’d heard a bunch of complaints of poor service all across my district,” said Rep. Rob Ecklund (DFL-International Falls) in a news release. “I am a Frontier customer myself, and the service has been lousy.”

Other customers had their complaints published in the Timberjay newspaper, which has been the unofficial meeting place for frustrated customers who cannot get satisfaction from Frontier.

“This has been the worst service experience of my life,” said Melissa Holmes, of Embarrass in northeastern Minnesota. “My whole neighborhood here on Wahlsten Road in Embarrass has had service issues with Frontier for decades. Repeated calls to the company go nowhere.”

The newspaper blamed Frontier’s wrong priorities in a scathing editorial last fall:

Prospects for an improvement in Frontier’s service quality appear unlikely given the increasingly tenuous financial condition of the company. Frontier went deeply in debt in early 2016, when it completed an $11 billion purchase of landline infrastructure formerly owned by Verizon in California, Texas, and Florida. The acquisition more than doubled the size of the company, but also prompted a major restructuring, which included significant layoffs.

Frontier officials had touted the acquisition at the time, arguing that the company knew how to make money from traditional landline infrastructure even as the industry is rapidly transitioning to wireless. But the company has yet to demonstrate it is up to the challenge and as complaints over poor service have mounted, the company has hemorrhaged customers, particularly in more populated regions, where customers often have viable alternatives.

In response, Frontier claims it updated its billing software and is making “process improvements” in the way it conducts business.

If you live in Minnesota and wish to share your views with the MPUC, you can visit their website, register, and comment until May 25, 2018.

The state’s initial investigation and report on Frontier is due on May 11.

KSTP-TV in Minneapolis-St. Paul reports Frontier is under investigation by the state telecom regulator for poor service. (2:21)

Charter Communications has introduced internet-delivered cable television packages that its cable TV subscribers have requested for years, including one offering a true a-la-carte lineup of network TV channels and the customer’s choice of 10 cable channels for $25 a month.

Spectrum Choice was soft-launched this week and is a companion to a larger internet-delivered package of TV services targeting cord-cutters called Spectrum Stream, which is also available in many areas.

Although Spectrum customers can visit the order page to sign up for Spectrum Choice immediately, when we tested it this afternoon we found the website was not able to complete an order. It turns out Spectrum is initially “hand-selecting” about 100,000 customers in selected areas for Spectrum Choice, but won’t disclose exactly where those areas are. We know from some reviews, it is available in parts of Ohio.

For now, would-be customers can try building their own package from at least 65 cable networks, including several networks Spectrum usually bundles into higher cost Silver and Gold packages. For example, Turner Classic Movies, Hallmark Movies and Mysteries, and FX Movie Channel are all available to choose. Spectrum Choice also offers all three major cable news networks as well as Spectrum News (where available). ESPN, ESPN II, FOX Sports, NBC Sports Network, and NFL Network are also available for sports fans. Even Music Choice is included.

Spectrum Choice customers are not tied down with a bloated package of channels, except for the included large bundle of local stations, which includes ABC, CBS, NBC, FOX, CW, MyNetworkTV, PBS, and independent/foreign language over the air stations. The availability of public television is a rarity among online cable TV alternatives. In most areas, digital subchannels like Grit and MeTV are also included, depending on what networks are provided by stations in your area. You will also get several shopping channels, C-SPAN I, II, and III, and local Public, Educational, and Government Access channels as seen on your local cable system.

If you visit their website can complete an order online, you are qualified to receive their service. If there is no option to move forward to complete an order, you are not qualified to sign up at this time, but check back later or call Spectrum and ask.

The service relies on the Spectrum TV app (available on iOS, Android, Roku, and Xbox One) and the Spectrum website to stream video programming to customers, and no set-top box is required. DVR service is not worth the effort or cost. It requires a traditional DVR set top box and you can only watch recorded shows on the television connected to the DVR. Be aware there are also restrictions viewing some channels outside of the home, just as Spectrum’s cable TV customers already understand:

Details (click the name of the package for more information):

Spectrum Choice TV

If you don’t mind Charter/Spectrum choosing your channel lineup, a second option offers more channels for about the same price.

• 7-day money back guarantee/trial, then $15 for the first month
• To get the service, you must have an internet-only plan or an internet + voice plan from Spectrum. You cannot be a current traditional cable TV subscriber
• After the first month, the service costs $25 per month for the first two years, including the Broadcast TV Surcharge, but excluding tax
• After 24 months, price increases to $30 a month
• Your assigned Spectrum TV username and password will also work on websites that authenticate you as a qualified cable TV customer
• Premium channels are $7.50 each for HBO, Showtime, The Movie Channel, Starz, and Starz Encore or bundle all-five for $15 a month for two years. Epix is also available a-la-carte.

Spectrum Stream TV

• $21.99 a month (not including $3 Broadcast TV Surcharge) for 25+ pre-selected channels including local stations and major basic cable networks
• All features included with Choice TV work similarly except the lineup is not a-la-carte. But you may get more channels at a comparable price.
• After two years, the price increases to $26.99. Starting in year three, the price rises again to $34.99.
• The same $15 promotion for five premium movie networks noted above applies, if interested.

Spectrum’s promotion of Stream TV. (1:00)