More than 36,000 Nebraska phone customers were left with blocked landline service, unable to reach 911 emergency services or many area business and government agencies after a switch failure resulted in “all circuits are busy” messages for hours on end.

Windstream’s service outage was unprecedented in Nebraska telephone history, lasting over 15 hours and impacting customers in at least 12 southeastern Nebraska counties.  Government agencies were forced to open emergency operations centers, police and fire officials asked people with emergencies to walk or drive to the nearest police station or firehouse, and some customers were left thinking the whole incident was a hacker April Fools Day attack on the telephone system.

The Nebraska Public Service Commission plans to hold hearings to investigate Windstream’s failure to maintain their network in good working order.  The Commission wants to know how the failure happened, why it took 15 hours to restore service, and what plans the company has to make sure it doesn’t happen again.

The Journal Star reported on what they’ve learned thus far about the outage:

Problems were worst in downtown Lincoln, including state and local government offices and the University of Nebraska-Lincoln. But they ultimately stretched to at least a dozen Southeast Nebraska counties.

Brad Hedrick, Windstream’s Nebraska vice president and general manager, said a number of operations and engineering technicians were doing a “root-cause analysis” Friday to try to find out what went wrong.

The outage came about 7 hours after Windstream updated the switch. But because the failure occurred so long after the update, Hedrick said, it’s unclear whether it played any role.

“We’re digging into all potential scenarios to make sure we didn’t overlook anything,” he said.

Windstream is not aware of a similar problem ever occurring before in its nationwide network, although Hedrick said he has heard of other companies having similar problems elsewhere.

Local emergency management officials said they had never experienced a service failure of similar magnitude.

“We’ve had, over the years, some minor outages, mostly geographical,” said Julie Righter, communications manager of Lancaster County’s 911 Center. “Nothing this widespread. This was multiple counties.

“It wasn’t just 911. It was every kind of phone.”

Righter said the 911 center was alerted to the problem by an alarm company employee.

“They called us and asked if we were able to call out, at which point our staff started making some test calls back and forth and we did discover we had problems.”

[flv width=”640″ height=”500″]http://www.phillipdampier.com/video/KOLN Windstream Outage Update 4-2-10.flv[/flv]

A complete wrap-up of the Windstream outage in Nebraska comes courtesy of KOLN-TV in Lincoln.  (16 minutes)

Included are interviews with three key players in the saga:

1. Commissioner Tim Schram – Public Service Commission
2. Brad Hedrick – Windstream Communications
3. Julie Righter – (Emergency) Communications Center

Windstream Communications customers in eastern Nebraska have spent much of today without access to emergency 911 services, and many were without their own landlines as well.  A switch failure in downtown Lincoln caused the outage impacting several counties starting at 7:45 Thursday morning.

Emergency services personnel were forced to rely on cell phones and amateur radio operators to process calls for emergency service, and several law enforcement personnel were staged in outage areas to assist with any calls for help.

Emergency dispatch centers were silent for much of today.  Those attempting to call 911 received a busy signal.

“You know there’s people out there that are going to need help at some point, there always is, and they’re unable to get that help,” Cass County dispatcher Deb Thiessen told KETV in Omaha.

“It’s very rare there’s an outage of this type,” said Cass County Chief Deputy Brad Lahm.

[flv]http://www.phillipdampier.com/video/KOLN-ABC Nebraska Windstream Suffers Major Outage 4-1-10.flv[/flv]

KOLN-TV and NTV report on today’s major Windstream outage.  (5 minutes)

[flv width=”640″ height=”380″]http://www.phillipdampier.com/video/KETV Omaha Windstream Failure No April Fools Joke 4-1-10.flv[/flv]

KETV-TV in Omaha also covered the outage in their viewing area. (2 minutes)

Comcast's usage meter is now available in 25 states

Comcast customers in at least 25 states have been notified that Comcast’s new usage measurement meter is now up and running.  Comcast introduced a 250 GB monthly usage limit in August 2008 after the Federal Communications Commission stopped the company from throttling usage-intensive file-trading applications.  Comcast has enforced the cap among those customers who regularly exceed it by wide margins, usually warning customers by phone or mail that they must reduce usage or face account suspension.  The usage meter application allows the company to direct customers to the self-measurement tool the company hopes will reduce the need for warnings.

Customers in Alabama, Arkansas, Connecticut, Colorado, Delaware, Florida, Georgia, Kansas, Maine, Maryland, Minnesota, Missouri, Nevada, New Hampshire, New York, Oregon, Pennsylvania, South Carolina, Tennessee, Utah, Vermont, Virginia, Washington, Washington, D.C., West Virginia, and Wisconsin should have already or will receive e-mail from the company officially notifying them about the launch of the usage meter.

Since the meter was introduced, broadband usage and pricing has increased for many customers, but the usage cap has not.  While generous by current standards, an inflexible usage limit will increasingly trap customers who use Comcast broadband service for high quality video streaming, file backups, or file trading activities which can consume considerable bandwidth.

Informally, Comcast has allowed some residential customers to purchase second accounts if they intend to blow past their usage allowance, because the company currently offers no official provisions for those who exceed the limit.

This AT&T customer was billed $12.95 in cramming charges. (Click image for more information)

AT&T and Verizon are among the top recipients of ill-gotten gains from so-called “cramming” incidents — customers who find unauthorized charges on their phone bill placed there by third party companies that maintain a cozy billing relationship with the two phone companies.

Boston-area resident Mike Cunningham paid $567 in phone charges billed “conveniently” to his Verizon Wireless phone bill.  Only he didn’t authorize them.

Cunningham recently sat down and reviewed nearly two years’ worth of dozen-page phone bills scrutinizing them for unauthorized charges.  Speaking to the Boston Globe, Cunningham didn’t initially notice the “enhanced voice mail’’ charges of $14.95 and $13.22 buried on pages five and six.  That’s not surprising, since many Verizon Wireless customers are now billed online and most customers don’t wade through multi-page online billing statements.

After Cunningham added several years of these monthly charges up, totaling $567, that got his attention.

Cunningham was billed by ILD Teleservices Inc., which said Cunningham’s grandson — then 10 years old — ordered its services while on a website that offers free video games. Cunningham said his grandson, who doesn’t have a cellphone, did not intentionally order voice mail — and knew nothing about it.

ILD generates an enormous number of complaints about unauthorized charges placed on consumer phone bills.  The company describes itself as a leading payment processor of online transactions between merchants and consumers, processing more than 120 million billing transactions per year totaling $500 million of third party charges placed on telephone bills.  Although the company claims to put its potential clients through a rigorous screening process, the avalanche of customer complaints, including the fact a 10-year old was able to be victimized by one of ILD’s clients, suggests otherwise.

Even worse, companies like AT&T and Verizon profit handsomely from fees paid by payment processors like ILD to gain the lucrative ability to charge customers’ phone bills for services, ordered or otherwise.  With a profit incentive to protect, consumers are getting the short end of the stick when calling Verizon or AT&T to complain.  More often than not they pass the buck (while keeping the change for themselves) back to the third party billing agency to try and secure refunds.

Only a staggering amount of potential earnings from such billing practices would seem enough to make risking the customer’s relationship with their phone company worthwhile.  After customers spend hours dealing with unruly and hostile customer service representatives working for such billing agencies, there is little chance that customer will be endeared to the phone company that put them through the nightmare in the first place.

Susan from Ambler, Pennsylvania is an excellent example:

“ILD Teleservices placed a charge on my Verizon phone bill for a service that was not requested, authorized or that would even work (ringtones for a land line!) on Feb 22, 2010. When I called Verizon to question the bill, they informed me the charge was not a Verizon charge but from a third party company,” she told Consumer Affairs.

David in Galt, California was billed by ILD on his AT&T phone bill for ordering a service over his home computer, an amazing feat considering he doesn’t have one.

“I received a charge of 14.95 on my AT&T phone bill from ILD Teleservices. They claimed that someone in the household went online and ordered the service. We did not have the capability to do that with no computer at the house,” he writes.

More than 3,000 complaints have been logged against ILD by Consumer Affairs, with customers highly annoyed that their phone companies refuse to stand by them when illegitimate charges show up on their phone bills. John from Wisconsin got no help from AT&T when identify theft allowed someone to add unwanted services to his phone bill under his wife’s name.

It turns out someone signed him up for TotalContactSolutions, a Florida-based company that charges $14.95 a month to alert up to 10 people with text or voice messages “during catastrophic events such as terrorist attacks or natural disasters.”  Perhaps the service will come in handy to contact those 10 friends and family members when you discover the charges on your phone bill and pass out on the floor.

John reports his credit rating is being terrorized by a company that refuses to refund the unauthorized charges unless he can prove, with an e-mail from AT&T no less, that nobody could have signed up for the service from John’s home.  Unfortunately for John, AT&T’s surveillance of its customers doesn’t extend that far, and the company refused to forgive the charges, instead threatening him with collections.

Unfortunately, your tax dollars are hard at work paying for state utility commissions, the Federal Communications Commission, and consumer service agencies to assist consumers who are victimized twice by cramming charges — once by Verizon or AT&T for allowing them on their bills in the first place, and a second time trying to deal with a third party company to reverse them.

A Boston Globe review of more than 200 cramming-related complaints from consumers — filed with the Massachusetts Department of Telecommunications and Cable and the attorney general since 2007 — found that state workers reviewing complaints sometimes spent hours trying to resolve a single one.

The case of Soren Jensen, a retired engineer who lives in Duxbury, is typical. Jensen said he spotted four charges of $9.99 apiece for text messages on his wife and son’s cellphone bills starting in 2008. There was a common thread: Jensen’s wife and son said they had both taken IQ tests online, entering their cellphone numbers to receive the results. He suspected they mistakenly signed up for text messaging in the process.

After reviewing the fine print on his bill and doing some online sleuthing, he found Verizon Wireless had an agreement with Solow, a computer gaming website that contacts players using a text message service that charges $9.99 per message.

Jensen said he called Verizon Wireless to complain, and the company agreed to remove one of the $9.99 charges.

“I just don’t get why Verizon doesn’t want to protect us, as the customer,’’ Jensen said. “Verizon should not allow this kind of stuff. It raises a lot of questions.’’

The answers aren’t difficult to find when you follow the money.  Both AT&T and Verizon collect plenty from fees charged to cramming companies and billing agencies for the right to bill their services directly on your phone bill.  Neither company will disclose exactly how much they profit from such arrangements, but they are clear about who is responsible when mystery fees turn up on your phone bill:  you are.

Both companies told the Boston Globe they “encourage customers to scrutinize their bills to make sure they are not improperly charged for services.”

And they make that very easy by labeling mystery fees with such helpful billing descriptions as “enhanced calling service” or “enhanced voicemail.”  One of Stop the Cap!‘s readers was billed for services described as an “enhanced recovery fee” and another for “customer support and assistance.”

Verizon claims complaints about third-party charges are “infrequent” and says customers can contact the company and block all third-party charges from their bills, something we strongly recommend you consider doing before being victimized.

AT&T won’t go that far.  It told the Globe:

AT&T, which also allows third-party billing, advises customers to direct their complaints to the company assessing the fee. Names of third-party vendors are disclosed on AT&T bills, the company said. “AT&T’s third-party billing contracts require service providers to address cramming complaints appropriately, including issuing credits if customers have been crammed,’’ an AT&T spokeswoman said in an e-mail.

But AT&T is still in the business of scaring its customers.  TotalContactSolutions maintains a required AT&T customer disclaimer on their website, which includes several states where AT&T can disconnect your phone line over billing disputes:

You have the right to dispute the Employee Notification Services charges billed on your local telephone bill. You are not legally responsible for Employee Notification Services charges incurred by minors or vulnerable adults without your consent. Your local telephone service will not be disconnected because you fail to pay a charge by Employee Notification Services, except that nonpayment of certain regulated telecommunications charges may result in disconnection of service in AL, FL, GA, KY, LA, SC, and TN.

And we know what phone company lobbied their way into obtaining the right to cut your service off if you don’t pay, don’t we?

In the end, Cunningham got refunds for the unauthorized fees on his Verizon Wireless bill, after the companies discovered a minor child was involved.

ILD claims to have sent Cunningham $1,000 in coupons as part of a settlement, something Cunningham claims is a lie.  Cunningham is better off without them.  The $1,000 in coupons ILD offered is suspiciously similar to an offer from another ILD client that promises that amount in grocery coupons you can print on your computer… if you sign-up for enhanced voicemail service for $12.95 a month.

Was this the $1,000 in "free coupons" offered to Mike Cunningham? If so, it comes with some very expensive strings attached.

Cunningham is so disgusted with Verizon Wireless for putting him through this ordeal, he wanted to cancel his service and move on.  But Verizon knows how to hold customers captive.  Instead of sending him a refund check for $567, they applied it as a credit that can only be redeemed by remaining a Verizon Wireless customer until the credit is exhausted.

“It’s like salt in the wound,’’ he told the Globe. “I can’t leave. I’m a captive audience.’’

Stop the Cap! notes the Federal Communications Commission is overwhelmed with cramming complaints that number well into the thousands every year.  Since telephone companies refuse to stand up for their customers, it is imperative that the FCC order phone companies to stop allowing all third-party billing unless and until a customer “opts-in” to such billing, in writing.  No third party “opt-in” requests should be permitted, and customers should not have to chase their phone companies to opt-out of a service that has a built-in profit incentive for funny business, fraud, and costly scams that cost customers enormous time and money to resolve.

[flv width=”640″ height=”500″]http://www.phillipdampier.com/video/CBS News Cramming Charges 2-23-08.flv[/flv]

The CBS Evening News ran this item about cramming charges and the problems they cause customers more than two years ago, interviewing a representative from Verizon Wireless.  Very little has changed as the money, and complaints, keep pouring in.  (3 minutes)

Verizon Communications is indefinitely finished expanding FiOS — its fiber to the home triple-play package of broadband, phone, and TV — to new cities across its service area.  In short, if your community isn’t already engaged in franchise negotiations with the telecommunications company, there is no fiber from Verizon in your immediate future.

The company said after spending $23 billion upgrading its aging copper wire phone network, it needs to finish construction and improve its reach in existing FiOS-wired communities.  When the company ceases FiOS construction, it hopes to pass 18 million customer homes across its multi-state service area.

The decision to stop expansion of advanced fiber optics threatens to leave hundreds of communities behind, too late to the party or simply too far down Verizon’s priority list.  Among important cities Verizon will pass up include Alexandria, Virginia, Boston and Baltimore.

That concerns city officials, especially in Baltimore which already considers itself on the disadvantaged list.

“My take on it is that Baltimore is not equipped for the future,” Rev. Johnny Golden, past president of the Interdenominational Ministerial Alliance and an advocate for improved access to technology in the city told the Baltimore Sun. “We have a decent broadband system for today, but it does not have the infrastructure to take us into the future where we need to go.”

Despite the fiber bypass, the company will continue negotiations with about a dozen communities where negotiations were already underway – mostly in New York, Massachusetts, and Pennsylvania.  Despite company spin, the decision to drop the shovels and wheel away the spools of fiber does represent a dramatic change of plans, evidenced by the company’s decision to bypass the aforementioned lucrative urban communities.

For cable companies like Comcast and Time Warner Cable, Verizon’s announcement brings a sigh of relief.  Both cable operators handily beat Verizon’s DSL offerings and are swiping increasing numbers of Verizon’s phone customers who are disconnecting their landline service in favor of cell phones or “digital phone” service from the cable companies.

Both Time Warner Cable and Comcast have also kept a larger percentage of their customers than Verizon hoped.

In markets where FiOS is available, Verizon has only achieved 25% penetration for television service and 28% for Internet, far below the 40 percent penetration Verizon CEO Ivan Seidenberg hoped to achieve.  The reasons consumers didn’t switch to Verizon FiOS vary, but include:

• Pricing was not always lower than what the incumbent cable operator offered, and in many cases prices for some service were higher once the promotional rate expired;
• Cable operators in competitive areas improved service, offered more aggressively priced bundles, and increased broadband speed;
• Many cable operators locked their customers into two year “price protection agreements” which hold customers in place until agreements expire (if they don’t auto-renew);
• Installation can prove disruptive because of the elaborate rewiring required in many homes;
• Consumers didn’t see enough compelling reasons to switch.

Seidenberg

Still, Verizon has future-proofed their fiber optic service areas and are better positioned to deliver extremely high broadband speed and HD offerings than their cable counterparts.  But that has never impressed short-term focused Wall Street.  Many in the financial press have attacked Verizon for the costly fiber upgrades they believe will not work for the short-term investor seeking immediate return from their Verizon stock purchase.  With the rumor mill predicting upcoming retirement for Seidenberg, his likely successors are hardly FiOS fanboys.

John Killian, Verizon’s current chief financial officer, is a short-term results man.  Samuel Greenholtz, an analyst with the Gerson Lehrman Group, doesn’t see Killian sharing much of Seidenberg’s visionary long term thinking.  Lowell McAdam, another prospect for the top job, is currently the president of the Wireless Services division, and would likely bring a wireless “solution” for broadband customers left off the FiOS list.  Neither man seems particularly interested in restarting the push for fiber in the future.

For 2010, capital expenses are flat or down across the company except in the Wireless Services division.  Verizon already declared copper phone wiring dead, and has elected to abandon its rural and suburban customers,  systematically sold off to America’s “rural phone companies” Frontier, CenturyLink, or Windstream.  Those still with Verizon but without FiOS will find the future of their landlines increasingly perilous.  Greenholtz notes Verizon has terminated another 1,200 line technicians and the company intends to spend two percent less on its copper wire network this year over last.

Greenholtz witnessed first hand what happens when a company starts to ignore its legacy network — his residential phone line quit working:

Having worked at Verizon and its predecessors for over 25 years, I expected a fix would be swift and trouble-free.  Wrong. I was offered a two-week appointment date for repair people to come out and look at the problem. I might add here that my wife does not use the cellular device that she carries around anymore than necessary and certainly never uses it when in close proximity to the hardwired set. By resorting to measures that I certainly would never have thought of using 10 years ago, I was able to get attention brought to the problem much quicker — and the issue has been resolved satisfactorily.

It seems that Verizon’s residential repair and maintenance has sunk to a new low.  Neighbors and other people on copper cabling are often experiencing problems. If there is static on the line, subscribers are frequently told nothing can be done to correct the situation because they need to replace the cable or do cable maintenance – but there is no budget available to do the work.   So, repairmen take the brunt of the public’s unhappiness with the service they are receiving. In contrast, when I spoke with some friends regarding FiOS and its maintenance issues, I found a much better response time to any difficulties the customers were experiencing.

Shockingly for a Wall Street-focused “expert network,” Greenholtz was allowed to offer his belief the only real solution to phone companies ignoring their undesirable customers is to regulate the heck out of them.

What can be done to cure the situation with residential landline services?   Unfortunately, it is going to have to come down to regulation.  Verizon, and no doubt, AT&T, has been doing what they want for many years now.  The PUCs have given them a lot of opportunities to offer advanced services that the commissions thought would spread throughout the serving areas, but they are increasingly realizing that is not in the plans.  They are going to have to force these companies to be responsive to the needs of the entire footprint, not just the Fortune 500 territories — and the nearby residential homes.

[Update 4/1/2010: While working on another story, I was amused to discover we had written about Mr. Greenholtz before, back on April 15th, when he was telling his readers “do-gooders” forced Time Warner Cable to attempt usage caps on customers.  I wonder if we would have heard something different from him had his broadband service faced an Internet Overcharging scheme.]

Blind deregulation and legislative-friendly handouts to companies like AT&T and Verizon have never resulted in better service for consumers.  They haven’t proven to save consumers any money either.  Ultimately, the decision to provide FiOS and U-verse came with investor consent, and when the economic downturn threatened the value of the stock and dividends, no deregulation or statewide franchise agreement is going to keep the fiber party from coming to a close.

[flv]http://www.phillipdampier.com/video/WSYR Syracuse Verizon FiOS Winds Down 3-26-10.flv[/flv]

WSYR-TV in Syracuse reports on the demise of Verizon FiOS’ expansion plans, which have a significant impact on central New York where many communities will be left behind.  One saving grace for New Yorkers ticked off at Albany — they’re now off the FiOS list indefinitely.  (2 minutes)