More than 36,000 Nebraska phone customers were left with blocked landline service, unable to reach 911 emergency services or many area business and government agencies after a switch failure resulted in “all circuits are busy” messages for hours on end.

Windstream’s service outage was unprecedented in Nebraska telephone history, lasting over 15 hours and impacting customers in at least 12 southeastern Nebraska counties.  Government agencies were forced to open emergency operations centers, police and fire officials asked people with emergencies to walk or drive to the nearest police station or firehouse, and some customers were left thinking the whole incident was a hacker April Fools Day attack on the telephone system.

The Nebraska Public Service Commission plans to hold hearings to investigate Windstream’s failure to maintain their network in good working order.  The Commission wants to know how the failure happened, why it took 15 hours to restore service, and what plans the company has to make sure it doesn’t happen again.

The Journal Star reported on what they’ve learned thus far about the outage:

Problems were worst in downtown Lincoln, including state and local government offices and the University of Nebraska-Lincoln. But they ultimately stretched to at least a dozen Southeast Nebraska counties.

Brad Hedrick, Windstream’s Nebraska vice president and general manager, said a number of operations and engineering technicians were doing a “root-cause analysis” Friday to try to find out what went wrong.

The outage came about 7 hours after Windstream updated the switch. But because the failure occurred so long after the update, Hedrick said, it’s unclear whether it played any role.

“We’re digging into all potential scenarios to make sure we didn’t overlook anything,” he said.

Windstream is not aware of a similar problem ever occurring before in its nationwide network, although Hedrick said he has heard of other companies having similar problems elsewhere.

Local emergency management officials said they had never experienced a service failure of similar magnitude.

“We’ve had, over the years, some minor outages, mostly geographical,” said Julie Righter, communications manager of Lancaster County’s 911 Center. “Nothing this widespread. This was multiple counties.

“It wasn’t just 911. It was every kind of phone.”

Righter said the 911 center was alerted to the problem by an alarm company employee.

“They called us and asked if we were able to call out, at which point our staff started making some test calls back and forth and we did discover we had problems.”

[flv width=”640″ height=”500″]http://www.phillipdampier.com/video/KOLN Windstream Outage Update 4-2-10.flv[/flv]

A complete wrap-up of the Windstream outage in Nebraska comes courtesy of KOLN-TV in Lincoln.  (16 minutes)

Included are interviews with three key players in the saga:

1. Commissioner Tim Schram – Public Service Commission
2. Brad Hedrick – Windstream Communications
3. Julie Righter – (Emergency) Communications Center