A homeless man’s mattress fire under a bridge in Massachusetts was enough to create the biggest telecommunications disaster for Verizon since the terrorist attacks of 9/11. The fire melted fiber and damaged copper cables that represented the backbone of Verizon’s landline network in the region, disrupting phone, cable, and broadband service for thousands of Verizon’s customers in northeastern Massachusetts in late August. Now that service has been restored, the damages from the outage and its ripple effects are still being calculated as questions are being raised about how the company handles its communications network.

The damage went far beyond an inconvenience for Verizon customers:

• Area businesses were ripe for plundering with Verizon-dependent alarm systems out of service;
• Cell towers went down if they were connected by Verizon’s fiber optic network;
• Local law enforcement communication systems ceased to function in areas where Verizon provided the vital link between the dispatch center and transmitting facilities miles away;
• Banks and other local businesses closed down because Verizon-based connectivity was inoperable. That left ATMs throughout the region out of service and credit card transactions often impossible to manage;
• 911 systems in several communities had to transfer emergency calls to other 911 centers miles away;
• Even the Registry of Motor Vehicles locked their doors and shut down while the outage persisted over several days.

[flv width=”640″ height=”380″]http://www.phillipdampier.com/video/NECN Verizon telecommunications outage causes problems 8-31-12.flv[/flv]
NECN News reports on Verizon’s enormous Massachusetts service outage and how it impacted the lives of affected customers. (3 minutes)

Verizon’s landline network ultimately failed its Massachusetts customers on a scale the company admits it has not seen in a long time.

“It’s the biggest problem we’ve faced, nationally, other than 9-11,” Bill Wilson, area manager for Verizon, told the Eagle-Tribune. “This is the biggest problem we’ve had in 20 years.”

At the heart of the problem is Verizon’s lack of redundancy in its landline network. With fiber optic cables managing a larger share of broadband traffic, phone calls, and even Verizon’s television service, the loss of even a single fiber cable can disrupt service for hundreds or thousands of customers, many more than would be affected by a damaged copper cable.

State Sen. Barry Finegold (D-Andover) is questioning Verizon’s decision not to have a backup plan in place.

“So if there’s a fire there’s a redundant system in place so if one fails there’s another to back it up,” Finegold said.

Customers, particular those working from home, wholeheartedly agreed.

“I am appalled,” Shela Horvitz, a Verizon FIOS customer who lost her phone, Internet and TV service for days as a result of the fire, told the newspaper. “Can you say, ‘Single Point of Failure?’”

[flv width=”640″ height=”380”]http://www.phillipdampier.com/video/Eagle Tribune Verizon Talks to Media About Outage 9-2-12.flv[/flv]

Verizon officials show reporters what the fire did to their network and how they are going to restore service after a fire on Lawrence Central Bridge caused mass outages in the region. From the Eagle-Tribune. (4 minutes)

Verizon blames the entire affair on homeless people, who they say should not have been sleeping on top of their wires. The idea of network redundancy for Verizon’s landline network? “Cost prohibitive,” say company officials.

Joseph Zukowski, vice president for government affairs for the phone company, said the problem was so rare, it was comparable to a 100-year storm. He compared the outage with a natural disaster.

“We have extensive security measures to make the network as secure as possible,” Zukowski said. “We restrict building access and access to our cables. Nowhere on the list is a homeless guy lighting a match on a mattress. We’re not laying blame, but the best thing would have been not to have the tent city there.”

Local police acknowledge the bridge where the cables cross the Merrimack River is a popular spot for the local homeless to congregate, and they have attempted to control the problem. But nobody ever told them Verizon’s vital regional communications network infrastructure was at ground zero of the mattress fire.

“Going forward, if the stuff that’s there is so important, it really needs to be secured so nobody can get access to it,” said police chief John Romero.

Verizon has promised refunds for all affected customers.

[flv width=”640″ height=”380″]http://www.phillipdampier.com/video/Verizon Lawrence MA Bridge Fire Verizon Restoration 2012 9-1-12.flv[/flv]

A Verizon-produced video illustrates how the company is repairing its damaged network. The heat from the fire on August 27th melted and fused both fiber and copper cables, and the protective casings that house the cables.

Verizon: “The work to restore service is complex, given that technicians are splicing thousands of individual copper and fiber-optic connections in a very confined area under the bridge. The conduit structure that holds the cables, which was protected by a metal cage, was destroyed and needs to be replaced. Verizon crews have been working in 24-hour shifts since the fire to restore service for customers and will continue to do so until every customer is back in service.”  (2 minutes)

Thousands of customers across northeast Massachusetts from Tewksbury to Rockport have been without Internet and phone service since early Monday after a mattress being used by a homeless man caught fire adjacent to Verizon’s copper and fiber optic cables on a bridge that crosses the Merrimack River.

”Your communications services may have been interrupted due to a fire in Lawrence that damaged a major cable providing service to your community,” Verizon’s dispatch manager Donna Powers wrote in letters sent to officials in the affected communities.

The outages are impacting communities including Gloucester, Manchester, Rockport, Essex, Littleton, Lawrence, Andover, North Andover and Tewksbury, and points in-between.

With high-capacity fiber circuits out of action, regional calls and certain Internet services were disrupted. Verizon is giving priority to restoring network operational and surveillance circuits, high-capacity fiber backbone circuits, and 911.

Verizon will not give a time frame when it expects to fully restore service, although the company indicated it is now rotating crews continuously to restore service to individual homes and businesses in the Lawrence and North Andover areas.

Lawrence Fire Chief Jack Bergeron said the problem started with a lit cigarette disposed on a vagrant’s mattress, which was on fire by the time firefighters arrived on scene. The mattress was on top of conduits that contained the copper and fiber cables, despite signs marked “danger” and “high-voltage.”

With no redundant backup facilities, a major outage can leave customers without service for days until repairs are completed.

ALEC Rock exposes the truth about how many of today’s bills are actually written and passed into law with the help of a shadowy, corporate-backed group known as the “American Legislative Exchange Council” (ALEC). Counted among its members are: AT&T, CenturyLink, Charter Communications, Comcast, FairPoint Communications, Sprint, Time Warner Cable, and Verizon. ALEC works on elected members of state legislatures to deregulate phone and cable service, eliminate consumer protection/oversight laws, ban publicly-owned broadband networks, and let phone companies walk away from providing rural phone service at will.  (2 minutes)

Some 30,000 Shaw customers, mostly in Calgary, spent much of yesterday without phone service, and many more western Canadians experienced Internet problems as a result of a small electrical fire at a Shaw Communications facility in downtown Calgary.

Described by a news report as “chaos,” the fire also brought down at least three Calgary radio stations, an area bank’s ATM network, hospital communications, government offices, and Calgary’s 311 government information service.

A transformer fire on the 13th floor at Shaw’s Calgary headquarters, combined with the sprinkler system that put the fire out, proved the old adage that water and electricity don’t mix. Calgary’s power company disconnected electrical service to the building, creating additional outages for corporate customers who use Shaw’s data center, also inside the building.

As a result of the fire and its impact, Calgary officials decided to activate the city’s municipal emergency plan.

Shaw’s Internet service problems were felt as far away as Vancouver, according to news reports. As of late this afternoon, Shaw’s website is still offline.

While phone service has largely been restored, everything is not back to normal at Shaw’s headquarters, where many workers were turned away as late as this afternoon. Several streets around the building remained closed as of this morning.

Calgary Emergency Management Agency director Bruce Burrell called the incident “a major telecommunications failure,” and noted it was felt across the province and beyond.

Shaw joined local and provincial officials to assess the incident and the fire department response, as well as reviewing its impact.

It has been a difficult week for Calgary and Alberta generally. On Monday, just as the Calgary Stampede Rodeo fired into high gear, Alberta’s electricity companies began rolling blackouts after six power generators went down.

Questions are being raised about the impact of both events.

[flv width=”616″ height=”380″]http://www.phillipdampier.com/video/CTV Shaw Service Disrupted 7-12-12.flv[/flv]

CTV Calgary covers the impact of the fire at Shaw Communications’ headquarters in Calgary, Alb. and the cleanup still ongoing.  (6 minutes)

NOAA caught this ominous derecho cloud front in La Porte, Ind on June 29. The same storm would later cut power for millions all the way to the eastern seaboard.

A series of severe thunderstorms accompanied by near-hurricane-force winds caused millions of customers in several Mid-Atlantic states to lose power and telecommunications services late Friday, and some are expected to remain without service until at least this coming weekend.

The storm, known as a “derecho,” uprooted trees, which in turn knocked down power lines and caused wind-related damage to buildings from Ohio to West Virginia, Virginia to Maryland, and even into North Carolina.

But the storm also is raising questions about the massive failures in commercial telecommunications systems that left entire 911 emergency response systems offline for days, wireless networks non-operational, cell phone systems overwhelmed, and broadband service, deemed a lower priority by emergency officials, down and offline.

Some of the biggest problems remain in and around the nation’s capital and in the states of West Virginia and Virginia, where inadequate infrastructure proved especially susceptible to the storm’s damaging winds.

D.C., Maryland, and northern Virginia

In northern Virginia, calls to 911 were met by silence over the weekend, thanks to a catastrophic failure of Verizon’s landline network. With primary lines down, Verizon’s backup 911 systems also failed, leaving millions with no access to emergency responders.

Fairfax County officials finally put the word out the best way to summon emergency help was to drive (through streets littered with debris and downed power lines) to the nearest fire or police station for assistance.

“It’s just not OK for the entire 911 system in the region to go down for the period of time that we were out, especially after an enormous emergency where people needed to make those calls the most,” Sharon Bulova, chairman of the Fairfax County Board of Supervisors, told the Associated Press.

Verizon spokesman Harry Mitchell was left flat-footed, promising an investigation into Verizon’s latest 911 failure, and called the storm as damaging as a hurricane. He urged local officials to “move forward” beyond the immediate criticism and help make progress to get service restored.

Many emergency response networks also depend on telecommunications services, including fiber cables, to reach transmission towers for radio dispatch and mobile data terminals. In northern Virginia, the city of Alexandria has been managing to handle emergency dispatch services for several counties.

With power lines down, cable and phone lines often went as well. In those cases, electric utilities have first priority to restore service, and then cable and phone companies can begin repairs of their own.

Since cable operators rely on power companies to supply electricity to their amplifiers and other equipment, Comcast and Cox, which dominate the region, are blaming most of their outages on power disruptions, and promise service will be restored when the power returns.

Verizon’s DSL and FiOS broadband networks were both disrupted by the storm, primarily because of downed lines and power losses.Even wireless networks, which some might suspect would be immune to downed lines, were also seriously affected by the storm. Cell towers connect to the provider’s network through fiber optic and T1 lines, and although backup power generators can maintain a cell tower for days in some cases, backhaul line cuts can leave cell towers useless.

In metro D.C., call completion problems were a problem during the storm and sometime after as local residents turned to cell phones to communicate. Over the weekend, customers in and around Richmond, Va., found Verizon Wireless useless for text messages because of a service disruption. As backup generators ran dry of fuel, some cell towers that survived the initial storm have been shutting down until maintenance crews arrive and refuel.

The harshest criticism has so far escaped phone and cable companies. Instead, local officials and residents remain focused on Pepco, the power utility serving the Washington area. Pepco has learned from previous storms to become a master of lowered expectations, and is promising to do its best to restore power a week or more after the storm was a memory.

West Virginia and western Virginia

The state of West Virginia, and western rural Virginia state, have illustrated what happens when deteriorating infrastructure is asked to withstand winds of up to 100mph. Frontier’s operations in West Virginia were hit especially hard. Landline networks in that state had been allowed to deteriorate for years by former owner Verizon Communications. Frontier had its hands full trying to keep up with repairs, calling in additional staff and trying to maintain landline service in some areas with the help of generators.

That job was made much harder by a rash of generator thefts that impacted the phone company, and local authorities are still looking for those responsible. At least one-third of all central switching offices operated by Frontier in West Virginia remain on generator power as of yesterday. As of July 3, the company reported it has 12,000 repair requests still waiting for action.

It was a similar story in the western half of Virginia where independent phone companies and Verizon were faced with an enormous number of downed trees and power lines, many in rural areas. More than 108,000 Virginia residents are still without power as of this afternoon, and many will not see it restored until the weekend.

Because the derecho swept across a large area encompassing the entire state, it has been difficult for utility crews to respond from unaffected areas to assist in repairs because the damage was so widespread. Logistically, just coordinating repair operations has proved difficult because cell service has been spotty (or networks have been jammed with calls) in some of the worst-affected areas.

“Derechos are nothing to fool with, but still this was not the most serious storm Virginia has ever dealt with, and the impacts on our telecommunications networks seem to indicate they’ve been allowed to fall apart over the last several years,” shares Stop the Cap! reader Edward Klein, who lives near Roanoke. “I think an investigation is needed to make sure utilities are spending enough money to keep these networks in good shape so this kind of thing doesn’t happen everytime a storm sweeps through.”