Verizon’s flooded headquarters on West St., lower Manhattan (The Wall Street Journal)

As cleanup efforts continue across New York, New Jersey, and Connecticut, some of America’s largest telecommunications companies are coming under increased scrutiny for being caught flat-footed after Hurricane Sandy roared across the tri-state region, causing damage Verizon’s chief technology officer now admits is worse than 9/11.

As of this morning, Verizon Wireless’ network is reportedly straining, particularly in Manhattan and Brooklyn, where cell service that worked immediately after the storm is now increasingly failing.

Verizon said 94% of its cell sites were operational after the storm, but some local officials in the area believe 94% of Verizon’s wireless network has now failed them when they need it the most.

Many telecom companies, particularly AT&T, are being criticized for excessive secrecy about the ongoing state of their networks post-Sandy. AT&T, which left its customers in the dark about service restoration as late as last night while asking customers to contribute $10 to the American Red Cross, finally mass e-mailed customers a statement devoid of much detail signed by Steve Hodges, president of AT&T’s northeast region.

“Restoring our wireless network is our top priority,” Hodges writes. “The vast majority of our cell sites in the Northeast are online and working. We are working issues in areas that were especially hard-hit, where flooding, power loss, transportation and debris all pose challenges. Our crews are working around the clock to restore network service to areas that were impacted by the storm. We will not stop until we repair all of the damage to our network and restore service back to its full capacity.”

The Federal Communications Commission correctly predicted the situation with mobile phones could get worse before it gets better, as backup power wears down and flooding persists. At a press conference held yesterday, FCC chairman Julius Genachowski revealed at least a quarter of all cell sites in areas damaged by Sandy were not operational. Those numbers were less optimistic that those provided by carriers.

The FCC this week activated the Disaster Information Reporting System, a central reporting point for telecommunications companies to update the agency regarding outages and other service disruptions. The FCC also alerted providers that in emergency circumstances, they can assist companies getting fuel for generators and help locate portable cell tower equipment for companies caught unaware.

AT&T’s belated letter to customers affected by Hurricane Sandy

Some may need the help.

New York State Assemblyman Alec Brook-Krasny and Brooklyn Borough President Marty Markowitz both reported Verizon Wireless’ outages are worsening in Brooklyn and midtown Manhattan.

Brooklyn Borough president Marty Markowitz

The Federal Emergency Management Agency (FEMA) today told Sen. Chuck Schumer the federal agency will reimburse New York for 100 percent of the costs incurred restoring power across the storm areas. But that may not expedite how quickly power returns.

Power restoration is expected to bring most cell towers back online. Worsening service is being attributed to battery backup or generator equipment exhausting on-hand fuel supplies, which usually keeps service up and running for up to three days. That means cell towers without power and unreachable by workers will have begun failing late Wednesday into today.

Damage assessments are further behind in New Jersey, the state that took the worst impact from Hurricane Sandy.

Stop the Cap! obtained some new figures from cell phone companies regarding the state of their networks:

• Verizon: Still holding to 94% operational in storm areas;
• AT&T: Declined to comment except to say “the vast majority” of their network is operational;
• T-Mobile: 80% operational in NYC, 90% operational in Washington, D.C.
• Sprint: 75% operational

[flv width=”384″ height=”228″]http://www.phillipdampier.com/video/WSJ Verizon Offices Damaged 11-1-12.mp4[/flv]

Verizon’s critical network takes another hit. “We’ve been here before,” says one Verizon executive, referring to the destruction from the 9/11 terrorist attacks which severely damaged the same facility on West Street now flooded out. (3 minutes)

Our readers report that cell service becomes spotty to non-existent in coastal New Jersey and Connecticut. In Manhattan anywhere south of 29th Street, readers report almost no signals at all.

Verizon’s damaged facilities include those on West and Broad Streets in Manhattan (circled).

Residents are trading tips about “magic spots” where cell service does suddenly pop up, and Gizmodo notes the only place in Alphabet City (the east side in southern Manhattan) to get service is on literally one street corner, where crowds congregate to make and receive calls.

The other salve for telecom withdrawal is the nearest pay phone.

Amusing stories of 20-somethings waiting in long lines only to be confounded by unfamiliar pay phones are appearing in the New York media. One radio station even aired basic instructions for members of the Millennial Generation that have never heard of inserting coins into telephones.

The biggest challenge for the city’s pay phone vendors is clearing them of coin overloads, something unheard of before the storm.

The often maligned pay phone has exposed the limits of the “more advanced” and expensive networks that were supposed to replace them. Despite claims of superiority for wireless service, northeast residents have once again discovered it has its limits:

• They don’t work during major weather events that knock out power and limit access to maintain backup generators;
• Cell networks are less capable of handling large call volumes, a problem made worse when cell phone refugees in other areas seek out remaining cell signals, further congesting the network;
• Wireless is just as susceptible to wireline or fiber failures on the ground. Cell towers typically connect to providers through wired backhaul circuits, which knock out cell service if they fail;
• Cell phone users need power to recharge their power-hungry smartphones. Batteries drain even faster searching for a weak or non-existent cell signal;

Hardest hit remains Verizon, which allowed reporters access inside damaged facilities to help New Yorkers better understand the scope of the problem.

[flv width=”384″ height=”228″]http://www.phillipdampier.com/video/WSJ Wireless Network Outages 11-1-12.mp4[/flv]

The Wall Street Journal takes a look at the state of the wireless communications networks across the northeastern U.S. and when service will be back.  (4 minutes)

Eleven years after the 9/11 terrorist attacks that took out Verizon’s West Street office when buildings collapsed at the nearby World Trade Center, Verizon is likely going to have to re-learn some lessons about catastrophe management as flood waters recede.

Verizon has deployed this 53-foot Emergency Mobile Communications Center for use by the Nassau County Office of Emergency Management that provides Internet and phone service.

The Wall Street Journal was able to obtain access inside the damaged facilities, and the reporter covering the event was left somewhat stunned by the scope of the damage.

In the middle of organized, yet chaotic recovery efforts was Verizon’s chief technology officer Tony Melone who had seen enough to declare the damage worse than 9/11.

The pictures of several feet of muddy water from the nearby Hudson River covering the lobby of the company’s headquarters on West Street said it all. The mostly salt water was an unwelcome guest in Verizon’s building, especially considering the five level basement below the lobby contains critical cables and telecommunications equipment. Almost four of those basement floors were completely flooded. After the water was pumped out, dampness and leaves from nearby trees remain littered on the floor.

One lesson learned after 9/11 was not to place critical phone switches below ground level. After reconstruction, the switches were moved to a higher floor and consequently were left undamaged. But while Verizon moved its backup generators upstairs, it left the pumps and fuel tanks that power them in the basement — leaving them inoperable.

This morning, passersby on West Street have to step around Verizon’s network of generators now running outside of the building, right next to large temporary fuel tanks to power them.

Verizon central offices in other parts of Manhattan, particularly further southeast on Broad Street, were never upgraded and are in worse shape, with electrical equipment damaged perhaps beyond repair. The force of the water was strong enough to bend the 86 year-old steel and bronze doors. Workers there are still trying to get water out of the building, shoving a pipe down an elevator shaft to facilitate pumping.

Verizon has some redundancy built into its network to protect its most valuable customers. That kept the landline phones working at the New York Stock Exchange, even though other landline and wireless customers will have  to wait longer for service to resume.

AT&T’s generator staging area near Meriden, Connecticut. (Credit: Brian Pernicone)

Some critics of the increasingly concentrated telecommunications landscape think Verizon and other companies have still not learned enough to prevent the kinds of service disruptions that will leave some customers without service for weeks.

It is hard to miss the bustle outside of Verizon’s offices damaged by the storm, watching flood water drain down the street. But things are murkier at cell phone providers who have been less than forthcoming about specific outage information and service restoration assessments.

Some have advocated the federal government step in and require cell phone service, now deemed essential by an increasing number of Americans, be protected with robust backup solutions to keep service up and running after catastrophic weather events.

After Hurricane Katrina, the FCC in 2007 tried to issue new rules that required a minimum of eight hours of backup power for all cell sites. The industry balked, predicting it would lead to “staggering and irreparable harm” for the cell companies. One wireless trade association warned their members might take several cell sites down if they were forced to provide backup power.

The CTIA Wireless Association and Sprint-Nextel sued the agency in federal court and the Bush Administration’s Office of Management and Budget eventually killed the proposed regulations.

T-Mobile and AT&T have cut an emergency deal to share their cellphone networks in areas affected by Superstorm Sandy. They’re trying to make it a little easier for customers to get a signal as carriers restore their networks. Some say companies should be forced to make their networks more resilient. National Public Radio’s Morning Edition has the story. (November 1, 2012) (3 minutes)
You must remain on this page to hear the clip, or you can download the clip and listen later.

T-Mobile and AT&T customers in the impact zone of Hurricane Sandy will share cellular networks through a mutual aid pact announced today by both companies.

The temporary roaming agreement will allow customers of AT&T and T-Mobile to use cell sites providing the strongest reception and the highest call completion rate in areas of New York and New Jersey where cell service has degraded.

T-Mobile said the temporary arrangement will provide a seamless roaming experience because both companies share GSM technology. There will be no additional charges for this service. We have no information as to whether this agreement will also cover texting and data services.

Earlier today, T-Mobile provided this update to customers about restoration progress:

T-Mobile network engineers are working as quickly as possible to restore service to areas affected by Hurricane Sandy. In Washington, D.C. the network is more than 90% operational. In New York City, the network is more than 80% operational. Restoration work continues in the harder hit areas of lower Manhattan, Staten Island, Long Island, coastal and Northern New Jersey, Connecticut and portions of Pennsylvania, including Philadelphia.

T-Mobile rapid response engineering teams have staged equipment throughout the areas most severely impacted and continue to make assessments regarding how quickly we may be able to begin restoration, and where it is needed most.

The darker the color, the higher percentage of CP&L customers without electric service. Areas in gray are served by other electric utilities.

Although less hard-hit than New Jersey, Connecticut residents will wait almost as long as customers further south for restoration of electric service, with AT&T getting its own services back up and running only after electric utility repair crews finish work.

Customers across the state are experiencing power outages that range from a handful of homes to near-complete blackouts that utility companies predict will take at least a week to repair. For a second day, AT&T continued to leave its customers in the dark, with nothing more than a general statement it continued to “assess the damage” to its wired and wireless networks that “have issues” across Connecticut.

In fact, AT&T’s most visible effort for Connecticut customers waiting for service is a website promotion asking for $10 donations for the American Red Cross, conveniently billed to your AT&T account.

“Why did I expect anything more from AT&T when they never seem to be terribly interested in customer service generally,” complains Stop the Cap! reader Bethany Johnson, also a U-verse customer e-mailing us from a friend’s phone on Verizon Wireless. “AT&T customer service won’t say anything to us and you can’t find a thing on their website with the same old statements on the news.”

Johnson says Connecticut Light & Power (CL&P) restored her electric service early this morning, but U-verse is out and her AT&T cell phone no longer has any signal from her home.

“When you call AT&T, one of their call centers answers and they just read out some statement that tells you nothing,” she says. “Verizon Wireless and Cablevision are falling all over each other trying to give us updates, but AT&T can’t be bothered.”

Johnson says her friends with Verizon Wireless seem to have weathered Hurricane Sandy better than she did, with much more sporadic AT&T cell service afflicting customers across Connecticut.

“My husband drives for a living and he says AT&T’s cell network as of today along the roads he travels really took a beating and he often can’t get in touch with me,” Johnson said. “AT&T says they have ‘issues’ in Connecticut and I am getting to the point where I am having an issue with them. Just tell us what is going on, we can take it.”

United Illuminating is reporting 137,983 customers without power; more than 43 percent of its service area. Nearly all of UI’s customers in the Southport section of Fairfield have no electric service, followed by Weston at 90 percent, Trumbull with nearly 85 percent, Monroe at nearly 82 percent and Bridgeport at 71 percent.

CL&P is reporting 351,910 or about 28 percent of its customers without power. In southwestern Connecticut, more than half of CL&P’s customers are in the dark in Darien, Greenwich, New Canaan, Westport, Wilton, Weston, Newtown, Monroe and Danbury.

Hurricane Sandy’s destructive forces of wind and water, combined with extensive electrical outages has wreaked havoc with telecommunications services from Maine to Virginia, leaving some customers potentially without service for weeks.

The storm has flooded Verizon‘s central switching offices in New York City, did extensive damage to Sprint’s wireless network and infrastructure, has left large sections of upstate and downstate New York without cable service, and clocks ticking for wireless cell customers using cell sites currently running on battery backup power.

Some of the worst problems are affecting Verizon’s landline and FiOS networks after the company lost two critical switching centers in Manhattan to extensive flooding. That has contributed to significant problems for Verizon customers across Manhattan, Queens, and Long Island. Further afield, Verizon customers without service can blame power outages and fallen trees that took out overhead wiring. Together, Verizon customers are experiencing significant problems with landline, broadband, and FiOS TV and Internet services in some areas.

Many Verizon Wireless cell sites are operating on battery backup units which maintain service for only a limited time. New York, New Jersey and Connecticut customers report increasing difficulty maintaining cell service signals as those battery backup units start to fail. Verizon engineering crews can restore undamaged cell sites with backup generators once permitted into storm-ravaged areas.

One of the hardest hit wireless carriers

Cablevision‘s business largely depends on areas that took a direct hit from Hurricane Sandy. Cablevision repair crews are encountering extensive power outages and damaged overhead wiring brought down during the storm in Connecticut and Long Island. Its service area closer to New York City has been primarily affected by power outages. Comcast said it was still starting an assessment process and was not prepared to report on the current state of its network, which operates in cities north and south of the New York City metro area.

While Time Warner Cable spokesman Alex Dudley reports little damage to Time Warner Cable’s systems, many remain offline from power interruptions, and Time Warner’s Twitter feed for upstate New York reports isolated outages in Portland, Maine and across upstate New York, primarily due to power losses or damage to infrastructure.

Sprint appears to be the hardest hit wireless carrier with widespread service outages, interruptions and call completion issues throughout the states of New York, New Jersey, Connecticut, Pennsylvania, Washington DC, Maryland, North Virginia and New England. Some customers far away from the worst-hit areas report trouble making and receiving calls on Sprint’s network. Many cell sites are also damaged.

AT&T is assessing damage to its landline operations in Connecticut, where it is the dominant phone company. Many AT&T cell phone sites, like Verizon, are operating on battery backup in power outage areas until AT&T can bring generators online to maintain service.

T-Mobile and MetroPCS report damage and service outages to their cellular networks as well, mostly from power outages.

Lyndhurst, NJ

Even old style communications networks were not spared from Hurricane Sandy. The Northeast Radio Watch reports a large number of broadcasters across the region off the air as of this morning:

• Outside of WOR (710), most New York City area AM stations are off the air. WOR survived the storm with its recently built three tower site located just above the flood waters. Chief engineer Tom Ray told NERW the water is 10 feet deep at WOR’s transmitter site in the Meadowlands. Many AM stations in New York favor transmitter locations in now-ravaged Lyndhurst and the Meadowlands. The result: indefinite absence of all-news WINS (1010) (it’s now back up — thanks to an update from Scott Fybush), which is now being heard on WXRK (92.3). Also missing: WLIB (1190), WSNR (620), WMCA (570), WNYC (820), WPAT (930), WNYM (970), WADO (1280) and WWRV (1330). FM outlets favor much higher transmitter locations, usually atop large skyscrapers, that escaped flood damage.
• WABC continues to air the audio portion of its broadcast on WEPN-AM (1050) and FM (98.7) for the benefit of those without power. WCBS studios are currently powered “by candlelight.”
• The Jersey shore’s FM outlets are mostly silent. Atlantic City was among the hardest hit, and some stations may be off the air for some time while rebuilding.
• Connecticut stations are also off the air. Powerhouse WICC (600) in Bridgeport has transmitters on Long Island Sound — a poor choice to withstand Sandy. It is likely underwater. Also gone: WGCH (1490 Greenwich), WAXB (850 Ridgefield) and WSHU (1260 Westport) and WALK-FM (97.5 Patchogue).

Repair crews for all concerned will likely only start assessing damage later today, but many will have to wait for power crews to complete work — they have first priority. Those lucky enough to see service restoration once power returns will be in far better shape than others who could wait weeks to get their Internet, television and phone service back.

Correction: Original story included reference to studio power knocked out at WOR-TV. That should have said WOR-AM (radio). 

NOAA caught this ominous derecho cloud front in La Porte, Ind on June 29. The same storm would later cut power for millions all the way to the eastern seaboard.

A series of severe thunderstorms accompanied by near-hurricane-force winds caused millions of customers in several Mid-Atlantic states to lose power and telecommunications services late Friday, and some are expected to remain without service until at least this coming weekend.

The storm, known as a “derecho,” uprooted trees, which in turn knocked down power lines and caused wind-related damage to buildings from Ohio to West Virginia, Virginia to Maryland, and even into North Carolina.

But the storm also is raising questions about the massive failures in commercial telecommunications systems that left entire 911 emergency response systems offline for days, wireless networks non-operational, cell phone systems overwhelmed, and broadband service, deemed a lower priority by emergency officials, down and offline.

Some of the biggest problems remain in and around the nation’s capital and in the states of West Virginia and Virginia, where inadequate infrastructure proved especially susceptible to the storm’s damaging winds.

D.C., Maryland, and northern Virginia

In northern Virginia, calls to 911 were met by silence over the weekend, thanks to a catastrophic failure of Verizon’s landline network. With primary lines down, Verizon’s backup 911 systems also failed, leaving millions with no access to emergency responders.

Fairfax County officials finally put the word out the best way to summon emergency help was to drive (through streets littered with debris and downed power lines) to the nearest fire or police station for assistance.

“It’s just not OK for the entire 911 system in the region to go down for the period of time that we were out, especially after an enormous emergency where people needed to make those calls the most,” Sharon Bulova, chairman of the Fairfax County Board of Supervisors, told the Associated Press.

Verizon spokesman Harry Mitchell was left flat-footed, promising an investigation into Verizon’s latest 911 failure, and called the storm as damaging as a hurricane. He urged local officials to “move forward” beyond the immediate criticism and help make progress to get service restored.

Many emergency response networks also depend on telecommunications services, including fiber cables, to reach transmission towers for radio dispatch and mobile data terminals. In northern Virginia, the city of Alexandria has been managing to handle emergency dispatch services for several counties.

With power lines down, cable and phone lines often went as well. In those cases, electric utilities have first priority to restore service, and then cable and phone companies can begin repairs of their own.

Since cable operators rely on power companies to supply electricity to their amplifiers and other equipment, Comcast and Cox, which dominate the region, are blaming most of their outages on power disruptions, and promise service will be restored when the power returns.

Verizon’s DSL and FiOS broadband networks were both disrupted by the storm, primarily because of downed lines and power losses.Even wireless networks, which some might suspect would be immune to downed lines, were also seriously affected by the storm. Cell towers connect to the provider’s network through fiber optic and T1 lines, and although backup power generators can maintain a cell tower for days in some cases, backhaul line cuts can leave cell towers useless.

In metro D.C., call completion problems were a problem during the storm and sometime after as local residents turned to cell phones to communicate. Over the weekend, customers in and around Richmond, Va., found Verizon Wireless useless for text messages because of a service disruption. As backup generators ran dry of fuel, some cell towers that survived the initial storm have been shutting down until maintenance crews arrive and refuel.

The harshest criticism has so far escaped phone and cable companies. Instead, local officials and residents remain focused on Pepco, the power utility serving the Washington area. Pepco has learned from previous storms to become a master of lowered expectations, and is promising to do its best to restore power a week or more after the storm was a memory.

West Virginia and western Virginia

The state of West Virginia, and western rural Virginia state, have illustrated what happens when deteriorating infrastructure is asked to withstand winds of up to 100mph. Frontier’s operations in West Virginia were hit especially hard. Landline networks in that state had been allowed to deteriorate for years by former owner Verizon Communications. Frontier had its hands full trying to keep up with repairs, calling in additional staff and trying to maintain landline service in some areas with the help of generators.

That job was made much harder by a rash of generator thefts that impacted the phone company, and local authorities are still looking for those responsible. At least one-third of all central switching offices operated by Frontier in West Virginia remain on generator power as of yesterday. As of July 3, the company reported it has 12,000 repair requests still waiting for action.

It was a similar story in the western half of Virginia where independent phone companies and Verizon were faced with an enormous number of downed trees and power lines, many in rural areas. More than 108,000 Virginia residents are still without power as of this afternoon, and many will not see it restored until the weekend.

Because the derecho swept across a large area encompassing the entire state, it has been difficult for utility crews to respond from unaffected areas to assist in repairs because the damage was so widespread. Logistically, just coordinating repair operations has proved difficult because cell service has been spotty (or networks have been jammed with calls) in some of the worst-affected areas.

“Derechos are nothing to fool with, but still this was not the most serious storm Virginia has ever dealt with, and the impacts on our telecommunications networks seem to indicate they’ve been allowed to fall apart over the last several years,” shares Stop the Cap! reader Edward Klein, who lives near Roanoke. “I think an investigation is needed to make sure utilities are spending enough money to keep these networks in good shape so this kind of thing doesn’t happen everytime a storm sweeps through.”