A traditional metrocell, designed to be installed on a utility pole or side of building.

AT&T’s wireless network expansion plans include more than 10,000 new HSPA+/LTE cell sites, 40,000 small “metrocells,” and 1,000 distributed antenna systems (DAS) that will improve network performance, broaden Wi-Fi service, and reduce traffic on its traditional cell tower network.

With much of urban and suburban America (and the roads that connect communities) already covered by cellular networks, AT&T has embarked on an effort to more efficiently manage its wireless traffic.

AT&T, the lowest-rated wireless carrier by Consumer Reports, has suffered from a reputation for dropped calls and inadequate network infrastructure investment. The company has sought to correct those mistakes with the implementation of its multi-billion dollar Project Velocity IP (VIP) program that will expand capacity and bring Wi-Fi to new places.

John Donovan, senior executive vice president of AT&T’s Technology and Network Operations division told attendees at the Citi Global Internet, Media & Communications conference in Las Vegas the company was shifting investment towards deploying small cell technology like “metrocells” that provides service to 32 or 64 concurrent users in a small geographic area. These fiber-fed, low-power small cells traditionally cover areas less than 1.2 miles wide, and can be hidden on utility poles or on buildings.

AT&T intends to leverage its U-verse fiber to the neighborhood network to provide much of the expanded network’s backhaul connectivity, at least in cities where AT&T provides landline service.

With an in-house fiber network, AT&T can more cheaply deploy expanded Wi-Fi that will help the company offload cellular data traffic. AT&T says customers will benefit because Wi-Fi use currently does not count against a customer’s monthly data usage allowance. With Wi-Fi accompanying new metrocell and DAS installations, AT&T customers will eventually see a much larger area of Wi-Fi service on their wireless devices, especially in urban areas.

AT&T’s fall announcement of a renewed push for U-verse compliments plans to expand its wireless network. In cities where AT&T is not the landline provider, the company often contracts with other telecom companies to handle traffic to and from cell sites.

Donovan noted a crucial key to the plan’s success is to demand a more seamless transition to and from Wi-Fi from device manufacturers, automatically switching customers off the cellular network in favor of Wi-Fi, where available. At present, customers make the choice. In the future, the device itself could ultimately become the final arbiter, choosing the strongest, most reliable wireless technology available automatically.

The company has not given up on traditional cell tower networks.

AT&T intends to expand its HSPA+ footprint to 300 million homes by the end of 2014. It reaches around 288 million homes at present, with LTE service available to around 170 million. The company intends to provide both its slower HSPA+ and faster LTE 4G service.

[flv width=”640″ height=”380″]http://www.phillipdampier.com/video/Metrocells.flv[/flv]

Alcatel-Lucent is a supplier of metrocell technology and produced this video explaining why offloading network traffic was important, particularly in large congested cities and at major event venues.  (2 minutes)

Phillip “Do you want to depend on AT&T for phone service that could be gone with the wind for weeks?” Dampier

Superstorm Sandy is getting credit for exposing the thin veneer of the “wireless future” some phone companies want to give their most rural customers after disconnecting their home phone lines in favor of wireless service.

Unfortunately for the providers selling you on the wireless revolution, reality intruded last month when Category 1 Hurricane Sandy arrived. In its wake, the storm obliterated a significant amount of wireless phone service for weeks in some of the most urbanized sections of the country, while leaving underground, traditional wired phone service largely untouched.

The storm that blew into the northeastern U.S. Oct. 29 left a legacy of interrupted or inadequate cell service that lasted more than two weeks. AT&T and Verizon Wireless reported their networks were not fully restored until Nov. 15. Sprint and T-Mobile are still addressing some issues with their networks as of today.

Although the storm was enormous in scope, it was only a Category 1 hurricane. It could have been much worse.

So where did things go wrong?

Although some sites lost their wired backhaul connection which connects the tower to the provider, the biggest problem was commercial power interruption. Without power, many providers were caught flat-footed with inadequate on-site backup plans to keep cell towers up and running until regular power could be restored.

The wireless industry fought tooth and nail against common sense regulations proposed by the Federal Communications Commission after Hurricane Katrina devastated infrastructure and power facilities in southern Louisiana and Mississippi.

The FCC proposed that every cell tower be equipped with on site battery backup equipment that could sustain service for a minimum of eight hours — sufficient time for power to be restored or company engineers to arrive with more robust generators.

Providers howled about the cost of outfitting the nation’s 200,000 cell sites with even a conservative amount of backup power. The cellular industry lobbying group and Sprint sued, calling it a wasteful and unnecessary mandate. The Bush Administration eventually dropped the whole matter in November 2008 as part of its war on “burdensome” regulation.

Since then, providers have been free to design their own emergency backup plans, or have none at all. Few have made those detailed plans public, giving customers information about how likely their cell phone will work in the event of a disaster.

Verizon Wireless has been the most aggressive, voluntarily adopting the proposed FCC standards and outfitting all of their cell sites with a minimum of eight hours of battery backup power. Other providers have backup facilities at some sites, often with lower capacity batteries that won’t last as long.

Sandy illustrated that even eight hours might be inadequate. Many cell sites were on generator power for more than a week, assuming engineers could regularly reach each tower with equipment and fuel.

Other cell sites could not be returned to service immediately because of major wind damage or flooding. Those that were in service were often overburdened by enormous call volumes.

Meanwhile, unless your landline provider’s central office was flooded, your phone line kept working during and after the storm, especially if your neighborhood wiring is buried underground.

In many cases, it was the only thing working, because traditional phone lines are independently powered and not dependent on electric service in your home to operate. That is what kept your dial tone humming even as your smartphone’s battery ran out.

Ironically, the network that performed the best through the storm is the same one AT&T and Verizon would like to phase out, starting in rural areas. AT&T wants to completely abandon wired service in its most rural service areas, where calling and waiting for emergency assistance is already a hindrance. AT&T plans to spend billions to bolster its rural cell tower network to cover the landline areas it wants to abandon, but those communities would be entirely dependent on the reliability of that network, because AT&T’s competitors are unlikely to build additional infrastructure to compete.

As Sandy just demonstrated, if high-profit Manhattan customers could not be assured of reliable cell phone service from any company that provide service there, how likely is it that a customer in rural Kansas will be in real trouble summoning help over AT&T’s wireless infrastructure in the event of a cell tower failure, wiping out the only telecommunications service available in nearby towns?

AT&T’s mobile charging van

AT&T’s effort to aid powerless areas of Brooklyn with mobile charging stations to help customers recharge dead cell phones fell flat when the company sent the trucks without the equipment needed to charge phones.

Timothy Stenovec reported from the Red Hook neighborhood:

In Coffey Park, just steps from where the National Guard was helping distribute food and water to residents, a large AT&T truck sat, two orange generators resting silently on the sidewalk next to it.

Despite the company’s intention for the vehicle to serve as a mobile power station, the truck was waiting on equipment necessary to charge phones, and had been turning people away all day.

Marie Reveron, who is 57 and has been without power since the storm, said she waited at the truck for more than two hours on Friday morning, expecting the equipment to arrive so she could charge her phone.

“Phone service is the most important thing, and now my phone is on its last, dying bar,” she told The Huffington Post. “Sometimes you have all the bars, and the phone won’t even work.”

[flv width=”640″ height=”380″]http://www.phillipdampier.com/video/ATT Charging Stations.flv[/flv]

AT&T is allowing the general public into area AT&T stores and portable charging centers to recharge their wireless equipment, at least when the equipment needed to do that shows up. (2 minutes)

As of Monday, Nov. 5, charging stations are available at the following locations. The stations are open to the general public.

Brooklyn:

• Red Hook East and West – Coffey Park at Richards Street
• Corner of Brighton Beach Avenue and Coney Island Avenue
• Surf Avenue Playground – West 25th Street and Surf Avenue

Manhattan:

• Fulton Street Houses – 419 West 17th Street between 9th and 10th Avenues
• Hamilton Fish – Pitt Street and East Houston Street

Queens:

• Hammel Playground – Beach 84th Street and Rockaway Beach Boulevard
• Conch Playground – Beach 44th Street and Rockaway Beach Boulevard
• Mott Avenue at Beach Channel Drive
• St. Francis de Sales Parish – 126-16 Rockaway Beach Boulevard at Beach 129th Boulevard

Staten Island:

• Midland Beach – Hunter Avenue and Father Capadanno Boulevard
• Parking Lot – Mill Road and New Dorp Lane

Other New York Locations:

• Floral Park (store) – 181 Jericho Turnpike

New Jersey Locations:

• Edgewater Square (store) – 75 River Road
• Watchung (store) – 1592 Route 22 East
• Point Pleasant Beach (The Wireless Experience – Authorized Retailer) – 3122 Route 88 and Highland Drive

The Star-Ledger reports things in New Jersey may be worse.

AT&T brought in hundreds of generators to power cell towers, according to company spokeswoman Ellen Webner, but she said keeping them topped off with fuel has been a challenge. Webner told the newspaper the company will talk to customers who want their bill adjusted for outage time.

[flv width=”640″ height=”380″]http://www.phillipdampier.com/video/ATT Generators.flv[/flv]

AT&T carefully tracks its generators now being deployed to cell sites still without power. But some critics wonder why generators are not on site before disaster strikes. (2 minutes)

[flv width=”640″ height=”380″]http://www.phillipdampier.com/video/ATT Portable Microwave Cell Tower.flv[/flv]

AT&T cannot easily bring back cell sites that lack backhaul connections to Verizon’s central offices, some still non-operational due to severe flood damage. AT&T shows off emergency equipment that can establish a temporary microwave backhaul link and restore cell service. (2 minutes)

Verizon’s flooded headquarters on West St., lower Manhattan (The Wall Street Journal)

As cleanup efforts continue across New York, New Jersey, and Connecticut, some of America’s largest telecommunications companies are coming under increased scrutiny for being caught flat-footed after Hurricane Sandy roared across the tri-state region, causing damage Verizon’s chief technology officer now admits is worse than 9/11.

As of this morning, Verizon Wireless’ network is reportedly straining, particularly in Manhattan and Brooklyn, where cell service that worked immediately after the storm is now increasingly failing.

Verizon said 94% of its cell sites were operational after the storm, but some local officials in the area believe 94% of Verizon’s wireless network has now failed them when they need it the most.

Many telecom companies, particularly AT&T, are being criticized for excessive secrecy about the ongoing state of their networks post-Sandy. AT&T, which left its customers in the dark about service restoration as late as last night while asking customers to contribute $10 to the American Red Cross, finally mass e-mailed customers a statement devoid of much detail signed by Steve Hodges, president of AT&T’s northeast region.

“Restoring our wireless network is our top priority,” Hodges writes. “The vast majority of our cell sites in the Northeast are online and working. We are working issues in areas that were especially hard-hit, where flooding, power loss, transportation and debris all pose challenges. Our crews are working around the clock to restore network service to areas that were impacted by the storm. We will not stop until we repair all of the damage to our network and restore service back to its full capacity.”

The Federal Communications Commission correctly predicted the situation with mobile phones could get worse before it gets better, as backup power wears down and flooding persists. At a press conference held yesterday, FCC chairman Julius Genachowski revealed at least a quarter of all cell sites in areas damaged by Sandy were not operational. Those numbers were less optimistic that those provided by carriers.

The FCC this week activated the Disaster Information Reporting System, a central reporting point for telecommunications companies to update the agency regarding outages and other service disruptions. The FCC also alerted providers that in emergency circumstances, they can assist companies getting fuel for generators and help locate portable cell tower equipment for companies caught unaware.

AT&T’s belated letter to customers affected by Hurricane Sandy

Some may need the help.

New York State Assemblyman Alec Brook-Krasny and Brooklyn Borough President Marty Markowitz both reported Verizon Wireless’ outages are worsening in Brooklyn and midtown Manhattan.

Brooklyn Borough president Marty Markowitz

The Federal Emergency Management Agency (FEMA) today told Sen. Chuck Schumer the federal agency will reimburse New York for 100 percent of the costs incurred restoring power across the storm areas. But that may not expedite how quickly power returns.

Power restoration is expected to bring most cell towers back online. Worsening service is being attributed to battery backup or generator equipment exhausting on-hand fuel supplies, which usually keeps service up and running for up to three days. That means cell towers without power and unreachable by workers will have begun failing late Wednesday into today.

Damage assessments are further behind in New Jersey, the state that took the worst impact from Hurricane Sandy.

Stop the Cap! obtained some new figures from cell phone companies regarding the state of their networks:

• Verizon: Still holding to 94% operational in storm areas;
• AT&T: Declined to comment except to say “the vast majority” of their network is operational;
• T-Mobile: 80% operational in NYC, 90% operational in Washington, D.C.
• Sprint: 75% operational

[flv width=”384″ height=”228″]http://www.phillipdampier.com/video/WSJ Verizon Offices Damaged 11-1-12.mp4[/flv]

Verizon’s critical network takes another hit. “We’ve been here before,” says one Verizon executive, referring to the destruction from the 9/11 terrorist attacks which severely damaged the same facility on West Street now flooded out. (3 minutes)

Our readers report that cell service becomes spotty to non-existent in coastal New Jersey and Connecticut. In Manhattan anywhere south of 29th Street, readers report almost no signals at all.

Verizon’s damaged facilities include those on West and Broad Streets in Manhattan (circled).

Residents are trading tips about “magic spots” where cell service does suddenly pop up, and Gizmodo notes the only place in Alphabet City (the east side in southern Manhattan) to get service is on literally one street corner, where crowds congregate to make and receive calls.

The other salve for telecom withdrawal is the nearest pay phone.

Amusing stories of 20-somethings waiting in long lines only to be confounded by unfamiliar pay phones are appearing in the New York media. One radio station even aired basic instructions for members of the Millennial Generation that have never heard of inserting coins into telephones.

The biggest challenge for the city’s pay phone vendors is clearing them of coin overloads, something unheard of before the storm.

The often maligned pay phone has exposed the limits of the “more advanced” and expensive networks that were supposed to replace them. Despite claims of superiority for wireless service, northeast residents have once again discovered it has its limits:

• They don’t work during major weather events that knock out power and limit access to maintain backup generators;
• Cell networks are less capable of handling large call volumes, a problem made worse when cell phone refugees in other areas seek out remaining cell signals, further congesting the network;
• Wireless is just as susceptible to wireline or fiber failures on the ground. Cell towers typically connect to providers through wired backhaul circuits, which knock out cell service if they fail;
• Cell phone users need power to recharge their power-hungry smartphones. Batteries drain even faster searching for a weak or non-existent cell signal;

Hardest hit remains Verizon, which allowed reporters access inside damaged facilities to help New Yorkers better understand the scope of the problem.

[flv width=”384″ height=”228″]http://www.phillipdampier.com/video/WSJ Wireless Network Outages 11-1-12.mp4[/flv]

The Wall Street Journal takes a look at the state of the wireless communications networks across the northeastern U.S. and when service will be back.  (4 minutes)

Eleven years after the 9/11 terrorist attacks that took out Verizon’s West Street office when buildings collapsed at the nearby World Trade Center, Verizon is likely going to have to re-learn some lessons about catastrophe management as flood waters recede.

Verizon has deployed this 53-foot Emergency Mobile Communications Center for use by the Nassau County Office of Emergency Management that provides Internet and phone service.

The Wall Street Journal was able to obtain access inside the damaged facilities, and the reporter covering the event was left somewhat stunned by the scope of the damage.

In the middle of organized, yet chaotic recovery efforts was Verizon’s chief technology officer Tony Melone who had seen enough to declare the damage worse than 9/11.

The pictures of several feet of muddy water from the nearby Hudson River covering the lobby of the company’s headquarters on West Street said it all. The mostly salt water was an unwelcome guest in Verizon’s building, especially considering the five level basement below the lobby contains critical cables and telecommunications equipment. Almost four of those basement floors were completely flooded. After the water was pumped out, dampness and leaves from nearby trees remain littered on the floor.

One lesson learned after 9/11 was not to place critical phone switches below ground level. After reconstruction, the switches were moved to a higher floor and consequently were left undamaged. But while Verizon moved its backup generators upstairs, it left the pumps and fuel tanks that power them in the basement — leaving them inoperable.

This morning, passersby on West Street have to step around Verizon’s network of generators now running outside of the building, right next to large temporary fuel tanks to power them.

Verizon central offices in other parts of Manhattan, particularly further southeast on Broad Street, were never upgraded and are in worse shape, with electrical equipment damaged perhaps beyond repair. The force of the water was strong enough to bend the 86 year-old steel and bronze doors. Workers there are still trying to get water out of the building, shoving a pipe down an elevator shaft to facilitate pumping.

Verizon has some redundancy built into its network to protect its most valuable customers. That kept the landline phones working at the New York Stock Exchange, even though other landline and wireless customers will have  to wait longer for service to resume.

AT&T’s generator staging area near Meriden, Connecticut. (Credit: Brian Pernicone)

Some critics of the increasingly concentrated telecommunications landscape think Verizon and other companies have still not learned enough to prevent the kinds of service disruptions that will leave some customers without service for weeks.

It is hard to miss the bustle outside of Verizon’s offices damaged by the storm, watching flood water drain down the street. But things are murkier at cell phone providers who have been less than forthcoming about specific outage information and service restoration assessments.

Some have advocated the federal government step in and require cell phone service, now deemed essential by an increasing number of Americans, be protected with robust backup solutions to keep service up and running after catastrophic weather events.

After Hurricane Katrina, the FCC in 2007 tried to issue new rules that required a minimum of eight hours of backup power for all cell sites. The industry balked, predicting it would lead to “staggering and irreparable harm” for the cell companies. One wireless trade association warned their members might take several cell sites down if they were forced to provide backup power.

The CTIA Wireless Association and Sprint-Nextel sued the agency in federal court and the Bush Administration’s Office of Management and Budget eventually killed the proposed regulations.

T-Mobile and AT&T have cut an emergency deal to share their cellphone networks in areas affected by Superstorm Sandy. They’re trying to make it a little easier for customers to get a signal as carriers restore their networks. Some say companies should be forced to make their networks more resilient. National Public Radio’s Morning Edition has the story. (November 1, 2012) (3 minutes)
You must remain on this page to hear the clip, or you can download the clip and listen later.

Verizon Communications is on high alert to monitor the potential impact of Hurricane Sandy on the company’s landline, FiOS, and wireless networks — primarily from line damage and extended power outages that could come as a consequence of the slow-moving Category 1 hurricane. Top wind speeds from Sandy have been upgraded this morning to 90mph, making the storm’s impact even more severe for residents along the Atlantic coastline.

Verizon retail outlets are stocking up on car phone chargers and universal charging devices to help customers who endure extended power outages, but some retail stores may close early or stay closed if local weather conditions warrant.

Non-essential construction projects and internal training programs have been suspended so the company can focus on network repairs, as needed.

Verizon wireline and wireless business units have activated national and regional command and control centers, enabling Verizon operations teams to monitor the storm’s progress and company operations, including network performance. Verizon has established communications with power and other service providers to ensure proper coordination in the event of storm damage. The company also has contacted vendors and other outside partners so that critical communications equipment and supplies can be prioritized, stocked and shipped as needed.

Company equipment — including poles, fiber-optic and copper cable, portable cell sites that can replace a damaged cell tower and mobile emergency generators that can be used when local electrical power fails — is being staged in and around the mid-Atlantic and Northeast regions.

Verizon is the dominant phone company and wireless provider in the northeastern U.S.

In addition, Verizon managers are communicating the company’s storm preparation efforts and coordinating pre-planned response activities with the public-safety community, as well as state, county and municipal agencies along the East Coast and the Midwest.

“Verizon Wireless stands ready to serve our customers, and I urge everyone first and foremost to stay safe,” said Dan Mead, president and CEO of Verizon Wireless. “We live and work in the towns and cities in the storm’s path, and we are dedicated to keeping our friends, families and neighbors connected in times like these. We prepare for situations like this year-round, and pride ourselves in our ability to be there for our customers when they count on us most.”

As Sandy’s track came more into focus, the company began communicating with its customers on Friday, posting consumer tips on various company websites, issuing a news release to media outlets in the threatened region and nationally, engaging customers through social media such as Twitter and Facebook, and sending emails to consumers, with key links for troubleshooting and reporting service problems.

Bob Mudge, president of Verizon’s Consumer and Mass Business division, said: “In addition to communicating with customers and ensuring that we will be working to keep the network operating and responding quickly to issues as they arise, we have reminded our employees of the need to work safely, be alert, and help our customers in any way they can. But our people know this well and are at their best in these critical situations when our customers depend on us the most.”

Mudge noted that even though Verizon technicians may be ready to repair storm-damaged Verizon facilities, they may have to wait for approval from local power companies, first-responders or law enforcement before beginning restoration work.

Customers may contact Verizon online at www.verizon.com/outage to report any wireline service-related issues; or call 1-800-VERIZON (1-800-837-4966). Business customers are advised to contact their regular customer service centers or account teams as needed.