A homeless man’s mattress fire under a bridge in Massachusetts was enough to create the biggest telecommunications disaster for Verizon since the terrorist attacks of 9/11. The fire melted fiber and damaged copper cables that represented the backbone of Verizon’s landline network in the region, disrupting phone, cable, and broadband service for thousands of Verizon’s customers in northeastern Massachusetts in late August. Now that service has been restored, the damages from the outage and its ripple effects are still being calculated as questions are being raised about how the company handles its communications network.

The damage went far beyond an inconvenience for Verizon customers:

• Area businesses were ripe for plundering with Verizon-dependent alarm systems out of service;
• Cell towers went down if they were connected by Verizon’s fiber optic network;
• Local law enforcement communication systems ceased to function in areas where Verizon provided the vital link between the dispatch center and transmitting facilities miles away;
• Banks and other local businesses closed down because Verizon-based connectivity was inoperable. That left ATMs throughout the region out of service and credit card transactions often impossible to manage;
• 911 systems in several communities had to transfer emergency calls to other 911 centers miles away;
• Even the Registry of Motor Vehicles locked their doors and shut down while the outage persisted over several days.

[flv width=”640″ height=”380″]http://www.phillipdampier.com/video/NECN Verizon telecommunications outage causes problems 8-31-12.flv[/flv]
NECN News reports on Verizon’s enormous Massachusetts service outage and how it impacted the lives of affected customers. (3 minutes)

Verizon’s landline network ultimately failed its Massachusetts customers on a scale the company admits it has not seen in a long time.

“It’s the biggest problem we’ve faced, nationally, other than 9-11,” Bill Wilson, area manager for Verizon, told the Eagle-Tribune. “This is the biggest problem we’ve had in 20 years.”

At the heart of the problem is Verizon’s lack of redundancy in its landline network. With fiber optic cables managing a larger share of broadband traffic, phone calls, and even Verizon’s television service, the loss of even a single fiber cable can disrupt service for hundreds or thousands of customers, many more than would be affected by a damaged copper cable.

State Sen. Barry Finegold (D-Andover) is questioning Verizon’s decision not to have a backup plan in place.

“So if there’s a fire there’s a redundant system in place so if one fails there’s another to back it up,” Finegold said.

Customers, particular those working from home, wholeheartedly agreed.

“I am appalled,” Shela Horvitz, a Verizon FIOS customer who lost her phone, Internet and TV service for days as a result of the fire, told the newspaper. “Can you say, ‘Single Point of Failure?’”

[flv width=”640″ height=”380”]http://www.phillipdampier.com/video/Eagle Tribune Verizon Talks to Media About Outage 9-2-12.flv[/flv]

Verizon officials show reporters what the fire did to their network and how they are going to restore service after a fire on Lawrence Central Bridge caused mass outages in the region. From the Eagle-Tribune. (4 minutes)

Verizon blames the entire affair on homeless people, who they say should not have been sleeping on top of their wires. The idea of network redundancy for Verizon’s landline network? “Cost prohibitive,” say company officials.

Joseph Zukowski, vice president for government affairs for the phone company, said the problem was so rare, it was comparable to a 100-year storm. He compared the outage with a natural disaster.

“We have extensive security measures to make the network as secure as possible,” Zukowski said. “We restrict building access and access to our cables. Nowhere on the list is a homeless guy lighting a match on a mattress. We’re not laying blame, but the best thing would have been not to have the tent city there.”

Local police acknowledge the bridge where the cables cross the Merrimack River is a popular spot for the local homeless to congregate, and they have attempted to control the problem. But nobody ever told them Verizon’s vital regional communications network infrastructure was at ground zero of the mattress fire.

“Going forward, if the stuff that’s there is so important, it really needs to be secured so nobody can get access to it,” said police chief John Romero.

Verizon has promised refunds for all affected customers.

[flv width=”640″ height=”380″]http://www.phillipdampier.com/video/Verizon Lawrence MA Bridge Fire Verizon Restoration 2012 9-1-12.flv[/flv]

A Verizon-produced video illustrates how the company is repairing its damaged network. The heat from the fire on August 27th melted and fused both fiber and copper cables, and the protective casings that house the cables.

Verizon: “The work to restore service is complex, given that technicians are splicing thousands of individual copper and fiber-optic connections in a very confined area under the bridge. The conduit structure that holds the cables, which was protected by a metal cage, was destroyed and needs to be replaced. Verizon crews have been working in 24-hour shifts since the fire to restore service for customers and will continue to do so until every customer is back in service.”  (2 minutes)

Phillip “The Company Paid by Suddenlink to Issue a Third Party Guarantee Makes All the Difference” Dampier

When Stop the Cap! and Broadband Reports reader Simon contacted us about Suddenlink’s fact-free usage measurement tool that managed to rack up nearly 23GB of usage for one West Virginia customer on the same day his service was out for most of the evening, he probably did not think one customer catching the cable company’s fingers in the usage cookie jar would make much difference.

But it did.

Suddenlink spokesman Pete Abel, initially responding to complaints about the usage tool’s accuracy, told Light Reading last week its meter was “consistently accurate, as was demonstrated in the tests we ran before we launched this program.”

Four days later, the company effectively disavowed that, put the meter’s built-in overlimit fee scheme on hold and plans to hire a third party company to “validate the accuracy of its system,” after finding it was faulty after all.

Suddenlink won’t say what is causing the inaccuracies, but blamed “unusual” circumstances for the problem. The company is now refunding customers billed overlimit fees of $10 per 50GB and waiving future charges until its system is reviewed and validated by “a trusted third party.”

Stop the Cap! believes that does not come close to satisfying the company’s responsibility to its customers for accurate billing.

Suddenlink has never demonstrated it actually needs an Internet Overcharging scheme with usage limits and overlimit fees. The company proves that when it claims only a “relatively small number of customers” were ever billed overlimit fees. With no demonstrable usage problem, the company’s need to implement its Project Imagine “Allowance Plan” is sorely lacking.

Easy as counting anyway we like.

Additionally, the accuracy of providers’ usage measurement tools has proven highly suspect, and not just with Suddenlink. All of the companies caught with inaccurate meters always strongly defend them, until overwhelming evidence suggests they should not. Even super-sized companies like Bell Canada (BCE) and AT&T have enforced usage limits with meters the companies later had to disavow. Suddenlink is only the latest.

The scale in your grocery store is checked and certified. So is the corner gas pump, your electric meter, water meter, and gas meter. Why should broadband usage be any different?

Consumers are right to suspect Suddenlink’s usage meter. No official regulatory body verifies the accuracy of usage measurement tools and whatever company Suddenlink chooses to “verify” its meter has a built in conflict of interest — it works for a company that depends on a certain result in its favor. Suddenlink clearly has no business in the usage measurement business when it insists on the accuracy of a meter it disavows just a few days later.

With only murky details available to consumers about what caused the problem and why Suddenlink did not see it until a customer managed to catch them in the act, there is little confidence the company will actually solve a problem it never realized it had. There is also nothing to assure us — “third party guarantee” or not — it cannot happen all over again.

Suddenlink customers need to reach out and tell Suddenlink its “Allowance Plan” is completely unacceptable. Tell the cable company you don’t want to worry about their unverifiable and proven-inaccurate metering program. Ask them why you should remain a customer when they spend time and money on a scheme that the company itself admits is not really needed — targeting just a small number of “heavy users.”

Suddenlink’s customer service team does not think much of customers who use their broadband service a lot, as this recent “Who’s On First” exchange illustrates:

Lisa (Suddenlink): “Well, you show heavy OVERUSAGE of the Internet, you drew 14GB of data yesterday.”

Customer: “Okay, let’s back up, explain to me how I drew 12GB of data when my power was off and I wasn’t home on June 30.”

Lisa: “I didn’t say anything about June 30.”

Customer:  “If you have sooo much faith in your meter, explain to me how I drew 12GBs of data on June 30, while I didn’t have power, and wasn’t home.”

Lisa:  “I didn’t say anything about June 30.”

Customer:  “I’m asking, how did I draw 12GB of data without power to my house?”

If Suddenlink has a problem with a handful of users creating problems for other subscribers on its broadband network, it has always reserved the right to contact those customers directly and work out the problem one on one. That is a far better solution than inconveniencing all of their customers with endless rounds of “usage roulette,” where the big winner could find themselves with Bill Shock from overlimit fees, whether they actually deserve them or not.

[flv]http://www.phillipdampier.com/video/CNBC Internet v. Cable 8-20-10.flv[/flv]

CNBC interviewed Suddenlink CEO Jerry Kent in August 2010 on how his company intends to deal with “invasive online video,” threatening to erode cable-TV profits. Kent proved Suddenlink doesn’t really need any extra money from overlimit fees — the days of big spending on capacity are over, but the money is nice to have anyway.  (8 minutes)

Stuck with snail slow DSL, spotty Wi-Fi, or usage-capped 3G and need faster access? A new project from Connectify is now attracting funding from the Kickstarter project to deliver a new broadband connection combiner that can turn up to 10 different wireless and wired connections into one super-sized broadband pipe.

Connectify’s Dispatch software will manage connections ranging from dialup to Ethernet through a hotspot application that can be run on a desktop PC. The Philadelphia company has been pushing the project primarily to the speed obsessed, demonstrating outdoor connections to multiple open Wi-Fi networks, 3G and 4G mobile broadband that when combined deliver more than 80Mbps of download speed. But the software may also prove useful as a connections management tool that can seamlessly switch from free Wi-Fi when your connection becomes intolerably slow to a different pipe — 3G or 4G wireless broadband — all without missing a beat.

With two weeks left in the Kickstarter campaign, Connectify has raised just over $33,000 of the $50,000 goal. The company recently sweetened the deal early investors get, perhaps to attract an additional burst of funding. Those investing $50 or more will receive a lifetime license with unlimited software upgrades forever.

Some mobile carriers are experimenting with similar technology, mostly to move customers automatically off of their mobile networks to Wi-Fi, where available. Others are experimenting with technology that would allow simultaneous connections to Wi-Fi and 4G networks, moving different types of data across one or both simultaneously.

[flv width=”640″ height=”380″]http://www.phillipdampier.com/video/Connectify Dispatch.mp4[/flv]

Introducing Connectify Dispatch, which can turn multiple slow broadband connections into one fast one. (4 minutes)

[flv width=”616″ height=”388″]http://www.phillipdampier.com/video/Ernestine We Dont Care.mp4[/flv]

The more things change, the more they stay the same. On September 18, 1976 Lily Tomlin’s “Ernestine” reminded Americans here at the Phone Company we handle 84 billion calls a year with a system consisting of a multibillion-dollar matrix that is so sophisticated even we can’t handle it. But that’s your problem, isn’t it? We don’t care. We don’t have to. We’re the Phone Company.  (1 minute)

Spending money to earn more money is a fiscally sound principle of doing business, but short term investors often decry increased spending as harmful to the value of a company’s stock and dividend payout. That is why Hawaiian Telcom (HawTel) earns mixed reviews from Wall Street about the company’s aggressive infrastructure improvement project, a fiber to the neighborhood network that intends to bring television, phone, and faster broadband service to an increasing number of Hawaiians.

HawTel’s stock price has bounced up, down, up, and then down again as investors digest the company’s ongoing effort to reinvent itself as a 21st century telecom company.

The Old HawTel

HawTel’s fiber buildout began on the island of Oahu in 2011, eventually passing 27,400 homes on the island. At the end of 2011, 1,600 (6%) of those homes signed up for the service. That’s an acceptable number, especially for a service barely promoted. HawTel does not mention the television service on its primary website, and approaches potential customers one-on-one with in-person and targeted mail marketing.

At the end of the second quarter or 2012, HawTel TV had 6,400 subscribers. The company hopes to have an additional 50,000 homes enabled for its TV service by the end of 2012, with the goal of enabling 240,000 households across Hawaii over the next five years. HawTel hopes to eventually capture 30% of the Hawaiian market.

HawTel’s principal competitor is Oceanic Time Warner Cable, which provides traditional cable service across the Hawaiian Islands. HawTel had been at a substantial disadvantage competing with Time Warner’s television package and faster broadband service. But the fiber upgrades are allowing at least some customers to purchase speeds up to 50/10Mbps, slightly faster than what the cable operator offers.

Time Warner has taken note of the phone company’s re-emergence as a strong competitor, targeting Oahu with special promotional offers that lock customers in place with triple play discounts designed to make it inconvenient to switch providers.

The New HawTel

Unfortunately for HawTel, fiber upgrades do not come cheap, and the company’s earnings have taken a hit.

Capital expenditures totaled $41.2 million for the six-months ended June 30, 2012, up from $35.4 million for the six-month period a year ago due primarily to investments in broadband network infrastructure and expansion of video enabled households.

Hawaiian Telcom reported an 18 percent decline in second quarter earnings, which it blamed primarily on broadband network expansion.

The company also announced it lost another 6% of traditional landline customers during the second quarter, but that was offset by expansion in its broadband and television service. For HawTel, the solution to ending landline losses is to upgrade their network to compete with the types of communications services consumers are interested in buying today.

But those plans can and do conflict with at least some stock traders who are interested primarily in short term financial results. Spending can cut into profits, so some analysts downgrade stocks of companies spending the most, even if only to compete more effectively down the road.

So far, HawTel executives have not been discouraged carrying their network expansion plans forward. In July, Hawaiian Telcom announced it would acquire Wavecom Solutions Corporation’s local exchange carrier business in a stock purchase transaction valued at $13 million.

Wavecom’s undersea fiber network

The acquisition would give Hawaiian Telcom access to Wavecom’s fiber optic network connecting the main Hawaiian islands. Wavecom, formerly known as Pacific Lightnet, Inc., serves more than 1,700 customers across Hawaii.

In an application with the Federal Communications Commission, HawTel officials said access to Wavecom’s 400-mile undersea telecommunications cable network will permit the company to expand and enhance its broadband and television services beyond Oahu to other Hawaiian islands, and help position the company to effectively compete with Time Warner.

[flv width=”640″ height=”380″]http://www.phillipdampier.com/video/Hawaiian Telcom TV Tour.flv[/flv]

Watch a HawTel-produced video tour of the company’s new TV service.  (4 minutes)