A panel electromechanical switch similar to those in use in New York until the 1970s. They were installed in the 1920s.

As late as the 1970s, New York Telephone (today Verizon) was still maintaining electromechanical panel switches in its telephone exchanges that were developed in the middle of World War I and installed in Manhattan between 1922-1930. Reliance on infrastructure 40-50 years old is nothing new for telephone companies across North America. A Verizon technician in New York City is just as likely to descend into tunnels constructed well before they were born as is a Bell technician in Toronto.

Slightly marring last week’s ambitious announcement Bell (Canada) was going to commence an upgrade to fiber to the home service across the Greater Toronto Area came word from a frank Bell technician in attendance who predicted Bell’s plans were likely to run into problems as workers deal with aging copper infrastructure originally installed by their fathers and grandfathers decades earlier.

The technician said some of the underground conduits he was working in just weeks earlier in Toronto’s downtown core were “easily 60-70 years old” and the existing optical fiber cables running through some of them were installed in the mid-1980s.

At least that conduit contained fiber. In many other cities, copper infrastructure from the 1960s-1980s is still in service, performing unevenly in some cases and not much at all in others.

Earlier this year, several hundred Verizon customers were without telephone service for weeks because of water intrusion into copper telephone cables, possibly amplified by the corrosive road salt dumped on New York streets to combat a severe winter. Verizon’s copper was down and out while its fiber optic network was unaffected. On the west coast, AT&T deals with similar outages caused by flooding. If that doesn’t affect service, copper theft might.

Fiber optic cable

Telephone companies fight to get their money’s worth from infrastructure, no matter how old it is. Western Electric first envisioned the panel switches used in New York City telephone exchanges until the end of the Carter Administration back in 1916. It was all a part of AT&T’s revolutionary plan to move to subscriber-dialed calls, ending an era of asking an operator to connect you to another customer.

AT&T engineer W.G. Blauvelt wrote the plan that moved New York to fully automatic dialing. By 1930, every telephone exchange in Manhattan was served by a panel switch that allowed customers to dial numbers by themselves. But Blauvelt could not have envisioned that equipment would still be in use fifty years later.

As demand for telephones grew, the phone company did not expand its network of panel switches, which were huge – occupying entire buildings – loud, and very costly to maintain. It did not replace them either. Instead, newer exchanges got the latest equipment, starting with more modern Crossbar #1 switches in 1938. In the 1950s, Crossbar #5 arrived and it became a hit worldwide. Crossbar #5 switches usually stood alone or worked alongside older switching equipment in fast growing exchanges. It occupied less space, worked well without obsessive maintenance, and was reliable.

It was not until the 1970s that the Bell System decided to completely scrap their electromechanical switches in favor of newer electronic technology. The advantages were obvious — the newer equipment occupied a fraction of the space and had considerably more capacity than older switches. That became critical in New York starting in the late 1960s when customer demand for additional phone lines exploded. New York Telephone simply could not keep up with and waiting lists often grew to weeks as technicians looked for spare capacity. The Bell System’s answer to this growth was a new generation of electronic switches.

The #1 ESS was an analog electronic switch first introduced in New Jersey in 1965. Although it worked fine in smaller and medium-sized communities, the switch’s software bugs were notorious when traffic on the exchange reached peak loads. It was clear to New York Telephone the #1 ESS was not ready for Manhattan until the bugs were squashed.

Bell companies, along with some independent phone companies that depended on the same equipment, moved cautiously to begin upgrades. It would take North American phone companies until August 2001 to retire what was reportedly the last electromechanical switch, serving the small community of Nantes, Quebec.

A notorious 1975 fire destroyed a phone exchange serving lower Manhattan. That was one way to guarantee an upgrade from New York Telephone.

On rare occasions, phone companies didn’t have much of a choice. The most notorious example of this was the Feb. 27, 1975 fire in the telephone exchange located at 204 Second Avenue and East 13th Street in New York. The five alarm fire destroyed the switching equipment and knocked out telephone service for 173,000 customers before 700 firefighters from 72 fire units managed to put the fire out more than 16 hours later. That fire is still memorialized today by New York firefighters because it injured nearly 300 of them. But the fire’s legacy continued for decades as long-term health effects, including cancer, from the toxic smoke would haunt those who fought it.

The New York Telephone building still stands and today also houses a street level Verizon Wireless retail store.

New York Telephone engineers initially rescued a decommissioned #1 Crossbar switch waiting to be melted down for scrap. It came from the West 18th Street office and was cleaned and repaired and put into emergency service until a #1 ESS switch originally destined for another central office was diverted. This part of Manhattan got its upgrade earlier for all the wrong reasons.

Throughout the Bell System in the 1970s and 80s, older switches were gradually replaced in favor of all electronic switches, especially the #5 ESS, introduced in 1982 and still widely in service today, serving about 50% of all landlines in the United States. Canadian telephone companies often favored telephone switches manufactured by Northern Telecom (Nortel), based in Mississauga, Ontario. They generally worked equally well as the American counterpart and are also in service in parts of the United States.

The legacy of more than 100 years of telephone service has made running old and new technology side by side nothing unusual for telephone companies. It has worked for them before, as has their belief in incremental upgrades. So Bell’s announcement it would completely blanket Toronto with all-fiber service is a departure from standard practice.

For Bell in Toronto, the gigabit upgrade will begin by pushing fiber cables through existing conduits that are also home to copper and fiber wiring still in service. If a conduit is blocked or lacks enough room to get new fiber cables through, the Bell technician predicted delays. It is very likely that sometime after fiber service is up and running, copper wire decommissioning will begin in Toronto. Whether those cables remain dormant underground and on phone poles for cost reasons or torn out and sold for scrap will largely depend on scrap copper prices, Bell’s budget, and possible regulator intervention.

But Bell’s upgrade will clearly be as important, if not more so, than the retirement of mechanical phone switches a few decades earlier. For the same reasons — decreased maintenance costs, increased capacity, better reliability, and the possibility to market new services for revenue generation make fiber just as good of an investment for Bell as electronic switches were in the 1970s and 1980s.

[flv]http://www.phillipdampier.com/video/ATT Reconnecting 170000 Phone Customers in NYC After a Major Fire 1975.mp4[/flv]

AT&T produced this documentary in the mid-1970s about how New York Telephone recovered from a fire that destroyed a phone exchange in lower Manhattan and wiped out service for 173,000 customers in 1975. The phone company managed to get service restored after an unprecedented three weeks. It gives viewers a look at the enormous size of old electromechanical switching equipment and masses of phone wiring. (22:40) 

All of the top-10 broadband rankings for accessibility, affordability, speed, and subscription rates have been awarded to countries in Europe and Asia, while the United States continues to fall further behind.

This week, the UN Broadband Commission issued its annual report on broadband and had little to say about developments in North America, where providers have maintained the status quo of delaying upgrades, raising prices, and limiting usage. As a result, other countries are rapidly outpacing North America, preparing the infrastructure to support the 21st century digital economy while officials in the U.S.A. and Canada cater primarily to the interests of large incumbent cable and telephone companies.

The United States has fallen from 20th to 24th place in wired broadband subscriptions, per capita. Virtually every country in western Europe now beats the United States, as does Hong Kong, Belarus, and New Zealand. Canada scored better, taking 14th place.

Only managing a meager 19th place, only 84.2% of Americans are online. Iceland has 96.5% of their population on the Internet, closely followed by the other northern European nations of Norway, Sweden and Denmark. Also scoring superior to the United States: Andorra, Bahrain, Qatar, and the United Arab Emirates. Canada did better than its southern neighbor as well, coming in at number 16.

With big profits to be made in wireless, large wireless phone companies like Verizon Wireless and AT&T helped the U.S. achieve its best rating — 10th place in wireless. But the countries that exceeded the United States did much better (Canada was not rated this year.)

With the arguable exception of wireless, the United States is no longer a world leader in broadband and continues a slow but steady decline in rankings as other countries leapfrog over the U.S.

At least 140 countries now have a National Broadband Plan in place, most maintaining stronger oversight over telecommunications infrastructure than the largely unregulated U.S. broadband marketplace. After reviewing broadband performance across most UN member states, the Broadband Commission for Digital Development recognized several traits common in countries where broadband has been particularly successful:

Competition is essential to promote enhanced broadband. A monopoly or duopoly (usually a telephone company and cable or wireless operator) is not enough to promote healthy broadband advancements. At least three, near-equal competitors are required to achieve the best upgrades and price competition. The presence of smaller competitors or those charging considerably different pricing had little effect on competition.

Countries with the best speeds have national policies promoting the installation of fiber optic technology, at least in multi-dwelling units and new developments. Although the cost of fiber and its installation can amount to as much as 80% of a broadband expansion project, many countries have been successful compelling competing providers to share a single fiber optic network (and its costs) to make the investment more affordable. In terms of ultra-high-speed broadband, there are still not many consumer apps and services that need Gigabit speeds, but such services are on their way. Experience shows that technology typically moves faster than most people anticipate – so countries and operators need to start planning now for the imminent broadband world.

A coherent regulatory foundation that emphasizes competition over regulation was the most effective policy. But regulatory frameworks must guarantee a level playing field among competitors and strong oversight to make sure competitors play fair. Regulation is not keeping pace with the changes in the market – Internet players offering equivalent voice and messaging services are, by and large, subject to relatively limited requirements (including consumer protection, privacy, interoperability, security, emergency calls, lawful intercept of customer data, universal service). Asymmetric regulation has resulted in an uneven competitive landscape for services. Governments and policy-makers need to review and update their regulatory frameworks to take into account evolving models of regulation.

Telecommunication and broadband access providers need to explore business arrangements with Internet content providers that will accelerate global investment in broadband infrastructure, to the mutual benefit of all, including end-consumers. Internet companies and Internet content providers need to contribute to investment in broadband infrastructure by debating interconnection issues and agreeing fees/revenue shares with other operators and broadband providers.

That last issue is now being hotly debated in the United States, where providers are seeking compensation from streaming video providers like Netflix, which now account for a substantial amount of Internet traffic.

AT&T’s Fiber Fairy Tale

Holding your breath waiting for AT&T’s GigaPower 1Gbps U-verse upgrade to arrive in a town near you is hazardous to your health.

Despite a blizzard of press releases promoting the forthcoming arrival of gigabit Internet access from AT&T, the fine print reveals as little as one percent of some communities will actually get the upgrades.

In Winston-Salem, N.C., city officials cannot even get a firm commitment from AT&T that it will deliver the faster service to the 63 businesses the city chose as early candidates for the fiber upgrade.

In June, the city and AT&T signed an agreement for gigabit broadband expansion using AT&T’s GigaPower U-verse platform. But AT&T largely gets to decide where, when and even if it will invest in upgraded service. The city did not impose many conditions beyond a requirement that AT&T provide up to 20 free Internet connections to community sites with a one-time installation cost of $300 to $500. Another 20 connections would be provided to small to mid-size businesses, with no obligation to buy services.

In response, AT&T said it would only commit to reviewing the city’s list and “make an effort to serve the proposed locations if they are in the vicinity of where service will be available.”

If those locations fall outside of AT&T’s plans, no gigabit fiber.

A significant indicator of the true extent of AT&T’s expansion plans is whether the company is allocating capital spending commensurate with the costs of running fiber optic cable to individual homes and businesses. So far, AT&T has not. With no obligation to deliver the service AT&T is implying it will offer, the company is free to wire a handful of technology parks, businesses, and new housing developments and claim to have met its commitment, despite the fact 99 percent of area residents have no access to the faster speeds.

For the benefit of low-income residents who lack affordable Internet access, AT&T also promised it would offer some lower-speed Internet connections in a limited number of apartment complexes in low-income areas.
Here are the sites nominated by the city of Winston-Salem for AT&T gigabit broadband. AT&T’s response: ‘Maybe.’

Community sites: Aids Care Service; Boys & Girls Clubs at New Walkertown Road and Reynolds Park; Brown & Douglas Neighborhood Center; Russell Recreation Center; Liberty CDC; Community Care Center; ElBuen Pastor; Forsyth Technical Community College’s Woodruff Center; Gateway YWCA; Knollwood Baptist Church; Little Creek Neighborhood Center; Malloy/Jordan East Winston Heritage Center; MLK Jr. Center; Reynolda Branch library; S.G. Atkins CDC; SciWorks; Sedge Garden Center; Shepherd’s Center; South Fork Center; Southside Library; United Metropolitan Church; Winston Lake YMCA.

Small- to mid-size businesses: Bellomy Research; Campus Partners; Carolina Liquid Chemistries Corp.; Center for Design Innovation; CML Microcircuits (USA); Computer Credit Inc.; Computing Solutions Group Inc.; COR365 Innovation Solutions; Dairy Fresh Inc.; DataChambers LLC; Davenport Transportation Consulting; Debbie’s Staffing Service; Eastridge Technology Inc.; Exhibit Works; Flywheel; IMG College; Interact 911; KeraNetics LLC; Key Services Inc.; Kings Plaza; MissionMode; Ocular Systems; Odigia; OnceLogix LLC; Out of Our Minds Animation Studios Inc.; Page’s Sporting Goods; PhoneTree; Piedmont Propulsion; Segmented Marketing Solutions Inc.; Small Footprint Inc.; SolidSpace LLC; Special Event Services; Sunrise Technologies Inc.; The Clearing House Payment Center; Triad Semiconductor; TrueLook; Voyss Solutions; Washington Perk site at Washington Park; West 3rd Street Media; West End Mill Works.

Source: City of Winston-Salem

AT&T’s Magic Fiber Fairy brings fiber to you, if you approve AT&T’s business agenda.

If it wins approval from regulators to buy satellite TV provider DirecTV, AT&T says it will have enough money to afford to expand its gigabit fiber network Gigapower U-verse to an extra two million homes.

That bit of non-sequitur was the highlight of AT&T’s regulatory filing with the Securities and Exchange Commission. AT&T claims money for the fiber expansion will come from anticipated savings from programming volume discounts AT&T will get combining DirecTV’s 20.3 million customers with AT&T’s 5.7 million U-verse TV subscribers.

AT&T expects cost synergies to exceed $1.6 billion annual run-rate by three years after closing.  These savings will begin in the first year after closing, ramp up over four years and grow with the addition of video subscribers thereafter.  It is anticipated that at least 40% of these total synergies will be realized by year two after closing.  These synergies are conservative and derived from items such as programming cost reductions, operational efficiencies and reductions in redundant broadcast infrastructure.  Programming cost reductions are the most significant part of the expected cost synergies.  At this time, AT&T’s U-verse content costs represent approximately 60% of its subscriber video revenues.  With the scale this transaction provides, we estimate AT&T’s U-verse content costs after the completion of the transaction will be reduced by approximately 20% or more as compared with our forecasted standalone content costs.

AT&T believes that despite perennially increasing programming costs, especially for popular over-the-air and cable networks, the 20 percent of anticipated savings will give AT&T enough money to vastly expand its fiber network.

“The economics of this transaction will allow the combined company to upgrade two million additional locations to high-speed broadband with Gigapower FTTP (fiber to the premise) and expand our high-speed broadband footprint to an additional 13 million locations where AT&T will be able to offer a pay TV and high-speed broadband bundle,” AT&T wrote.

On AT&T’s budget, the company can send you this really nice star ceiling kit, but it won’t pay for gigabit broadband.

Before announcing its intent to buy DirecTV, AT&T already promised to expand Gigapower U-verse to up to 100 cities, while telling investors it anticipated flat spending on network improvements. On Tuesday, AT&T went further and dramatically cut investments in its wireline network to a level that raised concerns for the financial security of several of its vendors, including those supplying fiber optic cable and equipment.

AT&T predicted savings from the merger will amount to $1.6 billion a year, but not until three years after the merger closes. There are questions whether this amount is enough to fund the kind of fiber expansion AT&T promises.

In 2012, AT&T committed to expanding U-verse to 8.5 million more customer locations at a cost of $6 billion. That investment paid for AT&T’s less-costly fiber to the neighborhood service. Based on AT&T’s figures, the cost to deploy fiber into each neighborhood, while still utilizing existing copper wiring to bring service into each home, was $705 per home or business.

AT&T Gigapower U-verse requires AT&T to spend considerably more to extend fiber service directly to each premises it intends to serve. Google is spending approximately $4,000 to reach each home with fiber optics in Kansas City. But AT&T’s math suggests it only has to spend about $800 per home (based on the $1.6 billion savings figure it expects to begin receiving in 2017) for decommissioning the remaining copper and extending U-verse fiber for each of two million customer homes passed. What does AT&T know that Google does not?

But wait. AT&T is also committing to use that $1.6 billion to expand traditional fiber to the neighborhood U-verse to 13 million additional homes as well. That means AT&T has a budget that limits it to $106 per home for a combined 15 million new locations passed. That amount is enough for a fiber optic star ceiling kit or a really nice fiber strand light fixture, but it isn’t nearly enough to bring gigabit broadband to AT&T customers.

One thing is certain: AT&T will not be passing on any cost savings to customers in the form of lower bills. AT&T’s proposed investment is a blatant appeal to regulators with promises of broadband expansion the company has already made and shows few signs of actually delivering.

Phone companies looking for a cheap way to increase broadband speeds are turning away from fiber optics and towards advanced forms of DSL that don’t bring cost objections from shareholders.

Whether your provider is AT&T or an ISP in Europe or Australia, financial pressure to improve broadband on the cheap is fueling research to wring the last kilobit out of decades-old copper phone wiring.

Alcatel-Lucent suggests VDSL2 Vectoring is one such technology that can enable download speeds up to 100Mbps using noise-cancelling technology to suppress interference.

But the advice doesn’t impress fiber optic fans who suggest any reliance on deteriorating copper phone lines simply postpones an inevitable fiber upgrade that could come at a higher cost down the road.

VDSL2 Vectoring and G.Fast are only as good as the copper wiring that extends to each customer. Up to 45 percent of North American wire pairs are in some state of disrepair.

Vectoring has been described as “pixie dust” by Australia’s former Communications Minister Stephen Conroy. Conroy was overseeing Australia’s switch to fiber service as part of the National Broadband Network. But a change in government has scrapped those plans in favor of a cheaper fiber to the neighborhood broadband upgrade advocated by the new Communications Minister Malcolm Turnbull that resembles AT&T’s U-verse.

“Malcolm can sprinkle pixie dust around and call it vectoring and he can do all that sort of stuff but he cannot guarantee upload speeds,” Conroy told Turnbull.

As with all forms of DSL, speed guarantees are extremely difficult to provide because the technology only performs as well as the copper wiring that connects a neighborhood fiber node to a customer’s home or office. Upload speeds are, in practical terms, significantly slower than download speeds with VDSL2. Turnbull expected download and upload speeds on Australia’s VDSL2 network to be around a ratio of 4:1, which means a customer who has a download speed of 25Mbps per second would receive an upload speed of around 6Mbps.

In the lab, VDSL2 Vectoring delivers promising results, with speeds as high as 100Mbps on the download side. DSL advocates are excited about plans to boost those speeds much higher, as much as 1,000Mbps, using G.Fast technology now under development and expected in 2015. VDSL2 Vectoring and G.Fast both require operators to minimize copper line lengths for best results. Unfortunately, dilapidated copper networks won’t work well regardless of the line length, and with many telephone companies cutting back upkeep budgets for the dwindling number of customers still using landlines, an estimated 15-45 percent of all line pairs are now in some state of disrepair.

Assuming lab-like conditions, G.Fast can deliver 500Mbps over copper lines less than 100 meters long and 200Mbps over lines between 100 and 200 meters in length.

G.Fast also allows for closer symmetrical speeds, so upload rates can come close or match download speeds.

This cabinet houses the connection between a fiber optic cable and copper phone wiring.

Providers prefer the copper-fiber approach primarily for cost reasons. There are estimates deploying a G.Fast-capable VDSL service to a home would cost around 70 percent less than fiber to the home service. Workers would not need to enter customer homes either, offering less-costly self-install options.

Telekom Austria and Swisscom are among providers committed to launching the technology. Both countries are mountainous and have many rural areas to serve. Fiber rich providers are also looking at the technology for rural customers too costly or too remote to service with fiber.

Critics question the real world performance of both VDSL2 Vectoring and G.Fast on compromised copper landline networks. Decades of repairs, deteriorating insulation, corroded wires, water ingress, and RF interference can all conspire to deliver a fraction of promised speeds.

Many critics also point to the required aggressive deployment of fiber/VDSL cabinets — unsightly and occasionally loud “lawn refrigerators” that sit either in the right of way in front of homes or hang from nearby utility poles. To get the fastest possible speeds, one cabinet may be needed for every four or five homes, depending on lot size. Australia’s VDSL network, without Vectoring or G.Fast requires at least 70,000 cabinets, each powered by the electric grid and temporary backup batteries that keep services running for 1-2 hours in the event of a power failure. The batteries need to be decommissioned periodically and, in some instances, have caused explosions.

The costs of electric consumption, backup batteries, infrastructure, and maintenance of copper lines must be a part of the cost equation before dismissing fiber to the home as too expensive.