Subscribe
When One Giant ISP Goes Down, An Entire Country Will Notice
Millions of Australians were completely cut off from the global Internet Thursday when Telstra, Australia’s primary Internet Service Provider, lost connectivity to all websites hosted outside of the country.
The outage – which occurred between 7:43am and 8:50am – affected all Telstra home and business ADSL broadband, cable and mobile internet customers nationwide, the company said.
A Telstra spokesman said a planned change in the hardware that controls the ebb and flow of its international internet traffic was the cause of yesterday’s nationwide outage. Telstra’s network service was restored by rolling back the change to its original settings and restarting the equipment.
“We’re continuing to work to understand what happened and why. What we understand from preliminary inquiries is that from 7.43am customers attempting to access websites hosted overseas, or Australian sites with content hosted overseas, received a network error,” the spokesman said.
The result was a torrent of phone calls from upset customers waiting on hold with Telstra customer service, assuming they got their call through, while sipping their morning coffee.
“We have commenced a detailed and thorough technical investigation into the incident. This may take some time to conduct to ensure we fully understand the issue and can put appropriate measures in place to maintain the integrity and operation of our network,” the spokesman said.
An investigation? Matt in Sydney was bemused with the entire experience: “I can’t believe the reported resolution was a restart. ‘Have you tried turning it off and on again? Is it plugged in?'”
“Have you tried rebooting?”
“Yes. I restarted three times like you always tell me.”
Why are they doing an upgrade at 7am and not at midnight? That way, if things go wrong, you have a decent amount of time to fix it before it affects yoour user base during prime time.
FAIL. As always, ISPs should start their upgrades around 1AM in the morning, when connections are least likely to be used, and if they good something up they would have ample time to fix it. At least this might explain why my latency to servers in Australia was unusually low (200ms on both my Verizon line and Frontier line) and the latency was unusually consistent last night as well. Normally it’s 50-100ms higher to Australia after 11PM at night.
But yay BSOD!
Joe: Many years ago that may have been true but today many of us are up well past midnight and the people coming off second shift are just coming on-line. It may be midnight here but on the other side of the world it’s noon. There is never a good time to change equipment of software anymore except you save labor costs on first shift. A group notice of a shutdown would have been a nice touch but 90% of the people would have never seen it in time and even if they saw it they would still be going… Read more »
My point being every minute of every day is now prime time.