[ad_۱]
Telstra has blamed the lack of a software update for a key time-keeping system for outage that caused nation-wide chaos last week, with its maintenance teams also unaware of a design change that affected how it would reset.
Ahead of an appearance by its chief executive, Vicki Brady, at a Senate inquiry over the mobile outage on Friday, Telstra provided a written submission that revealed the cause. In it, the company stated it didn’t lack redundancy in its network – but the redundancy also did not prevent the outage.
The submission confirmed reporting that one of Telstra’s network time protocol (NTP) servers, designed to ensure the systems had the correct time, had reset to be “۲۰۰۶”.
Telstra said it has three NTP servers, in Melbourne, Sydney and Perth.
During maintenance, the company shut down and restarted the server in Melbourne. Due to an “underlying software configuration” within the server, it restarted with the wrong ۲۰۰۶ date.
“Over the next few hours, the incorrect date rippled slowly across the network, causing authentication certificates in other servers to become invalid,” Telstra said.
“Customers were intermittently unable to authenticate onto the network (‘no service’), which affected their ability to place voice calls and use data across Telstra’s mobile network.”
Telstra said it had made an intentional design change to the equipment to fix an earlier fault – but that this had not been properly documented. This meant maintenance workers arriving at the Melbourne site early last Wednesday were not aware how the device would be reset.
A software update had also not been applied to the device, and if that had been done, the outage may not have occurred, Telstra said.
The submission also said when the Melbourne NTP server disconnected for maintenance, the other two worked as redundancy and backup as expected.
“The failure mode here was not inherently related to hardware, levels of redundancy, or the architecture of our network,” Telstra said.
But when the Melbourne server supplied an incorrect date once switched back on, “downstream systems used that date in security, authentication, session and policy-control processes.”
“The issue was therefore not simply the loss of one NTP server or redundancy in the design of the configuration of the three NTP servers, but the propagation and acceptance of erroneous date information by interconnected systems that rely on timing as a trust and ordering reference.”
Telstra also said it was taking full accountability for the outage.
“That is clearly unacceptable. If maintenance work can trigger this kind of outage, it suggests our controls were not good enough,” the company said.
“We are accountable for that, and our investigation will address why that design change was not documented, why the software update was not completed, and what needs to change in our controls so known risks are captured, prioritised and closed before they can affect customers.”
In the course of the outage, Telstra said ۵۸,۸۳۵ calls to triple zero successfully connected, while ۶۰۴ experienced an error.
Telstra also operates the triple-zero platform for all telcos, and the company said the platform doesn’t use the NTP servers for synchronisation, so it was unaffected by the incident. Fixed line callers on the NBN were also not affected.
On Friday, Brady and a number of other executives are due to appear before a Senate inquiry – established into last year’s Optus outage – to explain the outage that wrecked havoc for mobile services, transport systems, retailers and electric-vehicle charging.
The chair of the committee, the Greens senator Sarah Hanson Young, said the inquiry will seek to get to the truth of the outage and making “sure that we don’t keep accepting a system that leaves Australians vulnerable”.
