Hewlett Packard Enterprise (HPE) technicians have been burning the midnight oil with Australian Taxation Office (ATO) staff in a bid to help the agency complete a major replacement of its failed hardware.
“Specialist ATO and HPE technicians have worked through the night to restore our systems and online services,” the agency said early on February 3. “While there has been significant progress on their restoration plan, the process is highly complex".
Just a day earlier, the agency told Australians that it was “experiencing issues relating to the hardware faults that occurred in December,” as it worked to replace the affected hardware.
"We are replacing the affected hardware, but this process will take some time,” the ATO said in a statement on February 2.
The next day, however, problems continued to hamper the ATO and its systems.
“Work will continue around the clock with an immediate focus on restoring priority services – the Tax Agent, Business and BAS Portals, ATO online, the Australian Business Register (ABR), Standard Business Reporting (SBR), and Superannuation online services,” the agency said early on February 3.
“The performance of different systems may vary as this restoration process is undertaken,” it said.
Later in the day, the agency said it was still working with its partners from HPE to restore its systems and services, but due to “an unforeseen complexity in the system restoration process,” services were unlikely to be available before close of business on February 3.
“HPE continue to deploy their global resources to restore the ATO systems and the associated infrastructure,” the ATO said. “Once this restoration is complete, our staff will work to make ATO services available for the community.
“We fully acknowledge the significant impact this is having on our key stakeholders, who rely heavily on the availability of our systems and services. We apologise for the inconvenience and disruption this has caused.
“While everything is being done to restore our systems as quickly as possible, we cannot make services available to the community until the integrity of the system is confirmed,” it said.
On 4 February, the ATO revealed that it had commissioned a new Storage Area Network (SAN), which had arrived on ATO premises, and that it was working with HPE engineers to restore systems and services.
"This new system will contain vast amounts of ATO data and it is currently being configured to provide more reliability and stability," the agency said. "The nature and scope of the ATO’s SAN means that this process of replacing the affected hardware will take some time."
The ongoing technical issues stem from a major systems outage that hit the ATO on December 12, 2016 following an “unprecedented” failure of storage hardware that had been upgraded in November 2015 by HPE.
The technical issues took out some of the ATO’s core internal systems and public-facing services for days, with the agency confirming a full nine days after the initial outage that its business-critical systems were live once again.
It is understood that the affected hardware was two new HPE 3Par storage area network (SAN) units acquired by the ATO in 2015.
The agency has spent the better part of the past seven weeks working to restore its systems to full capacity, retrieve temporarily lost data, and replace the affected hardware.
HPE confirmed in late December that it had launched an internal investigation into the cause of the hardware failure.
On January 24, the ATO revealed that it had appointed professional services industry company, PricewaterhouseCoopers (PwC), to conduct an independent review of the incident.
Updated on 6 February.
