The outage that left Outlook.com and Hotmail users without access to their emails for almost 16 hours this week was due to overheating servers in the data center, Microsoft said.
The service disruption began at around 1.30 p.m. PDT on March 12 and affected access to Hotmail.com and Outlook.com, as well as some SkyDrive service users. The issue was fixed some 16 hours later, at around 5.40 a.m. PDT on March 13. At the time, Microsoft acknowledged the problems but did not issue an explanation.
Arthur de Haan, Microsoft’s vice president, explained in a blog post late on Wednesday that the outage happened after the company updated firmware on a core part of its systems.
He wrote: “This is an update that had been done successfully previously, but failed in this specific instance in an unexpected way. This failure resulted in a rapid and substantial temperature spike in the datacenter. This spike was significant enough before it was mitigated that it caused our safeguards to come in to place for a large number of servers in this part of the datacenter.”
Humans had to intervene!
The overheating issue prevented access to users’ mailboxes stored on the affected servers, and also prevented the system to switch to standby servers to allow continued access, de Haan explained, so the fix needed both software and human intervention.
“Requiring this kind of human intervention is not the norm for our services and added significant time to the restoration,” he said.
Microsoft apologized for the outage, but did not say exactly how many users were affected: “We sincerely apologize and regret the impact this outage had on all of you. Now that we’re through the resolution, we’re also hard at work on ensuring this doesn’t happen again.”
In case of future outages, De Haan wrote that “https://status.live.com is always the best and most reliable way to get real time information specific to any service issues that we are encountering, and when you are signed in, is customized based on the health of your specific account.”
Bad timing
The outage did not come at a good time for Microsoft. The revamped Outlook.com debuted in preview mode last July, heralded as the successor of Hotmail and a rival to Gmail and Yahoo Mail.
After extensive testing, Microsoft opened up Outlook.com for anyone who wanted to migrate from Hotmail, ahead of a complete migration of 360 million email accounts by the summer of 2013.
Sixty million Hotmail users migrated to the new service. However, a 16-hour outage is not the most reassuring signal to send to Outlook.com users, who also have been unable to access their account for several days in mid January and again in late February.