ARN

Back-up lessons from OVHcloud's catastrophic fire disaster

Organisations that failed to make sure their cloud data was backed up properly learned the consequences the hard way when an OVHcloud data centre burned last month.

European’s largest cloud provider, OVHcloud, suffered a catastrophic fire last month that destroyed one of its data centres and smoke-damaged a neighbouring one.

OVHcloud customers with data in the burned-out data centre who had their own disaster recovery measures in place or who purchased the off-site back-up and disaster-recovery services offered by OVHcloud have been able to resume operations. Those who did not lost data that will never come back.

Some losses were complete, such as those described on Twitter by rounq.com who is still waiting for back-ups and redundancy that he thought were already in place, according to his tweets. Companies that had some type of off-site back-up seemed to be up and running again, such as Centre Pompidou.

There also appear to be companies somewhere in the middle that have resumed operations but have also acknowledged losing data. One of them is Facepunch, providers of the dystopian game Rust that involves players creating their own virtual environments that are stored as files on a server. If the environments that they built were stored in the destroyed data centre, it appears to be gone.

'There is no cloud'

Nothing sums up the reality of cloud computing more than this: There is no cloud; there is only someone else’s computer. And if that someone else is your cloud provider, and its computers are not properly protected, it can harm you.

Remember that the cloud is not magic, and that nothing stops the fundamental laws of physics and chaos. It appears that OVHcloud had everything it needed to prevent and stop a data centre fire, but as it turned out, it wasn’t up to the task.

Remember that the cloud service you use, whether it is IaaS, PaaS, or SaaS, is just another data centre like one you might build yourself—only it’s in another place run by other people. They may be the best at what they do, but neither they nor the systems they employ are infallible. That is why you should always have back-up and disaster-recovery plans in place. Hope you never have to use them, but have them ready if you do.

Another part of the OVHcloud story that drives this home is how much time it’s taking the provider to bring additional capacity online.

They are the largest cloud provider in Europe, and they are still struggling to replace the lost computing and storage capacity they lost over a month ago. It looks like they’re doing their best, but they can build and provision servers only so fast. They’re just another company trying to build a data centre. There is no cloud; there is only someone else’s computer.

Follow the 3-2-1 rule

The incredibly basic 3-2-1 rule states that you should have at least three versions of your data on two different media, one of which is off-site. And the rule applies to data stored in a public cloud.

Many tweets from OVH customers said they stored their back-ups on another server in the same data centre that burned, which means their primary and back-up data were destroyed by the fire. Others felt it was OVH’s responsibility to protect their data from a data centre fire, so they made no provisions at all for back-ups.

At least one Twitter handle seemed unsympathetic toward those who would blame OVH. @kalle_sintonen has repeatedly told people that you get what you deserve when you pay for an inexpensive VM with no redundancy without also paying for an optional offsite back-up service offered by the provider.

The optional service that OVH offered made sure that back-ups were copied to another data centre. Customers who opted for that service have been able to order new servers and restore their operations from this other back-up.

Some customers did not use it, and, unless they provided other back-up, their data is gone. The consequences of ignoring the 3-2-1 rule are unforgiving when something like this happens.

Know whether service agreements ensure back-up

You should know how resources are backed up in your private data centre, but do you know how your cloud resources are protected? Are back-ups stored in redundant storage in an additional location than the resource that they are backing up? You need to know, and you that back-up needs to meet the requirements of the 3-2-1 rule.

Read your service agreement to see what protections it includes. Does it even mention back-up? Does it talk about disaster recovery? Most cloud contracts do not, and if they do, they specify that back-ups and DR are your responsibility, not theirs.

If they offer an optional back-up service, it is your responsibility to opt in. Make sure that everything you think they are providing is guaranteed in writing. Remember: if it’s not in writing, it doesn’t exist.

If your contract does include back-up, does it explicitly say how the vendor stores back-up data and whether it is stored in a completely different system in a completely different region and account?  How will their back-up system protect you in a disaster like the OVH fire?  Could the back-ups be in a neighbouring data centre that could be damaged by the same fire? If your provider doesn’t have good answers, demand better ones or change providers.

It appears that OVH is bending over backwards to help customers as much as it can. But in the end, if your primary and back-up data were both stored in the destroyed data centre, your data is gone forever.

Even if a customer stored an additional copy in the neighbouring data centre, it also might be gone due to smoke damage. OVH is literally scrubbing the insides of smoke-damaged computers to bring them back online, but some of these systems might not recover, and data stored on them could be gone forever.

Some cloud-storage vendors are transparent about data protection. If you search for back-up and recovery on their websites, you might find a webpage that states back-up of your data is your responsibility. Or it might detail what kind of back-ups they do provide and where that data is stored.

Other vendors are opaque and don't make that information readily available. Try getting a straightforward answer from them on whether back-up and recovery is your responsibility. If a vendor representative tells you that your data is safe and that you don’t need to worry about back-up and recovery, get the details in writing.

If you do have a back-up and DR service, make sure it protects against all disasters including electronic attacks. Something that physically destroys the data centre is not the only thing you need to worry about. You also need protection from threats like ransomware and hackers deleting your account.

Short of well-documented back-up and disaster-recovery protection included in your SLA, make sure you perform your own back-ups. It will cost some money, but it’s a good bet the companies that lost everything last month wish they had done so.