Amazon AWS Data Loss Shows Cloud Backups Are Crucial

We have certain grown accustomed to today’s “cloud age” where services we subscribe to are in the cloud, data is stored in the cloud, and last but not least, we store our enterprise business-critical data and resources in the cloud.  The public cloud giants that we know today have certainly become “household names” in the enterprise.  Amazon AWS, Microsoft Azure, and Google GCP supply the majority of cloud resources that we take advantage of. 

It seems this year that the cloud providers have had a relatively rough year in terms of stability, suffering very high-profile outages across the board.  Recently, over the Labor Day weekend in the United States, Amazon AWS suffered the worst kind of interruption in services – one where data was lost.  We don’t hear about that too often with cloud providers as in most people’s minds, they are infallible and incapable of losing your data. 

However, this recent event for Amazon certainly highlights that backups are one of the most critical components of protecting and serving out your data and services.  In this post, I will throw some general thoughts out there around hosting data in the cloud, why backups are important in the cloud, and some methodologies to think about when designing a data protection strategy.  Let’s take a look at Amazon AWS data loss shows cloud backups are crucial.

Why is Hosting Data in the Cloud Becoming Prevalent?

Things have happened so quickly it seems in the past decade to get where we are today with cloud becoming so prevalent.  Amazon AWS started this move many years ago en masse by enterprise customers.  We all remember the early days where there was a slight stigma and reservation about moving data to the cloud.  Security concerns, and other privacy issues plagued the cloud early on. 

However, after those first few formative years, cloud adoption caught on like wildfire.  It almost seemed like overnight, we as enterprise IT professionals and business leaders alike decided to start trusting the public cloud from many different fronts.  This included security, privacy, and just general high availability and stability of our data.  At that point, businesses and enterprise environments couldn’t move to the cloud fast enough.

Also, with the changing times and ever faster pace required by businesses today, housing infrastructure on-premises and providing the features that customers and business stake holders now require is becoming increasingly difficult.  There is really no way that customers housing infrastructure on-premises can compete with the scale, features, accessibility, and products offered by the giant public cloud providers today. 

Massive data centers and an almost mind-boggling array of cloud offered services and features are introduced in an almost continuous flood from the public cloud giants.  Additionally, Software-as-a-Service offerings from the public cloud providers such as Microsoft and Google by the likes of Office 365 and Google G Suite provide massive appeal to enterprise businesses today in terms of hosting corer business applications and services such as business productivity, email, and data storage. 

An additional consideration for cloud migration can be based off a misconception as such – that cloud providers are incapable of losing our data.  With massive world-class data centers, redundant servers, power, network connections, and other facilities amenities, your data is much safer in the cloud from a data loss perspective than it is on-premises.

While this is somewhat true when comparing the orders of magnitude difference in the hardware and redundancy measures put into place by public cloud providers when compared to on-premises enterprise data centers, public cloud provider service and availability is NOT infallible.

When it comes down to scrutinizing public cloud provider infrastructure when compared to on-premise data centers, they are using much of the same underlying physical server hardware.  This means that failures can and do happen.  When looking at what this means – while it may not happen very often, you CAN lose data in the public cloud due to hardware failure as the Amazon AWS Data Loss Shows Cloud Backups Are Crucial.

Amazon AWS Loses Customer Data

What happened in this very recent event where Amazon customer data was lost?  The Register ran a post just a few days ago at the time of this writing that detailed events that appear to have led up to the data loss event with a subset of Amazon AWS EC2 instances.  What happened?

A simple power loss event looks to have been the culprit.  It might seem hard to believe that an Amazon AWS data center could be subject to interrupted service due to a power loss event, however, this time apparently that was the case. 

In what appears to have been a cascading series of unfortunate events, power was lost to Amazon data centers.  When the power loss event happened, Amazon’s backup generators fired up, however, they detailed a failure of backup generators that led to physical Amazon AWS servers going down. 

The result of the hardware going down was corrupted customer data in the form of Amazon AWS EC2 instances and Elastic Block Store (EBS) volumes being corrupted to the point of unrecoverable data.  According to Register this led to a total of more than 1 TB of Amazon customer data in EBS storage vanishing. 

While a larger impact was felt initially, they were able to recover the majority of the systems and customer data.  The fallout that resulted as detailed by Amazon:

“This resulted in 7.5 per cent of all instances in that Availability Zone failing by 0610 PDT…Over the last few hours we have recovered most instances but still have 1.5 per cent of the instances in that Availability Zone remaining to be recovered. Similar impact existed to EBS and we continue to recover volumes within EBS. New instance launches in this zone continue to work without issue.”

At the end of the day, a handful of customers had data that was unrecoverable.  Wait, unrecoverable?  Are we saying that massively giant public cloud providers like Amazon can actually lose our data that is stored there?  YES.  Actually, this should come as no surprise.  If you consider the SLAs and agreements as well as the fine print associated with data housed with Amazon and other public cloud giants, they tell you in no uncertain terms they are NOT responsible for data loss that may result in housing your data and resources in their cloud data centers.

Unfortunately, with the power failure event suffered in the Labor Day weekend Amazon AWS debacle, customer data was lost.  This means EBS volume data as well as EC2 instances were lost.  This underscores an often-overlooked concept when dealing with public cloud environments – Backups are absolutely essential.  

If you are not backing up your public cloud environments, you are asking for data loss, PERIOD.  Public cloud providers are incredibly stable and resilient from a relative perspective; however, you would be taking a huge risk by not going the extra step of protecting your data that exists there.

Risks to your Public Cloud Data

Are problems from the public cloud vendor and the underlying hardware the only threat to your data that exists in the cloud?  No, and even more sobering, hardware failure is most likely the least of your worries to your data in the public cloud.  What are some of the common data loss threats that you need to protect against in cloud environments?

Data loss is most likely to happen at the hands of end users or administrators as well as from security threats.  How do these scenarios unfold?  First, let’s take a look at data loss at the hands of end users.

If you think about the most common use case for backups on-premises, you might first think of the scenario where an end user either accidentally deletes a file or intentionally deletes a file, mistaking it for a file they want to delete.  This of course will lead to data loss. 

Think too about a file that a user modifies and instead of performing a “Save As”, they perform a “Save” operation.  If a user deletes massive amounts of data from a document and saves it, data is lost at that point.   Simply because data is being accessed by end users in the public cloud instead of on-premises, this does not change the data loss possibilities.  In fact, many of these age-old scenarios that happen on-premises can happen in the cloud as well.

What about security threats to your data?  One of the most alarming threats to your data today is ransomware.  Ransomware slyly and unscrupulously uses encryption technology to encrypt your data so that you can no longer access it without the decryption key.  In order to obtain the decryption key, the attackers force you to pay for the key in some form of cryptocurrency.

Many business leaders and even IT operations personnel can be under the misconception that ransomware is an on-premises only problem.  However, this is simply not the case.  Ransomware can and does affect cloud resources.  There are several key points of attack for ransomware affecting public cloud resources.  What are these?

Your organization may be using file synchronization between your on-premises environment and your Software-as-a-Service offering such as Google G Suite and Microsoft Office 365 cloud-based storage.  If ransomware infects files on-premises and these changes are synchronized to your public cloud storage, your files in the cloud are now encrypted as well.

There are other attack vectors and services that ransomware can affect.  Kevin Mitnick, a well-known hacker turned “white hat” hacker recently demonstrated the dangers of what he referred to as “Ransomcloud” where ransomware can effectively be used to encrypt cloud-based email such as Microsoft’s Exchange Online offering that is part of the Office 365 public cloud.

Cloud Backups – Eggs in One Basket Effect

Some companies may make the great decision to start backing up public cloud environments but chose a backup solution that houses the backup of their cloud environment in the same environment as the production environment.  This is placing all your “eggs in one basket” and can be very dangerous to do. 

When looking for a cloud backup solution, you want to look for a vendor and solution that is able to backup your cloud data and store that data in different storage than the public cloud being protected, even in a different public cloud.  In this way you are diversifying your data so that there is little risk that you would lose both production as well as your backups.

Best Practice Cloud Backup Methodologies

When thinking about how to design backups of your public cloud environments such as public cloud Software-as-a-Service (SaaS) offerings, it is important to think about general backup best practice methodologies.  While there are certainly differences between protecting data on-premises vs the cloud, many aspects are quite similar. 

A longstanding backup best practice methodology regarding on-premises backups is the 3-2-1 backup rule.  What is this?  It states that you need to have at least (3) backup copies of your data stored on at least (2) different types of media, with at least (1) copy stored offsite.

The point of this methodology, again, is diversifying your data.  This means that you are lessening the chances of losing ALL copies of both backups and production data.  The more copies you have of your data, stored on different forms of media, and one stored in a completely different site altogether, the chances of losing all of these copies is next to impossible. 

Continuing this same line of thinking with your cloud data, you want to ensure you have more than one copy of your production data.  This by default means that you are going to want to have backups.  Along with backups, are those backups stored in the same environment?  As mentioned above, you don’t want to have both your production data and backup data exist in the same public cloud environment. 

Many cloud-to-cloud backup solutions that are available may have great features, however, often you will find they will only allow storing backups of the public cloud being protected in that same public cloud.  As demonstrated by the AWS data loss event over the Labor Day weekend, this is not a safe way to protect your data. 

In addition to being able to diversify your data, you want to have the following capabilities when it comes to being able to protect your data and services in the cloud:

• Automatic backups
• Multiple restore points with effective versioning
• Ability to restore alternative versions of files and data
• Ability to migrate data between different user accounts in the case of G Suite or Office 365
• Ability to download data from the cloud backups
• Restore deleted items
• Advanced reporting and search for backups that are taken from the cloud
• Retention control
• Secure backups that are encrypted both in-flight and at-rest

The above are some of the major features when you look at backups taken of cloud environments that you would most likely want to have to ensure data is properly protected and have the ability to restore data in the way you need.

There are many cloud-to-cloud backup solutions out on the market and several that would most likely fit the bill for protecting your data.  There are a few that I have used and recommend for protecting cloud environments.  One of these is Spinbackup which offers solutions for both G Suite and Office 365 environments.  It provides a unique approach to not only protecting your data but also in securing your environment from security threats like ransomware.  As ransomware is one of the primary culprits these days with data loss, having your cloud environment protected from ransomware by the data protection solution is brilliant.

Another great feature I like about Spinbackup compared to other products on the market is the ability it possesses to store backups across not only Amazon AWS, but also Google GCP and Microsoft Azure storage.  The data storage features are among the best I have seen on the market today for a cloud-to-cloud backup solution.

Wrapping Up

The cloud is certainly where IT infrastructure is heading.  Most likely, your business is either looking at cloud or already there.  Most IT operations folks have some type of cloud initiatives on the books.  As resilient and fully-featured as cloud environments are, your data is ultimately your responsibility.  The recent event with Amazon AWS over the Labor Day 2019 weekend demonstrates just how important backups of cloud data truly are. 

While it is rare and isolated in nature, losing data in the cloud is certainly possible.  You want to ensure that just as you backup your data on-premises, you also have a backup strategy for your data and services housed in the cloud.  This includes very popular Software-as-a-Service environments such as Google G Suite and Microsoft Office 365.

No matter which backup solution you choose, choose a backup solution that backs up the data and services you are using in the cloud, has the features that you need in a backup solution, and is able to store your backups in a way that you need such as granular restores and migration.  Be sure to check out Spinbackup as it is one of the best ones out there I have used.