For secure data backup, here’s how to do the 3-2-1 rule right

As the number of places where we store data increases, the basic concept of what is referred to as the 3-2-1 rule often gets forgotten. This is a problem, because the 3-2-1 rule is easily one of the most foundational concepts for designing data protection. It’s important to understand why the rule was created, and how it’s currently being interpreted in an increasingly tapeless world.

What is the 3-2-1 rule for backup?

The 3-2-1 rule says there should be at least three copies or versions of data stored on two different pieces of media, one of which is off-site. Let’s take a look at each of the three elements and what it addresses.

• 3 copies or versions: Having at least three different versions of your data over different periods of time ensures that you can recover from accidents that affect multiple versions. Any good backup system will have many more than three copies.
• 2 different media: You should not have both copies of your data on the same media. Consider, for example, Apple’s Time Machine. You can fool it using Disc Utility to split your hard drive into two virtual volumes, and then use Time Machine to backup the first volume to the “second” volume. If the primary drive fails, the backup will fail as well. This is why you always have the backup on different media than the original.
• 1 backup off-site: A speaker at a conference once said he didn’t like tapes because he put them in a box on top of a server, and they melted when the server caught fire. The problem wasn’t tape; the problem was he put his backups on top of his server. Your backup copies, or at least one version of them, should be stored in a different physical location than the thing you are backing up.

Mind the air gap

An air gap is a way of securing a copy of data by placing it on a machine on a network that is physically separate from the data it is backing up. It literally means there is a gap of air between the primary and the backup. This air gap accomplishes more than simple disaster recovery; it is also very useful for protecting against hackers.

If all backups are accessible via the same computers that might be attacked, it is possible that a hacker could use a compromised server to attack your backup server. By separating the backup from the primary via an air gap, you make it harder for a hacker to pull that off. It’s still not impossible, just harder.

Everyone wants an air gap. The discussion these days is how to accomplish an air gap without using tapes.Back in the days of tape backup, it was easy to provide an air gap. You made a backup copy of your data and put it in a box, then you handed it to an Iron Mountain driver. Instantly, there was a gap of air between your primary and your backup. It was close to impossible for a hacker to attack both the primary and the backup.

That is not to say it was impossible; it just made it harder. For hackers to attack your secondary copy, they needed to resort to a physical attack via social engineering. You might think that tapes stored in an off-site storage facility would be impervious to a physical attack via social engineering, but that is definitely not the case. (I have personally participated in white hat attacks of off-site storage facilities, successfully penetrated them and been left unattended with other people’s backups.) Most hackers don’t resort to physical attacks because they are just too risky, so air-gapping backups greatly reduces the risk that they will be compromised.

Faulty 3-2-1 implementations

Many things that pass for backup systems now do not pass even the most liberal interpretation of the 3-2-1 rule. A perfect example of this would be various cloud-based services that store the backups on the same servers and the same storage facility that they are protecting, ignoring the “2” and the “1” in this important rule.

For example, it is very common for customers of public cloud vendors to backup their systems by creating snapshots/images of the resources they are using. The images are typically stored in object storage in the same account that is running the primary systems. If hackers gain privileged access, they could easily delete both the primary and all secondary copies of the data. The 3-2-1 rule still applies to the cloud. Keep a copy somewhere else – in a different account, in a different availability zone – just keep it somewhere else.

The 3-2-1 rule is also ignored by a lot of people using hundreds of SaaS services. Consider, for example, the advent of Kubernetes and the reality that many people store their Kubernetes configuration in GitHub. Important backups are stored in a system that you may or may not be backing up. Consider other services like email providers or filesharing services where even the primary copy of your data is stored only in a third-party vendor’s platform. The backups in many of these services are simply additional copies of data in the same location. Be sure to ask your vendors how they would help you recover if your entire account was hacked by a third-party.

What about electronic air gaps?

A purist would say that the only way to have a true air gap is to put backups on removable media such as tape and then physically separate them from the primary. Others acknowledge that many companies have moved on from tape as a protection mechanism and might use it only for long-term storage if they use it at all. The question is how to make sure a hacker can’t access the primary and the secondary via an electronic hack.

The current best answer is to separate these two copies in as many ways as possible. Consider doing as many of the following as you can:

• Different storage – Use a different storage type than what you use for your primary storage. An attack designed for one will probably not work on another.
• Different environment – Use a backup system that isn’t directly reachable via your LAN.  That’s another way to prevent compromised on-prem servers from attacking your backups.
• Different OS – Use a backup server or service that runs on an OS other than Windows can go a long way. Most ransomware attacks have been against Windows.
• Different account – As much as possible, use completely different credentials in your backup and disaster-recovery systems. That way if an account is compromised, the credentials won’t work to attack your backups.
• Immutable storage – Some cloud vendors offer immutable storage, where backups sent there cannot be changed or deleted until the time you specify.  Even you can’t delete them.

The 3-2-1 rule is a good rule that has served the data-protection world well for a long time. Always ask how well you are complying with it; it just might save your bacon one day.