Short answer

A backup is not verified by reading its log, but by restoring it. Some backups are reported as fine by the system for months and then, when the moment comes, cannot take you back to the point you need. Continuity is built in layers: a fast local backup, a backup that a central machine ‘pulls’ and that the machine being backed up cannot delete, and an off-site backup with a restricted key. And a test restore in the calendar.

Key points

  • A backup is verified by restoring it, not by reading its log: some backups the system reports as fine cannot take you back to the point you need.
  • A large disk, a file server or a synchronised folder gives a sense of security that does not survive hardware failure, accidental deletion or an encryption attack.
  • Continuity is built in layers: a fast local backup, a backup that a central machine takes and the backed-up machine cannot delete, and another off-site.
  • Layer zero is being able to rebuild the machine from scratch with versioned procedures and configuration; if that depends on one person's memory, it is not a layer.
  • A weekly integrity check and a monthly test restore, with the date and result recorded, turn the backup from an act of faith into a fact.

A large disk is not a backup

Keeping data on a high-capacity disk, on a file server or in a synchronised folder gives a sense of security that does not survive a hardware failure, an accidental deletion or an attack that encrypts the machine. When a disk starts to fail, the right thing to do is stop writing to it and mount it read-only to rescue whatever can be saved.

An attack that encrypts a machine also encrypts everything that machine can reach. That is why the backup that matters is the one the machine cannot touch: a central station connects and takes the encrypted backup away, and the office machine does not hold that station’s credentials.

Layers, and what happens if

One way to explain backups to non-technical people is through scenarios. A folder is deleted by mistake: it is recovered from the previous day’s local backup. The disk fails: the machine is reinstalled from the configuration repository and the backup from the central station is restored. Fire or theft: a new machine, a rebuild and the data from the cloud in a European region.

Layer zero is being able to rebuild the machine from scratch using versioned procedures and configuration. If that depends on one person’s memory, it is not a layer.

Test restores, and failures reported as they are

A weekly integrity check and a monthly test restore, with the date and result recorded, change the conversation: the backup stops being an act of faith and becomes a fact. The trial runs uncover errors that the log does not show, such as backups marked as failed because a file changed during the backup even though the archive was created correctly.

Distinguishing warnings from failures, and alerting only on real failures, keeps the team from tuning out the alerts. And keeping the previous version of a script before fixing it makes it possible to roll back if the fix causes problems.

Written decisions and procedures with a way back

Every lasting decision deserves a short note: context, decision, exceptions and consequences. For example, that only configuration goes off-site until the client decides about their data, or that the cloud is used only with recorded authorisation. Years later, that note explains why things are the way they are.

A procedure is not ready unless it says how it is verified and how it is reversed. Objective, command, verification and rollback fit on half a page, and that half page is the difference between an operation and a gamble.

Frequently asked questions

How do I know whether a backup actually works?

By restoring it, not by reading its log. Some backups are reported as fine by the system for months and then, when the moment comes, cannot take you back to the point you need. A weekly integrity check and a monthly test restore, with the date and result recorded, turn the backup from an act of faith into a fact. Trial restores uncover errors that the log does not show.

How can we protect our backups from an attack that encrypts our computers?

Keep the backup that matters out of reach of the machine being backed up, because an attack that encrypts a machine also encrypts everything that machine can reach. A central station connects and takes the encrypted backup away, and the office machine does not hold that station's credentials. That layer sits alongside a fast local backup and an off-site backup with a restricted key.

How do we recover data if a folder is deleted by mistake or a disk fails?

Each scenario is covered by a different layer. A folder deleted by mistake is recovered from the previous day's local backup; if the disk fails, the machine is reinstalled from the configuration repository and the backup is restored from the central station. After a fire or theft, you start with a new machine, rebuild it and recover the data from the cloud. If a disk starts to fail, the right thing to do is stop writing to it and mount it read-only to rescue whatever can be saved.

What should an IT operations procedure include to be reliable?

Objective, command, verification and rollback: a procedure is not ready unless it says how it is verified and how it is reversed. All of that fits on half a page, and that half page is the difference between an operation and a gamble. In addition, every lasting decision deserves a short note with context, decision, exceptions and consequences, which years later explains why things are the way they are.