Leipzig University has been offline for days. What about you?

Leipzig University took its IT off the network after an attack. Days later, central services were still unreachable, and there was no date for when they would be back. That is the part that hurts in day to day operations. Not the attack itself, but the week afterwards, when nobody can say when what will run again.

The question for you is not whether your firewall holds. The question is what happens when it does not. And you write that answer down beforehand, not at half past six on a Saturday morning.

Three services, one order

Take half an hour and write down which three services keep your business running. At a metal fabricator with 60 people, that is typically the ERP system with the orders, the file storage with the drawings, and email. You do not need more than three to start with, and fewer is not enough.

Now the uncomfortable part. Those three will not come back at the same time. If everything has to be rebuilt, you work through them one after another, and you set the order now, not during the incident. For the metal fabricator that might mean file storage first, because otherwise the machines stand still, then the ERP system, and email last, because people can get by on the phone for two days. For a wholesaler the order is exactly the other way round, because without incoming orders nothing happens at all.

Each of the three services comes with two numbers that you set yourself. How old may the state be that you restore, one hour or one day? And how long may it take until it is back, four hours or three days? Both numbers cost money when they are small. That is why you set them per service and not as a blanket rule for everything.

Who decides on Saturday morning

In a real incident there are decisions no piece of technology can make. Do we take production off the network? Do we inform customers today or only once we know more? Do we restore last night's state and lose a day of bookings, or do we wait another two hours for an analysis?

A human being decides that, and that human has a name. Write it down, plus a deputy, plus both phone numbers. On paper, because if the file storage is gone, the emergency list inside it is gone too. The same logic applies to the technical side. Monitoring may report and propose, a major version jump or a rollback is approved by a human. Automation that intervenes in a running incident without approval makes situations worse, not better.

How to spot a backup that is not there

A backup that has never been restored is a claim. Among administrators this is a running theme, and the recurring point in discussions about backup tools is always the same. A tool is judged not by the backup but by the restore, and that is exactly the part that hardly anyone practises regularly.

Three checks you can start without deep technical knowledge:

• Get one file back. Name any file from last week and have it pulled out of the backup. If that takes longer than an hour, or if someone has to look up how it works first, you know where you stand.
• Have a whole service brought up on replacement hardware. Once a year, with a stopwatch running. The time that comes out of it is your real number. The one in the quote is not.
• Ask who can delete the backups. If the account an attacker uses to encrypt your servers can also reach the backups, you do not have any. One copy has to sit off site and be unreachable or unchangeable for the running system.

Three copies, two different media, one off site. That is old, unspectacular and still holds.

Us, one example from the plan

One point from our own recovery plan, because it is rarely written down anywhere. The first service we restore is the password safe. Without it nobody gets to the credentials for everything else, and the whole order would be blocked. For that, two passwords sit on paper in two places, and the safe itself is backed up along the pattern of three copies, two media, one off site. On the same infrastructure run our cloud, mail server, Git server and this website. Check which service plays that role in your company. It is almost never the one you name first.

Three ways, not two

If you are now noticing that you do not have the answers, there are three ways, and they are often reduced to two.

You can do it yourself. For that you need someone in house who checks backups, installs updates and is reachable at night. You can rent a service per head per month. Then recovery sits with the provider, and you read along in their status updates, the way Leipzig University is currently being watched from the outside.

In between lies the third way. Open building blocks run on a server that belongs to you, maintained by someone who is paid for it. The data stays with you, the service provider is replaceable, and on call duty is in the contract. That is the category we work in.

Sovereignty is not an all or nothing question here. You decide how deep it goes. A cloud AI for text suggestions is a legitimate choice, a local model on your own hardware is too, and it costs more. Anyone publishing on Instagram sends data to Instagram, and no server location changes that. The difference lies in whether you can switch providers without losing your data.

Start small. Three services, one order, two numbers per service, one name on paper. That is half an hour of work and the difference between a bad Saturday and a bad week.