Web host down for two days: what to do now and how to keep backups your provider can't take down with it

At 04:50 Moscow time on November 2, Beget, a Russian hosting provider, posted a short message in its channel: a "major incident". A few hours later it added that the problem affected every layer of its architecture and that engineers were reconfiguring infrastructure by hand inside the data centers. In the afternoon it published an official force majeure notice. By Beget's own count, its cloud servers host more than 450,000 websites.

As of November 3, dedicated servers are back, DNS (the NS servers) is the stated top priority, email has no timeline, and the company won't name a date for everything else. Two of my clients, an online store and a content site, have both their hosting and their domain registration at Beget. Nothing of theirs works yet.

Update, November 3. Beget names a cyberattack as the cause, and the Coordination Center for .RU, which runs the zone, says the same. Replying to customers in its official social media group, Beget says its legal position on services and consequences will be set out after the investigation ends.

Update, November 3, 15:15 Moscow time. Beget published a breakdown of the situation. By its account, the attackers got access to the management layer of the entire architecture and then ran data-wiping scripts. File backups of the hosting and cloud are intact, but customers can't get to them yet: more than 1,000 servers are being checked by hand for hidden vulnerabilities and malicious scripts. By a preliminary estimate, almost all shared hosting sites can be restored. In the cloud (VPS, S3, Kubernetes) all data was lost, and binary VPS backups are gone entirely; file backups of VPS remain. For .ru, .рф and .su domains registered with Beget and running on its DNS, A, MX and TXT records can be edited again. Beget recommends changing every password and access key that may have been stored on its servers.

None of this is a dig at Beget. Any provider can have an outage like this. Below: what to do when your web host goes down, depending on what you still have access to, why the host's own backups barely help in that moment, and how to spend one day building a backup setup that doesn't depend on the provider that just failed.

In short. When your web host goes down, check two things first: who your domain registrar is and where your latest backup lives. If you have a copy outside the failed provider and control of your domain, move now. If you have no copy, or the domain is stuck with the same provider, put up a contact page if you can and wait. For next time, keep a copy at a different provider that your web server can write but cannot delete, and measure how long a restore takes. That is the 3-2-1-1-0 rule, covered below with scripts.

What to do when your web host goes down and stays down

What you do depends on what you can still reach. Start with the domain.

Where your DNS lives and who your registrar is

Run a whois lookup on your domain and look at two fields: the registrar and the name servers.

• Name servers at the failed host, registrar somewhere else. Fixable. Log in to the registrar, point the name servers at a working DNS provider and recreate your records there. You still need somewhere to run the site, but you control the domain.
• Registrar and host are the same company. Beget, for example, is also an accredited registrar for .RU domains, and plenty of hosts elsewhere sell domains too. If their control panel is down, there is nowhere to change the name servers. Even a fresh backup won't help quickly: the site comes up on a new server, and the domain keeps pointing at the old one. That is where my clients are right now. On November 3 Beget brought back DNS record editing for .ru, .рф and .su domains on its name servers, so an A record can now point the site at a new server. Management of international domains is not back yet.

Moving a domain to another registrar is a job for a quiet week. The setup to aim for is simple: domain at one company, hosting at another.

If your domain expires while the registrar is down

Beget's customers are already asking about this one. For .RU domains the registration rules are explicit:

• You get 30 days after expiry. During this priority renewal period only the previous holder can renew, and only through their registrar. After it ends, the registration is cancelled (clauses 4.5 and 4.9).
• You can't switch registrars at that point. A transfer is not allowed once the registration has expired (7.8), and before that a transfer needs an AuthInfo code from the current registrar (7.5).
• The registry can pause the clock. The rules let the Coordination Center suspend the priority renewal period while a registrar is inactive, until it resumes work or the domains move to another registrar (4.5). As of November 3, the Coordination Center said it was helping Beget restore its infrastructure and was ready to take further support measures. Later that day Beget said it had agreed with the Coordination Center that .ru, .рф and .su domains with expiring registrations would not be removed from the zone and would keep working. The Coordination Center has not announced this itself yet.

What to do now: check the expiry date in whois, keep records of every renewal attempt (screenshots, dated support tickets), and if the date has passed, write to both the registrar and the registry. Other zones have their own grace periods, so look up yours before you need it, and renew early: for .RU you can file the renewal up to 60 days ahead (4.4).

Scenario 1: you have a recent backup outside the host

The good case. Rent a VPS or hosting from another provider, set up the stack, restore files and the database dump, test the site by IP or through your hosts file, then switch DNS. Plan around the size of the site. A large store on 1C-Bitrix (a CMS widely used for online stores in Russia) with a catalog of 100,000+ products takes me about one working day to move. A small WordPress site goes faster.

Before you switch DNS, run through forms, cart and checkout on the new server.

Scenario 2: you have the code but no recent database

If the code sits in a developer's Git repository, it redeploys anywhere: it holds no state. A store won't work without its database, because products, orders and customers live there. What you can do is put up a temporary storefront on the same templates: contacts, the main pages, and an inquiry form that sends to email or Telegram. That keeps ad traffic from landing on a connection error while the provider works on recovery.

Scenario 3: you have nothing

Everything lived in one hosting account. Pause your ads first, or the budget keeps paying for an error page. Then put a contact page on any working host, if you control the domain.

You can restore a site from the Wayback Machine, partly. It keeps snapshots of the pages it managed to crawl, with no database, cart or customer accounts. For a content site that is a way to get the text of the main pages back. The old advice to check Google's cache no longer works: Google retired the cache: operator in September 2024.

Why your host's backups don't save you

Because when the host is down, those backups are down with it. In the same notice Beget says file backups exist for the vast majority of shared hosting and cloud resources, but restoring from them "will take a long time" because of the volume of data. So the copies exist, and you get them when your turn comes in the provider's queue.

The cloud is worse off. In its November 3 breakdown Beget says VPS, S3 and Kubernetes data was lost and binary VPS backups are gone entirely. If your site's backup sat in the same provider's S3, it no longer exists.

Engineers call this a single point of failure. When the site, its backups, DNS, email and the domain all sit with one provider behind one login, one big outage cuts you off from all of it at once.

Two related traps:

• Replication instead of backups. A second database server that copies changes in real time saves you when a disk dies. An accidentally dropped table or encrypted data reaches the replica just as fast as any other change.
• The web server can delete its own backups. If the storage key on the production server is allowed to delete, a compromised server means compromised backups. The fix is below.

RPO vs RTO: how much data and time you can afford to lose

RPO and RTO are the two numbers to agree on with whoever runs your site, in place of a vague promise of "reliability": how much data and how much time the business can lose in an outage. Both are defined in NIST's contingency planning guide, SP 800-34.

• RPO (Recovery Point Objective) is the point in time to which data must be recovered after an outage. Put simply, how many recent changes you can live without. Nightly dump, server dies at 6 pm: every order and signup from that day is gone.
• RTO (Recovery Time Objective) is how long recovery can take before it starts hurting the business. That covers everything: download the archive, restore, test, switch DNS. Having a backup says nothing about RTO, and the Beget case shows it: the copies exist, the timeline doesn't.

You shrink RPO by copying more often. PostgreSQL has a sharper tool than a nightly dump: continuous archiving of the write-ahead log (WAL) lets you restore to any point in time after the base backup. Only completed WAL segments get archived, so archive_timeout caps how old unarchived data can get. The docs call a value of about a minute reasonable.

You shrink RTO one way only: restore from a backup on a clean server and time it.

Three common setups side by side:

• Backups inside your hosting account. Unavailable whenever the host is. RPO is whatever the host configured, RTO is whatever the host manages.
• A copy at another provider, protected from deletion. Survives a host outage and a server compromise. RPO depends on how often you copy, RTO on how well you've rehearsed the restore.
• A hot standby (a second server with automatic failover). Shortest downtime, highest cost, and it doesn't protect against deleted data on its own: you still need backups.

The 3-2-1-1-0 backup rule for a website

Veeam extended the classic 3-2-1 rule to 3-2-1-1-0 by adding one immutable copy and zero recovery errors. For an ordinary website or store it looks like this:

1. Three copies. The live site plus at least two backups.
2. Two different media. For example, the host's backups plus object storage.
3. One copy offsite. At a different provider, with a different login and a different contract, for example Amazon S3.
4. One immutable copy. It can't be deleted or overwritten until its retention period ends, even with full access to the web server.
5. Zero errors on restore. Backups get checked by restoring them, on a schedule.

S3 Object Lock: a copy no user can delete

Amazon S3 and many S3-compatible services make backups immutable with Object Lock. Retention comes in two modes, plus legal hold:

• Governance: users with the s3:BypassGovernanceRetention permission can override or remove the lock.
• Compliance: no user can overwrite or delete the object version, including the AWS account root user. The retention period can be extended but not shortened. AWS notes that the only way to delete such an object early is to delete the whole account.
• Legal hold: no expiry date, stays until someone removes it.

Object Lock only works on versioned buckets. If someone uploads a file under the same name, S3 creates a new version and the locked one stays where it is. Once Object Lock is enabled on a bucket, it can't be disabled.

Compliance mode has a flip side: for the whole retention period you can't delete those copies, and you pay to store them. Pick the period to match your RPO and your budget. AWS suggests testing your settings in governance mode before you switch to compliance.

One more consequence: a write-only key can still upload junk, and that junk gets locked for the full period too. Your old backups stay intact, but the storage bill grows, so keep an eye on bucket size as well.

How to protect backups from ransomware

Data also gets lost after a break-in, and a break-in can go unnoticed for a long time: see this WordPress case with backdoors in four layers. Whoever gets into the server gets every key stored on it, including the backup storage key. If that key can delete, ransomware wipes the backups along with the site. So the protection rests on two things:

• A write-only key on the server. An IAM policy that allows s3:PutObject and nothing else. With default retention on the bucket, uploads get locked automatically, so the server needs no retention permissions either.
• Object Lock in compliance mode. Covers the case where a broader key leaks: a locked version can't be deleted before its date.

A separate admin account manages the bucket, and its keys never touch the production server.

The scripts: bucket, write-only key, backup and checks

A working minimum for a MySQL site (WordPress, WooCommerce, Bitrix) with backups in Amazon S3, using AWS CLI v2. Replace everything marked TODO. Treat the scripts as a starting point: run them against a test bucket in governance mode before they touch production data.

Step 1. A bucket with Object Lock. Run this from an admin machine, not from the web server.

# Admin credentials: this profile never goes on the web server
export AWS_PROFILE=backup-admin
BUCKET=example-backups      # TODO: your bucket name
REGION=eu-central-1         # TODO: your region (for us-east-1, drop --create-bucket-configuration)

# Object Lock is enabled at creation; once on, it can't be turned off
aws s3api create-bucket --bucket "$BUCKET" --region "$REGION" \
  --create-bucket-configuration LocationConstraint="$REGION" \
  --object-lock-enabled-for-bucket
# Object Lock already turns versioning on; this makes it explicit and is harmless
aws s3api put-bucket-versioning --bucket "$BUCKET" \
  --versioning-configuration Status=Enabled

# Every new object version is locked for 30 days.
# TODO: start with GOVERNANCE, test, then switch to COMPLIANCE
aws s3api put-object-lock-configuration --bucket "$BUCKET" \
  --object-lock-configuration \
  '{"ObjectLockEnabled":"Enabled","Rule":{"DefaultRetention":{"Mode":"COMPLIANCE","Days":30}}}'

Step 2. Two IAM users: one can only upload, one can only read.

# backup-writer: its key goes on the web server. PutObject only, no delete
cat > writer.json <<'EOF'
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": "s3:PutObject",
    "Resource": "arn:aws:s3:::example-backups/*"
  }]
}
EOF
aws iam create-user --user-name backup-writer
aws iam put-user-policy --user-name backup-writer \
  --policy-name upload-only --policy-document file://writer.json
aws iam create-access-key --user-name backup-writer

# backup-reader: for the age check (step 4) and restores (step 5), kept off the web server
cat > reader.json <<'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::example-backups" },
    { "Effect": "Allow", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::example-backups/*" }
  ]
}
EOF
aws iam create-user --user-name backup-reader
aws iam put-user-policy --user-name backup-reader \
  --policy-name read-only --policy-document file://reader.json
aws iam create-access-key --user-name backup-reader

The secret from create-access-key is shown once. On the web server it goes into its own profile in ~/.aws/credentials as [backup-writer].

Step 3. The backup script on the web server. The database password lives in ~/.my.cnf, not in the script. The archive is encrypted with a public GPG key and the private key stays off the server, so a stolen copy is useless without it.

#!/usr/bin/env bash
set -euo pipefail
# cron runs with a minimal PATH, and aws cli often lives in /usr/local/bin
export PATH=/usr/local/bin:/usr/bin:/bin

BUCKET=example-backups                 # TODO
SITE_DIR=/var/www/example.com          # TODO: site root
DB_NAME=example_db                     # TODO
GPG_RECIPIENT=backup@example.com       # TODO: public key imported on the server
STAMP=$(date -u +%Y-%m-%dT%H-%M-%SZ)
TMP=$(mktemp -d); trap 'rm -rf "$TMP"' EXIT

# --single-transaction: a consistent InnoDB snapshot without locking tables
mysqldump --single-transaction --routines --triggers "$DB_NAME" | gzip > "$TMP/db.sql.gz"

# Skip caches and Bitrix's own backup archives: the CMS rebuilds caches, archives bloat the copy.
# The site is live and files change mid-archive: tar exit code 1 (file changed) is not a failure
tar -czf "$TMP/files.tar.gz" --warning=no-file-changed -C "$SITE_DIR" \
  --exclude='./bitrix/cache' --exclude='./bitrix/managed_cache' --exclude='./bitrix/backup' \
  --exclude='./wp-content/cache' . || [ $? -eq 1 ]

tar -cf - -C "$TMP" db.sql.gz files.tar.gz \
  | gpg --batch --encrypt --recipient "$GPG_RECIPIENT" --trust-model always -o "$TMP/backup.tar.gpg"

# Object Lock buckets reject uploads without a checksum, so set one explicitly
# instead of relying on the CLI version's default.
# backup-writer can upload; it can't delete older copies
aws s3 cp "$TMP/backup.tar.gpg" "s3://$BUCKET/example.com/$STAMP.tar.gpg" \
  --checksum-algorithm CRC32 --profile backup-writer

Run it nightly with cron (crontab -e):

# 03:30 every night; output goes to a log you can read when something breaks
30 3 * * * /opt/backup/backup.sh >> /var/log/site-backup.log 2>&1

Step 4. Check the age of the latest backup. The most dangerous failure is the silent one: the site runs fine, and the backup stopped a week ago because a key was revoked or the disk filled up. Run this check somewhere other than the web server, so it keeps working when the server doesn't.

#!/usr/bin/env bash
set -euo pipefail
export PATH=/usr/local/bin:/usr/bin:/bin

BUCKET=example-backups; PREFIX=example.com/   # TODO
MAX_AGE_H=26                                  # one backup a day plus slack for a slow dump
# Bot token and chat ID live in a separate file with mode 600, not in the script
source /etc/backup-alert.env                  # TODO: TG_TOKEN=... and TG_CHAT=...

alert() {
  curl -sf "https://api.telegram.org/bot$TG_TOKEN/sendMessage" \
    -d chat_id="$TG_CHAT" --data-urlencode text="$1" > /dev/null
}

# A revoked key or no network should alert too, not exit quietly under set -e.
# Contents || `[]` handles an empty prefix, where max_by would fail
LAST=$(aws s3api list-objects-v2 --bucket "$BUCKET" --prefix "$PREFIX" \
  --query 'max_by(Contents || `[]`, &LastModified).LastModified' --output text \
  --profile backup-reader) \
  || { alert "Could not check backups for $PREFIX"; exit 1; }

# An empty bucket is an incident too
if [ -z "$LAST" ] || [ "$LAST" = "None" ]; then alert "No backups at all for $PREFIX"; exit 1; fi

AGE_H=$(( ( $(date +%s) - $(date -d "$LAST" +%s) ) / 3600 ))
[ "$AGE_H" -gt "$MAX_AGE_H" ] && alert "Backup for $PREFIX is $AGE_H hours old" || true

Step 5. Test the restore. This is the "0" in the rule. Once a month, or after big changes, pull the latest backup onto a clean server, decrypt it with the private key, restore it and open the site. Time it from start to a working site: that number is your real RTO.

aws s3 cp s3://example-backups/example.com/<file>.tar.gpg . --profile backup-reader
gpg --decrypt <file>.tar.gpg | tar -xf -          # needs the private key
mysql -e 'CREATE DATABASE example_db_restore'     # a separate database, never production
gunzip -c db.sql.gz | mysql example_db_restore
tar -xzf files.tar.gz -C /var/www/restore          # TODO: test site root

What to ask whoever runs your site, today

Even if your site is up right now, ask four questions:

1. Where are the backups and credentials if the hosting dashboard stops loading? Including the domain registrar, DNS and email.
2. Who notices if the backup fails three nights in a row? And how do they find out.
3. Can someone who gets into the web server delete the backups? With the keys already stored on that server.
4. When did anyone last restore the project from a backup on a clean server, and how long did it take? If never, you don't know your RTO.

How I can help

If your site is down along with your host, or the answers above made you uneasy, I can step in:

• Emergency migration. I bring the site up at another provider from whatever you still have, move the database and files, and switch DNS.
• Backups, done for you. Encrypted copies to external storage with a write-only key and Object Lock, backup age checks with Telegram alerts, and a test restore with a measured RTO.
• An audit. I map your single points of failure: domain, DNS, email, backups, access.

Details are on the website support page. A related story about analytics services that go down too: how I ran backup analytics on Umami while Yandex Metrica was out.

FAQ

What should I do if my web host is down and my backups are with them? Wait for the provider to recover, and meanwhile put up a contact page if you control the domain. Without a copy, there is no recent site to restore. Beget, for example, says file backups exist for the vast majority of resources, but restoring from them will take a long time.

What if my domain expires while my registrar is down? For .RU domains you have 30 days of priority renewal after expiry, renewal goes only through your registrar, and a transfer to another registrar is not allowed during that time. Keep records of your renewal attempts and contact both the registrar and the Coordination Center, which can suspend that period while a registrar is inactive.

Can I restore my website from the Wayback Machine? Partly. It keeps snapshots of the pages it crawled: text and layout, with no database, cart or customer accounts. Google's cache is no longer an option, since the cache: operator was retired in 2024.

What is the difference between replication and backup? A replica repeats every change on the primary almost immediately, including accidental deletes and encryption. A backup is a snapshot from a specific moment that you can go back to after that kind of mistake.

What are RPO and RTO in simple terms? RPO is how much recent data you can afford to lose, such as a day of orders with nightly backups. RTO is how long the site can take to recover before it starts hurting the business.

How do I protect backups from deletion and ransomware? Keep a copy at a different provider, give the web server a write-only key (an IAM policy with only s3:PutObject), and turn on S3 Object Lock in compliance mode: until the retention date, no user can delete a locked copy, including the account root user.