Disaster Recovery Runbook

When something is on fire, this is the doc you open. Keep it current.

TL;DR (the first 60 seconds)

1. Check /status — is the platform up? If yes, the issue is local.
2. Check /api/readyz — Supabase, /home, Pods.
3. Check /dashboard/box — CPU, memory, swap.
4. Check pm2 logs hostkingpro --lines 200 — recent errors.
5. Check Cloudflare — is the origin server reachable?

If you got here, you're past triage. Continue reading for the procedures.

RPO / RTO targets

• RPO (Recovery Point Objective): 6 hours for Supabase data (nightly pg_dump), 1 hour for site assets (filesystem snapshot taken every hour).
• RTO (Recovery Time Objective): 4 hours for full platform restore from S3 backups. 30 minutes for a single tenant restoration.

Failure modes

Mode 1: One site is broken

Symptoms: 502 from a specific hostname, /dashboard/sites/[id]/logs shows app crash, systemd unit is in failed state.

Procedure:

# 1. Find the tenant home
ls /home/hk_usr_<system_username>

# 2. Check systemd status
sudo systemctl status hostking-<system_username>.service

# 3. If the unit is in failed state, restart it
sudo systemctl reset-failed hostking-<system_username>.service
sudo systemctl start hostking-<system_username>.service

# 4. If the unit keeps failing, check the journal for the actual error
sudo journalctl -u hostking-<system_username>.service -n 200 --no-pager

# 5. If the app is fundamentally broken, redeploy from the last good git ref
hostking redeploy <site_id>

Mode 2: One Pod is unhealthy

Symptoms: All sites on a Pod return 502, Pod's /status page fails, scheduler marks it draining.

Procedure:

# 1. Identify the Pod
supabase db query "SELECT id, pod_identifier, status FROM pods WHERE status != 'active'"

# 2. Move sites off the unhealthy Pod to a healthy one
#    The Pod scheduler should do this automatically. If it doesn't:
supabase db query "UPDATE site_assets SET pod_id = (SELECT id FROM pods WHERE status = 'active' LIMIT 1) WHERE pod_id = '<unhealthy-pod-id>'"

# 3. Mark the unhealthy Pod as draining (so no new sites land there)
supabase db query "UPDATE pods SET status = 'draining' WHERE id = '<unhealthy-pod-id>'"

# 4. The Pod scheduler will continue to migrate sites off. Watch the audit log:
hostking audit --target pod --target-id <pod-id> --days 1

Mode 3: Database is corrupted

Symptoms: supabase db query returns 500, deploys are stuck in queued, /api/readyz reports supabase.ok = false.

Procedure:

# 1. Check the Supabase status page first. If Supabase itself is down, wait.
#    https://status.supabase.com

# 2. If the issue is local to our project (ref: ofqmkkydnoagxryymozv),
#    we have nightly backups at s3://hostkingpro-backups/supabase/

# 3. Restore from the latest backup. Find the most recent:
aws s3 ls s3://hostkingpro-backups/supabase/ | tail -5

# 4. Restore (this drops the project and re-creates it from the dump):
LATEST=$(aws s3 ls s3://hostkingpro-backups/supabase/ | tail -1 | awk '{print $4}')
aws s3 cp s3://hostkingpro-backups/supabase/$LATEST /tmp/restore.sql.gz
gunzip /tmp/restore.sql.gz
# Pipe into psql — see the Supabase docs for the connection string
psql "$(cat ~/.config/supabase/db-url)" < /tmp/restore.sql

Mode 4: The entire box is gone

Symptoms: SSH doesn't respond, the box is unreachable.

Procedure:

1. Spin up a fresh EC2 instance with the same AMI.
2. Run ./infra/scripts/provision-new-box.sh (this is a script that doesn't exist yet — add it to Phase 4 backlog).
3. Restore from S3 backups (the box will pull down everything during the provision step).
4. Update DNS (we use Cloudflare; the script does this automatically).
5. Verify all sites are back up via the smoke test: ./scripts/smoke-test.sh.

Postmortem template

After every incident, write a postmortem in docs/postmortems/<date>-<short-name>.md. Use this template:

# <Date>: <One-line summary>

## Impact

How many users were affected? For how long? What was the blast radius?

## Timeline

- HH:MM — first alert fired
- HH:MM — on-call paged
- HH:MM — root cause identified
- HH:MM — mitigation applied
- HH:MM — full restore

## Root cause

One paragraph. No jargon. Pretend you're explaining it to a customer.

## What went well

- The dashboards were already up-to-date
- We had recent backups
- The on-call rotation worked

## What went poorly

- The alert fired 10 minutes after the actual outage
- We didn't have a runbook for this exact failure mode
- The fix took longer than it should have

## Action items

- [ ] <Person> — <Action> (ETA: <Date>)
- [ ] <Person> — <Action> (ETA: <Date>)

Contacts

• Cloudflare support: account #TODO (kept in 1Password).
• Supabase support: paid tier, response within 4 business hours.
• AWS support: Business tier, response within 1 hour for production-impact issues.
• On-call rotation: see the team's Slack #oncall channel.

References

• Architecture overview
• Performance audit
• Supabase status
• Cloudflare status