Emergency Rollback Protocol: Step-by-Step Disaster Recovery Checklist When a Live VPS Migration Fails

 

From an unaffected client or monitoring host, distinguish DNS errors from TLS, upstream, web-server and database failures. A 502 Bad Gateway may be an unhealthy upstream rather than a broken DNS change; a 503 may reflect maintenance or capacity controls.

dig +short A example.com @1.1.1.1

dig +short A example.com @8.8.8.8

curl -Ik --connect-timeout 5 https://example.com/

curl -Ik --resolve example.com:443:NEW_VPS_IP https://example.com/

curl -Ik --resolve example.com:443:OLD_VPS_IP https://example.com/

Replace example.com and the IP placeholders. The --resolve option routes that specific curl request to an IP while retaining the hostname for HTTP/TLS; it does not change public DNS. Check expected content and certificate validity, not just HTTP status.

2. Protect data before changing where users write

Record the last known consistent replication position, backup/snapshot ID, and write ownership. On e-commerce or logged-in sites, orders, payments, queued jobs, sessions, uploads, and background writes matter as much as database tables. Preserve logs and snapshots from both machines. Following a detailed migrate website to VPS guide can help administrators plan data transfers, verify database consistency, and reduce the risk of losing critical information during server migration.

Never simply “unlock the old database” while requests may still write to the new database. Doing so creates two writable copies (split-brain). First fence the failed side from accepting writes, confirm replication or reconcile any transactions already received there, and only then designate the old database as the sole writer. If exact reconciliation is not yet possible, favor maintenance mode over accepting conflicting transactions.

For MySQL, read_only and super_read_only are controls to understand, not a universal incident toggle. Privileges, topology, application database users and managed-host restrictions differ. Use the documented failover procedure for your replication design. Analyze binary logs/GTIDs before replay; blind replay can duplicate or corrupt transactions.

3. Revert production routing carefully

If you use a load balancer or reverse proxy, switching its origin to a verified healthy host may be more predictable than waiting for recursive DNS caches. If your cutover changed an A/AAAA record, change those records back to the old VPS IP in the authoritative DNS zone; do not alter the nameservers unless the migration actually changed DNS delegation.

dig +short NS example.com

dig +short A example.com @AUTHORITATIVE_NS

dig +short AAAA example.com @AUTHORITATIVE_NS

dig +short A example.com @1.1.1.1

A DNS revert does not instantly purge cached answers in all recursive resolvers, client devices or active connections. Keep the new VPS safe for residual traffic: proxy it to the selected writer, serve maintenance responses, or preserve a consistent read-only state as your architecture allows. Review CDN proxy settings and AAAA records too.

4. Bring the origin VPS back into service

After verifying data currency and single-writer ownership, check the origin web service, application workers, TLS, disk space and dependencies. On systemd-based Ubuntu/Debian installations, these checks are typical:

sudo systemctl status nginx --no-pager

sudo nginx -t

sudo systemctl status apache2 --no-pager

sudo apachectl configtest

sudo journalctl -u nginx -n 80 --no-pager

sudo journalctl -u apache2 -n 80 --no-pager

sudo tail -n 80 /var/log/nginx/error.log

Use the commands for the web server actually installed; paths and units vary. Reload a valid configuration with sudo systemctl reload nginx (or apache2). Restart only if indicated by a confirmed failure and after accounting for active sessions; avoid blanket service restarts during an unresolved database incident.

5. Decide between rollback and forward-fix

Choose rollback when…	Choose forward-fix when…
Old host is healthy, data is current, and one-writer fencing is proven	New host has a narrow, reversible web configuration or service issue
Critical checkout/login flows fail and old path is recoverable	Returning would discard unreplicated writes or uploads
Recovery procedure has been tested and owners approve	Data reconciliation makes a quick revert more hazardous

6. Validate the recovered production path

·         Check authoritative and multiple recursive DNS answers, while expecting temporary differences.

·         Test home page, authenticated login, checkout/write operation (with an approved test account), API routes, scheduled jobs and emails.

·         Verify the active writer and database replication health. Compare recent order IDs, timestamps, uploads and queue offsets from both sides.

·         Confirm SSL certificate, application logs, 5xx rate, CPU/memory, disk capacity and alerts.

·         Notify stakeholders of the actual restored state; continue monitoring through DNS cache decay.

7. Preserve evidence and document the incident

 

Record the cutover timestamp, first symptoms, abort decision, DNS changes with timestamps, database fencing/reconciliation, service changes, health-check results, and data-loss assessment. Keep snapshots and logs until the investigation is complete. For WooCommerce stores, review WooCommerce sessions and cart data to understand how customer sessions are stored and assess whether active shopping carts were affected during migration. A blameless post-incident review should produce specific changes to the next migration plan, including a tested rollback drill and pre-cutover acceptance criteria.