Server outage

The dedicated server hosting the Iceshrimp infrastructure has experienced a catastrophic disk failure.

All data is safe, and we're working to restore service as quickly as possible.

Apologies for any inconvenience caused.

- zotan

Incident timeline

2026-05-10 03:51 CEST - Server crashes due to disk failure, all services go down.
2026-05-10 11:00 CEST - Investigation starts. Disk failure identified as cause of the downtime.
2026-05-10 11:51 CEST - Hosting provider has reconnected the affected drive and it appears to be working again, an offsite backup is running.
2026-05-10 12:04 CEST - The backup has completed, and all data is safe, even if the drive dies fully now.
2026-05-10 12:12 CEST - Steps are being taken to ensure a single disk failure cannot take down the server in the future.
2026-05-10 12:38 CEST - The file system is being migrated to ZFS, as I have lost trust in the crash safety of BTRFS.