Homelab
Incidents
Outages and their blameless post-mortems: timeline, root cause, fixes.
Post-mortem: the WAF ran out of memory, then out of disk
A phone's automatic backup got the WAF killed 139 times in one night; every kill left an upload file behind, until the disk filled up and large responses failed on every site.
ReadPost-mortem: the nightly backup cut by the hosting provider's maintenance
A maintenance of the remote storage cut the backup upload and left the restic repository locked; nothing was lost, but the following nights would have failed too.
ReadPost-mortem: two WAF outages in the same morning
An invalid option stopped CrowdSec (1 h outage), then large file uploads exhausted the WAF's memory (25 min); uploads fixed and a new alert added.
ReadPost-mortem: the whole household banned by the WAF
Intrusion tests run from the home network got its public address banned; the whole family received 403s on the sign-in portal.
Read