#Monitoring
Login watch, and the WAF again
An alert when login attempts come from an address never seen to succeed, without noise from 5G; dashboards fixed; and a night when the WAF was killed 139 times.
Security audit, fleet updates and email
A black-box and grey-box audit, schedulable updates for the whole fleet with vulnerability watch, readable logs, and emails that finally get past the filters.
ADR 0023: Fleet updates and security watch
Each machine publishes its update status, a nightly watch compares versions and known vulnerabilities, and maintenance is launched in two clicks in Semaphore (announcement, then scheduled update).
ADR 0020: Centralised logs with VictoriaLogs, agentless collection
Logs from every machine reach VictoriaLogs on the monitoring VM through mechanisms already in place (journald, syslog), and can be searched in Grafana.
ADR 0014: Monitoring with Prometheus, a custom public NOC behind SSO
Prometheus collects, Alertmanager notifies by email and on the phone (ntfy), and a custom page publishes a NOC viewable after authentication.
Monitoring, a custom NOC and automation
The homelab monitors itself, sends alerts to the phone, shows its status on a home-made NOC and is driven from Semaphore.
Runbook: responding to an alert
Where alerts arrive, and the first steps for each one.