Fix alert recovery, bound backup job history and expand data volumes

docs/alert-recovery.md

@@ -0,0 +1,27 @@
+# Alert recovery
+
+Backup success is the CronJob's `lastSuccessfulTime`, not the continued
+existence of an older failed Job. Backup rules must match CronJob info and
+success metrics explicitly on `(namespace,cronjob)`; info has extra schedule
+labels. Test both recovery and real failures with
+`python3 tests/run-alert-recovery.py` (PyYAML and promtool or Podman required).
+
+Backup Job metadata and pod logs expire one day after completion or failure.
+This does not delete backup files. Stale-backup rules continue to alert when
+successful backups are missing, even after failed Job records expire.
+During recovery, retire old failed Jobs only after confirming a newer success
+for their owning CronJob. Do not mark a backup successful manually.
+
+The CPU-capacity alert preserves the upstream total-capacity check for a
+non-HA control plane and applies largest-node failure capacity only to HA
+control planes. Requests remain based on the review in `cpu-sizing.md`.
+This does not promise failover for workloads pinned to arschrock.
+
+On odroid-h3-04, persistent journald storage is bounded using
+`bootstrap/host-config/journald-retention.conf`, installed as
+`/etc/systemd/journald.conf.d/homelab-retention.conf`. This host configuration
+requires installation through SSH; Argo does not manage host systemd files.
+The node's small system disk also holds active container images, so deleting
+unused images alone may free nothing. Never remove Ceph monitor database files
+to make space. Ceph crash reports may be archived after reviewing the cause
+and verifying it has stopped; archival retains the reports for investigation.