# Alert recovery

Backup success is the CronJob's `lastSuccessfulTime`, not the continued
existence of an older failed Job. Backup rules must match CronJob info and
success metrics explicitly on `(namespace,cronjob)`; info has extra schedule
labels. Test both recovery and real failures with
`python3 tests/run-alert-recovery.py` (PyYAML and promtool or Podman required).

Backup Job metadata and pod logs expire one day after completion or failure.
This does not delete backup files. Stale-backup rules continue to alert when
successful backups are missing, even after failed Job records expire.
During recovery, retire old failed Jobs only after confirming a newer success
for their owning CronJob. Do not mark a backup successful manually.

The CPU-capacity alert preserves the upstream total-capacity check for a
non-HA control plane and applies largest-node failure capacity only to HA
control planes. Requests remain based on the review in `cpu-sizing.md`.
This does not promise failover for workloads pinned to arschrock.

On odroid-h3-04, persistent journald storage is bounded using
`bootstrap/host-config/journald-retention.conf`, installed as
`/etc/systemd/journald.conf.d/homelab-retention.conf`. This host configuration
requires installation through SSH; Argo does not manage host systemd files.
The node's small system disk also holds active container images, so deleting
unused images alone may free nothing. Never remove Ceph monitor database files
to make space. Ceph crash reports may be archived after reviewing the cause
and verifying it has stopped; archival retains the reports for investigation.