28 lines
1.6 KiB
Markdown
28 lines
1.6 KiB
Markdown
# Alert recovery
|
|
|
|
Backup success is the CronJob's `lastSuccessfulTime`, not the continued
|
|
existence of an older failed Job. Backup rules must match CronJob info and
|
|
success metrics explicitly on `(namespace,cronjob)`; info has extra schedule
|
|
labels. Test both recovery and real failures with
|
|
`python3 tests/run-alert-recovery.py` (PyYAML and promtool or Podman required).
|
|
|
|
Backup Job metadata and pod logs expire one day after completion or failure.
|
|
This does not delete backup files. Stale-backup rules continue to alert when
|
|
successful backups are missing, even after failed Job records expire.
|
|
During recovery, retire old failed Jobs only after confirming a newer success
|
|
for their owning CronJob. Do not mark a backup successful manually.
|
|
|
|
The CPU-capacity alert preserves the upstream total-capacity check for a
|
|
non-HA control plane and applies largest-node failure capacity only to HA
|
|
control planes. Requests remain based on the review in `cpu-sizing.md`.
|
|
This does not promise failover for workloads pinned to arschrock.
|
|
|
|
On odroid-h3-04, persistent journald storage is bounded using
|
|
`bootstrap/host-config/journald-retention.conf`, installed as
|
|
`/etc/systemd/journald.conf.d/homelab-retention.conf`. This host configuration
|
|
requires installation through SSH; Argo does not manage host systemd files.
|
|
The node's small system disk also holds active container images, so deleting
|
|
unused images alone may free nothing. Never remove Ceph monitor database files
|
|
to make space. Ceph crash reports may be archived after reviewing the cause
|
|
and verifying it has stopped; archival retains the reports for investigation.
|