2.9 KiB
Cluster backups
All jobs run on arschrock and write beneath /mnt/backup/kubernetes. Each job uses hostPath.type: Directory and checks that its mounted destination is bcachefs before writing. Database and object store credentials are mounted from their existing namespace Secrets. The public GitOps repository contains no credential values.
PostgreSQL
The host job uses the local PostgreSQL Unix socket and peer authentication. In-cluster PostgreSQL jobs run in their database namespace and use that server's service DNS name and existing Secret. Servers with a superuser credential get a globals dump and custom-format dump of every connectable database; instances with app-only credentials dump the configured application database. Dumps are written under a timestamped directory. Each database archive is checked with pg_restore --list, and the complete marker is written only after the full run succeeds. The globals dump omits role password hashes; restore must reset database passwords from the current Secrets. A directory without complete is incomplete and must not be used for restore.
One-time host preparation for PostgreSQL: sudo install -d -o 962 -g 962 -m 0700 /mnt/backup/kubernetes/postgresql/arschrock. UID 962 is the host postgres account. The PostgreSQL backups are not automatically pruned.
CephFS and S3
Each CephFS CronJob mounts one existing CephFS PVC read-only and mirrors it to its own pre-created bcachefs directory with rsync --archive --numeric-ids --inplace --partial --delete. It refuses to run unless /source is a CephFS mount and /backup is bcachefs. These are mirrors, so source deletions and changes propagate to the mirror; they do not provide historical snapshots.
The RGW job syncs all 12 current buckets with rclone sync. It reads Rook-generated per-bucket object user Secrets in rook-ceph; the nextcloud-new bucket uses the existing nextcloud object user. The destination is a local mirror, and remote deletions propagate. The first complete CephFS and S3 runs are needed before declaring these data sets backed up. The combined S3 run has a 48-hour deadline because the buckets total roughly 527 GB; its CronJob forbids overlapping runs.
Schedules and alerts
PostgreSQL runs daily between 02:15 and 06:30 Europe/Vienna; CephFS mirrors run between 07:00 and 12:30; RGW starts at 13:00. CronJobs forbid overlapping runs of the same backup. Prometheus alerts on failed backup Jobs, daily backup CronJobs without a success in 25 hours, and the S3 mirror without a success in 48 hours. A missing /mnt/backup mount fails the pod setup or the in-container filesystem check and therefore cannot write into the underlying directory.
Valkey/Redis persistence is not included yet. etcd and RBD remain deferred. Existing files and old scripts directly under /mnt/backup are preserved. No restore has been performed; the PostgreSQL archive checks validate archive readability, not a full restore rehearsal.