# Cluster backups

All jobs run on arschrock and write beneath `/mnt/backup/kubernetes`. Each job uses `hostPath.type: Directory` and checks that its mounted destination is bcachefs before writing. Database and object store credentials are mounted from their existing namespace Secrets. The public GitOps repository contains no credential values.

## PostgreSQL

The host job uses the local PostgreSQL Unix socket and peer authentication. In-cluster PostgreSQL jobs run in their database namespace and use that server's service DNS name and existing Secret. Servers with a superuser credential get a globals dump and custom-format dump of every connectable database; instances with app-only credentials dump the configured application database. Dumps are written under a timestamped directory. Each database archive is checked with `pg_restore --list`, and the `complete` marker is written only after the full run succeeds. The globals dump omits role password hashes; restore must reset database passwords from the current Secrets. A directory without `complete` is incomplete and must not be used for restore.

One-time host preparation for PostgreSQL: `sudo install -d -o 962 -g 962 -m 0700 /mnt/backup/kubernetes/postgresql/arschrock`. UID 962 is the host `postgres` account. The PostgreSQL backups are not automatically pruned.

The Mumble PostgreSQL backup uses CloudNativePG in the `mumble` namespace and
writes to `/mnt/backup/kubernetes/postgresql/mumble`. Its daily 04:00
Europe/Vienna CronJob runs on `arschrock`, reads the generated
`mumble-postgresql-superuser` Secret, and backs up globals plus every
connectable database. A completed run on 2026-10-25 was verified. The retired
`default-mumble` backup schedule was removed after cutover; its prior files, if
any, remain untouched on bcachefs.

## CephFS and S3

Each CephFS CronJob mounts one existing CephFS PVC read-only and mirrors it to its own pre-created bcachefs directory with `rsync --archive --numeric-ids --inplace --partial --delete`. It refuses to run unless `/source` is a CephFS mount and `/backup` is bcachefs. These are mirrors, so source deletions and changes propagate to the mirror; they do not provide historical snapshots.

The RGW job syncs all 12 current buckets with `rclone sync`. It reads Rook-generated per-bucket object user Secrets in `rook-ceph`; the `nextcloud-new` bucket uses the existing `nextcloud` object user. The destination is a local mirror, and remote deletions propagate. The first complete CephFS and S3 runs are needed before declaring these data sets backed up. The combined S3 run has a 48-hour deadline because the buckets total roughly 527 GB; its CronJob forbids overlapping runs.

## Schedules and alerts

PostgreSQL runs daily between 02:15 and 06:30 Europe/Vienna; CephFS mirrors run between 07:00 and 12:30; RGW starts at 13:00. CronJobs forbid overlapping runs of the same backup. Prometheus alerts on failed backup Jobs, daily backup CronJobs without a success in 25 hours, and the S3 mirror without a success in 48 hours. A missing `/mnt/backup` mount fails the pod setup or the in-container filesystem check and therefore cannot write into the underlying directory.

Valkey/Redis persistence is not included yet. etcd and RBD remain deferred. Existing files and old scripts directly under `/mnt/backup` are preserved. No restore has been performed; the PostgreSQL archive checks validate archive readability, not a full restore rehearsal.