|
|
|
@@ -1,11 +1,53 @@
|
|
|
|
|
# Mumble cutover and recovery
|
|
|
|
|
# Mumble
|
|
|
|
|
|
|
|
|
|
The published `mumble` chart owns the application and its optional Bitnami PostgreSQL subchart. This directory contains only cluster-specific values. The initial Argo sync used `mumble.enabled=false` to bring up an independent `mumble-postgresql` Service/StatefulSet/PVC while the legacy Mumble Deployment kept using the shared database. After a final dump/restore with Mumble stopped, `mumble.enabled=true` adopted the existing Service and Deployment. The old shared database and its PVC were not changed or removed.
|
|
|
|
|
Argo CD deploys Mumble and its CloudNativePG cluster in the `mumble` namespace.
|
|
|
|
|
The published `brunner.ninja/mumble` chart owns the workload, TCP/UDP Services,
|
|
|
|
|
probes, configuration rendering, and optional PostgreSQL modes. GitOps selects
|
|
|
|
|
the chart version and supplies this cluster's addresses, registration details,
|
|
|
|
|
Secret names, and PostgreSQL endpoint.
|
|
|
|
|
|
|
|
|
|
The final dump is retained outside Git at `/home/feedc0de/migration-backups/mumble-2026-10-19/final-cutover.dump` on `arschrock` (mode 0600). All 14 Mumble table row counts matched after restore, including 33 users and 18 channels. The current `default/mumble` Deployment uses `mumble-postgresql.default.svc.cluster.local`. Its `Recreate` strategy avoids two server processes using the same database. If rollback is needed, freeze writes again and account for any writes made in the new database; blindly pointing Mumble back at the old copy would lose them.
|
|
|
|
|
## PostgreSQL and credentials
|
|
|
|
|
|
|
|
|
|
`default/mumble-secrets` contains the application's database, SuperUser, and registry passwords. `default/mumble-postgres` is generated once by the source repository's guarded bootstrap helper: a random PostgreSQL administrator password plus the existing Mumble database password. Neither Secret is in this public repository. Preserve and back up these Secrets with the new PostgreSQL PVC. The certificate remains `default/brunner-ninja-tls`, managed separately by cert-manager. No Mumble PVC exists; durable Mumble state is in PostgreSQL.
|
|
|
|
|
`mumble-postgresql` is a one-instance CloudNativePG cluster using PostgreSQL
|
|
|
|
|
18.6 and an 8 GiB Ceph RBD claim. Mumble connects to
|
|
|
|
|
`mumble-postgresql-rw.mumble.svc.cluster.local`. The `mumble-secrets` Secret in
|
|
|
|
|
this namespace supplies both Mumble's application credentials and the
|
|
|
|
|
CloudNativePG bootstrap username/password. It was copied from the prior
|
|
|
|
|
`default/mumble-secrets` by the guarded source-repository helper; neither
|
|
|
|
|
Secret's values are stored in this public repository. The TLS Secret
|
|
|
|
|
`brunner-ninja-tls` is produced by the namespace-local Certificate and
|
|
|
|
|
`letsencrypt-dns` ClusterIssuer.
|
|
|
|
|
|
|
|
|
|
The Service retains TCP/UDP port 64738 and the MikroTik operator annotations. Kubernetes/MetalLB should allocate the LoadBalancer address; the operator follows it. The old `192.168.0.14` pin is not a GitOps value. Do not delete the Service to change addressing. During a PostgreSQL restore, pause Argo auto-sync or scale the Mumble Application only with a recorded recovery plan, then resume once data is consistent.
|
|
|
|
|
## Networking
|
|
|
|
|
|
|
|
|
|
The `service.additionalServices.funkfeuer` chart value creates `mumble-funkfeuer`, exposing TCP and UDP 64738 on `10.12.4.170` (Funkfeuer/OLSR) and `192.168.0.2` (the `arschrock` LAN address). It uses cluster-wide endpoint routing to reach the ready Mumble pod even when that pod is on an Odroid. The original `mumble` LoadBalancer Service retains its `Local` traffic policy, LAN address, and MikroTik forwarding. Funkfeuer clients need an OLSR route to `10.12.4.170`; both direct entry points depend on `arschrock` being online.
|
|
|
|
|
The LoadBalancer Service exposes TCP/UDP 64738 and retains the MikroTik
|
|
|
|
|
operator annotations; Kubernetes dynamically allocated `192.168.0.14` after
|
|
|
|
|
the move. Do not pin that address. The chart's `mumble-funkfeuer` Service keeps
|
|
|
|
|
the direct addresses `10.12.4.170` and `192.168.0.2`. Both Services select the
|
|
|
|
|
Mumble Pod in this namespace.
|
|
|
|
|
|
|
|
|
|
## Migration and recovery
|
|
|
|
|
|
|
|
|
|
On 2026-10-25, the old Mumble Deployment was stopped before the final database
|
|
|
|
|
dump. The final custom-format dump is at
|
|
|
|
|
`/mnt/backup/mumble-migration-20261025T065619Z/final-cutover.dump` on
|
|
|
|
|
`arschrock` (mode 0600), and its archive was validated with `pg_restore --list`.
|
|
|
|
|
All 14 application table counts matched after restore, including 34 users,
|
|
|
|
|
18 channels, and 6,536 log rows. The server was then started against the new
|
|
|
|
|
database. The pre-cutover snapshot is in the same directory.
|
|
|
|
|
|
|
|
|
|
The first complete scheduled-format backup was also verified on 2026-10-25:
|
|
|
|
|
`/mnt/backup/kubernetes/postgresql/mumble/2026-10-25T06-59-24Z/`. The Job ran
|
|
|
|
|
on `arschrock`, wrote a `complete` marker, and its database archive passed
|
|
|
|
|
`pg_restore --list`. The daily backup CronJob refuses to write unless the
|
|
|
|
|
`/mnt/backup` host mount is bcachefs.
|
|
|
|
|
|
|
|
|
|
The old `default/data-mumble-postgresql-0` PVC is retained for rollback (UID
|
|
|
|
|
`6d0c591f-9aae-4932-8f94-ebf5ec1032a8`, PV
|
|
|
|
|
`pvc-6d0c591f-9aae-4932-8f94-ebf5ec1032a8`). Its StatefulSet is retired after
|
|
|
|
|
the new cluster and backup were verified; do not delete this PVC or its PV
|
|
|
|
|
without an explicit recovery decision. The old `default/mumble-secrets` and
|
|
|
|
|
`default/mumble-postgres` Secrets are likewise retained as cold rollback
|
|
|
|
|
material. To roll back, quiesce the new Mumble writer, preserve a fresh dump of
|
|
|
|
|
new writes, then restore/reconnect the old database before starting the old
|
|
|
|
|
server. Never run both Mumble servers as writers at once.
|
|
|
|
|
