From 3d1eb51b6fa28d5803429933a6ce6d07cdcc160a Mon Sep 17 00:00:00 2001 From: 0xFEEDC0DE64 Date: Sat, 24 Oct 2026 19:03:11 +0200 Subject: [PATCH] Record Quay namespace cutover and rollback --- apps/quay/README.md | 74 +++++++++++++++++++++++++++------------------ 1 file changed, 44 insertions(+), 30 deletions(-) diff --git a/apps/quay/README.md b/apps/quay/README.md index fc2d4f5..73fd0a6 100644 --- a/apps/quay/README.md +++ b/apps/quay/README.md @@ -1,36 +1,50 @@ # Quay registry -The Quay Application consumes the separately published `quay` chart. The -chart owns Quay's registry and mirror Deployments, Service, Ingress, and -optional PostgreSQL and Valkey subcharts. These values choose the existing -hostname, private configuration Secret, database storage class, and existing -Valkey instance for this cluster. +The `quay` Argo Application consumes the separately published Quay chart in +the `quay` namespace. This installation uses a three-instance CloudNativePG +cluster (`quay-postgresql`, managed by the separate `quay-postgresql` Argo +Application) and the Quay chart's standalone, persistent Valkey dependency. +The chart also supports its embedded Bitnami PostgreSQL dependency and an +external/shared PostgreSQL server for other installations. The registry and +mirror both use the private `quay/quay-config-next` Secret. CloudNativePG +generates `quay/quay-postgresql-app`; `quay/quay-valkey` is a cluster-managed +random password Secret. Neither password nor Quay's encryption/S3 credentials +is stored in this public repository. -The Rook `CephObjectStoreUser/rook-ceph/quay` is adopted by the same Argo -Application without changing its name or spec. Its existing `quay` bucket is -**not** recreated or pruned by Helm or Argo. At migration inventory it had -5,416 objects and bucket ID -`e326b22c-970e-4ff6-bf65-499f3bab92bf.900401.2`. Keep the Ceph user Secret, -bucket, and RGW data during any rollback. +The Rook `CephObjectStoreUser/rook-ceph/quay` and existing `quay` bucket are +unchanged. S3 objects were **not copied**: the new Quay uses the same bucket, +endpoint, credentials, and persistent Quay encryption keys. Redis data was +not copied from the old shared Valkey because Quay uses it for transient build +logs/events; copying the shared database would also import unrelated services' +keys. No build was running at cutover. Quay's durable action logs are in +PostgreSQL. If an in-flight task was missed, retry it rather than restoring +the shared cache into this dedicated Valkey. -`default/secret/quay-config` contains Quay's persistent encryption keys and -points at the former shared PostgreSQL Service. It is retained as rollback -data. `quay-config-next` contains the same keys and S3 credentials with only -`DB_URI` changed to `quay-postgresql.default.svc.cluster.local`; neither -Secret's contents belongs in this public repository. `quay-postgres` holds the -dedicated PostgreSQL passwords. Restore those Secrets from the encrypted -administrator backup before restoring this installation in a new cluster. +On 2026-10-24 both old Quay writers were stopped before the final PostgreSQL +18.6 dump. The dump was restored into CloudNativePG and verified against the +source counts (repository 71, tag 572, manifest 1092, imagestorage 5346, +logentry3 7102). The public UI, registry auth challenge, and an anonymous +image manifest and blob were tested after the switch. The old +`default/data-quay-postgresql-0` PVC remains for cold rollback: UID +`1decb0df-e74d-41fe-bfa5-d3d498175f27`, PV +`pvc-1decb0df-e74d-41fe-bfa5-d3d498175f27`. The old PostgreSQL workload +and old Quay workloads/services were removed, but the PVC and old +`default/quay-postgres` and `default/quay-config-next` Secrets were retained. -The cutover stopped both old Quay Deployments, took a final PostgreSQL dump, -restored it into `quay-postgresql`, and enabled the registry with -`quay-config-next`. The final dump is stored outside Git at -`/home/feedc0de/migration-backups/quay/final-cutover-2026-10-20.dump` on the -administrator host. The old shared `quay` database and its PVC remain for -rollback; do not drop or overwrite them until off-cluster backups and actual -image pulls and pushes have been verified. The old direct Gitea deployment -writer and its kubeconfig were removed. Argo CD is the only normal writer. +Local, mode-0600 migration backups are at +`/home/feedc0de/quay-migration-2026-10-24/` on arschrock: +`quay-final.dump`, `quay-new-secrets.json`, and `quay-old-secrets.json`. +The directory is mode 0700. These are **not off-host backups**; copy them to a +secure off-host backup system before considering the old PVC disposable. -The Argo Application intentionally has pruning disabled. In particular, -removing the adopted Rook object-user resource from Git must never silently -delete the credentials to the existing image bucket. Review data-bearing -resources individually before enabling pruning. +Rollback must be one writer at a time. Stop the new `quay` and +`quay-repomirror` Deployments and remove/disable the new Ingress first. Recreate +the old Bitnami PostgreSQL chart in `default` using the retained PVC and +`quay-postgres` Secret, then restore the prior Quay chart values and Ingress +with the old config Secret. If the new registry accepted pushes after cutover, +dump its current database and migrate those writes back before switching; +the cold old PVC alone would omit them. Keep the same Ceph bucket throughout. + +The Quay Argo Application intentionally has pruning disabled so the adopted +Rook object user cannot be silently deleted. Review data-bearing resources +individually before enabling pruning. Argo CD is the only normal writer.