Record Quay namespace cutover and rollback

apps/quay/README.md

@@ -1,36 +1,50 @@
 # Quay registry
 
-The Quay Application consumes the separately published `quay` chart. The
-chart owns Quay's registry and mirror Deployments, Service, Ingress, and
-optional PostgreSQL and Valkey subcharts. These values choose the existing
-hostname, private configuration Secret, database storage class, and existing
-Valkey instance for this cluster.
+The `quay` Argo Application consumes the separately published Quay chart in
+the `quay` namespace. This installation uses a three-instance CloudNativePG
+cluster (`quay-postgresql`, managed by the separate `quay-postgresql` Argo
+Application) and the Quay chart's standalone, persistent Valkey dependency.
+The chart also supports its embedded Bitnami PostgreSQL dependency and an
+external/shared PostgreSQL server for other installations. The registry and
+mirror both use the private `quay/quay-config-next` Secret. CloudNativePG
+generates `quay/quay-postgresql-app`; `quay/quay-valkey` is a cluster-managed
+random password Secret. Neither password nor Quay's encryption/S3 credentials
+is stored in this public repository.
 
-The Rook `CephObjectStoreUser/rook-ceph/quay` is adopted by the same Argo
-Application without changing its name or spec. Its existing `quay` bucket is
-**not** recreated or pruned by Helm or Argo. At migration inventory it had
-5,416 objects and bucket ID
-`e326b22c-970e-4ff6-bf65-499f3bab92bf.900401.2`. Keep the Ceph user Secret,
-bucket, and RGW data during any rollback.
+The Rook `CephObjectStoreUser/rook-ceph/quay` and existing `quay` bucket are
+unchanged. S3 objects were **not copied**: the new Quay uses the same bucket,
+endpoint, credentials, and persistent Quay encryption keys. Redis data was
+not copied from the old shared Valkey because Quay uses it for transient build
+logs/events; copying the shared database would also import unrelated services'
+keys. No build was running at cutover. Quay's durable action logs are in
+PostgreSQL. If an in-flight task was missed, retry it rather than restoring
+the shared cache into this dedicated Valkey.
 
-`default/secret/quay-config` contains Quay's persistent encryption keys and
-points at the former shared PostgreSQL Service. It is retained as rollback
-data. `quay-config-next` contains the same keys and S3 credentials with only
-`DB_URI` changed to `quay-postgresql.default.svc.cluster.local`; neither
-Secret's contents belongs in this public repository. `quay-postgres` holds the
-dedicated PostgreSQL passwords. Restore those Secrets from the encrypted
-administrator backup before restoring this installation in a new cluster.
+On 2026-10-24 both old Quay writers were stopped before the final PostgreSQL
+18.6 dump. The dump was restored into CloudNativePG and verified against the
+source counts (repository 71, tag 572, manifest 1092, imagestorage 5346,
+logentry3 7102). The public UI, registry auth challenge, and an anonymous
+image manifest and blob were tested after the switch. The old
+`default/data-quay-postgresql-0` PVC remains for cold rollback: UID
+`1decb0df-e74d-41fe-bfa5-d3d498175f27`, PV
+`pvc-1decb0df-e74d-41fe-bfa5-d3d498175f27`. The old PostgreSQL workload
+and old Quay workloads/services were removed, but the PVC and old
+`default/quay-postgres` and `default/quay-config-next` Secrets were retained.
 
-The cutover stopped both old Quay Deployments, took a final PostgreSQL dump,
-restored it into `quay-postgresql`, and enabled the registry with
-`quay-config-next`. The final dump is stored outside Git at
-`/home/feedc0de/migration-backups/quay/final-cutover-2026-10-20.dump` on the
-administrator host. The old shared `quay` database and its PVC remain for
-rollback; do not drop or overwrite them until off-cluster backups and actual
-image pulls and pushes have been verified. The old direct Gitea deployment
-writer and its kubeconfig were removed. Argo CD is the only normal writer.
+Local, mode-0600 migration backups are at
+`/home/feedc0de/quay-migration-2026-10-24/` on arschrock:
+`quay-final.dump`, `quay-new-secrets.json`, and `quay-old-secrets.json`.
+The directory is mode 0700. These are **not off-host backups**; copy them to a
+secure off-host backup system before considering the old PVC disposable.
 
-The Argo Application intentionally has pruning disabled. In particular,
-removing the adopted Rook object-user resource from Git must never silently
-delete the credentials to the existing image bucket. Review data-bearing
-resources individually before enabling pruning.
+Rollback must be one writer at a time. Stop the new `quay` and
+`quay-repomirror` Deployments and remove/disable the new Ingress first. Recreate
+the old Bitnami PostgreSQL chart in `default` using the retained PVC and
+`quay-postgres` Secret, then restore the prior Quay chart values and Ingress
+with the old config Secret. If the new registry accepted pushes after cutover,
+dump its current database and migrate those writes back before switching;
+the cold old PVC alone would omit them. Keep the same Ceph bucket throughout.
+
+The Quay Argo Application intentionally has pruning disabled so the adopted
+Rook object user cannot be silently deleted. Review data-bearing resources
+individually before enabling pruning. Argo CD is the only normal writer.