ci: make rgw-multisite-testing wait for multisite readiness

The rgw-multisite-testing canary job has been chronically flaky
(roughly a third of recent canary runs, including on master and
release branches). Job logs and collected artifacts show two recurring
mechanisms, neither of which is addressed by bumping timeouts:

1. Every recent failure of the "write an object" step died within
   ~16s: the very first "s3cmd mb" against the primary zone got
   ECONNREFUSED three times across the 10s retry window. The RGW
   process never crashed; RGWs pause their HTTP frontends to reload
   the realm whenever the RGW configuration period changes ("rgw
   realm reloader: Pausing frontends for realm update..."), which
   keeps happening shortly after the second zone joins the zonegroup,
   exactly when the test starts writing. While paused, the readiness
   probe also times out, the endpoint is removed from the Service, and
   connections to the ClusterIP are refused.

2. The "kubectl wait --for=condition=available deployment/..."
   barrier added previously is a no-op for these deployments: with one
   replica and maxUnavailable=1, the availability threshold is zero
   ready pods, so the condition is always true.

Changes:

- Wait on RGW pod readiness instead of deployment availability.
- Add a "wait for multisite sync to be established" step that polls
  "radosgw-admin sync status" on both clusters until metadata and
  data sync report caught up, mirroring the checkpoints ceph's own
  multisite QA performs before asserting replication.
- Probe both RGW endpoints over HTTP until several consecutive
  requests succeed before writing, to ride out realm-reload pauses.
- Replace the fixed 3-attempt s3cmd retries with time-budgeted
  retries.
- Replace the s3cmd read polling loop and its SIGUSR1 timer (which was
  never cancelled on success, silently shrinking the second
  direction's time budget) with "radosgw-admin bucket sync
  checkpoint", the marker-based barrier used by ceph QA, followed by
  a single get and diff.
- Dump multisite diagnostics (pod state, RGW logs, sync status,
  period) when the job fails.

Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>

.github/workflows/canary-integration-test.yml

@@ -1937,9 +1937,15 @@ jobs:
         run: |
           tests/scripts/github-action-helper.sh wait_for cephcluster my-cluster rook-ceph
           tests/scripts/github-action-helper.sh wait_for cephcluster my-cluster rook-ceph-secondary
-          # wait for the RGW deployments to be ready to serve traffic
-          kubectl -n rook-ceph wait --for=condition=available deployment/rook-ceph-rgw-multisite-store-a --timeout=300s
-          kubectl -n rook-ceph-secondary wait --for=condition=available deployment/rook-ceph-rgw-zone-b-multisite-store-a --timeout=300s
+          # Wait for the RGW pods to be ready. With a single replica and maxUnavailable=1, the
+          # RGW deployments count as "available" even with zero ready pods, so wait on the pods.
+          kubectl -n rook-ceph wait --for=condition=Ready pod -l rook_object_store=multisite-store --timeout=300s
+          kubectl -n rook-ceph-secondary wait --for=condition=Ready pod -l rook_object_store=zone-b-multisite-store --timeout=300s
+
+      - name: wait for multisite sync to be established
+        shell: bash --noprofile --norc -eo pipefail -x {0}
+        run: |
+          tests/scripts/github-action-helper.sh wait_for_multisite_sync_established
 
       - name: write an object to one cluster, read from the other
         shell: bash --noprofile --norc -eo pipefail -x {0}
@@ -1962,6 +1968,12 @@ jobs:
           tests/scripts/github-action-helper.sh wait_for_operator_log_message "${not_committed_msg} ${ns_name_primary_object_zone}" 600
           tests/scripts/github-action-helper.sh wait_for_operator_log_message "${not_committed_msg} ${ns_name_secondary_object_zone}" 600
 
+      - name: dump multisite diagnostics on failure
+        if: failure()
+        shell: bash --noprofile --norc -eo pipefail -x {0}
+        run: |
+          tests/scripts/github-action-helper.sh dump_multisite_diagnostics
+
       - name: collect common logs
         if: always()
         uses: ./.github/workflows/collect-logs