rgw: add option to disable synchronization traffic

Some users want to deploy two CephObjectStores for a single Zone. The first
configures RGWs to process the synchronization of the data, while the second
CephObjectStore configures the client RGWs.

Currently, this can be implemented by setting the RGW option
'rgw_run_sync_thread' in the 'rook-config-override' ConfigMap, though it is not
really user friendly.

Ref: https://docs.ceph.com/en/latest/radosgw/config-ref/#confval-rgw_run_sync_thread

This commit adds a new option in the CephObjectStore CRD as defined in issue
https://github.com/rook/rook/issues/12272. The new option
'disableMultisiteSyncTraffic' determine whether the operator should disable the
multisite sync threads for the RGWs.

If the option is set to 'false', or if the option is not specified, the operator
does nothing. This ensures that the multisite sync threads will not be enabled
for users that disabled explicitely the multisite sync threads either manually
or with the 'rook-config-override' ConfigMap.

This commit also recommends to use two objectstore when scaling Ceph Objectstore
Multisite replication, with one objectstore configured with disabled replication
traffic.

Signed-off-by: Lucas Henry <polyedre@disroot.org>
(cherry picked from commit 393d09347e)

Documentation/Storage-Configuration/Object-Storage-RGW/ceph-object-multisite.md

@@ -186,6 +186,46 @@ object-multisite-pull-realm.yaml (with changes) in the [examples](https://github
 kubectl create -f object-multisite-pull-realm.yaml
 ```
 
+## Scaling a Multisite
+
+Scaling the number of gateways that run the synchronization thread to 2 or more can increase the latency of the
+replication of each S3 object. The recommended way to scale a mutisite configuration is to dissociate the gateway dedicated
+to the synchronization from gateways that serve clients.
+
+The two types of gateways can be deployed by creating two CephObjectStores associated with the same CephObjectZone. The
+objectstore that deploys the gateway dedicated to the synchronization must have `spec.gateway.instances` set to `1`,
+while the objectstore that deploys the client gateways have multiple replicas and should disable the synchronization
+thread on the gateways by setting `spec.gateway.disableMultisiteSyncTraffic` to `true`.
+
+```yaml
+---
+apiVersion: ceph.rook.io/v1
+kind: CephObjectStore
+metadata:
+  name: replication
+  namespace: rook-ceph
+spec:
+  gateway:
+    port: 80
+    instances: 1
+    disableMultisiteSyncTraffic: false
+  zone:
+    name: zone-a
+---
+apiVersion: ceph.rook.io/v1
+kind: CephObjectStore
+metadata:
+  name: clients
+  namespace: rook-ceph
+spec:
+  gateway:
+    port: 80
+    instances: 5
+    disableMultisiteSyncTraffic: true
+  zone:
+    name: zone-a
+```
+
 ## Multisite Cleanup
 
 Multisite configuration must be cleaned up by hand. Deleting a realm/zone group/zone CR will not delete the underlying Ceph realm, zone group, zone, or the pools associated with a zone.