From 20c302f72da4a39663c551b2c0e0ccb186443d34 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?S=C3=A9bastien=20Han?= Date: Wed, 14 Apr 2021 12:19:36 +0200 Subject: [PATCH 1/2] ceph: use replicaset instead of deployments for rgw instances MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Since Pacific, we can deploy multiple object gateway using the same keying and they will appear in the service map separetly. So from now on and on upgrades to Pacific Rook will remove all extra deployments to only keep 1. In this single deployment the number of replica will be set the desired `instances` count from the CRD spec. You will now see gateways like this on Pacific: ``` rgw: 3 daemons active (1 hosts, 1 zones) ``` See an upgrade operator logs from Octopus to Pacific: ``` 2021-04-14 10:05:16.883145 I | ceph-object-controller: setting multisite settings for object store "my-store" 2021-04-14 10:05:17.006492 I | ceph-object-controller: Multisite for object-store: realm=my-store, zonegroup=my-store, zone=my-store 2021-04-14 10:05:17.006516 I | ceph-object-controller: multisite configuration for object-store my-store is complete 2021-04-14 10:05:17.006523 I | ceph-object-controller: creating object store "my-store" in namespace "rook-ceph" 2021-04-14 10:05:17.006533 I | cephclient: getting or creating ceph auth key "client.rgw.my.store.a" 2021-04-14 10:05:17.298176 I | ceph-object-controller: object store "my-store" deployment "rook-ceph-rgw-my-store-a" started 2021-04-14 10:05:17.307913 I | ceph-object-controller: object store "my-store" deployment "rook-ceph-rgw-my-store-a" already exists. updating if needed 2021-04-14 10:05:17.317441 I | op-k8sutil: updating deployment "rook-ceph-rgw-my-store-a" after verifying it is safe to stop 2021-04-14 10:05:17.317463 I | op-mon: checking if we can stop the deployment rook-ceph-rgw-my-store-a 2021-04-14 10:05:25.552648 I | op-k8sutil: finished waiting for updated deployment "rook-ceph-mgr-a" 2021-04-14 10:05:25.552662 I | op-mon: checking if we can continue the deployment rook-ceph-mgr-a 2021-04-14 10:05:25.555077 I | op-mgr: setting services to point to mgr "a" 2021-04-14 10:05:25.608535 I | op-osd: start running osds in namespace "rook-ceph" 2021-04-14 10:05:25.608587 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2021-04-14 10:05:25.618392 I | op-osd: start provisioning the OSDs on PVCs, if needed 2021-04-14 10:05:25.620512 I | op-osd: no storageClassDeviceSets or volumeSources are defined to configure OSDs on PVCs 2021-04-14 10:05:25.620542 I | op-osd: start provisioning the OSDs on nodes, if needed 2021-04-14 10:05:25.628416 I | op-osd: 1 of the 1 storage nodes are valid 2021-04-14 10:05:25.760702 I | op-k8sutil: Removing previous job rook-ceph-osd-prepare-minikube to start a new one 2021-04-14 10:05:25.773213 I | op-k8sutil: batch job rook-ceph-osd-prepare-minikube still exists 2021-04-14 10:05:26.595673 I | op-mgr: successful modules: prometheus 2021-04-14 10:05:27.129569 I | op-mgr: successful modules: mgr module(s) from the spec 2021-04-14 10:05:27.776720 I | op-k8sutil: batch job rook-ceph-osd-prepare-minikube deleted 2021-04-14 10:05:27.781881 I | op-osd: started OSD provisioning job for node "minikube" 2021-04-14 10:05:27.786693 I | op-osd: OSD orchestration status for node minikube is "starting" 2021-04-14 10:05:28.617937 I | op-mgr: successful modules: balancer 2021-04-14 10:05:28.706307 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2021-04-14 10:05:28.710245 I | op-osd: updating OSD 0 on node "minikube" 2021-04-14 10:05:31.596511 I | op-mgr: the dashboard secret was already generated 2021-04-14 10:05:31.926125 I | op-mgr: setting ceph dashboard "admin" login creds 2021-04-14 10:05:32.190343 E | op-mgr: failed modules: "dashboard". failed to initialize dashboard: failed to set login credentials for the ceph dashboard: failed to set login creds on mgr: failed to complete command for set dashboard creds: Invalid command: unused arguments: ["P.c(b6V$0pfK#)'70c5z"] dashboard set-login-credentials : Set the login credentials. Password read from -i Traceback (most recent call last): File "/usr/bin/ceph", line 1310, in retval = main() File "/usr/bin/ceph", line 1256, in main outf.write(outbuf) TypeError: a bytes-like object is required, not 'str' . 2021-04-14 10:05:35.665419 I | op-k8sutil: finished waiting for updated deployment "rook-ceph-rgw-my-store-a" 2021-04-14 10:05:35.665553 I | op-mon: checking if we can continue the deployment rook-ceph-rgw-my-store-a 2021-04-14 10:05:35.675293 I | ceph-object-controller: config map "rook-ceph-rgw-my-store-mime-types" for object store "my-store" already exists, not overwriting 2021-04-14 10:05:35.691549 I | ceph-object-controller: found more rgw deployments 3 than desired 3 in object store "my-store", scaling down 2021-04-14 10:05:35.691580 I | op-k8sutil: removing deployment rook-ceph-rgw-my-store-c if it exists 2021-04-14 10:05:35.698382 I | op-k8sutil: Removed deployment rook-ceph-rgw-my-store-c 2021-04-14 10:05:35.704568 I | op-k8sutil: "rook-ceph-rgw-my-store-c" still found. waiting... 2021-04-14 10:05:44.881129 I | op-osd: OSD orchestration status for node minikube is "orchestrating" 2021-04-14 10:05:44.882149 I | op-osd: OSD orchestration status for node minikube is "completed" 2021-04-14 10:05:45.756762 I | op-k8sutil: "rook-ceph-rgw-my-store-c" still found. waiting... 2021-04-14 10:05:45.782039 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2021-04-14 10:05:45.785188 I | op-osd: updating OSD 1 on node "minikube" 2021-04-14 10:05:54.768851 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2021-04-14 10:05:54.773191 I | op-osd: updating OSD 2 on node "minikube" 2021-04-14 10:05:55.843879 I | op-k8sutil: "rook-ceph-rgw-my-store-c" still found. waiting... 2021-04-14 10:06:05.255136 I | op-osd: finished running OSDs in namespace "rook-ceph" 2021-04-14 10:06:05.255155 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2021-04-14 10:06:05.560201 W | ceph-cluster-controller: upgrade orchestration completed but somehow we still have more than one Ceph version running. map[ceph version 15.2.9 (357616cbf726abb779ca75a551e8d02568e15b17) octopus (stable):3 ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable):6]: 2021-04-14 10:06:05.885001 I | op-k8sutil: "rook-ceph-rgw-my-store-c" still found. waiting... 2021-04-14 10:06:08.321209 I | exec: timeout waiting for process radosgw-admin to return. Sending interrupt signal to the process 2021-04-14 10:06:10.645672 I | ceph-spec: object "rook-ceph-rgw-my-store-c" matched on delete, reconciling 2021-04-14 10:06:11.940994 I | op-k8sutil: confirmed rook-ceph-rgw-my-store-c does not exist 2021-04-14 10:06:11.972935 I | ceph-spec: object "rook-ceph-rgw-my-store-c-keyring" matched on delete, reconciling 2021-04-14 10:06:11.974394 I | ceph-object-controller: deleting rgw CephX key and configuration in centralized mon database for "rook-ceph-rgw-my-store-c" 2021-04-14 10:06:13.721964 I | ceph-object-controller: successfully deleted rgw config for "client.rgw.my.store.c" in mon configuration database 2021-04-14 10:06:13.721989 I | cephclient: deleting ceph auth "client.rgw.my.store.c" 2021-04-14 10:06:14.073700 I | ceph-object-controller: completed deleting rgw CephX key and configuration in centralized mon database for "rook-ceph-rgw-my-store-c" 2021-04-14 10:06:14.073724 I | op-k8sutil: removing deployment rook-ceph-rgw-my-store-b if it exists 2021-04-14 10:06:14.083950 I | op-k8sutil: Removed deployment rook-ceph-rgw-my-store-b 2021-04-14 10:06:14.090226 I | op-k8sutil: "rook-ceph-rgw-my-store-b" still found. waiting... 2021-04-14 10:06:24.174846 I | op-k8sutil: "rook-ceph-rgw-my-store-b" still found. waiting... 2021-04-14 10:06:30.505574 I | ceph-spec: object "rook-ceph-rgw-my-store-b" matched on delete, reconciling 2021-04-14 10:06:32.201263 I | op-k8sutil: confirmed rook-ceph-rgw-my-store-b does not exist 2021-04-14 10:06:32.219559 I | ceph-object-controller: deleting rgw CephX key and configuration in centralized mon database for "rook-ceph-rgw-my-store-b" 2021-04-14 10:06:32.220219 I | ceph-spec: object "rook-ceph-rgw-my-store-b-keyring" matched on delete, reconciling 2021-04-14 10:06:33.821969 I | ceph-object-controller: successfully deleted rgw config for "client.rgw.my.store.b" in mon configuration database 2021-04-14 10:06:33.821992 I | cephclient: deleting ceph auth "client.rgw.my.store.b" 2021-04-14 10:06:34.182091 I | ceph-object-controller: completed deleting rgw CephX key and configuration in centralized mon database for "rook-ceph-rgw-my-store-b" 2021-04-14 10:06:34.188659 I | ceph-object-controller: successfully scaled down rgw deployments to 1 in object store "my-store" 2021-04-14 10:06:34.188676 I | ceph-object-controller: enabling rgw dashboard 2021-04-14 10:06:49.487902 I | exec: timeout waiting for process radosgw-admin to return. Sending interrupt signal to the process 2021-04-14 10:06:49.494605 W | ceph-object-controller: failed to enable dashboard for rgw. failed to create user "dashboard-admin": failed to create s3 user: signal: interrupt 2021-04-14 10:06:49.494676 I | ceph-object-controller: created object store "my-store" in namespace "rook-ceph" ``` Running pods: ``` ... ... rook-ceph-rgw-my-store-a-6756bdbbd4-7265v 1/1 Running 1 41m rook-ceph-rgw-my-store-a-6756bdbbd4-l5ccr 1/1 Running 1 41m rook-ceph-rgw-my-store-a-6756bdbbd4-m679c 1/1 Running 1 41m ... ... ``` Signed-off-by: Sébastien Han --- PendingReleaseNotes.md | 1 + cluster/charts/rook-ceph/templates/resources.yaml | 2 +- cluster/examples/kubernetes/ceph/crds.yaml | 2 +- pkg/apis/ceph.rook.io/v1/types.go | 2 +- pkg/operator/ceph/object/rgw.go | 4 ++++ pkg/operator/ceph/object/spec.go | 4 ++++ 6 files changed, 12 insertions(+), 3 deletions(-) diff --git a/PendingReleaseNotes.md b/PendingReleaseNotes.md index eb7490ef2..d3e22dddd 100644 --- a/PendingReleaseNotes.md +++ b/PendingReleaseNotes.md @@ -34,3 +34,4 @@ v1.6... up updates and upgrades for larger Ceph clusters * Disable CSI GRPC metrics by default * Monitors failover can be [disabled](Documentation/ceph-mon-health.md#failing-over-a-monitor), this is useful for monitor going under planned maintenance where automatic failover is not desired +* RGW: Rook has started using deployment's replicatset functionality instead of deploying multiple deployments for each rgw diff --git a/cluster/charts/rook-ceph/templates/resources.yaml b/cluster/charts/rook-ceph/templates/resources.yaml index dbb9d0397..6a3fd9fe4 100644 --- a/cluster/charts/rook-ceph/templates/resources.yaml +++ b/cluster/charts/rook-ceph/templates/resources.yaml @@ -6394,7 +6394,7 @@ spec: nullable: true type: array instances: - description: The number of pods in the rgw replicaset. If "allNodes" is specified, a daemonset is created. + description: The number of pods in the rgw replicaset. format: int32 minimum: 1 type: integer diff --git a/cluster/examples/kubernetes/ceph/crds.yaml b/cluster/examples/kubernetes/ceph/crds.yaml index 0849be8ca..4ebf12264 100644 --- a/cluster/examples/kubernetes/ceph/crds.yaml +++ b/cluster/examples/kubernetes/ceph/crds.yaml @@ -6397,7 +6397,7 @@ spec: nullable: true type: array instances: - description: The number of pods in the rgw replicaset. If "allNodes" is specified, a daemonset is created. + description: The number of pods in the rgw replicaset. format: int32 minimum: 1 type: integer diff --git a/pkg/apis/ceph.rook.io/v1/types.go b/pkg/apis/ceph.rook.io/v1/types.go index fad83f2c3..aa33bf673 100755 --- a/pkg/apis/ceph.rook.io/v1/types.go +++ b/pkg/apis/ceph.rook.io/v1/types.go @@ -1062,7 +1062,7 @@ type GatewaySpec struct { // +optional SecurePort int32 `json:"securePort,omitempty"` - // The number of pods in the rgw replicaset. If "allNodes" is specified, a daemonset is created. + // The number of pods in the rgw replicaset. // +kubebuilder:validation:Minimum=1 Instances int32 `json:"instances"` diff --git a/pkg/operator/ceph/object/rgw.go b/pkg/operator/ceph/object/rgw.go index 1249a8c8d..a2a1625a3 100644 --- a/pkg/operator/ceph/object/rgw.go +++ b/pkg/operator/ceph/object/rgw.go @@ -85,6 +85,10 @@ func (c *clusterConfig) startRGWPods(realmName, zoneGroupName, zoneName string) // start a new deployment and scale up desiredRgwInstances := int(c.store.Spec.Gateway.Instances) + // If running on Pacific we force a single deployment and later set the deployment replica to the "instances" value + if c.clusterInfo.CephVersion.IsAtLeastPacific() { + desiredRgwInstances = 1 + } for i := 0; i < desiredRgwInstances; i++ { var err error diff --git a/pkg/operator/ceph/object/spec.go b/pkg/operator/ceph/object/spec.go index 2451845ad..42e02d11a 100644 --- a/pkg/operator/ceph/object/spec.go +++ b/pkg/operator/ceph/object/spec.go @@ -58,6 +58,10 @@ func (c *clusterConfig) createDeployment(rgwConfig *rgwConfig) (*apps.Deployment return nil, err } replicas := int32(1) + // On Pacific, we can use the same keyring and have dedicated rgw instances reflected in the service map + if c.clusterInfo.CephVersion.IsAtLeastPacific() { + replicas = c.store.Spec.Gateway.Instances + } d := &apps.Deployment{ ObjectMeta: metav1.ObjectMeta{ Name: rgwConfig.ResourceName, From 274447c539ed11a09614f58cd01d3f17f4615928 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?S=C3=A9bastien=20Han?= Date: Wed, 14 Apr 2021 17:23:49 +0200 Subject: [PATCH 2/2] ceph: add anti-affinity rules to rgw pods MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Ensure in the default examples that gateways will run on different nodes. Signed-off-by: Sébastien Han --- .../kubernetes/ceph/object-openshift.yaml | 24 ++++++++++++------- cluster/examples/kubernetes/ceph/object.yaml | 17 ++++++++++++- 2 files changed, 31 insertions(+), 10 deletions(-) diff --git a/cluster/examples/kubernetes/ceph/object-openshift.yaml b/cluster/examples/kubernetes/ceph/object-openshift.yaml index 07fba4426..7ba94661f 100644 --- a/cluster/examples/kubernetes/ceph/object-openshift.yaml +++ b/cluster/examples/kubernetes/ceph/object-openshift.yaml @@ -54,16 +54,22 @@ spec: # securePort: 443 # The number of pods in the rgw deployment instances: 1 - # The affinity rules to apply to the rgw deployment or daemonset. + # The affinity rules to apply to the rgw deployment placement: - # nodeAffinity: - # requiredDuringSchedulingIgnoredDuringExecution: - # nodeSelectorTerms: - # - matchExpressions: - # - key: role - # operator: In - # values: - # - rgw-node + podAntiAffinity: + preferredDuringSchedulingIgnoredDuringExecution: + - weight: 100 + podAffinityTerm: + labelSelector: + matchExpressions: + - key: app + operator: In + values: + - rook-ceph-rgw + # topologyKey: */zone can be used to spread RGW across different AZ + # Use in k8s cluster if your cluster is v1.16 or lower + # Use in k8s cluster is v1.17 or upper + topologyKey: kubernetes.io/hostname # topologySpreadConstraints: # tolerations: # - key: rgw-node diff --git a/cluster/examples/kubernetes/ceph/object.yaml b/cluster/examples/kubernetes/ceph/object.yaml index 3d1a24bf5..2b59c94d4 100644 --- a/cluster/examples/kubernetes/ceph/object.yaml +++ b/cluster/examples/kubernetes/ceph/object.yaml @@ -54,8 +54,23 @@ spec: # securePort: 443 # The number of pods in the rgw deployment instances: 1 - # The affinity rules to apply to the rgw deployment or daemonset. + # The affinity rules to apply to the rgw deployment. placement: + podAntiAffinity: + preferredDuringSchedulingIgnoredDuringExecution: + - weight: 100 + podAffinityTerm: + labelSelector: + matchExpressions: + - key: app + operator: In + values: + - rook-ceph-rgw + # topologyKey: */zone can be used to spread RGW across different AZ + # Use in k8s cluster if your cluster is v1.16 or lower + # Use in k8s cluster is v1.17 or upper + topologyKey: kubernetes.io/hostname + # A key/value list of annotations # nodeAffinity: # requiredDuringSchedulingIgnoredDuringExecution: # nodeSelectorTerms: