forked from rook/rook
Sébastien Han
20c302f72d
Since Pacific, we can deploy multiple object gateway using the same
keying and they will appear in the service map separetly. So from now on
and on upgrades to Pacific Rook will remove all extra deployments to
only keep 1. In this single deployment the number of replica will be set
the desired `instances` count from the CRD spec.
You will now see gateways like this on Pacific:
```
rgw: 3 daemons active (1 hosts, 1 zones)
```
See an upgrade operator logs from Octopus to Pacific:
```
2021-04-14 10:05:16.883145 I | ceph-object-controller: setting multisite settings for object store "my-store"
2021-04-14 10:05:17.006492 I | ceph-object-controller: Multisite for object-store: realm=my-store, zonegroup=my-store, zone=my-store
2021-04-14 10:05:17.006516 I | ceph-object-controller: multisite configuration for object-store my-store is complete
2021-04-14 10:05:17.006523 I | ceph-object-controller: creating object store "my-store" in namespace "rook-ceph"
2021-04-14 10:05:17.006533 I | cephclient: getting or creating ceph auth key "client.rgw.my.store.a"
2021-04-14 10:05:17.298176 I | ceph-object-controller: object store "my-store" deployment "rook-ceph-rgw-my-store-a" started
2021-04-14 10:05:17.307913 I | ceph-object-controller: object store "my-store" deployment "rook-ceph-rgw-my-store-a" already exists. updating if needed
2021-04-14 10:05:17.317441 I | op-k8sutil: updating deployment "rook-ceph-rgw-my-store-a" after verifying it is safe to stop
2021-04-14 10:05:17.317463 I | op-mon: checking if we can stop the deployment rook-ceph-rgw-my-store-a
2021-04-14 10:05:25.552648 I | op-k8sutil: finished waiting for updated deployment "rook-ceph-mgr-a"
2021-04-14 10:05:25.552662 I | op-mon: checking if we can continue the deployment rook-ceph-mgr-a
2021-04-14 10:05:25.555077 I | op-mgr: setting services to point to mgr "a"
2021-04-14 10:05:25.608535 I | op-osd: start running osds in namespace "rook-ceph"
2021-04-14 10:05:25.608587 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s"
2021-04-14 10:05:25.618392 I | op-osd: start provisioning the OSDs on PVCs, if needed
2021-04-14 10:05:25.620512 I | op-osd: no storageClassDeviceSets or volumeSources are defined to configure OSDs on PVCs
2021-04-14 10:05:25.620542 I | op-osd: start provisioning the OSDs on nodes, if needed
2021-04-14 10:05:25.628416 I | op-osd: 1 of the 1 storage nodes are valid
2021-04-14 10:05:25.760702 I | op-k8sutil: Removing previous job rook-ceph-osd-prepare-minikube to start a new one
2021-04-14 10:05:25.773213 I | op-k8sutil: batch job rook-ceph-osd-prepare-minikube still exists
2021-04-14 10:05:26.595673 I | op-mgr: successful modules: prometheus
2021-04-14 10:05:27.129569 I | op-mgr: successful modules: mgr module(s) from the spec
2021-04-14 10:05:27.776720 I | op-k8sutil: batch job rook-ceph-osd-prepare-minikube deleted
2021-04-14 10:05:27.781881 I | op-osd: started OSD provisioning job for node "minikube"
2021-04-14 10:05:27.786693 I | op-osd: OSD orchestration status for node minikube is "starting"
2021-04-14 10:05:28.617937 I | op-mgr: successful modules: balancer
2021-04-14 10:05:28.706307 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort
2021-04-14 10:05:28.710245 I | op-osd: updating OSD 0 on node "minikube"
2021-04-14 10:05:31.596511 I | op-mgr: the dashboard secret was already generated
2021-04-14 10:05:31.926125 I | op-mgr: setting ceph dashboard "admin" login creds
2021-04-14 10:05:32.190343 E | op-mgr: failed modules: "dashboard". failed to initialize dashboard: failed to set login credentials for the ceph dashboard: failed to set login creds on mgr: failed to complete command for set dashboard creds: Invalid command: unused arguments: ["P.c(b6V$0pfK#)'70c5z"]
dashboard set-login-credentials <username> : Set the login credentials. Password read from -i <file>
Traceback (most recent call last):
File "/usr/bin/ceph", line 1310, in <module>
retval = main()
File "/usr/bin/ceph", line 1256, in main
outf.write(outbuf)
TypeError: a bytes-like object is required, not 'str'
.
2021-04-14 10:05:35.665419 I | op-k8sutil: finished waiting for updated deployment "rook-ceph-rgw-my-store-a"
2021-04-14 10:05:35.665553 I | op-mon: checking if we can continue the deployment rook-ceph-rgw-my-store-a
2021-04-14 10:05:35.675293 I | ceph-object-controller: config map "rook-ceph-rgw-my-store-mime-types" for object store "my-store" already exists, not overwriting
2021-04-14 10:05:35.691549 I | ceph-object-controller: found more rgw deployments 3 than desired 3 in object store "my-store", scaling down
2021-04-14 10:05:35.691580 I | op-k8sutil: removing deployment rook-ceph-rgw-my-store-c if it exists
2021-04-14 10:05:35.698382 I | op-k8sutil: Removed deployment rook-ceph-rgw-my-store-c
2021-04-14 10:05:35.704568 I | op-k8sutil: "rook-ceph-rgw-my-store-c" still found. waiting...
2021-04-14 10:05:44.881129 I | op-osd: OSD orchestration status for node minikube is "orchestrating"
2021-04-14 10:05:44.882149 I | op-osd: OSD orchestration status for node minikube is "completed"
2021-04-14 10:05:45.756762 I | op-k8sutil: "rook-ceph-rgw-my-store-c" still found. waiting...
2021-04-14 10:05:45.782039 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort
2021-04-14 10:05:45.785188 I | op-osd: updating OSD 1 on node "minikube"
2021-04-14 10:05:54.768851 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort
2021-04-14 10:05:54.773191 I | op-osd: updating OSD 2 on node "minikube"
2021-04-14 10:05:55.843879 I | op-k8sutil: "rook-ceph-rgw-my-store-c" still found. waiting...
2021-04-14 10:06:05.255136 I | op-osd: finished running OSDs in namespace "rook-ceph"
2021-04-14 10:06:05.255155 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph"
2021-04-14 10:06:05.560201 W | ceph-cluster-controller: upgrade orchestration completed but somehow we still have more than one Ceph version running. map[ceph version 15.2.9 (357616cbf726abb779ca75a551e8d02568e15b17) octopus (stable):3 ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable):6]:
2021-04-14 10:06:05.885001 I | op-k8sutil: "rook-ceph-rgw-my-store-c" still found. waiting...
2021-04-14 10:06:08.321209 I | exec: timeout waiting for process radosgw-admin to return. Sending interrupt signal to the process
2021-04-14 10:06:10.645672 I | ceph-spec: object "rook-ceph-rgw-my-store-c" matched on delete, reconciling
2021-04-14 10:06:11.940994 I | op-k8sutil: confirmed rook-ceph-rgw-my-store-c does not exist
2021-04-14 10:06:11.972935 I | ceph-spec: object "rook-ceph-rgw-my-store-c-keyring" matched on delete, reconciling
2021-04-14 10:06:11.974394 I | ceph-object-controller: deleting rgw CephX key and configuration in centralized mon database for "rook-ceph-rgw-my-store-c"
2021-04-14 10:06:13.721964 I | ceph-object-controller: successfully deleted rgw config for "client.rgw.my.store.c" in mon configuration database
2021-04-14 10:06:13.721989 I | cephclient: deleting ceph auth "client.rgw.my.store.c"
2021-04-14 10:06:14.073700 I | ceph-object-controller: completed deleting rgw CephX key and configuration in centralized mon database for "rook-ceph-rgw-my-store-c"
2021-04-14 10:06:14.073724 I | op-k8sutil: removing deployment rook-ceph-rgw-my-store-b if it exists
2021-04-14 10:06:14.083950 I | op-k8sutil: Removed deployment rook-ceph-rgw-my-store-b
2021-04-14 10:06:14.090226 I | op-k8sutil: "rook-ceph-rgw-my-store-b" still found. waiting...
2021-04-14 10:06:24.174846 I | op-k8sutil: "rook-ceph-rgw-my-store-b" still found. waiting...
2021-04-14 10:06:30.505574 I | ceph-spec: object "rook-ceph-rgw-my-store-b" matched on delete, reconciling
2021-04-14 10:06:32.201263 I | op-k8sutil: confirmed rook-ceph-rgw-my-store-b does not exist
2021-04-14 10:06:32.219559 I | ceph-object-controller: deleting rgw CephX key and configuration in centralized mon database for "rook-ceph-rgw-my-store-b"
2021-04-14 10:06:32.220219 I | ceph-spec: object "rook-ceph-rgw-my-store-b-keyring" matched on delete, reconciling
2021-04-14 10:06:33.821969 I | ceph-object-controller: successfully deleted rgw config for "client.rgw.my.store.b" in mon configuration database
2021-04-14 10:06:33.821992 I | cephclient: deleting ceph auth "client.rgw.my.store.b"
2021-04-14 10:06:34.182091 I | ceph-object-controller: completed deleting rgw CephX key and configuration in centralized mon database for "rook-ceph-rgw-my-store-b"
2021-04-14 10:06:34.188659 I | ceph-object-controller: successfully scaled down rgw deployments to 1 in object store "my-store"
2021-04-14 10:06:34.188676 I | ceph-object-controller: enabling rgw dashboard
2021-04-14 10:06:49.487902 I | exec: timeout waiting for process radosgw-admin to return. Sending interrupt signal to the process
2021-04-14 10:06:49.494605 W | ceph-object-controller: failed to enable dashboard for rgw. failed to create user "dashboard-admin": failed to create s3 user: signal: interrupt
2021-04-14 10:06:49.494676 I | ceph-object-controller: created object store "my-store" in namespace "rook-ceph"
```
Running pods:
```
...
...
rook-ceph-rgw-my-store-a-6756bdbbd4-7265v 1/1 Running 1 41m
rook-ceph-rgw-my-store-a-6756bdbbd4-l5ccr 1/1 Running 1 41m
rook-ceph-rgw-my-store-a-6756bdbbd4-m679c 1/1 Running 1 41m
...
...
```
Signed-off-by: Sébastien Han <seb@redhat.com>