Commit Graph
2792 Commits
Author SHA1 Message Date
Sébastien Han 8c589a3785 ceph: silence harmless errors
Let's not look at misleading errors if the operator is still
initializing.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-07-01 15:37:08 +02:00
Jiffin Tony Thottan 9e3cf68d04 test: ci test for TLS objectstore
Extend the object store smoke test to include TLS configurations.

Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
2021-07-01 18:50:27 +05:30
Sébastien Han b68cac34cb Merge pull request #8222 from sp98/update-block-pool-channels
ceph: update blockPoolChannel before starting the mirror monitoring
2021-07-01 14:10:17 +02:00
Santosh Pillai 8d48a9e10a ceph: update blockPoolChannel before starting the mirror monitoring
update blockPoolChannel monitoringRunning field to `true` before
starting the goroutine to monitor mirroring

Signed-off-by: Santosh Pillai <sapillai@redhat.com>
2021-07-01 15:27:37 +05:30
Jiffin Tony Thottan 560f44b07e ceph: enable debug for adminops client if the rook loglevel <= debug
The debug for adminOps client can be enabled by setting `Debug` flag.
In this PR, it is enabled for OBC and cephobjectstore not healthchecker.
Also add similar changes while calling `NewS3Agent` in the bucket
provisioner code path.

Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
2021-07-01 11:36:51 +05:30
Santosh Pillai 4041868e2e ceph: hybrid Storage Pools
Creates hybrid crush rule for choosing Primary OSD for
high performing SSD devices and remaining OSD for low performance HDD devices.

Signed-off-by: Santosh Pillai <sapillai@redhat.com>
2021-07-01 11:16:54 +05:30
Blaine Gardner c22f545ebf ceph: block delete object store when buckets exist
Block deletion of CephObjectStore resources when buckets exist in the
object store.

Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
2021-06-29 14:31:39 -06:00
Yuji Ito d7c46d70d3 ceph: fix overwriting shared livenessProbe
When the Operator set the livenessProbe of the POD, it overwrote the contents of the shared
desiredProbe. Therefore, the wrongly configured livenessProbe was applied to other PODs,
resulting in a CrashLoop. In this PR, I created another instance to avoid overwriting the
desiredProbe.

Closes: https://github.com/rook/rook/issues/8196
Signed-off-by: Yuji Ito <llamerada.jp@gmail.com>
2021-06-28 09:54:33 +00:00
Sébastien Han 507f913376 Merge pull request #8191 from leseb/fix-loop-break
ceph: fix ineffective break statement
2021-06-25 17:12:53 +02:00
Sébastien Han 23298ba62d ceph: fix ineffective break statement
The `break` statement is not necessary, especially since we don't want
to break out of the outerloop.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-06-25 16:23:23 +02:00
Travis Nielsen 259d046ff8 Merge pull request #8190 from Rakshith-R/upd/csi-driver-registrar
ceph: update csi node-driver-registrar to latest release
2021-06-24 12:38:39 -06:00
Rakshith R 2baf50771e ceph: update csi node-driver-registrar to latest release
This commit updates csi node-driver-registrar to latest
release v2.2.0.
https://github.com/kubernetes-csi/node-driver-registrar/releases

Signed-off-by: Rakshith R <rar@redhat.com>
2021-06-24 10:22:51 +05:30
Sébastien Han 153f1d661c ceph: append additional info in the rbd-mirror bootstrap peer token
We know append additional information to the rbd-mirror bootstrap peer
token. It is useful for disaster recovery scenario where the other
cluster is reading the peer token and needs to know the pool_id as well
as the namespace.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-06-23 18:49:39 +02:00
Travis Nielsen b1c1e296dd ceph: evict a mon if colocated with another mon
In a production cluster we never should have two mons running on
the same node. However, if two mons have ended up on the same node,
whether from a bug or some other unintentional event, the operator
will evict one of them and failover to a new mon. This only applies
when allowMultiplePerNode is false.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-06-23 09:29:04 -06:00
Sébastien Han b108442bb8 Merge pull request #8173 from leseb/do-not-leak-kek
ceph: do not leak key encryption key
2021-06-22 16:09:00 +02:00
Travis Nielsen 6b0c1e9a7b Merge pull request #8140 from Rakshith-R/mv-csi-funcs
ceph: read and validate CSI params in Go routine
2021-06-22 07:36:29 -06:00
Sébastien Han bcd2511a72 ceph: do not leak key encryption key
When evaluating the potential "warning" key from the json output we were
not hiding stdout (1) from the output, so the key was leaked. Also the
warning check for the key was not perfect, it only checked for the
command to succeed, but we should introspect it since the "warning" key
is always present unlike the "error" one.
We now only print the warning or the error is present.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-06-22 15:04:13 +02:00
Sébastien Han e6929ce308 Merge pull request #7913 from abursavich/extract-kk
Remove dependency on k8s.io/kubernetes
2021-06-22 09:22:41 +02:00
Travis Nielsen 27857432af Merge pull request #8147 from travisn/stretch-node-assign
ceph: Mons in stretch cluster should be assigned to a node when using dataDirHostPath
2021-06-21 10:06:55 -06:00
Andy Bursavich 11f0add706 build: update golangci-lint version and nolint comment format
Signed-off-by: Andy Bursavich <abursavich@gmail.com>
2021-06-21 08:50:04 -07:00
Travis Nielsen b1ff9f4436 Merge pull request #8157 from leseb/vault-cacert-capath
ceph: use cacert if no client cert/key are present
2021-06-21 09:50:01 -06:00
Travis Nielsen 80a05e70d4 Merge pull request #8114 from subhamkrai/cronjob-v1beta1-v1
ceph: update cronjob from v1beta1 to v1
2021-06-21 09:29:17 -06:00
Sébastien Han eb0c986196 ceph: use cacert if no client cert/key are present
We have seen issues with fullchain.pem files containing one self-signed
certificate. At the same time, the failing configuration was not
providing any client certificates or keys.

The error from curl was `"self signed certificate in the chain"`.

The openssl binary is not available so we cannot even c_rehash the
certificate directory when using `--capath`, however this worked already
with a fully signed fullchain.pem along with a client certificate and
key. So we cannot only use `--cacert` which will break that use case.

This is becoming really obscur on why using `--cacert` solves this
problem. I just can't wait to use the `vault` binary, that's all I can
say...

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-06-21 16:06:25 +02:00
Travis Nielsen ac73d1eb6c Merge pull request #8139 from thotz/tlsrookoperatorcrash
ceph: initialise httpclient for bucketchecker and objectstoreuser
2021-06-21 07:42:37 -06:00
Rakshith R 66c9ccab86 ceph: read and validate CSI params in Go routine
This commit moves SetParams and ValidateCSIParam func
into go routine/lock, since it modifies/reads global CSIParam
variable.

Signed-off-by: Rakshith R <rar@redhat.com>
2021-06-21 15:44:54 +05:30
Jiffin Tony Thottan 867474e405 ceph: initialise httpclient for bucketchecker and objectstoreuser
For the TLS communication for AdminOps Api, httpclient is required and
filled with TLS certs, currently it is set to nil pointer in
`buckethealthchecker` and `cephobjectstoreuser`.

Thanks @Krast76 finding the issue even proposing the fix.

Fixes: #8132
Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
2021-06-21 10:59:58 +05:30
Travis Nielsen a7066770f7 ceph: mons in stretch cluster assign to node
When the mons are using the dataDirHostPath and not a volumeClaimTemplate,
they need to be permanently assigned to a node. The node assignment
was missing in stretch clusters when the dataDirHostPath was being
used. This resulted in mons that were moving to other nodes for example
during a node drain, which would cause the mon to lose its backing store.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-06-18 15:25:27 -06:00
subhamkrai f82fc46b97 ceph: update cronjob from v1beta1 to v1
this commit update cronjob from v1beta1 to v1

Signed-off-by: subhamkrai <srai@redhat.com>
2021-06-18 08:35:02 +05:30
Travis Nielsen dd3b4738bd ceph: activate osd container should use correct dataDirHostPath
The active osd container was always using /var/lib/rook for the
configuration of the OSD. If the dataDirHostPath is anything other
than the default, the config would show up in the wrong place.
Changing this back to the dataDirHostPath does not appear to affect
any upgrade scenarios as the activate container will simply start
extracting the config to the correct place.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-06-17 06:59:31 -06:00
Travis Nielsen c961d9f7ac Merge pull request #8117 from degorenko/device-class-for-present-osds
ceph: set device class for already present osd deployments
2021-06-16 12:07:11 -06:00
Travis Nielsen 2bcb7aa045 Merge pull request #8130 from subhamkrai/remove-force
ceph: remove --force when creating filesystem
2021-06-16 09:06:45 -06:00
subhamkrai 5d5bdbbbda ceph: remove --force when creating filesystem
we should not use --force when creating filesystem(fs)
as it will recreate fs which will leads to data loss
when `preserveFilesystemOnDelete` is set to 'true'.

this commit removes --force argument when create fs.

Signed-off-by: subhamkrai <srai@redhat.com>
2021-06-16 13:39:49 +05:30
Denis Egorenko 6dbb902674 ceph: set device class for already present osd deployments
Add ability to set device class for existing osd deployments,
based on its crush map device class.

Related-Issue: rook#8028
Signed-off-by: Denis Egorenko <degorenko@mirantis.com>
2021-06-15 13:31:37 +04:00
Madhu Rajanna 08997434ac ceph: update csi sidecar images to latest release
This commit updates all the kubernetes sidecar
images to latest available release version.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-06-15 08:09:27 +05:30
Satoru Takeuchi 02d84f2c0f Merge pull request #8109 from BlaineEXE/close-cephcluster-cleanup-channel
ceph: handle CephCluster stopCleanupCh better
2021-06-11 10:14:04 +09:00
Travis Nielsen c6e51fad42 Merge pull request #8089 from travisn/secure-global-id-reclaim
ceph: Disable insecure global id if no insecure clients
2021-06-10 10:50:36 -06:00
Blaine Gardner a5c29a3847 ceph: handle CephCluster stopCleanupCh better
Investigate a possible memory leak around CephCluster cleanup's
stopCleanupCh channel. Add more close() statements and comment that Go's
garbage collector is able to close the channel when there are no more
referents to the channel.

Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
2021-06-10 10:50:30 -06:00
Blaine Gardner 102bd29e79 Merge pull request #8012 from BlaineEXE/implement-dependencies
Ceph: Implement dependencies
2021-06-10 10:37:45 -06:00
Blaine Gardner 21e290e003 ceph: implement dependencies for CephCluster
Implement the first step of `design/ceph/resource-dependencies.md` to
add dependency checking when deleting a CephCluster.

Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
2021-06-10 10:06:10 -06:00
Travis Nielsen b1f4411fb4 ceph: disable insecure global id if no insecure clients
In the latest Ceph releases starting with v16.2.1, all clients are recommended
to be updated so they will have a security fix to connect with a secure
global ID. A health warning will be raised if any insecure clients are connected
and another health warning is raised if insecure clients are still allowed.
Rook will now disable allowing the insecure clients if the health warning
is not being raised to indicate that there are insecure clients still connected.
This means that upgraded clusters will not have this disabled until all the
daemons are updated.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-06-10 10:05:30 -06:00
Rakshith R f489aba8f2 ceph: set CSI_ENABLE_HOST_NETWORK default value to true
This commit changes CSI_ENABLE_HOST_NETWORK default value to true
since it was observed that cephfsvolume gets blocked when csi cephfs
nodeplugin is restarted and nodeunpublish call also hangs when
cephfs nodeplugin is using pod networking.

Updates: #8085

Signed-off-by: Rakshith R <rar@redhat.com>
2021-06-10 15:33:49 +05:30
Santosh Pillai 49dc03e8eb ceph: compare ceph commit hash for upgrade check
Currently we only check the Major, Minor, Extra and Build number in the ceph version to determine if all daemons should be upgraded or not. This PR also checks the commit hash for upgrading all the daemons.
This will be useful in cases where build number has not changed due to hotfix. For example:
existing ceph version - ceph version 14.2.11-139.el8cp (b8e1f91c99491fb2e5ede748a1c0738ed158d0f5) nautilus (stable)
new ceph version with hotfix - ceph version 14.2.11-139.0.hotfix.bz1959254.el8cp (5c0dc966af809fd1d429ec7bac48962a746af243) nautilus (stable)

Signed-off-by: Santosh Pillai <sapillai@redhat.com>
2021-06-10 12:21:37 +05:30
Sébastien Han 677c93f3aa ceph: operate over object store pools concurrently
We now create and delete pools in parallel.
Before this patch, the deletion:

```
2021-06-08 13:48:26.557328 I | cephclient: no images/snapshosts present in pool "my-store.rgw.control"
2021-06-08 13:48:26.557383 I | cephclient: purging pool "my-store.rgw.control" (id=1)
2021-06-08 13:48:27.030021 I | ceph-object-controller: done disabling the dashboard api secret key
2021-06-08 13:48:28.800661 I | cephclient: purge completed for pool "my-store.rgw.control"
2021-06-08 13:48:29.085744 I | cephclient: no images/snapshosts present in pool "my-store.rgw.meta"
2021-06-08 13:48:29.085762 I | cephclient: purging pool "my-store.rgw.meta" (id=3)
2021-06-08 13:48:30.835487 I | cephclient: purge completed for pool "my-store.rgw.meta"
2021-06-08 13:48:31.128349 I | cephclient: no images/snapshosts present in pool "my-store.rgw.log"
2021-06-08 13:48:31.128368 I | cephclient: purging pool "my-store.rgw.log" (id=4)
2021-06-08 13:48:32.864376 I | cephclient: purge completed for pool "my-store.rgw.log"
2021-06-08 13:48:33.180923 I | cephclient: no images/snapshosts present in pool "my-store.rgw.buckets.index"
2021-06-08 13:48:33.181049 I | cephclient: purging pool "my-store.rgw.buckets.index" (id=5)
2021-06-08 13:48:34.892163 I | cephclient: purge completed for pool "my-store.rgw.buckets.index"
2021-06-08 13:48:35.179952 I | cephclient: no images/snapshosts present in pool "my-store.rgw.buckets.non-ec"
2021-06-08 13:48:35.179994 I | cephclient: purging pool "my-store.rgw.buckets.non-ec" (id=6)
2021-06-08 13:48:37.376067 I | cephclient: purge completed for pool "my-store.rgw.buckets.non-ec"
2021-06-08 13:48:37.665201 I | cephclient: no images/snapshosts present in pool "my-store.rgw.buckets.data"
2021-06-08 13:48:37.665218 I | cephclient: purging pool "my-store.rgw.buckets.data" (id=8)
2021-06-08 13:48:39.394717 I | cephclient: purge completed for pool "my-store.rgw.buckets.data"
2021-06-08 13:48:39.725528 I | cephclient: no images/snapshosts present in pool ".rgw.root"
2021-06-08 13:48:39.725567 I | cephclient: purging pool ".rgw.root" (id=7)
2021-06-08 13:48:41.424628 I | cephclient: purge completed for pool ".rgw.root"
```

It took 15sec to cleanup...
Now with this patch:

```
2021-06-08 14:15:25.621472 I | cephclient: no images/snapshosts present in pool "my-store.rgw.buckets.index"
2021-06-08 14:15:25.621570 I | cephclient: purging pool "my-store.rgw.buckets.index" (id=12)
2021-06-08 14:15:25.668708 I | cephclient: no images/snapshosts present in pool "my-store.rgw.log"
2021-06-08 14:15:25.668729 I | cephclient: purging pool "my-store.rgw.log" (id=11)
2021-06-08 14:15:25.693002 I | cephclient: no images/snapshosts present in pool "my-store.rgw.meta"
2021-06-08 14:15:25.693050 I | cephclient: purging pool "my-store.rgw.meta" (id=10)
2021-06-08 14:15:25.698830 I | cephclient: no images/snapshosts present in pool "my-store.rgw.control"
2021-06-08 14:15:25.698854 I | cephclient: purging pool "my-store.rgw.control" (id=9)
2021-06-08 14:15:25.701732 I | cephclient: no images/snapshosts present in pool "my-store.rgw.buckets.non-ec"
2021-06-08 14:15:25.701758 I | cephclient: purging pool "my-store.rgw.buckets.non-ec" (id=13)
2021-06-08 14:15:25.702816 I | cephclient: no images/snapshosts present in pool ".rgw.root"
2021-06-08 14:15:25.702836 I | cephclient: purging pool ".rgw.root" (id=14)
2021-06-08 14:15:25.717144 I | cephclient: no images/snapshosts present in pool "my-store.rgw.buckets.data"
2021-06-08 14:15:25.717212 I | cephclient: purging pool "my-store.rgw.buckets.data" (id=15)
2021-06-08 14:15:26.393246 I | ceph-object-controller: done disabling the dashboard api secret key
```

It tool around 1sec.

The creation before this patch:

```

2021-06-08 16:47:10.669484 I | ceph-spec: adding finalizer "cephobjectstore.ceph.rook.io" on "my-store"
2021-06-08 16:47:10.677253 E | ceph-object-controller: failed to set object store "rook-ceph/my-store" status to "Progressing". failed to update object "my-store" status: Operation cannot be fulfilled on cephobjectstores.ceph.rook.io "my-store": the object has been modified; please apply your changes to the latest version and try again
2021-06-08 16:47:10.682231 I | op-mon: parsing mon endpoints: b=10.111.63.108:6789,c=10.108.123.222:6789,a=10.100.223.211:6789
2021-06-08 16:47:10.985002 I | ceph-object-controller: reconciling object store deployments
2021-06-08 16:47:11.002364 I | ceph-object-controller: ceph object store gateway service running at 10.99.31.5
2021-06-08 16:47:11.002384 I | ceph-object-controller: reconciling object store pools
2021-06-08 16:47:14.336086 I | cephclient: setting pool property "compression_mode" to "none" on pool "my-store.rgw.control"
2021-06-08 16:47:16.115072 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-minikube": the object has been modified; please apply your changes to the latest version and try again
2021-06-08 16:47:16.138224 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-minikube": the object has been modified; please apply your changes to the latest version and try again
2021-06-08 16:47:16.391914 I | cephclient: creating replicated pool my-store.rgw.control succeeded
2021-06-08 16:47:16.391943 I | cephclient: setting pool property "pg_num_min" to "8" on pool "my-store.rgw.control"
2021-06-08 16:47:20.390562 I | cephclient: setting pool property "compression_mode" to "none" on pool "my-store.rgw.meta"
2021-06-08 16:47:22.423269 I | cephclient: creating replicated pool my-store.rgw.meta succeeded
2021-06-08 16:47:22.423294 I | cephclient: setting pool property "pg_num_min" to "8" on pool "my-store.rgw.meta"
2021-06-08 16:47:26.502874 I | cephclient: setting pool property "compression_mode" to "none" on pool "my-store.rgw.log"
2021-06-08 16:47:28.555252 I | cephclient: creating replicated pool my-store.rgw.log succeeded
2021-06-08 16:47:28.555308 I | cephclient: setting pool property "pg_num_min" to "8" on pool "my-store.rgw.log"
2021-06-08 16:47:32.619546 I | cephclient: setting pool property "compression_mode" to "none" on pool "my-store.rgw.buckets.index"
2021-06-08 16:47:34.663020 I | cephclient: creating replicated pool my-store.rgw.buckets.index succeeded
2021-06-08 16:47:34.663050 I | cephclient: setting pool property "pg_num_min" to "8" on pool "my-store.rgw.buckets.index"
2021-06-08 16:47:38.712886 I | cephclient: setting pool property "compression_mode" to "none" on pool "my-store.rgw.buckets.non-ec"
2021-06-08 16:47:40.748430 I | cephclient: creating replicated pool my-store.rgw.buckets.non-ec succeeded
2021-06-08 16:47:40.748467 I | cephclient: setting pool property "pg_num_min" to "8" on pool "my-store.rgw.buckets.non-ec"
2021-06-08 16:47:44.830659 I | cephclient: setting pool property "compression_mode" to "none" on pool ".rgw.root"
2021-06-08 16:47:46.875834 I | cephclient: creating replicated pool .rgw.root succeeded
2021-06-08 16:47:46.875863 I | cephclient: setting pool property "pg_num_min" to "8" on pool ".rgw.root"
2021-06-08 16:47:50.933717 I | cephclient: setting pool property "compression_mode" to "none" on pool "my-store.rgw.buckets.data"
2021-06-08 16:47:52.969461 I | cephclient: creating replicated pool my-store.rgw.buckets.data succeeded
2021-06-08 16:47:52.969549 I | ceph-object-controller: setting multisite settings for object store "my-store"
2021-06-08 16:47:53.585737 I | ceph-object-controller: Multisite for object-store: realm=my-store, zonegroup=my-store, zone=my-store
2021-06-08 16:47:53.585777 I | ceph-object-controller: multisite configuration for object-store my-store is complete
2021-06-08 16:47:53.585787 I | ceph-object-controller: creating object store "my-store" in namespace "rook-ceph"
2021-06-08 16:47:53.585802 I | cephclient: getting or creating ceph auth key "client.rgw.my.store.a"
2021-06-08 16:47:53.930416 I | ceph-object-controller: setting rgw config flags
2021-06-08 16:47:53.931363 I | op-config: setting "client.rgw.my.store.a"="rgw_enable_usage_log"="true" option to the mon configuration database
2021-06-08 16:47:54.201200 I | op-config: successfully set "client.rgw.my.store.a"="rgw_enable_usage_log"="true" option to the mon configuration database
2021-06-08 16:47:54.201225 I | op-config: setting "client.rgw.my.store.a"="rgw_zone"="my-store" option to the mon configuration database
2021-06-08 16:47:54.454589 I | op-config: successfully set "client.rgw.my.store.a"="rgw_zone"="my-store" option to the mon configuration database
2021-06-08 16:47:54.454614 I | op-config: setting "client.rgw.my.store.a"="rgw_zonegroup"="my-store" option to the mon configuration database
2021-06-08 16:47:54.710218 I | op-config: successfully set "client.rgw.my.store.a"="rgw_zonegroup"="my-store" option to the mon configuration database
2021-06-08 16:47:54.710237 I | op-config: setting "client.rgw.my.store.a"="rgw_log_nonexistent_bucket"="true" option to the mon configuration database
2021-06-08 16:47:54.969049 I | op-config: successfully set "client.rgw.my.store.a"="rgw_log_nonexistent_bucket"="true" option to the mon configuration database
2021-06-08 16:47:54.969069 I | op-config: setting "client.rgw.my.store.a"="rgw_log_object_name_utc"="true" option to the mon configuration database
2021-06-08 16:47:55.255491 I | op-config: successfully set "client.rgw.my.store.a"="rgw_log_object_name_utc"="true" option to the mon configuration database
2021-06-08 16:47:55.255631 I | ceph-object-controller: object store "my-store" deployment "rook-ceph-rgw-my-store-a" started
2021-06-08 16:47:55.279316 I | ceph-object-controller: enabling rgw dashboard
2021-06-08 16:47:55.361208 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-minikube": the object has been modified; please apply your changes to the latest version and try again
2021-06-08 16:47:56.225904 I | ceph-object-controller: setting the dashboard api secret key
2021-06-08 16:47:56.225984 I | ceph-object-controller: created object store "my-store" in namespace "rook-ceph"
2021-06-08 16:47:56.589452 I | ceph-object-controller: starting rgw healthcheck
2021-06-08 16:47:56.650045 I | ceph-object-controller: done setting the dashboard api secret key
```

It took 46sec, and I've seen it taking almost a 1min sometimes.
Now with this patch:

```
2021-06-08 16:51:35.259558 I | ceph-spec: adding finalizer "cephobjectstore.ceph.rook.io" on "my-store"
2021-06-08 16:51:35.270524 E | ceph-object-controller: failed to set object store "rook-ceph/my-store" status to "Progressing". failed to update object "my-store" status: Operation cannot be fulfilled on cephobjectstores.ceph.rook.io "my-store": the object has been modified; please apply your changes to the latest version and try again
2021-06-08 16:51:35.274387 I | op-mon: parsing mon endpoints: b=10.111.63.108:6789,c=10.108.123.222:6789,a=10.100.223.211:6789
2021-06-08 16:51:35.599023 I | ceph-object-controller: reconciling object store deployments
2021-06-08 16:51:35.607256 I | ceph-object-controller: ceph object store gateway service running at 10.104.254.110
2021-06-08 16:51:35.607315 I | ceph-object-controller: reconciling object store pools
2021-06-08 16:51:39.337735 I | cephclient: setting pool property "compression_mode" to "none" on pool "my-store.rgw.buckets.non-ec"
2021-06-08 16:51:40.343290 I | cephclient: setting pool property "compression_mode" to "none" on pool ".rgw.root"
2021-06-08 16:51:40.346335 I | cephclient: setting pool property "compression_mode" to "none" on pool "my-store.rgw.buckets.index"
2021-06-08 16:51:40.362670 I | cephclient: setting pool property "compression_mode" to "none" on pool "my-store.rgw.meta"
2021-06-08 16:51:40.363445 I | cephclient: setting pool property "compression_mode" to "none" on pool "my-store.rgw.log"
2021-06-08 16:51:40.364321 I | cephclient: setting pool property "compression_mode" to "none" on pool "my-store.rgw.control"
2021-06-08 16:51:41.412870 I | cephclient: creating replicated pool my-store.rgw.buckets.non-ec succeeded
2021-06-08 16:51:41.412904 I | cephclient: setting pool property "pg_num_min" to "8" on pool "my-store.rgw.buckets.non-ec"
2021-06-08 16:51:42.460439 I | cephclient: creating replicated pool my-store.rgw.control succeeded
2021-06-08 16:51:42.460503 I | cephclient: setting pool property "pg_num_min" to "8" on pool "my-store.rgw.control"
2021-06-08 16:51:42.470666 I | cephclient: creating replicated pool my-store.rgw.meta succeeded
2021-06-08 16:51:42.470727 I | cephclient: setting pool property "pg_num_min" to "8" on pool "my-store.rgw.meta"
2021-06-08 16:51:42.473029 I | cephclient: creating replicated pool .rgw.root succeeded
2021-06-08 16:51:42.473064 I | cephclient: setting pool property "pg_num_min" to "8" on pool ".rgw.root"
2021-06-08 16:51:42.473895 I | cephclient: creating replicated pool my-store.rgw.buckets.index succeeded
2021-06-08 16:51:42.473923 I | cephclient: setting pool property "pg_num_min" to "8" on pool "my-store.rgw.buckets.index"
2021-06-08 16:51:42.478625 I | cephclient: creating replicated pool my-store.rgw.log succeeded
2021-06-08 16:51:42.478671 I | cephclient: setting pool property "pg_num_min" to "8" on pool "my-store.rgw.log"
2021-06-08 16:51:46.606905 I | cephclient: setting pool property "compression_mode" to "none" on pool "my-store.rgw.buckets.data"
2021-06-08 16:51:48.627320 I | cephclient: creating replicated pool my-store.rgw.buckets.data succeeded
2021-06-08 16:51:48.627368 I | ceph-object-controller: setting multisite settings for object store "my-store"
2021-06-08 16:51:49.325086 I | ceph-object-controller: Multisite for object-store: realm=my-store, zonegroup=my-store, zone=my-store
2021-06-08 16:51:49.325108 I | ceph-object-controller: multisite configuration for object-store my-store is complete
2021-06-08 16:51:49.325121 I | ceph-object-controller: creating object store "my-store" in namespace "rook-ceph"
2021-06-08 16:51:49.325134 I | cephclient: getting or creating ceph auth key "client.rgw.my.store.a"
2021-06-08 16:51:49.657898 I | ceph-object-controller: setting rgw config flags
2021-06-08 16:51:49.657920 I | op-config: setting "client.rgw.my.store.a"="rgw_log_object_name_utc"="true" option to the mon configuration database
2021-06-08 16:51:49.917898 I | op-config: successfully set "client.rgw.my.store.a"="rgw_log_object_name_utc"="true" option to the mon configuration database
2021-06-08 16:51:49.917917 I | op-config: setting "client.rgw.my.store.a"="rgw_enable_usage_log"="true" option to the mon configuration database
2021-06-08 16:51:50.184939 I | op-config: successfully set "client.rgw.my.store.a"="rgw_enable_usage_log"="true" option to the mon configuration database
2021-06-08 16:51:50.184970 I | op-config: setting "client.rgw.my.store.a"="rgw_zone"="my-store" option to the mon configuration database
2021-06-08 16:51:50.463799 I | op-config: successfully set "client.rgw.my.store.a"="rgw_zone"="my-store" option to the mon configuration database
2021-06-08 16:51:50.463821 I | op-config: setting "client.rgw.my.store.a"="rgw_zonegroup"="my-store" option to the mon configuration database
2021-06-08 16:51:50.720953 I | op-config: successfully set "client.rgw.my.store.a"="rgw_zonegroup"="my-store" option to the mon configuration database
2021-06-08 16:51:50.720999 I | op-config: setting "client.rgw.my.store.a"="rgw_log_nonexistent_bucket"="true" option to the mon configuration database
2021-06-08 16:51:50.974875 I | op-config: successfully set "client.rgw.my.store.a"="rgw_log_nonexistent_bucket"="true" option to the mon configuration database
2021-06-08 16:51:50.975024 I | ceph-object-controller: object store "my-store" deployment "rook-ceph-rgw-my-store-a" started
2021-06-08 16:51:51.019820 I | ceph-object-controller: enabling rgw dashboard
2021-06-08 16:51:51.980117 I | ceph-object-controller: created object store "my-store" in namespace "rook-ceph"
2021-06-08 16:51:51.980199 I | ceph-object-controller: setting the dashboard api secret key
2021-06-08 16:51:52.399997 I | ceph-object-controller: done setting the dashboard api secret key
2021-06-08 16:51:52.435936 I | ceph-object-controller: starting rgw healthcheck
```

It took 17sec.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-06-09 16:09:22 +02:00
Sébastien Han e17fa5be51 Merge pull request #7998 from leseb/replace-radosgw-admin-cli-with-goceph
ceph: stop using radosgw-admin CLI for s3 user management
2021-06-09 16:01:14 +02:00
Sébastien Han 90bea8a560 ceph: stop using radosgw-admin CLI for s3 user management
We have been having many issues with external mode with Ceph version
mismatching. The operator would have a Ceph version different than the
external cluster. The `radosgw-admin` was used to interact with S3
users, even a small version delta would cause the command to coredump.
After checking with the rgw core team it appears Rook was misusing the
CLI and the admin ops API should be used instead.
So this patch is the first introduction of go-ceph in Rook to consume
the rgw admin ops API instead of the `radosgw-admin` CLI, **only** for
user management in this initial commit.
Later we can do more such as bucket operation, zone management etc.

Closes: https://github.com/rook/rook/issues/7924
Signed-off-by: Sébastien Han <seb@redhat.com>
2021-06-09 11:08:22 +02:00
Sébastien Han 3f0b42fd59 Merge pull request #8086 from BlaineEXE/duplicate-delete-reconcile-events
ceph: fix detection of delete event reconciliation
2021-06-09 10:36:43 +02:00
Blaine Gardner 386eeb7eb9 ceph: fix detection of delete event reconciliation
Fix a bug where delete events on Rook-Ceph CRs would be detected
multiple times if deletion was blocked. Old method used pointer
comparison instead of using the (metav1.Time) Equal() method.

Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
2021-06-08 16:48:53 -06:00
Travis Nielsen 06fbf29b3c Merge pull request #8028 from degorenko/device-classes-resource-limits
ceph: add ability to set resource limit for OSDs based on device classes
2021-06-08 09:01:14 -06:00
Denis Egorenko 8a556fd847 ceph: add ability to set resource limit for OSDs based on device classes
Currently it is not possible to set resource limits based on device classes
for different OSDs. Now adding an ability to use predefined keys for main
cluster spec Resource section to reflect resource limits for different OSDs
with different device classes.

Closes: https://github.com/rook/rook/issues/8007
Signed-off-by: Denis Egorenko <degorenko@mirantis.com>
2021-06-08 17:46:41 +04:00
Travis Nielsen a8764a6df1 Merge pull request #8043 from Rakshith-R/cleanup-csi-statefulset
ceph: remove obsolete statefulset functions
2021-06-03 11:06:49 -06:00