A new variable is added to rook-ceph-operator-config
ConfigMap to allow using loop devices for osd.
This feature is intended to be used for testing purposes only.
Signed-off-by: Shinya Hayashi <shinya-hayashi@cybozu.co.jp>
The mons that are out of quorum may cause ceph commands to timeout
or fail unnecessarily trying to connect to a mon that is no longer
online. Now the mon health check will update the mon endpoints configmap
when a mon is detected out of quorum. This also means that if the
operator is restarted during a mon failover, the failed mon will no
longer remain in the ceph.conf, thus allowing the quorum to be more
likely to respond to the mons that are still in quorum.
The update for mons out of quorum only applies if other mons are in
quorum. If quorum is down, the configmap will not keep track of the offline
mons since too many are offline.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Adding two new entries(userID/userKey) in the cephClient secret, so that the csi driver can easily consume
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
External CephObjectStores already have endpoints defined by
spec.gateway.externalRgwEndpoints, and if the external store is
configured with TLS (HTTPS), the store's certificates will likely not
accept connections intended for the Service endpoint Rook creates. Some
users might not be able to easily add the service endpoint to their
certificates. Therefore, don't even bother creating a Service for
external clusters.
This does introduce a few issues. The Service seems to have been
initially created to allow multiple external RGW endpoints to be
addressable via a single address in Rook. For all connections to an
external CephObjectStore with multiple endpoints, simply choose an
endpoint at random. Random selection will prevent Rook from failing to
create buckets or users on an external store if one of the external
store's endpoints fails.
The latest OBC library (lib-bucket-provisioner) allows updating the
endpoints on ObjectBuckets after they are created. This allows Rook
users to change endpoints on external CephObjectStores without breaking
all existing OBCs. It requires implementation of the new GetUserID()
library call, requires updating Provision() and Grant() calls to be
idempotent, and it requires removing the Update() call.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
The monstore has a SetAll method for applying multiple settings
that was applying them individually. Now we use the assimilate
config option to make this more efficient.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The crash collector does not have the command line arguments
to run as ceph user id 167, so we set the security context
to run as the ceph user in the main crash collector
container.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The osd arguments had a few duplicate lines that could be
factored out and thus simplified for code maintainability.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Removing the finalizers from cluster resources can intermittently fail
in the CI if other updates are made at similar times. The controller
runtime will retry after the failure, but if something external
removes the finalizer on the cephcluster CR, the deletion will not
continue with the removal of finalizers from the other configmap
and secret critical resources.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
If the host network setting is changed in the cluster CR,
the existing mons must continue using the same network
settings or else the operator would update their pod
specs with the incorrect settings and cause mon quorum
to go down. Now the network setting is preserved so
the admin could switch between host and non-host network
configuration without reinstalling.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
This updates the cephcsi to latest 3.7.2
release and also updates sidecar to the latest
release in some missing files.
Note:- sidecar images are upto date in most of
the places, in some places it was missing, This
PR updates it in missing place.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
The `periodWillChange()` implementation to detect map keys to ignore.
Previously, we relied on the `GoString()` method to output the same
format always to determine the current path to the
map[string]interface{} node, but that may change depending on the underlying
implementation of go-cmp.
Instead, parse the go-cmp `Path` step by step and construct a JSON path
to the node in a stable format.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
Remove the health checker for CephObjectStore. The liveness and
readiness probes go through the same code paths in RGW as creating
buckets without as much affect on the storage backend.
Full discussion: https://github.com/rook/rook/issues/11031
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
The omap generator sidecar is the one who
sets the pvc name and clustername in the metadata
if the pv is re-attached to a new pvc or if the
clustername changes, for that reason we need to
set these flags on the omap generator sidecar
also.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
Rook currently knows rbd pools only if they defined through CephBlockPool, so it overrides the config
mgr/prometheus/rbd_stats_pools which may have already been set for pools external to Rook. Check config
before running set command and if there some existing pools append them to enableStatsForCephBlockPools.
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
Run the ganesha-rados-grace command in a remote pod when multus
networking is enabled.
Signed-off-by: parth-gr <paarora@redhat.com>
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
If the port is greater than 1024, for backward compatibility the port and
targetPort should be the same value. If the port is less than 1024,
the internal port must use a higher port number. In that case, the internal
port will be the default port numbers and only the public port will be
the desired port in the cluster CR.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Introduce a new env variable ROOK_CSI_IMAGE_PULL_POLICY in rook operator configmap which should be used to
customize the imagePullPolicy for the csi driver and imagePullPolicy property in cephVersionSpec for ceph pods.
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
The sssd.conf may refer to additional files, like a CA bundle or TLS
certificates. These files need to be made available in the SSSD sidecar.
With the new `sssdGenericFiles` reference to a VolumeSource and a
`MountPath`, a provided `sssd.conf` can use the additional files.
Signed-off-by: Niels de Vos <ndevos@redhat.com>
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
if Multus is enabled the clusterinfo should be updated with
network as multus as to run the ceph cmds in remote
executor
Signed-off-by: parth-gr <paarora@redhat.com>
On removing or setting params to false in operator config map, make sure
to set CSIParam to their appropriate defaults, since we are not restarting the
operator we cannot just assume these params/variables/flags are defaulted,
everytime we need to makesure some defaults are specified.
This is what happens without these changes:
* Set CSI_ENABLE_METADATA to true, this will tune CSIParam.CSIEnableMetadata
to true from false
* Create a PVC and check the metadata is added to the rbd/cephfs volumes
* Now Set CSI_ENABLE_METADATA to false, we assume CSIParam.CSIEnableMetadata
will be set to false, but that is not happening because the earlier value
of `CSIParam.CSIEnableMetadata = true` still persist.
(we are not restarting operator to assume value of CSIParam.CSIEnableMetadata
to be reset to false)
This happens with CSI_ENABLE_OMAP_GENERATOR and many other flags too,
fixing them all with these changes.
Thanks to Madhu for debugging the issue and testing it along.
Credit: Madhu Rajanna <madhupr007@gmail.com>
Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
OBC provisioner ignores insecure skip flag even if it is declared for
TLS auth, hence bucket creation was failing
Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
I guesstimate that around 50% of the operator log volume when set to log
level `DEBUG` are messages about cm's which aren't relevant to rook.
E.g.:
```
2022-09-16 23:03:05.430839 D | ceph-cluster-controller: hot-plug cm watcher: only reconcile on hot plug cm changes, this "ingress-controller-leader" cm is handled by another watcher
2022-09-16 23:03:05.643255 D | ceph-cluster-controller: hot-plug cm watcher: only reconcile on hot plug cm changes, this "fleet-agent-lock" cm is handled by another watcher
2022-09-16 23:03:05.799526 D | ceph-cluster-controller: hot-plug cm watcher: only reconcile on hot plug cm changes, this "cattle-controllers" cm is handled by another watcher
```
Yes, it is `DEBUG` and noise is expected. However, there is no message
about checking the status of the cm's rook actually needs to watch and
these messages don't seem very useful. I have been resorting to piping
the operators logs through `| grep -v 'hot-plug cm watcher'` to cut down
the noise.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
This option open-up volumes and volumemounts
for user customization of the rbd,cephfs,nfs
plugin daemonset.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
Sebastian Hasler
Travis Nielsen
parth-gr
Shinya Hayashi
Avan Thakkar
Blaine Gardner
Satoru Takeuchi
subhamkrai
Madhu Rajanna
Humble Chirammal
Niels de Vos
Prasanna Kumar Kalever
Jiffin Tony Thottan
Joshua Hoblitt
zhucan