Ceph will use a random nonce for the msgr daemons as long as
they are running as PID 1 or have the environment variable
CEPH_USE_RANDOM_NONCE set to indicate running in a containerized
environment. While the Ceph daemons in Rook are expected to
run in PID 1, there are some environments where the container
runtime may choose to run something else such as
/usr/bin/pod as PID 1. The side effect then is that ceph uses
the pid as the nonce and causes problems when the daemons
restart with the same PID.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
This commit implements this corner case during osd-prepare job.
```
The encrypted block is not opened, this is an extreme corner case
The OSD deployment has been removed manually AND the node rebooted
So we need to re-open the block to re-hydrate the OSDInfo.
Handling this case would mean, writing the encryption key on a
temporary file, then call luksOpen to open the encrypted block and
then call ceph-volume to list against the opened encrypted block.
We don't implement this, yet and return an error.
```
When underlying PVC for osd are CSI provisioned, the encrypted device
is closed when PVC is unmounted due to osd pod being deleted.
Therefore, this may occur more frequently and needs to be handled.
This commit implements the fix for the same.
Signed-off-by: Rakshith R <rar@redhat.com>
We can use LVM logical volumes and loop devices as OSD iff we specify
them in the `devices` field. If we use filters like `deviceFilter`,
these devices are not picked with a log message which let users
to specify the exact device name. It's better to mention which field
users should use in this case.
Closes: https://github.com/rook/rook/issues/11353
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
When mirror logs are enabled, `ceph osd pool get <poolname> all` command returns peer pool info along
with the debug logs. This causes error while parsing the logs for json
data. Using --out-file flag saves the json data in the file which can be
parsed easily while logging will still be written to stdout/stderr.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
The default value of maxUnavailable of daemonset is 1.
Sometimes the upgrade can be slow due to this setting.
So I add a parameter to change the value for rbd daemonset.
Signed-off-by: Shinya Hayashi <shinya-hayashi@cybozu.co.jp>
Co-authored-by: Toshikuni Fukaya <toshikuni-fukaya@cybozu.co.jp>
Co-authored-by: Yuji Ito <llamerada.jp@gmail.com>
The fsid, username, and user secrets were from legacy and no
longer needed on the osd daemon or mon pods. This removes the
unnecessary env vars from the pod specs.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
A new variable is added to rook-ceph-operator-config
ConfigMap to allow using loop devices for osd.
This feature is intended to be used for testing purposes only.
Signed-off-by: Shinya Hayashi <shinya-hayashi@cybozu.co.jp>
For the upgrade case, Rook can fail to set status information
on CephObjectStores after CRDs are updated but before the operator is
updated. To fix this, merely allow the slices to be null in the
CephObjectStore's status.endpoints.
This PR seems to be aggravating the helm filesystem upgrade test. Allow
30 more seconds for CRDs to be done before bailing. In debugging, the
filesystem often needed only an extra 3 seconds to succeed.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
The mons that are out of quorum may cause ceph commands to timeout
or fail unnecessarily trying to connect to a mon that is no longer
online. Now the mon health check will update the mon endpoints configmap
when a mon is detected out of quorum. This also means that if the
operator is restarted during a mon failover, the failed mon will no
longer remain in the ceph.conf, thus allowing the quorum to be more
likely to respond to the mons that are still in quorum.
The update for mons out of quorum only applies if other mons are in
quorum. If quorum is down, the configmap will not keep track of the offline
mons since too many are offline.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Adding two new entries(userID/userKey) in the cephClient secret, so that the csi driver can easily consume
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
External CephObjectStores already have endpoints defined by
spec.gateway.externalRgwEndpoints, and if the external store is
configured with TLS (HTTPS), the store's certificates will likely not
accept connections intended for the Service endpoint Rook creates. Some
users might not be able to easily add the service endpoint to their
certificates. Therefore, don't even bother creating a Service for
external clusters.
This does introduce a few issues. The Service seems to have been
initially created to allow multiple external RGW endpoints to be
addressable via a single address in Rook. For all connections to an
external CephObjectStore with multiple endpoints, simply choose an
endpoint at random. Random selection will prevent Rook from failing to
create buckets or users on an external store if one of the external
store's endpoints fails.
The latest OBC library (lib-bucket-provisioner) allows updating the
endpoints on ObjectBuckets after they are created. This allows Rook
users to change endpoints on external CephObjectStores without breaking
all existing OBCs. It requires implementation of the new GetUserID()
library call, requires updating Provision() and Grant() calls to be
idempotent, and it requires removing the Update() call.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
The monstore has a SetAll method for applying multiple settings
that was applying them individually. Now we use the assimilate
config option to make this more efficient.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The crash collector does not have the command line arguments
to run as ceph user id 167, so we set the security context
to run as the ceph user in the main crash collector
container.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The osd arguments had a few duplicate lines that could be
factored out and thus simplified for code maintainability.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Removing the finalizers from cluster resources can intermittently fail
in the CI if other updates are made at similar times. The controller
runtime will retry after the failure, but if something external
removes the finalizer on the cephcluster CR, the deletion will not
continue with the removal of finalizers from the other configmap
and secret critical resources.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
If the host network setting is changed in the cluster CR,
the existing mons must continue using the same network
settings or else the operator would update their pod
specs with the incorrect settings and cause mon quorum
to go down. Now the network setting is preserved so
the admin could switch between host and non-host network
configuration without reinstalling.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
This updates the cephcsi to latest 3.7.2
release and also updates sidecar to the latest
release in some missing files.
Note:- sidecar images are upto date in most of
the places, in some places it was missing, This
PR updates it in missing place.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
The `periodWillChange()` implementation to detect map keys to ignore.
Previously, we relied on the `GoString()` method to output the same
format always to determine the current path to the
map[string]interface{} node, but that may change depending on the underlying
implementation of go-cmp.
Instead, parse the go-cmp `Path` step by step and construct a JSON path
to the node in a stable format.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
When the osd is purged, properly log the host name
that is being purged and whether the host was
removed from the crush map.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Remove the health checker for CephObjectStore. The liveness and
readiness probes go through the same code paths in RGW as creating
buckets without as much affect on the storage backend.
Full discussion: https://github.com/rook/rook/issues/11031
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
The omap generator sidecar is the one who
sets the pvc name and clustername in the metadata
if the pv is re-attached to a new pvc or if the
clustername changes, for that reason we need to
set these flags on the omap generator sidecar
also.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
Rook currently knows rbd pools only if they defined through CephBlockPool, so it overrides the config
mgr/prometheus/rbd_stats_pools which may have already been set for pools external to Rook. Check config
before running set command and if there some existing pools append them to enableStatsForCephBlockPools.
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
currently, osdSafeToDestroy check is done before
marking osd out which prevents ceph from rebalancing.
With this changes, we are moving check for
osdSafeToDestroy after osd is marked out.
Signed-off-by: subhamkrai <srai@redhat.com>
Travis Nielsen
Jiffin Tony Thottan
Satoru Takeuchi
Rakshith R
Blaine Gardner
Santosh Pillai
Sebastian Hasler
parth-gr
Avan Thakkar
subhamkrai
Madhu Rajanna