After mon failover is initiated, there was a time window where if the operator
was restarted, the new mon is started and has joined quorum, but the operator
does not believe the mon should be in quorum after the operator restart.
The operator was mistakenly removing the extra mon prematurely, sometimes
causing quorum to be lost if another mon was also down at the same time.
If the mon does not come back online, steps to recover quroum would need
to be followed from the disaster guide. Now the expected list of mons
will be updated immediately during mon failover if the operator successfully
created the new mon deployment, thus removing the window where restarting
the operator can cause quorum loss.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Changing the livenessprobe handler is not possible anymore for the
simple reason that it does not make sense. Practicly speaking we just
cannot change its value since it is propapaged to all the daemons. The
exec handler contains the socket name of the daemon, because daemons
have different names we cannot possibly make it generic since it will
apply to all.
Closes: https://github.com/rook/rook/issues/7842
Signed-off-by: Sébastien Han <seb@redhat.com>
If the cluster is external we want to periodically rehydrate the mgr
endpoint. This handles the scenarion where the active manager changes,
so we need to update the endpoint with the new IP address.
The create-external-cluster-resources.py script now requires an extra
permission to query the manager service so Rook can discover the active
one and its IP.
Testing:
```
[leseb@tarox~/go/src/github.com/rook/rook][external-active-mgr-change] minikube kubectl -- exec -n rook-ceph deploy/rook-ceph-tools -ti -- ceph mgr stat
{
"epoch": 37,
"available": true,
"active_name": "b",
"num_standby": 1
}
[leseb@tarox~/go/src/github.com/rook/rook][external-active-mgr-change] kubectl -n rook-ceph-external get ep
NAME ENDPOINTS AGE
rook-ceph-mgr-external 172.17.0.12:9283 3m10s
[leseb@tarox~/go/src/github.com/rook/rook][external-active-mgr-change] k scale --replicas=0 deployment rook-ceph-mgr-b
deployment.apps/rook-ceph-mgr-b scaled
[leseb@tarox~/go/src/github.com/rook/rook][external-active-mgr-change] minikube kubectl -- exec -n rook-ceph deploy/rook-ceph-tools -ti -- ceph mgr stat
{
"epoch": 40,
"available": true,
"active_name": "a",
"num_standby": 0
}
[leseb@tarox~/go/src/github.com/rook/rook][external-active-mgr-change] kubectl -n rook-ceph-external get ep
NAME ENDPOINTS AGE
rook-ceph-mgr-external 172.17.0.13:9283 3m55s
```
Signed-off-by: Sébastien Han <seb@redhat.com>
there was confusion regarding crash collector pods
in pending state which was basically due to secret
`rook-ceph-crash-collector-keyring` was not created
till mons are in quorum. this commit creates crash
collector pods after secret `rook-ceph-crash-collector-keyring`
is created.
Signed-off-by: subhamkrai <srai@redhat.com>
This commit removes redundant `RBDProvisionerSTSTemplatePath` and
`CephFSProvisionerSTSTemplatePath` variables which were forgetten to
be removed in https://github.com/rook/rook/pull/5982 when csi support
for k8s 1.13 was removed.
Signed-off-by: Rakshith R <rar@redhat.com>
We don't need to set log_file to empty string since "log_to_file" is set
to False. This is probably an old leftover/hack we were doing in the
past when "log_to_file" and other related options were not there.
Because of this line
https://github.com/ceph/ceph/blob/master/src/perfglue/heap_profiler.cc#L99,
Ceph looks for the log_file conf option. In our case it was empty, so
the logging would default to the current directory which points to the
container runtime root when not set and we don't have permission to
write there.
We now force it to Ceph's default so that existing cluster will get the
fix after upgrading.
Just removing the option allows us to get dump in /var/log/ceph.
Phew, what a bug!
Signed-off-by: Sébastien Han <seb@redhat.com>
This PR has following changes around OSD pdbs
- Detect node drains more reliably. When OSD is down and OSD pod is not scheduled to any node or if the scheduled node is not ready, then assume that node is draining.
- Don't set no-out flag on the failure domain if the OSD is down due to reasons other than node drain (say disk failure)
- update unit tests
- update design doc to reflect above changes
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
In some clusters, the schema validation is failing for the default
object.yaml example that the value must be of a type string null.
While the null value shouldnt be strictly needed, we need working
examples by default.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
A user with unbalanced system (e.g., some OSDs use a partition which
resides on a device along-side OS partition) may want to have a
fine-grained control on the overall load of specific OSD. While osd
weight affects write-load, primary-affinity affects read-load.
This patch allow user to define osd's primary-affinity (similar to
'DeviceClass' and 'InitialWeight'), which is sent to the osd (via mon)
as part of deployment sequence, using 'ceph osd primary-affinity' command.
Valid values are within the range [0.0, 1.0). If not set, default value
is 1.
ROOK issue: https://github.com/rook/rook/issues/7773
Signed-off-by: Shachar Sharon <ssharon@redhat.com>
The unit test TestConfigureCVDevices was failing intermittently
due to comparing the first element of the slice of two elements.
The slice ordering is not guaranteed, which caused a failure
intermittently. The test only compares the critical values in the test
and no need to check elements of the slice.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Events like node drain can take more than 10-15 minutes. If the node is not update the default monTimeOut of 10 minutes, then rook will attempt to fail over the mon. This failover won't work as the node is still down.
This PR retries once before the mon failover if the mon pod is not scheduled
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
This commit updates external-snapshotter version to
v4.0.0 which supports snapshots v1.
Rook now defaults to enabling RBD and CephFS snapshotter
for K8s >= v1.17 and disabling it for K8s <= v1.16.
Supporting changes in documents and examples yaml files
are made.
Signed-off-by: Rakshith R <rar@redhat.com>
The deviceClass was only being set for LVM mode OSDs and was
missed for the raw mode for non-PVC. Now the deviceClass will be set
for all expected scenarios.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
currently, recreating erasure codedpPool with
different Settings show error and creation
failed.
Adding `--force` to the erasure coded profile
command works file with above condition.
Signed-off-by: subhamkrai <srai@redhat.com>
The VolumeSource type was only used internally when loading
the storageClassDeviceSets. The volumeSource element is never
expected to be specified in the CephCluster CR, so it is now
removed from the schema. The VolumeSource type is renamed to
deviceSet for clarity.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Because of the recent CRD changes made, the mon count had a minimum of
1, making the configuration of the external cluster impossible. The
operator would fail to add the finalizer:
```
2021-04-29 16:57:38.429451 E | ceph-cluster-controller: failed to reconcile. failed to add finalizer: failed to add finalizer "cephcluster.ceph.rook.io" on "test-external": CephCluster.ceph.rook.io "test-external" is invalid: spec.mon.count: Invalid value: 0: spec.mon.count in body should be greater than or equal to 1
```
We now fixed the CRD as well as adding the CR status.
Signed-off-by: Sébastien Han <seb@redhat.com>
The schema in v1.6 is more strict and we are seeing errors in the csv
upgrades where nullable values are not allowed for the pool settings.
We now allow the nullable settings again to fix the upgrade scenario.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
when store.Status was not initialized it caused
crashes when assigning store.Status.Info["endpoint"]
to objContext.Endpoint.
This commits makes sure store.Status is not Nil
before setting objContext.Endpoint and
objContext.SecureEndpoint in the ObjectStoreUser
controller.
Closes: https://github.com/rook/rook/issues/6916
Signed-off-by: Ali Maredia <amaredia@redhat.com>
set fsPreparedForUpgrade to true for upgrading mds, to
make sure that max_mds is set back to active count
after all mds agent gets upgraded
Signed-off-by: shenjiatong <yshxxsjt715@gmail.com>
When DEBUG logging is enabled in the operator, there are a number of overwhelming messages in the log that are very overwhelming and don't seem useful, which makes debug mode difficult to use
Updated and Cutted down the messages that are not useful in debug mode
Closes: https://github.com/rook/rook/issues/7499
Signed-off-by: parth-gr <paarora@redhat.com>
Add `SecuritySpec` to specify kms details for rgw in `ObjectStoreSpec` than using existing
`SecuritySpec` used for `ClusterSpec`. The vault secret engines conflict with OSD and RGW
kms encrytpion
Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
The ValidateConnectionDetails() contains `clusterSpec` assuming `SecuritySpec` only
part to `cephCluster` CRD. But we may need to validate same for `SecuritySpec` in
`cephObjectStore` CRD. Hence modifying the signature of the api.
Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
OBC operations will fail if RGW only enabled HTTPS. With this change, OBC operations
will use needed TLS certs to succeed the operation.
Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
Rook operator sends S3 to RGW server for bucket operations performed by healthchecker,
bucketprovisioner etc. Add api to fetch cacert if the TLS enabled for RGW.
Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
Allow metadata on VolumeClaimTemplates in StorageClassDeviceSets to be
persisted. The latest changes to controller-gen allows us to embed
objectmeta for embedded types.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
The ceph upgrade blocks if the ceph heatlh is HEALTH_ERR. If the user
wants to force the upgrade despite the health error, they will need
to set skipUpgradeChecks: true in the cephcluster CR. This message
will hopefully make that setting more discoverable.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The nodeAffinity for portable OSDs is needed to ensure the OSDs are
tied to the same zone or other failure domain where the OSD was
prepared and where it is expected to run. During upgrade the topology
affinity was not being detected since the osd prepare job is skipped
in that case. Now the topology will be detected from the running
OSD daemon pod's node.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Travis Nielsen
Sébastien Han
subhamkrai
Rakshith R
Santosh Pillai
Shachar Sharon
shenjiatong
Ali Maredia
Blaine Gardner
parth-gr
Jiffin Tony Thottan
Blaine Gardner
Satoru Takeuchi