earlier, to change the operator log level we need
to change the deployment which requires a restart
the operator pod.
Now, we are changing the log level dynamically
by reading the configmap. For, backward compatibility
we still load from env var.
Signed-off-by: subhamkrai <srai@redhat.com>
radosgw-admin may output logs before and after JSON when this command succeeds. We should
get rid of these logs before unmarshaling output. Although the logs before JSON were
treated in PR7354, the logs after JSON weren't.
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
Ceph v16 will be the default in Rook v1.7. Update references in the
master branch from v15 (currently v15.2.11) to v16 (currently v16.2.4).
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
`rbd mirror pool peer bootstrap create` will use cluster fsid by default, so specifying --site-name shouldn't be needed at all.
Signed-off-by: Sébastien Han <seb@redhat.com>
The 'ceph-volume raw list' command used to list OSDs on nodes by the
rook-ceph-osd-prepare- jobs return user-friendly device names (e.g.,
/dev/sda) rather than persistent device names. Friendly device names
often change when systems reboot, and even persistent device names can
change under certain circumstances (e.g., a disk controller is replaced
with a different one). To prevent OSD Pods created for raw OSDs on
nodes from crashing repeatedly if the device it was created with is
renamed, scan the list of OSDs on the host with 'ceph-volume raw list'
to find the right OSD to activate in the OSD pod's OSD activation init
container.
Fixes#7918
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
If the monitors addresses change, we must update the peer secret with
the new addresses. For this, each time the configmap
"rook-ceph-mon-endpoints" is updated, we trigger a reconcile on the
CephBlockPool CRs, which will update the token secret.
Signed-off-by: Sébastien Han <seb@redhat.com>
The rook.io/v1 package was only an internal implementation detail and
does not have any CRDs that rely on it. The CRD deserialization should
handle the change in internal types without any issue. This separation
gives more flexibility for the storage providers to implement exactly
what is needed for their storage provider instead of forcing to use the
same types and risk affecting another storage provider.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Removing the site's name helps us reducing the size of the rbd-mirror
socket path which under certain circumstances could exceed the system
limit.
Also, the namespace information is not that relevant, the cluster FSID
is.
Signed-off-by: Sébastien Han <seb@redhat.com>
If the ceph cli outputs an error with "error calling conf_read_file"
this means that the operator has not written its ceph configuration
file. Thus ceph cli commands will fail, so we can just ignore that since
the operator will soon write this file in its initialization sequence.
Signed-off-by: Sébastien Han <seb@redhat.com>
With this change, the Rook ceph cluster can have only a public network or
only a cluster network or both can be specified. Prior to this commit if
only a public network was specified, it was copied to the cluster network.
Signed-off-by: rohan47 <rohgupta@redhat.com>
After mon failover is initiated, there was a time window where if the operator
was restarted, the new mon is started and has joined quorum, but the operator
does not believe the mon should be in quorum after the operator restart.
The operator was mistakenly removing the extra mon prematurely, sometimes
causing quorum to be lost if another mon was also down at the same time.
If the mon does not come back online, steps to recover quroum would need
to be followed from the disaster guide. Now the expected list of mons
will be updated immediately during mon failover if the operator successfully
created the new mon deployment, thus removing the window where restarting
the operator can cause quorum loss.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Changing the livenessprobe handler is not possible anymore for the
simple reason that it does not make sense. Practicly speaking we just
cannot change its value since it is propapaged to all the daemons. The
exec handler contains the socket name of the daemon, because daemons
have different names we cannot possibly make it generic since it will
apply to all.
Closes: https://github.com/rook/rook/issues/7842
Signed-off-by: Sébastien Han <seb@redhat.com>
A directory is created under dataDirHostPath for each OSD. In host-based cluster,
its name is constructed from kernel device name like _dev_sdb though user specifies
udev persistent device name. We should use unique name among all OSDs.
Closes: https://github.com/rook/rook/issues/7878
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
If the cluster is external we want to periodically rehydrate the mgr
endpoint. This handles the scenarion where the active manager changes,
so we need to update the endpoint with the new IP address.
The create-external-cluster-resources.py script now requires an extra
permission to query the manager service so Rook can discover the active
one and its IP.
Testing:
```
[leseb@tarox~/go/src/github.com/rook/rook][external-active-mgr-change] minikube kubectl -- exec -n rook-ceph deploy/rook-ceph-tools -ti -- ceph mgr stat
{
"epoch": 37,
"available": true,
"active_name": "b",
"num_standby": 1
}
[leseb@tarox~/go/src/github.com/rook/rook][external-active-mgr-change] kubectl -n rook-ceph-external get ep
NAME ENDPOINTS AGE
rook-ceph-mgr-external 172.17.0.12:9283 3m10s
[leseb@tarox~/go/src/github.com/rook/rook][external-active-mgr-change] k scale --replicas=0 deployment rook-ceph-mgr-b
deployment.apps/rook-ceph-mgr-b scaled
[leseb@tarox~/go/src/github.com/rook/rook][external-active-mgr-change] minikube kubectl -- exec -n rook-ceph deploy/rook-ceph-tools -ti -- ceph mgr stat
{
"epoch": 40,
"available": true,
"active_name": "a",
"num_standby": 0
}
[leseb@tarox~/go/src/github.com/rook/rook][external-active-mgr-change] kubectl -n rook-ceph-external get ep
NAME ENDPOINTS AGE
rook-ceph-mgr-external 172.17.0.13:9283 3m55s
```
Signed-off-by: Sébastien Han <seb@redhat.com>
there was confusion regarding crash collector pods
in pending state which was basically due to secret
`rook-ceph-crash-collector-keyring` was not created
till mons are in quorum. this commit creates crash
collector pods after secret `rook-ceph-crash-collector-keyring`
is created.
Signed-off-by: subhamkrai <srai@redhat.com>
This commit removes redundant `RBDProvisionerSTSTemplatePath` and
`CephFSProvisionerSTSTemplatePath` variables which were forgetten to
be removed in https://github.com/rook/rook/pull/5982 when csi support
for k8s 1.13 was removed.
Signed-off-by: Rakshith R <rar@redhat.com>
We don't need to set log_file to empty string since "log_to_file" is set
to False. This is probably an old leftover/hack we were doing in the
past when "log_to_file" and other related options were not there.
Because of this line
https://github.com/ceph/ceph/blob/master/src/perfglue/heap_profiler.cc#L99,
Ceph looks for the log_file conf option. In our case it was empty, so
the logging would default to the current directory which points to the
container runtime root when not set and we don't have permission to
write there.
We now force it to Ceph's default so that existing cluster will get the
fix after upgrading.
Just removing the option allows us to get dump in /var/log/ceph.
Phew, what a bug!
Signed-off-by: Sébastien Han <seb@redhat.com>
This PR has following changes around OSD pdbs
- Detect node drains more reliably. When OSD is down and OSD pod is not scheduled to any node or if the scheduled node is not ready, then assume that node is draining.
- Don't set no-out flag on the failure domain if the OSD is down due to reasons other than node drain (say disk failure)
- update unit tests
- update design doc to reflect above changes
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
In some clusters, the schema validation is failing for the default
object.yaml example that the value must be of a type string null.
While the null value shouldnt be strictly needed, we need working
examples by default.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
A user with unbalanced system (e.g., some OSDs use a partition which
resides on a device along-side OS partition) may want to have a
fine-grained control on the overall load of specific OSD. While osd
weight affects write-load, primary-affinity affects read-load.
This patch allow user to define osd's primary-affinity (similar to
'DeviceClass' and 'InitialWeight'), which is sent to the osd (via mon)
as part of deployment sequence, using 'ceph osd primary-affinity' command.
Valid values are within the range [0.0, 1.0). If not set, default value
is 1.
ROOK issue: https://github.com/rook/rook/issues/7773
Signed-off-by: Shachar Sharon <ssharon@redhat.com>
The unit test TestConfigureCVDevices was failing intermittently
due to comparing the first element of the slice of two elements.
The slice ordering is not guaranteed, which caused a failure
intermittently. The test only compares the critical values in the test
and no need to check elements of the slice.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Events like node drain can take more than 10-15 minutes. If the node is not update the default monTimeOut of 10 minutes, then rook will attempt to fail over the mon. This failover won't work as the node is still down.
This PR retries once before the mon failover if the mon pod is not scheduled
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
This commit updates external-snapshotter version to
v4.0.0 which supports snapshots v1.
Rook now defaults to enabling RBD and CephFS snapshotter
for K8s >= v1.17 and disabling it for K8s <= v1.16.
Supporting changes in documents and examples yaml files
are made.
Signed-off-by: Rakshith R <rar@redhat.com>
The deviceClass was only being set for LVM mode OSDs and was
missed for the raw mode for non-PVC. Now the deviceClass will be set
for all expected scenarios.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
currently, recreating erasure codedpPool with
different Settings show error and creation
failed.
Adding `--force` to the erasure coded profile
command works file with above condition.
Signed-off-by: subhamkrai <srai@redhat.com>
The VolumeSource type was only used internally when loading
the storageClassDeviceSets. The volumeSource element is never
expected to be specified in the CephCluster CR, so it is now
removed from the schema. The VolumeSource type is renamed to
deviceSet for clarity.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Because of the recent CRD changes made, the mon count had a minimum of
1, making the configuration of the external cluster impossible. The
operator would fail to add the finalizer:
```
2021-04-29 16:57:38.429451 E | ceph-cluster-controller: failed to reconcile. failed to add finalizer: failed to add finalizer "cephcluster.ceph.rook.io" on "test-external": CephCluster.ceph.rook.io "test-external" is invalid: spec.mon.count: Invalid value: 0: spec.mon.count in body should be greater than or equal to 1
```
We now fixed the CRD as well as adding the CR status.
Signed-off-by: Sébastien Han <seb@redhat.com>
The schema in v1.6 is more strict and we are seeing errors in the csv
upgrades where nullable values are not allowed for the pool settings.
We now allow the nullable settings again to fix the upgrade scenario.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Jiffin Tony Thottan
Sébastien Han
subhamkrai
Satoru Takeuchi
Blaine Gardner
Blaine Gardner
Travis Nielsen
Nitin Goyal
rohan47
Rakshith R
Santosh Pillai
Shachar Sharon