If the cluster is not successfully initialized during the onAdd event,
the cluster will not be in a properly initialized state. The admin
will likely need to update the CR, in which case the onUpdate event would
be triggered. In this case we need to call the same code path as the
onAdd event to properly initialize the cluster.
Signed-off-by: travisn <tnielsen@redhat.com>
With the 1.1 release we should not be creating bluestore OSDs
with rook's legacy partitioning scheme. All new bluestore OSDs
should be created with ceph-volume. Therefore, the miminum
version allowed is a version of mimic supporting ceph-volume.
This commit also adds consistent version checking between
the Add and Update methods for the operator handling the
CR events.
Signed-off-by: travisn <tnielsen@redhat.com>
Range checks for CR values are done statically in the CRD
for the number of mons. There is no need to check in the code
if the mon count is out of range.
Signed-off-by: travisn <tnielsen@redhat.com>
Instead of requring csi instances for every ceph cluster that rook
manages, maintain one config map for all ceph clusters. This allows
for one set of csi drivers per rook operator.
The config is intialized as empty when Ceph CSI components are first
created and then updated every time a cluster's mons change.
As before clusters are identified by the k8s namespace the ceph cluster
is running in.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
Create a mutex at the rook operator level that will be used when
updating the shared-between-clusters config map that Ceph CSI consumes.
Pass the mutex down to the function that will update the CM.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
Add a new function that, given existing config string, key, and current
mons, will return an updated version of the Ceph CSI config string
having updated the data with the new mons.
Add a unit test function for the update function.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
- Added code to support StorageClassDeviceSet spec provided in the cluster-on-pvc.yaml
- The code reads the StorageClassDeviceSet spec and creates pvc based on the ‘count’ field for each device set.
- OSD prepare job is started for each PVC which activates the ceph-volume on each PVC
- Finally OSD is started on each of the PVC device.
Co-authored-by: rohan47 <rohgupta@redhat.com>
Co-authored-by: Ashish Ranjan <aranjan@redhat.com>
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
when the root of a filesystem mode pvc is mounted as the base data directory
for the monitor it will not be empty: `lost+found` from ext4 will be present.
this causes the mon mkfs to fail because it expects an empty directory.
the solution is to mount under a directory in the source volume.
Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
the life cycle post-start hook is used to chown the ceph daemon log
directory. however, hooks are not guaranteed to run before a container
entry point.
"Kubernetes sends the postStart event immediately after the Container is
created. There is no guarantee, however, that the postStart handler is
called before the Container’s entrypoint is called"
Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
The CSI driver requires the ceph key for administrating
the cluster. The key is stored in a secret and specified in the
storage class. The operator will now automatically generate
a secret named rook-ceph-csi that can be specified by
default in the storage class instead of requiring the
admin to create it.
Signed-off-by: travisn <tnielsen@redhat.com>
a helper flag indicating the exitence of an empty zone (empty = no
monitor pods) is computed when checking the health of the cluster in
order to determine if mons from overloaded zones have some new location
they can be scheduled. the computation of this flag did not check that
the empty zone contains valid nodes that pods can be scheduled on to.
this lead to a mon creation loop.
fixes: #3545
Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
Replace usage of the custom osd copybins command with the new top-level
copy-binaries command introduced with cmd-reporter.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
In order to cleanup all cluster resources in case of cluster deletion
all resources have an ownerreference to the cluster crd instance.
Because the name of the reference was incorrectly set to the name of the
namespace the cluster resources can unintentionally be deleted by the
kubernetes garbage collector after a kubernetes cluster anomaly.
Signed-off-by: René Kalff <r.kalff@intermax.nl>
this adds the access for rook to create monitoring resources
across namespaces and provides servicemonitor config to
monitor the current service being created
Signed-off-by: Umanga Chapagain <chapagainumanga@gmail.com>
The status messages on the CRs had much duplicated code
for logging an error if the status update failed.
Now any failures around the status update are logged
consistently within the updateClusterStatus method
so there is no need to return an error and duplicate
the error handling everywhere.
Signed-off-by: travisn <tnielsen@redhat.com>
Because the manager is running under the 'ceph' user the cephfs
subvolume plugin can not create directories anymore. It fails with a
permission denied because of the usage of the 'ceph' user, root is need
here, hence the special startup line to go over this.
Closes: https://github.com/ceph/ceph-csi/issues/486
Signed-off-by: Sébastien Han <seb@redhat.com>
Currently the ClusterController logs out a message like:
```
Cluster %s is not ready. Skipping orchestration.rook-ceph
```
without the "placeholders" fullfilled. This is because the code uses the
logger.Info() instead of logger.Infof() function.
This change fixes that to use Infof and therefore corrects the log
message.
Signed-off-by: Alexander Trost <galexrt@googlemail.com>
Before each pod starts, the daemon changes the ownership of
/var/log/ceph which is bindmounted on a hostPath.
On OpenShift envionments, the container must be privileged in order to
successfully change the owernship.
Closes: https://github.com/rook/rook/issues/3485
Signed-off-by: Sébastien Han <seb@redhat.com>
OpenShift environment requires privileged containers when interacting
with the host filesystem, typically via dataDirHostPath.
The init container that changes the ownership of dataDirHostPath was
missing the privileged flag and users were getting permission denied
errors on the chown command.
Fixes: https://github.com/rook/rook/issues/3485
Signed-off-by: Sébastien Han <seb@redhat.com>
At the beginning of every container startup we now change the ownership
of every known ceph directories to ceph:ceph.
This will allow daemons to start properly as well as allowing daemon to
writes their logs on files.
This is needed because ceph daemon now run as 'ceph' instead of 'root'
Signed-off-by: Sébastien Han <seb@redhat.com>
We do not determine the daemon type and name by string parsing anymore
but passing that info directly in the update call, it's more reliable.
Also removing a bunch of unused functions.
Signed-off-by: Sébastien Han <seb@redhat.com>
This commit makes OnAdd() and OnUpdate() consistent in behavior. They
will do their best detecting if we can upgrade, continue an upgrade or
refuse to do one. It also refuses to downgrade to any version.
It also load ClusterInfo earlier in the controller so that we can
compare the spec version versus the cluster running one.
Signed-off-by: Sébastien Han <seb@redhat.com>
If we successfully changed all the Ceph version of all the daemon we
consider the upgrade complete and successfully, thus we display a
message.
Signed-off-by: Sébastien Han <seb@redhat.com>
Do not consider clusterInfo Ceph version since it's the one picked by
the spec image.
The CephVersion provided here needs to be the current running version
of Ceph for when it is later passed to OkToStop and checked with
IsAtLeast. If you are upgrading from a pre 14.2.1 version to a
version 14.2.1 or newer, IsAtLeast will incorrectly return true and
then fail trying to run mon ok-to-stop on a mon pod running a pre 14.2.1 version.
Signed-off-by: Sébastien Han <seb@redhat.com>
These were used on a previous patch as a debug, let's remove them. They
are not intended to be displayed.
Signed-off-by: Sébastien Han <seb@redhat.com>
When a cluster is updated with a different image version, this triggers
a serialized restart of all the pods. Prior to this commit, no safety
check were performed and rook was hoping for the best outcome.
Now before doing restarting a daemon we check it can be restarted. Once
it's restarted we also check we can pursue with the rest of the
platform. For instance, with monitors we check that they are in quorum,
for OSD we check that PGs are clean and for MDS we make sure they are
all active.
Fixes: https://github.com/rook/rook/issues/2889
Signed-off-by: Sébastien Han <seb@redhat.com>
travisn
John Mulligan
rohan47andAshish Ranjan
ethan
Noah Watkins
Michael Vollman
Blaine Gardner
René Kalff
Umanga Chapagain
Sébastien Han
Alexander Trost
xieyanker
Rohan CJ