Now, the CephBlockPool CRD is managed with the controller-runtime.
So the watcher is outside of the main controller reconciliation loop of
CephCluster which brings numerous benefit such as:
* having its own reconciliation loop
* won't block anything from the main CephCluster controller loop
* fast than waiting for CephCluster loop to completion
Partially close: https://github.com/rook/rook/issues/1981
Signed-off-by: Sébastien Han <seb@redhat.com>
The rook types used across the storage providers moved from the v1alpha2
package to the v1 package. This commit points the packages at their new
location. Implementation is expected to remain unchanged.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
This commit is to handle all those unhandled errors which raises the gosec warning.
Fixed G104: Unhandled Errors are handled now
Signed-off-by: Nizamudeen <nia@redhat.com>
When the cluster is upgraded, the child controllers are notified with the
callback to the ParentClusterChanged() method. These should be made
in a separate goroutine so that the cluster controller is not blocked
on the child controllers and also so child controllers aren't blocked
on each other for the upgrade.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Fixed conditions getting resetted after the operator restart.
Did the changes which required to implement conditions on the rook ceph cluster
Conditions will eliminate the current status.State and incorporates a type which
provides much more description to the current status of the cluster.
Signed-off-by: Nizamudeen <nia@redhat.com>
We now always perform ok-to-stop change when the deployment is going to
be updated. This could be for various reasons like change in the spec or
a new Ceph image.
In any case, if Ceph resources are going to be restarted this must be
done carefully.
Also trimmed the isUpgrade variable, the upcoming commit will remove the
need of that variable.
Keeping isUpgrade from the controller code since we need it to detect
whether an orchestration upgrade should be triggered or not (based on
the ceph cluster status).
Also, we keep it on the child controllers so we don't go into an
orchestration phase if the cluster image hasn't changed. The child does
not need to be updated/go into its orchestration phase because the
cluster CR changed.
Closes: https://github.com/rook/rook/issues/4379
Signed-off-by: Sébastien Han <seb@redhat.com>
A new CRD option `continueUpgradeAfterChecksEvenIfNotHealthy` is added.
When upgrading, Rook goes OSD by OSD and then waits for PGs to be clean
before proceeding to the next OSD. Currently, Rook waits for 5 hours but
there might be circumstances where PGs need more time to settle.
Thus setting `continueUpgradeAfterChecksEvenIfNotHealthy` to true will
pursue the upgrade process, even if PGs are not 100% active+clean.
Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1786029
Signed-off-by: Sébastien Han <seb@redhat.com>
The isUpgrade flag was being set every time the operator
was started. This caused the upgrade checks to be run every
time the operator restarted. Thus, if any mons were down,
you likely would get stuck on the upgrade checks since another
mon could not be stopped. We need to rely on the upgrade being
detected either by a different image name from the cluster CR
update, or from different versions of the daemons running
in case the operator was restarted in the middle of an upgrade.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
When using the "errors" package, using `%+v` (extended format),
each Frame of the error's StackTrace will be printed in detail.
Let's only print `%v` to print the error.
If the error has a Cause it will be printed recursively.
Basically `%+v` has been replaced with `%v` for all `error` type
interfaces, whether the logger is Info, Warning or Error.
Signed-off-by: Sébastien Han <seb@redhat.com>
As of 14.2.5, the crash collector pod will use the client.crash user
instead of the client.admin keyring to post crash dumps.
Closes: https://github.com/rook/rook/issues/4388
Signed-off-by: Sébastien Han <seb@redhat.com>
If editing the cluster CR we must update the spec to reflect its new
content. Now the spec is updated so we can update with an unsupported
image as well as setting allowUnsupported: true.
It was not possible before we had to restart the operator, now we can do
it as an update.
Signed-off-by: Sébastien Han <seb@redhat.com>
We recently discovered a race condition which happens not to update the
child CRDs when CR image changes.
The race happens under the following sequence:
* orchestration onAdd() is called when the operator image restarts
or its image updated
* the orchestration then goes into processing mon/mgr/osd
* during this the cluster CR is updated so onUpdate() is called,
however, onAdd() is not done yet for child CRDs (mds, rgw)
* onUpdate() runs against the cluster CR and updates the cluster images
* by the time the child CRDs reached onUpdate() from their respective
ParentClusterChanged() methods, the image is already up to date
so we were exiting since our check for updating
these resources are based on the cluster image version
To fix this, we changed the comparison, but using isUpgrade instead and
slightly changed the way isUpgrade operates. So now, isUpgrade is true
even if the Ceph version is identical which allows us to verify whether
or not the image changed.
Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1775624
Signed-off-by: Sébastien Han <seb@redhat.com>
The osd-prepare jobs should not inherit osd resources since this might
cause scheduling issues.
Instead, we can now use the new CRD property `prepareosd` in the resource section.
Refer to the documentation for more details.
Closes: https://github.com/rook/rook/issues/2502
Signed-off-by: Sébastien Han <seb@redhat.com>
We now differentiate the cases where:
* we only consume the external cluster
* we consume the external cluster as well as creating stateless
resources in Kubernetes (bootstrap mds,rgw, nfs)
This is mostly controlled via the image property spec. If not defined,
not extra CRs won't be able to be created.
Now the external cluster feature supports Ceph cluster as of Luminous 12.2.
Signed-off-by: Sébastien Han <seb@redhat.com>
A job is started to detect the Ceph version. This job now allows setting
the node affinity and tolerations with the same setting that is
specified for the mon placement. No new placement spec is required, but
we will simply use the same spec that the mons use.
Signed-off-by: travisn <tnielsen@redhat.com>
On rare occasion, the ceph checks might not be 100% correct and the user
might decide to force an upgrade anyway.
Correctly, if we are in an upgrade, we perform `ok-to-stop` check for
each daemon, if one of them fails, we retry and eventually fail.
They are corner-cases where users would like to force the upgrade anyway.
Enhance the new CR property: "skipUpgradeChecks: true".
Closes: https://github.com/rook/rook/issues/3872
Signed-off-by: Sébastien Han <seb@redhat.com>
We can now enable via the cluster CR any manager module with a
setting under the new mgr element
mgr:
modules:
- name: pg_autoscaler
enabled: true
Co-authored-by: Sébastien Han <seb@redhat.com>
Co-authored-by: travisn <tnielsen@redhat.com>
Signed-off-by: travisn <tnielsen@redhat.com>
After further consideration and working with the feature for a few
weeks, the `configOverride` functionality seems to be more complexity
than is worth it for the expected benefit. Users still have the most
preferred option of setting configs in Ceph's config database using
`ceph cofig set` or the dashboard, and the override ConfigMap still
exists for fallback scenarios. The config database can still be used
by Rook for setting necessary configs, but the user has no way of
telling Rook to do so.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
Reduce the duplication of detecting the version of the external cluster
by refactoring to a single method that retrieves and validates the
version
Signed-off-by: travisn <tnielsen@redhat.com>
Add various upgrade tests so that the local cluster does not fail if
an upgrade is available on the external cluster.
Also, add better upgrades handling on update and on add. We now rely as
much as possible on a unique function that works both local and external
cluster 'detectAndValidateCephVersion()'. So the code will act the same
way wether a CR update is triggered or if the operator is restarted.
The health check now reports any topology changes on the external
cluster such as odd number of mons and low number. It will also detect
if the external cluster has been upgraded and will log that information,
helping the admin in his/her decision to trigger an image upgrade.
Finally, we prevent the creation of new CRs if the external cluster
version has a higher major number than the local one.
Closes: https://github.com/rook/rook/issues/3731
Signed-off-by: Sébastien Han <seb@redhat.com>
From now on, Rook will only perform check before upgrades when there is
an actual upgrade. So if the Ceph image changed and a new version is
desired Rook will go through all the daemons and update them one by one
and perform checks in between.
Closes: https://github.com/rook/rook/issues/3583
Signed-off-by: Sébastien Han <seb@redhat.com>
ceph: Use the new NetworkSpec api
ceph: Change all tests to use the NetworkSpec
ceph: Remove TypeMeta field from NetworkSpec
ceph: Change IsHost network func signature
Signed-off-by: giovanism <giovanism@outlook.co.id>
Re-adapt controller code with the latest changes. Split External and not
external more accuratly in initializeCluster.
Signed-off-by: Sébastien Han <seb@redhat.com>
When configuring an external cluster the orchestration of discover and all the ceph
daemons will be skipped. The operator will still watch for the creation of
crds for this namespace. When a filesystem, object, object user, or
ganesha crd are created, the operator will simply print an error
to the log.
Signed-off-by: travisn <tnielsen@redhat.com>
Signed-off-by: Sébastien Han <seb@redhat.com>
When configuring an external cluster the orchestration of discover and all the ceph
daemons will be skipped. The operator will still watch for the creation of
crds for this namespace. When a filesystem, object, object user, or
ganesha crd are created, the operator will simply print an error
to the log.
Signed-off-by: travisn <tnielsen@redhat.com>
Signed-off-by: Sébastien Han <seb@redhat.com>
The upgrade changes for #2901 added the check for the correct version
of the ceph image before continuing with an upgrade. This commit is
to refactor that change to work with the new code path to validate
the ceph version.
Signed-off-by: travisn <tnielsen@redhat.com>
If the cluster is not successfully initialized during the onAdd event,
the cluster will not be in a properly initialized state. The admin
will likely need to update the CR, in which case the onUpdate event would
be triggered. In this case we need to call the same code path as the
onAdd event to properly initialize the cluster.
Signed-off-by: travisn <tnielsen@redhat.com>
With the 1.1 release we should not be creating bluestore OSDs
with rook's legacy partitioning scheme. All new bluestore OSDs
should be created with ceph-volume. Therefore, the miminum
version allowed is a version of mimic supporting ceph-volume.
This commit also adds consistent version checking between
the Add and Update methods for the operator handling the
CR events.
Signed-off-by: travisn <tnielsen@redhat.com>
Create a mutex at the rook operator level that will be used when
updating the shared-between-clusters config map that Ceph CSI consumes.
Pass the mutex down to the function that will update the CM.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
In order to cleanup all cluster resources in case of cluster deletion
all resources have an ownerreference to the cluster crd instance.
Because the name of the reference was incorrectly set to the name of the
namespace the cluster resources can unintentionally be deleted by the
kubernetes garbage collector after a kubernetes cluster anomaly.
Signed-off-by: René Kalff <r.kalff@intermax.nl>
This commit makes OnAdd() and OnUpdate() consistent in behavior. They
will do their best detecting if we can upgrade, continue an upgrade or
refuse to do one. It also refuses to downgrade to any version.
It also load ClusterInfo earlier in the controller so that we can
compare the spec version versus the cluster running one.
Signed-off-by: Sébastien Han <seb@redhat.com>
This commit does:
* remove dead code SetCrushTunables()
* stop creating initial crushmap
There is no need to create an initial crushmap, since Ceph natively does
that for us.
We don't delete the existing configmap containing the initial crush map
of already deployed cluster, perhaps people are using for whatever
reason. It does not harm to keep it around.
Closes: https://github.com/rook/rook/issues/3138
Signed-off-by: Sébastien Han <seb@redhat.com>
Use cmd-reporter to detect Ceph version.
The job to detect Ceph version needs to run in a security account context
capable of getting/creating ConfigMaps, so a new security account, role,
and role binding are created for the cmd-runner jobs. In the future, the
Ceph mgr may create jobs in this service account for running
`ceph-volume` on nodes, and this also keeps Rook operators controlling
multiple Ceph clusters independent.
Fixes#2686 - ceph version job not working correctly with graylog
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
The check for whether to set the OwnerReferences is no longer
necessary. When passing the correct value to the apiVersion,
the OwnerReferences can be successfully set for OpenShift.
Signed-off-by: travisn <tnielsen@redhat.com>
The ceph controller will delete and recreate the detec-version job, when
an already existing job is found. This helps avoid cluster creation from
getting hung up on waiting for an already existing/stale job to finish.
Fixes#3081.
Signed-off-by: Kaushal M <kshlmster@gmail.com>
The CephCluster CR contains settings that are needed by other
CRs to configure the Ceph daemons. When the CephCluster CR
is updated, the updates will now be passed on to each of the
CR controllers to ensure the daemons are updated properly
without requiring an operator restart.
When calling the controllers from another controller,
we ensure that only a single goroutine is handling
CRs at any given time to prevent contention across
multiple CRs of the same type.
Signed-off-by: travisn <tnielsen@redhat.com>
Allow the user to set annotations in the CRDs of the following operators which will then be set on
the resulting Pods (Deployments, StatefulSets and so on).
Operators:
* Cassandra
* Ceph
* CockroachDB
* EdgeFS
* Minio
* NFS
The Annotations related structures have been added to the
rook.io/v1alpha2 pkg.
Signed-off-by: Alexander Trost <galexrt@googlemail.com>
It's quite convinient to expose /var/log/ceph so that we can decide to
activate logs locally on the machine and see what's going on.
This is only a placeholder when a daemon is stuck crashlooping and we
want to allow administrator to gather log files.
We still do not log on file but this can be activated via a config
option passed to the centralized config option store.
For some daemons, which typically do not store any data (rgw, rbd-mirror
and mds) we had to propagate dataDirHostPath from the cluster spec to
each creation call so that the bindmount can happen.
Fixes: https://github.com/rook/rook/issues/2881
Signed-off-by: Sébastien Han <seb@redhat.com>
When instanciating an osd, we now pass clusterInfo which basically
represents a collection of information about a particular Ceph cluster. Rook uses information
// about the cluster to configure daemons to connect to the desired cluster.
From clusterInfo, we use the CephVersion which helps us putting
conditions for our code execution.
Signed-off-by: Sébastien Han <seb@redhat.com>
Sébastien Han
Travis Nielsen
Nizamudeen
d-luu
Blaine Gardner
giovanism
John Mulligan
René Kalff
Umanga Chapagain
DL186038
Kaushal M
Alexander Trost