This PR adds a `settings` option in the `Module` struct
which can help users to futher customize the `balancer`
module with a differet mode.
Signed-off-by: sp98 <sapillai@redhat.com>
(cherry picked from commit 1ad37ecce9)
fix UpdateActiveMgrLabel to retry label update on failure. Previously,
the function always returned the current manager, even if update
failed, leading to inconsistent labels. This fix ensures retries on
failures.
Fixes: https://github.com/rook/rook/issues/13601
Signed-off-by: Redouane Kachach <rkachach@ibm.com>
(cherry picked from commit ee7c71c89e)
The pg_autoscaler is always enabled by ceph and cannot be disabled.
In a previous PR the pg_autoscaler config was removed from the main example.
Now the remaining config for the pg_autoscaler is removed as well.
Signed-off-by: travisn <tnielsen@redhat.com>
Fixes: #13167
Previously, the mgr did not honor the flag
ContinueUpgradeAfterChecksEvenIfNotHealthy
from the cluster spec. Only osd, mds, and rgw did.
To render the update behavior correct and complete across the daemons, this
change implements the honoring of the flag for the mgr.
Signed-off-by: Michael Adam <obnox@samba.org>
During certain maintenance tasks the admin will own running
operations on the ceph mgr, rgw, mds and rbd-mirror daemons
and the operator should not interfere with those operations.
Co-authored-by: gauravsitlani <gaurav.sitlani@live.com>
Signed-off-by: subhamkrai <srai@redhat.com>
Several messages were showing up in the operator log
more often than necessary, so let's change them to
debug level.
Signed-off-by: travisn <tnielsen@redhat.com>
While two mgrs is considered sufficient, more mgr daemons are
now allowed in case the admin wants even more fault tolerance
to the active mgr going down, in case multiple mgrs go down.
Up to five mgr pods will be allowed. All mgrs will be in
standby mode except one active mgr. All mgr pods have a sidecar
that will update their respective pod specs with the active
or passive label.
Signed-off-by: travisn <tnielsen@redhat.com>
ceph-exporter's ServiceMonitor and Service CRD contain different port
name, which results in no metrics being collected by prometheus.
this commit makes GetCephMonitor configurable, which we use to set
consistent port names in ServiceMonitor and Service for ceph-exporter.
Signed-off-by: Divyansh Kamboj <dkamboj@redhat.com>
If the ceph version does not suppport the exporter, the operator
will attempt to delete the service monitor related to the exporter
to ensure it does not exist for a node. If the expected rbac does
not exist, this will cause unnecessary errors since there would anyway
be no service monitor to delete. So we ignore the error of deleting
the service monitor for the exporter.
The context is also passed to the exporter so a new kubeconfig does
not need to be initiated and cause unnecessary logging about
invalid options for the config.
Signed-off-by: travisn <tnielsen@redhat.com>
When deploying a cluster with Helm Chart, if the namespace is not the default value `rook-ceph`, and if the monitoring feature is enabled, then the generated ServiceMonitor's `rook_cluster` selector now follows the namespace, not the hard-coded value `rook-ceph`.
Signed-off-by: Ho Kim <ho.kim@ulagbulag.io>
The prometheus mgr module and ceph exporter can now be optionally
disabled by the monitoring.metricsDisabled setting in the
CephCluster CR. These will not be disabled by default, rather
than the mgr module being disabled by default from v1.11.4.
Signed-off-by: travisn <tnielsen@redhat.com>
The active and standby labels are updated on the mgr pods
by the mgr sidecar. In the case of a single mgr, there is
not sidecar, so the dashboard and other mgr services were
not available when there was a single mgr. Now the mgr
pod will default to the active mgr status label so that
the single mgr case will succeed. In the case of two mgrs,
the sidecar will immediately update the standby mgr to
remove the active status label.
Signed-off-by: travisn <tnielsen@redhat.com>
With the new mgr HA implementation (based on readiness probe) we
don't need the mgr sidecar (live-watch) anymore. This commit is
intended to remove all the related code.
Signed-off-by: Redouane Kachach <rkachach@redhat.com>
After an update, the mgr pod will be restarted. Since the
readiness probe will indicate the standby mgr pod is never
ready, we need to check simply for the running mgr pod
and not whether or not it is ready.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
This fixes the the MultiClusterDeploySuite CI failure. The operator
is timing out waiting for the mgr deployments to be
ready, according to the WaitForDeploymentToStart() method. After
the reconcile times out after about five minutes, the next
reconcile succeeds since the wait is only done for new mgr
deployments.
Closes: https://github.com/rook/rook/issues/11685
Signed-off-by: Redouane Kachach <rkachach@redhat.com>
When the mgr services are updated by the operator to set the
active mgr properly, skip services that do not currently
have a selector set.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
With octopus coming to end of life, we remove support from
Rook for deploying Ceph Octopus and assume a min version of
Pacific v16. Any checks for octopus or earlier are removed
from the reconciles since they are obsolete.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
In new deployments we always want to disable the insecure global id.
If the mgr deployment is newly created, we know it's a new cluster
and the mons are in quorum, so we can go ahead and disable the
insecure global IDs.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The prometheus rules had been previously created if the cephcluster CR
setting monitoring.enabled was set to true. The rules were not customizable
and therefore not flexible enough. Now the rules are installed by the helm
chart. To customize the rules, a post-processor can be applied to the helm
chart.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
We can eliminate the OSD creation code for Ceph < v15.2.0 because Nautilus(v14.2.z)
is no longer supported. In other words, we can now expect that all newly-created
OSDs on PVC are raw mode.
Please note that the existing lvm mode OSDs on PVCs still be supported after this change.
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
When the active mgr changes, there may be user-created services
that also need to have their selector updated to point to the
new active mgr. Now the operator will update any service in the
rook operator namespace that has the label app=rook-ceph-mgr
to point to the active mgr.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
This commit adds context parameter to k8sutil service functions. By
this, we can handle cancellation during API call of service resource.
Signed-off-by: Yuichiro Ueno <y1r.ueno@gmail.com>
This commit adds context parameter to k8sutil prometheus functions. By
this, we can handle cancellation during API call of prometheus resource.
Signed-off-by: Yuichiro Ueno <y1r.ueno@gmail.com>
The templates for the mgr-generated ServiceMonitor and PrometheusRule objects included the labels
prometheus and team, making it impossible to override them as user.
This adds a new method `OverwriteApplyToObjectMeta` to
pkg/apis/ceph.rook.io/v1.Labels, which, contrary to the existing
`ApplyToObjectMeta` method, overwrites existing labels.
Closes: https://github.com/rook/rook/issues/8502
Signed-off-by: Mara Sophie Grosch <littlefox@lf-net.org>
This commit adds context parameter to k8sutil deployment functions. By
this, we can handle cancellation during API call of deployment resource.
Signed-off-by: Yuichiro Ueno <y1r.ueno@gmail.com>
On Pacific, Ceph's default is "upmap", so we should let it be
like this.
This lets the user change the mode is desired.
On Octopus though, Rook continues to force the mode to "upmap".
Closes: https://github.com/rook/rook/issues/9062
Signed-off-by: Sébastien Han <seb@redhat.com>
This commit is a large refactor on how the operator starts, stops and
how it starts various sub-components such as the ceph-csi driver. It
also refines the way we cancel orchestrations. We don't use breakpoints
anymore but send our self a SIGUP to reload our controller runtime
manager.
The reload will happen under different circonstances like:
* a new adminission controller secret is created/deleted/changed
* a CephCluster CR is edited
As mentioned earlier, the csi driver now has its own controller, just
like flex. It reacts to change in the operator config map for particular
ROOK_CSI_ fields.
A second new controller for the operator's general config has been
created, it manages:
* the logging level
* the ceph CLI command timeout
* the discovery daemon
The operator reacts much more rapidly to cancellation events by stopping
the manager's context and reloading it.
Signed-off-by: Sébastien Han <seb@redhat.com>
This commit adds 'managedBy' label to the ceph metrics if the
cephcluster is managed by another higher Resource. The optional
label shall be added as part of cephcluster CRs monitoring labels.
Signed-off-by: Anmol Sachan <anmol13694@gmail.com>
When there are multiple mgr daemons, the mgr sidecar owns reconciling
the services for the active mgr. However, for efficiency the sidecar
only reconciles the services if the active mgr changes. Now the operator
will also reconcile the mgr services to ensure that services are re-created
after being deleted, or otherwise in an incorrect state.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
If the cluster is external we want to periodically rehydrate the mgr
endpoint. This handles the scenarion where the active manager changes,
so we need to update the endpoint with the new IP address.
The create-external-cluster-resources.py script now requires an extra
permission to query the manager service so Rook can discover the active
one and its IP.
Testing:
```
[leseb@tarox~/go/src/github.com/rook/rook][external-active-mgr-change] minikube kubectl -- exec -n rook-ceph deploy/rook-ceph-tools -ti -- ceph mgr stat
{
"epoch": 37,
"available": true,
"active_name": "b",
"num_standby": 1
}
[leseb@tarox~/go/src/github.com/rook/rook][external-active-mgr-change] kubectl -n rook-ceph-external get ep
NAME ENDPOINTS AGE
rook-ceph-mgr-external 172.17.0.12:9283 3m10s
[leseb@tarox~/go/src/github.com/rook/rook][external-active-mgr-change] k scale --replicas=0 deployment rook-ceph-mgr-b
deployment.apps/rook-ceph-mgr-b scaled
[leseb@tarox~/go/src/github.com/rook/rook][external-active-mgr-change] minikube kubectl -- exec -n rook-ceph deploy/rook-ceph-tools -ti -- ceph mgr stat
{
"epoch": 40,
"available": true,
"active_name": "a",
"num_standby": 0
}
[leseb@tarox~/go/src/github.com/rook/rook][external-active-mgr-change] kubectl -n rook-ceph-external get ep
NAME ENDPOINTS AGE
rook-ceph-mgr-external 172.17.0.13:9283 3m55s
```
Signed-off-by: Sébastien Han <seb@redhat.com>
CephCluster.Spec.Labels takes a key "monitoring". This is used
to inject labels into all the monitoring resources created by
the Rook operator.
Signed-off-by: Umanga Chapagain <chapagainumanga@gmail.com>
It's better to validate ownerReferences when setting them. In addition, we should use
controllerrutil.Set{Controller,Owner}Reference, that have such validation, as possible.
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
If the count of mgr daemons is reduced, we remove the extra one.
The check is made during each reconcile.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
In Pacific the simple command to retrieve the active mgr is with
ceph mgr stat instead of retrieving the full ceph mgr dump.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The mgr daemon may be failed over by ceph if the active mgr is not
responding and the standby mgr is available. If the active mgr changes
the services for the dashboard and metrics will be updated with a
label selector for the new active mgr. The services cannot direct
traffic to the standby mgr or else they will be incorrectly redirected
to the active mgr.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The active mgr should match the labels on the services that
are available for prometheus and the dashboard. The selector
labels must be updated whenever there is a new active mgr.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Artem Torubarov
sp98
Redouane Kachach
travisn
Michael Adam
gauravsitlani
Redouane Kachach
Divyansh Kamboj
Ho Kim
xiaobaowen
Satoru Takeuchi
Sébastien Han
Yuichiro Ueno
Mara Sophie Grosch
parth-gr
Anmol Sachan
Lars Lehtonen
Umanga Chapagain
ushen