Commit Graph
44 Commits
Author SHA1 Message Date
Sébastien Han 3c1c3409b1 Merge pull request #8668 from Muyan0828/fix/osd-full-alert-query
ceph: fix CephOSDCriticallyFulla and CephOSDNearFull Alert Query
2021-09-09 19:01:06 +02:00
chengli f2c9793fca ceph: fix CephOSDCriticallyFulla and CephOSDNearFull Alert Query
Add hostname to CephOSDCriticallyFull and CephOSDNearFull rules's expr

Signed-off-by: chengli <chengli@alauda.io>
2021-09-09 21:58:14 +08:00
Anmol Sachan a1efcd7617 ceph: fix CephMonQuorumAtRisk Alert Query
The updated query will work for single mon deployment and will work better for
deployments with five or more mons.

Signed-off-by: Anmol Sachan <anmol13694@gmail.com>
2021-09-07 21:27:45 +05:30
Travis Nielsen 66dba4fd41 ceph: add rbac for mgr to create service monitor
When there are two mgr daemons, a sidecar of the mgr will update the
service monitor depending on the active mgr daemon. When there is only
a single mgr, the operator will create the service monitor. Now, we
add rbac for the mgr to create these resources.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-06-11 16:45:51 -06:00
Travis Nielsen ac20ab133a Merge pull request #7525 from umangachapagain/pool-quota-alerts
ceph: add pool quota usage alerts
2021-04-12 11:16:31 -06:00
Umanga Chapagain a34b928b2e ceph: add pool quota usage alerts
add quota usage alerts for pools with quota set.

Signed-off-by: Umanga Chapagain <chapagainumanga@gmail.com>
2021-04-12 19:51:57 +05:30
Sébastien Han 74c71880f1 ceph: add monitoring rules for pacific
New release, new files!

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-04-12 12:37:25 +02:00
Travis Nielsen c78e6a9e20 ceph: set active mgr label on mgr services
Other resources such as the service monitor need to match
the labels on the services to the active mgr, thus the active
mgr name is now set on the labels for the metrics service
and dashboard service.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-03-18 16:37:01 -06:00
Anmol Sachan f84cbb47fe ceph: add osd slow ops alert
This commit adds an alert that notifies the users about OSD
requests taking too long to process.

Signed-off-by: Anmol Sachan <anmol13694@gmail.com>
2021-03-16 15:40:28 +05:30
subhamkrai acfbc78392 ceph: ignore the monitoring folder from yamllint
this commit adds the monitoring folder to
yamllint ignore list.

Signed-off-by: subhamkrai <srai@redhat.com>
2021-03-16 12:33:37 +05:30
subhamkrai b46e31dbee ci: add gh action for yaml linter
this commit adds gh action for yaml linter.

Signed-off-by: subhamkrai <srai@redhat.com>
2021-03-11 08:23:18 +05:30
Anmol Sachan 1532070a3e ceph: add osd flapping alert
This PR adds an alert that notifies the users if an OSD restarts
for 5 times under 5 minutes. Each restart corresponds to 2 changes
in the value of ceph_osd_up metric.

Signed-off-by: Anmol Sachan <anmol13694@gmail.com>
2021-03-05 16:22:16 +05:30
Anmol Sachan 136231a71f ceph: fixed prometheus query to avoid many-to-many match error
Ceph_daemon label includes references to a replaced OSD that cause a prometheus
ruleset to fail due many-to-many match error. The changes to this PR filters the
extra matching ceph_daemon time-series.

Signed-off-by: Anmol Sachan <anmol13694@gmail.com>
2021-03-01 17:26:05 +05:30
Anmol Sachan fdb88edb6a ceph: updated thresholds of OSD capacity alerts and the device_class
This commit updates the OSD capacity thresholds to match the cluster capacity alert thresholds.
Also updated device_class info to provide info about which type of storage needs to be added.

Signed-off-by: Anmol Sachan <anmol13694@gmail.com>
2020-12-07 20:11:01 +05:30
Alexander Trost be16bc455f docs: operator use rbac.authorization.k8s.io/v1
This replaces any uses of `rbac.authorization.k8s.io/v1beta1` with
`rbac.authorization.k8s.io/v1` affecting RBAC objects like ClusterRole
and ClusterRoleBindings, etc. to make it consistent.
This is done as some RBAC objects used the `v1beta1` and others already
using `v1`.

Signed-off-by: Alexander Trost <galexrt@googlemail.com>
2020-08-20 12:08:32 +02:00
Anmol Sachan c35f31bcf2 ceph: removed the storage provisioners usage rules
In the interest of having the metrics avaliable generically for all the storage provisioners and just not ceph, the superseding metrics will be added to Openshift directly. Thus these metrics wont be required in rook.

Signed-off-by: Anmol Sachan <anmol13694@gmail.com>
2020-08-10 16:48:24 +05:30
Travis Nielsen 99d2131209 ceph: symlink prometheus manifests in same dir
The prometheus manifests should be symlinked in the same directory
instead of from the root of the repo for building across all
platforms

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-08-05 15:55:16 -06:00
Anmol Sachan a42616967d ceph: updated the PV count query for telemeter
Signed-off-by: Anmol Sachan <anmol13694@gmail.com>
2020-07-27 20:18:13 +05:30
Anmol Sachan f3eea6ebb7 ceph: added rules for PVC requested and used capacity for telemeter
This commit adds recording rules to monitor aggregated (requested and used) capacity of the PVCs for block and file type PVCs, via the telemeter.

Signed-off-by: Anmol Sachan <anmol13694@gmail.com>
2020-07-27 20:16:12 +05:30
Sébastien Han 6a643e9522 ceph: add monitoring rules for octopus
Add monitoring rules for octopus clusters.

Closes: https://github.com/rook/rook/issues/5897
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-07-27 10:16:49 +02:00
Anmol Sachan 6bf3aa9a9a ceph: added prometheus rule file for alerting on external cluster
Signed-off-by: Anmol Sachan <anmol13694@gmail.com>
2020-07-17 11:29:51 +02:00
Anmol Sachan ff1a8ad31e ceph: updated PVC utilizaztion alerts
Updated warning and critical alerts for all the PVC's created by rook-ceph provisioners. Added the recommendation to expand the PVCs.

Closes: BZ1849717
Signed-off-by: Anmol Sachan <anmol13694@gmail.com>
2020-06-25 13:03:01 +05:30
Anmol Sachan 624ca40817 ceph: added PVC utilization alerts
Added warning and critical alerts for all the PVC's created by rook-ceph provisioners. This is required to notify a admin timely to check on PVC utilization and take a required action to prevent workloads from failing.

Closes: BZ1849723
Signed-off-by: Anmol Sachan <anmol13694@gmail.com>
2020-06-23 21:06:59 +05:30
Anmol Sachan 7cebf15a7d ceph: added CephClusterReadOnly alert and updated utilization alerts
Added a new critical alert CephClusterReadOnly which notifies when the cluster becomes read-only
Updated the warning and critical alert to accomodate the new alert

Closes: BZ1809248
Signed-off-by: Anmol Sachan <anmol13694@gmail.com>
2020-06-19 14:07:51 +05:30
Satoru Takeuchi a5ea2858c9 manifests: remove unnecessary namespace fields
There are many namespace fields in ClusterRole{,Binding}. However,
ClusterRole{,Binding} are not namespaced. So we can remove these.

Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
2020-03-18 15:58:31 +00:00
Anmol Sachan caf79f971f ceph: Fixed CephMonHighNumberOfLeaderChanges alert
The alert was showing wrong POD and hostname in the
alert description. Fixed the query for the same by
adding the right labels from ceph_mon_metadata query
to the original query.

Signed-off-by: Anmol Sachan <anmol13694@gmail.com>
2020-01-14 13:32:03 +05:30
Sébastien Han a888ed5d7c Merge pull request #4485 from anmolsachan/osd_alerts
ceph: Added OSD utilization alerts
2019-12-12 18:24:27 +01:00
Anmol Sachan 99e961f72a ceph: Added OSD utilization alerts
Signed-off-by: Anmol Sachan <anmol13694@gmail.com>
2019-12-12 22:40:26 +05:30
Anmol Sachan 353621452b ceph: Improve storage utilization alert description and message.
Signed-off-by: Anmol Sachan <anmol13694@gmail.com>
2019-12-12 14:51:27 +05:30
Anmol Sachan 09af832085 ceph: Updated storage utilization thresholds and timings
This PR mainly takes a proactive approach to
alert the user to take action when the cluster
utilizartion is getting close to the maximum capacity.
Since ceph stops I/O at 95% utilization, the user should
be warned before it.

Signed-off-by: Anmol Sachan <anmol13694@gmail.com>
2019-11-10 23:44:53 +05:30
Umanga Chapagain ddb006152b Changed description for CephDataRecoveryTakingTooLong alert
Previous alert description was confusing. This makes it more
generic and user understandable.

Signed-off-by: Umanga Chapagain <chapagainumanga@gmail.com>
2019-09-13 13:58:51 +05:30
Umanga Chapagain 332c9993b1 Fixed HighMonLeaderChanges Alert
rounded the value of MonLeaderChange Rate to 2 decimal points (e.g. 2.44)
previously, we had 16 decimal points (e.g. 2.4406779661016946).
This makes the alert notification more sensible and readable.

Signed-off-by: Umanga Chapagain <chapagainumanga@gmail.com>
2019-09-09 10:34:00 +05:30
Madhu Rajanna a8589af337 Implement grpc metrics for cephcsi
grpc metrics is made option in rook
this can be disabled/enabled with ROOK_CSI_ENABLE_GRPC_METRCS
variable in operator.yaml
this will help admin who want to check the grpc metrics
of the ceph-csi plugins

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-08-30 15:01:45 +05:30
Madhu Rajanna 5587169fe9 Fix namespace selector in csi-livenees-monitoring
csi services are created in rook-ceph namespace

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-08-30 13:16:55 +05:30
Madhu Rajanna 53e33d4e08 Fix invalid port name in service monitoring
http-metrics1 port was not the service port name
defined for liveness

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-08-30 13:16:55 +05:30
Umanga Chapagain d92a04e8ed fixes: alerting rules and recording rules
- updated query for latency
- updated severity level for mgr down alert

Signed-off-by: Umanga Chapagain <chapagainumanga@gmail.com>
2019-08-27 13:50:51 +05:30
Daniel-Pivonka 665172a9ea ceph: deploy liveness sidecar with csi
Signed-off-by: Daniel-Pivonka <dpivonka@redhat.com>
2019-08-19 14:26:47 -04:00
Umanga Chapagain d30c06860d fixes permissions and servicemonitor configurations
this adds the access for rook to create monitoring resources
across namespaces and provides servicemonitor config to
monitor the current service being created

Signed-off-by: Umanga Chapagain <chapagainumanga@gmail.com>
2019-07-26 10:53:18 +05:30
Umanga Chapagain 732f3030d0 integrate prometheus based monitoring with rook-ceph
> add new field to cluster spec
> deploy servicemonitor
> deploy prometheusRule

Signed-off-by: Umanga Chapagain <chapagainumanga@gmail.com>
2019-07-08 11:40:17 +05:30
Shubhendu 577728bb03 Added latest prometheus alerting rules for ceph
New rules added:
- CephMgrIsAbsent
- CephMgrIsMissingReplicas
- CephMdsMissingReplicas
- CephMonHighNumberOfLeaderChanges
- CephOSDVersionMismatch
- CephMonVersionMismatch

Signed-off-by: Shubhendu <shtripat@redhat.com>
2019-06-06 10:04:45 +05:30
Travis Nielsen 46fadc792b Merge pull request #2996 from umangachapagain/ceph-mixins
Prometheus Rule for Ceph
2019-04-18 09:52:54 -06:00
Umanga Chapagain 361508ca7f Adding Prometheus based alerting for Ceph
Signed-off-by: Umanga Chapagain <chapagainumanga@gmail.com>
2019-04-18 19:37:00 +05:30
Kaushal M 2e274b306a ceph: Use aggregated ClusterRoles
The ClusterRoles in the manifests (and helm chart) for the ceph provider
have been updated to use [aggregated ClusterRoles][1].

All ClusterRoles have been split into 2 ClusterRoles.
An aggregated ClusterRole without any rules, named the same, but with an
aggregationRule that matches labels in the format
'rbac.ceph.rook.io/aggregate-to-<ClusterRole name>: "true"'.
A second ClusterRole which contains the existing rules named
'<ClusterRole name>-rules', with label
'rbac.ceph.rook.io/aggregate-to-<ClusterRole name>: "true"'.

These two ClusterRoles give the same behaviour as the previous
ClusterRole.

Additional rules to a ClusterRole can now be provided by creating a new
ClusterRole with the correct label. This should help make updates
simpler.

Fixes #2634.

[1]: https://kubernetes.io/docs/reference/access-authn-authz/rbac/#aggregated-clusterroles

Signed-off-by: Kaushal M <kshlmster@gmail.com>
2019-04-17 13:00:21 +05:30
Alexander Trost 097f640bbb doc: rename monitoring doc for Ceph
doc: move Ceph mon example files to ceph example subfolder

Signed-off-by: Alexander Trost <galexrt@googlemail.com>
2018-12-07 19:33:27 +01:00