docs: improve descriptions in localrules.yaml
Parallel upstream changes via https://github.com/ceph/ceph/pull/47284 Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
This commit is contained in:
1 parent
de448d31f9
commit
dc53497d37
2 files changed
+338
-297
No files matched your search
@@ -10,9 +10,9 @@ groups:
|
||||
type: ceph_default
|
||||
oid: 1.3.6.1.4.1.50495.1.2.1.2.1
|
||||
annotations:
|
||||
summary: Cluster is in an ERROR state
|
||||
summary: Cluster is in the ERROR state
|
||||
description: >
|
||||
Ceph in HEALTH_ERROR state for more than 5 minutes.
|
||||
The cluster state has been HEALTH_ERROR for more than 5 minutes.
|
||||
Please check "ceph health detail" for more information.
|
||||
|
||||
- alert: CephHealthWarning
|
||||
@@ -22,9 +22,9 @@ groups:
|
||||
severity: warning
|
||||
type: ceph_default
|
||||
annotations:
|
||||
summary: Cluster is in a WARNING state
|
||||
summary: Cluster is in the WARNING state
|
||||
description: >
|
||||
Ceph has been in HEALTH_WARN for more than 15 minutes.
|
||||
The cluster state has been HEALTH_WARN for more than 15 minutes.
|
||||
Please check "ceph health detail" for more information.
|
||||
|
||||
- name: mon
|
||||
@@ -41,7 +41,7 @@ groups:
|
||||
summary: Monitor quorum is at risk
|
||||
description: |
|
||||
{{ $min := query "floor(count(ceph_mon_metadata) / 2) +1" | first | value }}Quorum requires a majority of monitors (x {{ $min }}) to be active
|
||||
Without quorum the cluster will become inoperable, affecting all connected clients and services.
|
||||
Without quorum the cluster will become inoperable, affecting all services and connected clients.
|
||||
|
||||
The following monitors are down:
|
||||
{{- range query "(ceph_mon_quorum_status == 0) + on(ceph_daemon) group_left(hostname) (ceph_mon_metadata * 0)" }}
|
||||
@@ -55,10 +55,10 @@ groups:
|
||||
type: ceph_default
|
||||
annotations:
|
||||
documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#mon-down
|
||||
summary: One of more ceph monitors are down
|
||||
summary: One or more monitors down
|
||||
description: |
|
||||
{{ $down := query "count(ceph_mon_quorum_status == 0)" | first | value }}{{ $s := "" }}{{ if gt $down 1.0 }}{{ $s = "s" }}{{ end }}You have {{ $down }} monitor{{ $s }} down.
|
||||
Quorum is still intact, but the loss of further monitors will make your cluster inoperable.
|
||||
{{ $down := query "count(ceph_mon_quorum_status == 0)" | first | value }}{{ $s := "" }}{{ if gt $down 1.0 }}{{ $s = "s" }}{{ end }}There are {{ $down }} monitor{{ $s }} down.
|
||||
Quorum is still intact, but the loss of an additional monitor will make your cluster inoperable.
|
||||
|
||||
The following monitors are down:
|
||||
{{- range query "(ceph_mon_quorum_status == 0) + on(ceph_daemon) group_left(hostname) (ceph_mon_metadata * 0)" }}
|
||||
@@ -73,11 +73,15 @@ groups:
|
||||
oid: 1.3.6.1.4.1.50495.1.2.1.3.2
|
||||
annotations:
|
||||
documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#mon-disk-crit
|
||||
summary: Disk space on at least one monitor is critically low
|
||||
summary: Filesystem space on at least one monitor is critically low
|
||||
description: |
|
||||
The free space available to a monitor's store is critically low (<5% by default).
|
||||
You should increase the space available to the monitor(s). The
|
||||
default location for the store sits under /var/lib/ceph. Your monitor hosts are;
|
||||
The free space available to a monitor's store is critically low.
|
||||
You should increase the space available to the monitor(s). The default directory
|
||||
is /var/lib/ceph/mon-*/data/store.db on traditional deployments, and under
|
||||
/var/lib/rook/mon-*/data/store.db on the mon pod's worker node for Rook.
|
||||
Look for old, rotated versions of *.log and MANIFEST*. Do NOT touch any *.sst files.
|
||||
Also check any other directories under /var/lib/rook and other directories on the
|
||||
same filesystem, often /var/log and /var/tmp are culprits. Your monitor hosts are;
|
||||
{{- range query "ceph_mon_metadata"}}
|
||||
- {{ .Labels.hostname }}
|
||||
{{- end }}
|
||||
@@ -93,8 +97,12 @@ groups:
|
||||
summary: Disk space on at least one monitor is approaching full
|
||||
description: |
|
||||
The space available to a monitor's store is approaching full (>70% is the default).
|
||||
You should increase the space available to the monitor store. The
|
||||
default location for the store sits under /var/lib/ceph. Your monitor hosts are;
|
||||
You should increase the space available to the monitor(s). The default directory
|
||||
is /var/lib/ceph/mon-*/data/store.db on traditional deployments, and under
|
||||
/var/lib/rook/mon-*/data/store.db on the mon pod's worker node for Rook.
|
||||
Look for old, rotated versions of *.log and MANIFEST*. Do NOT touch any *.sst files.
|
||||
Also check any other directories under /var/lib/rook and other directories on the
|
||||
same filesystem, often /var/log and /var/tmp are culprits. Your monitor hosts are;
|
||||
{{- range query "ceph_mon_metadata"}}
|
||||
- {{ .Labels.hostname }}
|
||||
{{- end }}
|
||||
@@ -107,14 +115,15 @@ groups:
|
||||
type: ceph_default
|
||||
annotations:
|
||||
documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#mon-clock-skew
|
||||
summary: Clock skew across the Monitor hosts detected
|
||||
summary: Clock skew detected among monitors
|
||||
description: |
|
||||
The ceph monitors rely on a consistent time reference to maintain
|
||||
quorum and cluster consistency. This event indicates that at least
|
||||
one of your mons is not sync'd correctly.
|
||||
Ceph monitors rely on closely synchronized time to maintain
|
||||
quorum and cluster consistency. This event indicates that time on at least
|
||||
one mon has drifted too far from the lead mon.
|
||||
|
||||
Review the cluster status with ceph -s. This will show which monitors
|
||||
are affected. Check the time sync status on each monitor host.
|
||||
Review cluster status with ceph -s. This will show which monitors
|
||||
are affected. Check the time sync status on each monitor host with
|
||||
"ceph time-sync-status" and the state and peers of your ntpd or chrony daemon.
|
||||
|
||||
- name: osd
|
||||
rules:
|
||||
@@ -156,7 +165,7 @@ groups:
|
||||
oid: 1.3.6.1.4.1.50495.1.2.1.4.2
|
||||
annotations:
|
||||
documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#osd-down
|
||||
summary: An OSD has been marked down/unavailable
|
||||
summary: An OSD has been marked down
|
||||
description: |
|
||||
{{ $num := query "count(ceph_osd_up == 0)" | first | value }}{{ $s := "" }}{{ if gt $num 1.0 }}{{ $s = "s" }}{{ end }}{{ $num }} OSD{{ $s }} down for over 5mins.
|
||||
|
||||
@@ -176,10 +185,10 @@ groups:
|
||||
documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#osd-nearfull
|
||||
summary: OSD(s) running low on free space (NEARFULL)
|
||||
description: |
|
||||
One or more OSDs have reached their NEARFULL threshold
|
||||
One or more OSDs have reached the NEARFULL threshold
|
||||
|
||||
Use 'ceph health detail' to identify which OSDs have reached this threshold.
|
||||
To resolve, either add capacity to the cluster, or delete unwanted data
|
||||
Use 'ceph health detail' and 'ceph osd df' to identify the problem.
|
||||
To resolve, add capacity to the affected OSD's failure domain, restore down/out OSDs, or delete unwanted data.
|
||||
- alert: CephOSDFull
|
||||
expr: ceph_health_detail{name="OSD_FULL"} > 0
|
||||
for: 1m
|
||||
@@ -189,12 +198,13 @@ groups:
|
||||
oid: 1.3.6.1.4.1.50495.1.2.1.4.6
|
||||
annotations:
|
||||
documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#osd-full
|
||||
summary: OSD(s) is full, writes blocked
|
||||
summary: OSD full, writes blocked
|
||||
description: |
|
||||
An OSD has reached it's full threshold. Writes from all pools that share the
|
||||
An OSD has reached the FULL threshold. Writes to pools that share the
|
||||
affected OSD will be blocked.
|
||||
|
||||
To resolve, either add capacity to the cluster, or delete unwanted data
|
||||
Use 'ceph health detail' and 'ceph osd df' to identify the problem.
|
||||
To resolve, add capacity to the affected OSD's failure domain, restore down/out OSDs, or delete unwanted data.
|
||||
- alert: CephOSDBackfillFull
|
||||
expr: ceph_health_detail{name="OSD_BACKFILLFULL"} > 0
|
||||
for: 1m
|
||||
@@ -205,10 +215,11 @@ groups:
|
||||
documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#osd-backfillfull
|
||||
summary: OSD(s) too full for backfill operations
|
||||
description: |
|
||||
An OSD has reached it's BACKFILL FULL threshold. This will prevent rebalance operations
|
||||
completing for some pools. Check the current capacity utilisation with 'ceph df'
|
||||
An OSD has reached the BACKFILL FULL threshold. This will prevent rebalance operations
|
||||
from completing.
|
||||
Use 'ceph health detail' and 'ceph osd df' to identify the problem.
|
||||
|
||||
To resolve, either add capacity to the cluster, or delete unwanted data
|
||||
To resolve, add capacity to the affected OSD's failure domain, restore down/out OSDs, or delete unwanted data.
|
||||
- alert: CephOSDTooManyRepairs
|
||||
expr: ceph_health_detail{name="OSD_TOO_MANY_REPAIRS"} == 1
|
||||
for: 30s
|
||||
@@ -217,7 +228,7 @@ groups:
|
||||
type: ceph_default
|
||||
annotations:
|
||||
documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#osd-too-many-repairs
|
||||
summary: OSD has hit a high number of read errors
|
||||
summary: OSD reports a high number of read errors
|
||||
description: |
|
||||
Reads from an OSD have used a secondary PG to return data to the client, indicating
|
||||
a potential failing disk.
|
||||
@@ -231,7 +242,7 @@ groups:
|
||||
summary: Network issues delaying OSD heartbeats (public network)
|
||||
description: |
|
||||
OSD heartbeats on the cluster's 'public' network (frontend) are running slow. Investigate the network
|
||||
for any latency issues on this subnet. Use 'ceph health detail' to show the affected OSDs.
|
||||
for latency or loss issues. Use 'ceph health detail' to show the affected OSDs.
|
||||
- alert: CephOSDTimeoutsClusterNetwork
|
||||
expr: ceph_health_detail{name="OSD_SLOW_PING_TIME_BACK"} == 1
|
||||
for: 1m
|
||||
@@ -242,7 +253,7 @@ groups:
|
||||
summary: Network issues delaying OSD heartbeats (cluster network)
|
||||
description: |
|
||||
OSD heartbeats on the cluster's 'cluster' network (backend) are running slow. Investigate the network
|
||||
for any latency issues on this subnet. Use 'ceph health detail' to show the affected OSDs.
|
||||
for latency or loss issues. Use 'ceph health detail' to show the affected OSDs.
|
||||
- alert: CephOSDInternalDiskSizeMismatch
|
||||
expr: ceph_health_detail{name="BLUESTORE_DISK_SIZE_MISMATCH"} == 1
|
||||
for: 1m
|
||||
@@ -253,8 +264,8 @@ groups:
|
||||
documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#bluestore-disk-size-mismatch
|
||||
summary: OSD size inconsistency error
|
||||
description: |
|
||||
One or more OSDs have an internal inconsistency between the size of the physical device and it's metadata.
|
||||
This could lead to the OSD(s) crashing in future. You should redeploy the effected OSDs.
|
||||
One or more OSDs have an internal inconsistency between metadata and the size of the device.
|
||||
This could lead to the OSD(s) crashing in future. You should redeploy the affected OSDs.
|
||||
- alert: CephDeviceFailurePredicted
|
||||
expr: ceph_health_detail{name="DEVICE_HEALTH"} == 1
|
||||
for: 1m
|
||||
@@ -263,14 +274,14 @@ groups:
|
||||
type: ceph_default
|
||||
annotations:
|
||||
documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#id2
|
||||
summary: Device(s) have been predicted to fail soon
|
||||
summary: Device(s) predicted to fail soon
|
||||
description: |
|
||||
The device health module has determined that one or more devices will fail
|
||||
soon. To review the device states use 'ceph device ls'. To show a specific
|
||||
soon. To review device status use 'ceph device ls'. To show a specific
|
||||
device use 'ceph device info <dev id>'.
|
||||
|
||||
Mark the OSD as out (so data may migrate to other OSDs in the cluster). Once
|
||||
the osd is empty remove and replace the OSD.
|
||||
Mark the OSD out so that data may migrate to other OSDs. Once
|
||||
the OSD has drained, destroy the OSD, replace the device, and redeploy the OSD.
|
||||
- alert: CephDeviceFailurePredictionTooHigh
|
||||
expr: ceph_health_detail{name="DEVICE_HEALTH_TOOMANY"} == 1
|
||||
for: 1m
|
||||
@@ -280,12 +291,12 @@ groups:
|
||||
oid: 1.3.6.1.4.1.50495.1.2.1.4.7
|
||||
annotations:
|
||||
documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#device-health-toomany
|
||||
summary: Too many devices have been predicted to fail, unable to resolve
|
||||
summary: Too many devices are predicted to fail, unable to resolve
|
||||
description: |
|
||||
The device health module has determined that the number of devices predicted to
|
||||
fail can not be remediated automatically, since it would take too many osd's out of
|
||||
the cluster, impacting performance and potentially availabililty. You should add new
|
||||
OSDs to the cluster to allow data to be relocated to avoid the data integrity issues.
|
||||
The device health module has determined that devices predicted to
|
||||
fail can not be remediated automatically, since too many OSDs would be removed from the
|
||||
cluster to ensure performance and availabililty. Prevent data
|
||||
integrity issues by adding new OSDs so that data may be relocated.
|
||||
- alert: CephDeviceFailureRelocationIncomplete
|
||||
expr: ceph_health_detail{name="DEVICE_HEALTH_IN_USE"} == 1
|
||||
for: 1m
|
||||
@@ -294,15 +305,15 @@ groups:
|
||||
type: ceph_default
|
||||
annotations:
|
||||
documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#device-health-in-use
|
||||
summary: A device failure is predicted, but unable to relocate data
|
||||
summary: Device failure is predicted, but unable to relocate data
|
||||
description: |
|
||||
The device health module has determined that one or more devices will fail
|
||||
soon, but the normal process of relocating the data on the device to other
|
||||
OSDs in the cluster is blocked.
|
||||
|
||||
Check the the cluster has available freespace. It may be necessary to add
|
||||
more disks to the cluster to allow the data from the failing device to
|
||||
successfully migrate.
|
||||
Ensure that the cluster has available free space. It may be necessary to add
|
||||
capacity to the cluster to allow the data from the failing device to
|
||||
successfully migrate, or to enable the balancer.
|
||||
|
||||
- alert: CephOSDFlapping
|
||||
expr: |
|
||||
@@ -316,13 +327,13 @@ groups:
|
||||
oid: 1.3.6.1.4.1.50495.1.2.1.4.4
|
||||
annotations:
|
||||
documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds
|
||||
summary: Network issues are causing OSD's to flap (mark each other out)
|
||||
summary: Network issues are causing OSDs to flap (mark each other down)
|
||||
description: >
|
||||
OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was
|
||||
marked down and back up at {{ $value | humanize }} times once a
|
||||
minute for 5 minutes. This could indicate a network issue (latency,
|
||||
packet drop, disruption) on the clusters "cluster network". Check the
|
||||
network environment on the listed host(s).
|
||||
marked down and back up {{ $value | humanize }} times once a
|
||||
minute for 5 minutes. This may indicate a network issue (latency,
|
||||
packet loss, MTU mismatch) on the cluster network, or the public network if no cluster network
|
||||
is deployed. Check network stats on the listed host(s).
|
||||
|
||||
- alert: CephOSDReadErrors
|
||||
expr: ceph_health_detail{name="BLUESTORE_SPURIOUS_READ_ERRORS"} == 1
|
||||
@@ -335,7 +346,7 @@ groups:
|
||||
summary: Device read errors detected
|
||||
description: >
|
||||
An OSD has encountered read errors, but the OSD has recovered by retrying
|
||||
the reads. This may indicate an issue with the Hardware or Kernel.
|
||||
the reads. This may indicate an issue with hardware or the kernel.
|
||||
# alert on high deviation from average PG count
|
||||
- alert: CephPGImbalance
|
||||
expr: |
|
||||
@@ -343,14 +354,14 @@ groups:
|
||||
(
|
||||
(ceph_osd_numpg > 0) - on (job) group_left avg(ceph_osd_numpg > 0) by (job)
|
||||
) / on (job) group_left avg(ceph_osd_numpg > 0) by (job)
|
||||
) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata > 0.30
|
||||
) * on (ceph_daemon) group_left(hostname) ceph_osd_metadata > 0.30
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
type: ceph_default
|
||||
oid: 1.3.6.1.4.1.50495.1.2.1.4.5
|
||||
annotations:
|
||||
summary: PG allocations are not balanced across devices
|
||||
summary: PGs are not balanced across OSDs
|
||||
description: >
|
||||
OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} deviates
|
||||
by more than 30% from average PG count.
|
||||
@@ -367,13 +378,11 @@ groups:
|
||||
oid: 1.3.6.1.4.1.50495.1.2.1.5.1
|
||||
annotations:
|
||||
documentation: https://docs.ceph.com/en/latest/cephfs/health-messages#cephfs-health-messages
|
||||
summary: Ceph filesystem is damaged.
|
||||
summary: CephFS filesystem is damaged.
|
||||
description: >
|
||||
The filesystems metadata has been corrupted. Data access
|
||||
may be blocked.
|
||||
|
||||
Either analyse the output from the mds daemon admin socket, or
|
||||
escalate to support
|
||||
Filesystem metadata has been corrupted. Data may be inaccessible.
|
||||
Analyze metrics from the MDS daemon admin socket, or
|
||||
escalate to support.
|
||||
- alert: CephFilesystemOffline
|
||||
expr: ceph_health_detail{name="MDS_ALL_DOWN"} > 0
|
||||
for: 1m
|
||||
@@ -383,10 +392,10 @@ groups:
|
||||
oid: 1.3.6.1.4.1.50495.1.2.1.5.3
|
||||
annotations:
|
||||
documentation: https://docs.ceph.com/en/latest/cephfs/health-messages/#mds-all-down
|
||||
summary: Ceph filesystem is offline
|
||||
summary: CephFS filesystem is offline
|
||||
description: >
|
||||
All MDS ranks are unavailable. The ceph daemons providing the metadata
|
||||
for the Ceph filesystem are all down, rendering the filesystem offline.
|
||||
All MDS ranks are unavailable. The MDS daemons managing metadata
|
||||
are down, rendering the filesystem offline.
|
||||
- alert: CephFilesystemDegraded
|
||||
expr: ceph_health_detail{name="FS_DEGRADED"} > 0
|
||||
for: 1m
|
||||
@@ -396,11 +405,11 @@ groups:
|
||||
oid: 1.3.6.1.4.1.50495.1.2.1.5.4
|
||||
annotations:
|
||||
documentation: https://docs.ceph.com/en/latest/cephfs/health-messages/#fs-degraded
|
||||
summary: Ceph filesystem is degraded
|
||||
summary: CephFS filesystem is degraded
|
||||
description: >
|
||||
One or more metadata daemons (MDS ranks) are failed or in a
|
||||
damaged state. At best the filesystem is partially available,
|
||||
worst case is the filesystem is completely unusable.
|
||||
at worst the filesystem is completely unusable.
|
||||
- alert: CephFilesystemMDSRanksLow
|
||||
expr: ceph_health_detail{name="MDS_UP_LESS_THAN_MAX"} > 0
|
||||
for: 1m
|
||||
@@ -409,11 +418,11 @@ groups:
|
||||
type: ceph_default
|
||||
annotations:
|
||||
documentation: https://docs.ceph.com/en/latest/cephfs/health-messages/#mds-up-less-than-max
|
||||
summary: Ceph MDS daemon count is lower than configured
|
||||
summary: MDS daemon count is lower than configured
|
||||
description: >
|
||||
The filesystem's "max_mds" setting defined the number of MDS ranks in
|
||||
The filesystem's "max_mds" setting defines the number of MDS ranks in
|
||||
the filesystem. The current number of active MDS daemons is less than
|
||||
this setting.
|
||||
this value.
|
||||
- alert: CephFilesystemInsufficientStandby
|
||||
expr: ceph_health_detail{name="MDS_INSUFFICIENT_STANDBY"} > 0
|
||||
for: 1m
|
||||
@@ -422,11 +431,11 @@ groups:
|
||||
type: ceph_default
|
||||
annotations:
|
||||
documentation: https://docs.ceph.com/en/latest/cephfs/health-messages/#mds-insufficient-standby
|
||||
summary: Ceph filesystem standby daemons too low
|
||||
summary: Ceph filesystem standby daemons too few
|
||||
description: >
|
||||
The minimum number of standby daemons determined by standby_count_wanted
|
||||
is less than the actual number of standby daemons. Adjust the standby count
|
||||
or increase the number of mds daemons within the filesystem.
|
||||
The minimum number of standby daemons required by standby_count_wanted
|
||||
is less than the current number of standby daemons. Adjust the standby count
|
||||
or increase the number of MDS daemons.
|
||||
- alert: CephFilesystemFailureNoStandby
|
||||
expr: ceph_health_detail{name="FS_WITH_FAILED_MDS"} > 0
|
||||
for: 1m
|
||||
@@ -436,11 +445,11 @@ groups:
|
||||
oid: 1.3.6.1.4.1.50495.1.2.1.5.5
|
||||
annotations:
|
||||
documentation: https://docs.ceph.com/en/latest/cephfs/health-messages/#fs-with-failed-mds
|
||||
summary: Ceph MDS daemon failed, no further standby available
|
||||
summary: MDS daemon failed, no further standby available
|
||||
description: >
|
||||
An MDS daemon has failed, leaving only one active rank without
|
||||
further standby. Investigate the cause of the failure or add a
|
||||
standby daemon
|
||||
An MDS daemon has failed, leaving only one active rank and no
|
||||
available standby. Investigate the cause of the failure or add a
|
||||
standby MDS.
|
||||
- alert: CephFilesystemReadOnly
|
||||
expr: ceph_health_detail{name="MDS_HEALTH_READ_ONLY"} > 0
|
||||
for: 1m
|
||||
@@ -450,13 +459,13 @@ groups:
|
||||
oid: 1.3.6.1.4.1.50495.1.2.1.5.2
|
||||
annotations:
|
||||
documentation: https://docs.ceph.com/en/latest/cephfs/health-messages#cephfs-health-messages
|
||||
summary: Ceph filesystem in read only mode, due to write error(s)
|
||||
summary: CephFS filesystem in read only mode due to write error(s)
|
||||
description: >
|
||||
The filesystem has switched to READ ONLY due to an unexpected
|
||||
write error, when writing to the metadata pool
|
||||
error when writing to the metadata pool.
|
||||
|
||||
Either analyse the output from the mds daemon admin socket, or
|
||||
escalate to support
|
||||
Analyze the output from the MDS daemon admin socket, or
|
||||
escalate to support.
|
||||
|
||||
- name: mgr
|
||||
rules:
|
||||
@@ -469,11 +478,11 @@ groups:
|
||||
oid: 1.3.6.1.4.1.50495.1.2.1.6.1
|
||||
annotations:
|
||||
documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#recent-mgr-module-crash
|
||||
summary: A mgr module has recently crashed
|
||||
summary: A manager module has recently crashed
|
||||
description: >
|
||||
One or more mgr modules have crashed and are yet to be acknowledged by the administrator. A
|
||||
crashed module may impact functionality within the cluster. Use the 'ceph crash' commands to
|
||||
investigate which module has failed, and archive it to acknowledge the failure.
|
||||
One or more mgr modules have crashed and have yet to be acknowledged by an administrator. A
|
||||
crashed module may impact functionality within the cluster. Use the 'ceph crash' command to
|
||||
determine which module has failed, and archive it to acknowledge the failure.
|
||||
- alert: CephMgrPrometheusModuleInactive
|
||||
expr: up{job="ceph"} == 0
|
||||
for: 1m
|
||||
@@ -482,16 +491,16 @@ groups:
|
||||
type: ceph_default
|
||||
oid: 1.3.6.1.4.1.50495.1.2.1.6.2
|
||||
annotations:
|
||||
summary: Ceph's mgr/prometheus module is not available
|
||||
summary: The mgr/prometheus module is not available
|
||||
description: >
|
||||
The mgr/prometheus module at {{ $labels.instance }} is unreachable. This
|
||||
could mean that the module has been disabled or the mgr itself is down.
|
||||
could mean that the module has been disabled or the mgr daemon itself is down.
|
||||
|
||||
Without the mgr/prometheus module metrics and alerts will no longer
|
||||
function. Open a shell to ceph and use 'ceph -s' to to determine whether the
|
||||
mgr is active. If the mgr is not active, restart it, otherwise you can check
|
||||
the mgr/prometheus module is loaded with 'ceph mgr module ls' and if it's
|
||||
not listed as enabled, enable it with 'ceph mgr module enable prometheus'
|
||||
function. Open a shell to an admin node or toolbox pod and use 'ceph -s' to to determine whether the
|
||||
mgr is active. If the mgr is not active, restart it, otherwise you can determine
|
||||
the mgr/prometheus module status with 'ceph mgr module ls'. If it is
|
||||
not listed as enabled, enable it with 'ceph mgr module enable prometheus'.
|
||||
|
||||
- name: pgs
|
||||
rules:
|
||||
@@ -503,11 +512,10 @@ groups:
|
||||
type: ceph_default
|
||||
oid: 1.3.6.1.4.1.50495.1.2.1.7.1
|
||||
annotations:
|
||||
summary: One or more Placement Groups are inactive
|
||||
summary: One or more placement groups are inactive
|
||||
description: >
|
||||
{{ $value }} PGs have been inactive for more than 5 minutes in pool {{ $labels.name }}.
|
||||
Inactive placement groups aren't able to serve read/write
|
||||
requests.
|
||||
Inactive placement groups are not able to serve read/write requests.
|
||||
- alert: CephPGsUnclean
|
||||
expr: ceph_pool_metadata * on(pool_id,instance) group_left() (ceph_pg_total - ceph_pg_clean) > 0
|
||||
for: 15m
|
||||
@@ -516,10 +524,10 @@ groups:
|
||||
type: ceph_default
|
||||
oid: 1.3.6.1.4.1.50495.1.2.1.7.2
|
||||
annotations:
|
||||
summary: One or more platcment groups are marked unclean
|
||||
summary: One or more placement groups are marked unclean
|
||||
description: >
|
||||
{{ $value }} PGs haven't been clean for more than 15 minutes in pool {{ $labels.name }}.
|
||||
Unclean PGs haven't been able to completely recover from a previous failure.
|
||||
{{ $value }} PGs have been unclean for more than 15 minutes in pool {{ $labels.name }}.
|
||||
Unclean PGs have not recovered from a previous failure.
|
||||
- alert: CephPGsDamaged
|
||||
expr: ceph_health_detail{name=~"PG_DAMAGED|OSD_SCRUB_ERRORS"} == 1
|
||||
for: 5m
|
||||
@@ -529,13 +537,12 @@ groups:
|
||||
oid: 1.3.6.1.4.1.50495.1.2.1.7.4
|
||||
annotations:
|
||||
documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#pg-damaged
|
||||
summary: Placement group damaged, manual intervention needed
|
||||
summary: Placement group damaged; manual intervention needed
|
||||
description: >
|
||||
During data consistency checks (scrub), at least one PG has been flagged as being
|
||||
damaged or inconsistent.
|
||||
Scrubs have flagged at least one PG as damaged or inconsistent.
|
||||
|
||||
Check to see which PG is affected, and attempt a manual repair if necessary. To list
|
||||
problematic placement groups, use 'rados list-inconsistent-pg <pool>'. To repair PGs use
|
||||
problematic placement groups, use 'ceph health detail' or 'rados list-inconsistent-pg <pool>'. To repair PGs use
|
||||
the 'ceph pg repair <pg_num>' command.
|
||||
- alert: CephPGRecoveryAtRisk
|
||||
expr: ceph_health_detail{name="PG_RECOVERY_FULL"} == 1
|
||||
@@ -546,11 +553,11 @@ groups:
|
||||
oid: 1.3.6.1.4.1.50495.1.2.1.7.5
|
||||
annotations:
|
||||
documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#pg-recovery-full
|
||||
summary: OSDs are too full for automatic recovery
|
||||
summary: OSDs are too full for recovery
|
||||
description: >
|
||||
Data redundancy may be reduced, or is at risk, since one or more OSDs are at or above their
|
||||
'full' threshold. Add more capacity to the cluster, or delete unwanted data.
|
||||
- alert: CephPGUnavilableBlockingIO
|
||||
Data redundancy is at risk since one or more OSDs are at or above the
|
||||
'full' threshold. Add capacity to the cluster, restore down/out OSDs, or delete unwanted data.
|
||||
- alert: CephPGUnavailableBlockingIO
|
||||
# PG_AVAILABILITY, but an OSD is not in a DOWN state
|
||||
expr: ((ceph_health_detail{name="PG_AVAILABILITY"} == 1) - scalar(ceph_health_detail{name="OSD_DOWN"})) == 1
|
||||
for: 1m
|
||||
@@ -560,10 +567,10 @@ groups:
|
||||
oid: 1.3.6.1.4.1.50495.1.2.1.7.3
|
||||
annotations:
|
||||
documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#pg-availability
|
||||
summary: Placement group is unavailable, blocking some I/O
|
||||
summary: PG is unavailable, blocking I/O
|
||||
description: >
|
||||
Data availability is reduced impacting the clusters ability to service I/O to some data. One or
|
||||
more placement groups (PGs) are in a state that blocks IO.
|
||||
Data availability is reduced, impacting the cluster's ability to service I/O. One or
|
||||
more placement groups (PGs) are in a state that blocks I/O.
|
||||
- alert: CephPGBackfillAtRisk
|
||||
expr: ceph_health_detail{name="PG_BACKFILL_FULL"} == 1
|
||||
for: 1m
|
||||
@@ -573,7 +580,7 @@ groups:
|
||||
oid: 1.3.6.1.4.1.50495.1.2.1.7.6
|
||||
annotations:
|
||||
documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#pg-backfill-full
|
||||
summary: Backfill operations are blocked, due to lack of freespace
|
||||
summary: Backfill operations are blocked due to lack of free space
|
||||
description: >
|
||||
Data redundancy may be at risk due to lack of free space within the cluster. One or more OSDs
|
||||
have breached their 'backfillfull' threshold. Add more capacity, or delete unwanted data.
|
||||
@@ -587,10 +594,10 @@ groups:
|
||||
documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#pg-not-scrubbed
|
||||
summary: Placement group(s) have not been scrubbed
|
||||
description: |
|
||||
One or more PGs have not been scrubbed recently. The scrub process is a data integrity
|
||||
feature, protectng against bit-rot. It checks that objects and their metadata (size and
|
||||
attributes) match across object replicas. When PGs miss their scrub window, it may
|
||||
indicate the scrub window is too small, or PGs were not in a 'clean' state during the
|
||||
One or more PGs have not been scrubbed recently. Scrubs check metadata integrity,
|
||||
protecting against bit-rot. They check that metadata
|
||||
is consistent across data replicas. When PGs miss their scrub interval, it may
|
||||
indicate that the scrub window is too small, or PGs were not in a 'clean' state during the
|
||||
scrub window.
|
||||
|
||||
You can manually initiate a scrub with: ceph pg scrub <pgid>
|
||||
@@ -606,10 +613,11 @@ groups:
|
||||
description: |
|
||||
The number of placement groups per OSD is too high (exceeds the mon_max_pg_per_osd setting).
|
||||
|
||||
Check that the pg_autoscaler hasn't been disabled for any of the pools, with 'ceph osd pool autoscale-status'
|
||||
Check that the pg_autoscaler has not been disabled for any pools with 'ceph osd pool autoscale-status',
|
||||
and that the profile selected is appropriate. You may also adjust the target_size_ratio of a pool to guide
|
||||
the autoscaler based on the expected relative size of the pool
|
||||
(i.e. 'ceph osd pool set cephfs.cephfs.meta target_size_ratio .1')
|
||||
('ceph osd pool set cephfs.cephfs.meta target_size_ratio .1') or set the pg_autoscaler
|
||||
mode to "warn" and adjust pg_num appropriately for one or more pools.
|
||||
- alert: CephPGNotDeepScrubbed
|
||||
expr: ceph_health_detail{name="PG_NOT_DEEP_SCRUBBED"} == 1
|
||||
for: 5m
|
||||
@@ -620,9 +628,9 @@ groups:
|
||||
documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#pg-not-deep-scrubbed
|
||||
summary: Placement group(s) have not been deep scrubbed
|
||||
description: |
|
||||
One or more PGs have not been deep scrubbed recently. Deep scrub is a data integrity
|
||||
feature, protectng against bit-rot. It compares the contents of objects and their
|
||||
replicas for inconsistency. When PGs miss their deep scrub window, it may indicate
|
||||
One or more PGs have not been deep scrubbed recently. Deep scrubs
|
||||
protect against bit-rot. They compare data
|
||||
replicas to ensure consistency. When PGs miss their deep scrub interval, it may indicate
|
||||
that the window is too small or PGs were not in a 'clean' state during the deep-scrub
|
||||
window.
|
||||
|
||||
@@ -640,9 +648,9 @@ groups:
|
||||
annotations:
|
||||
summary: Root filesystem is dangerously full
|
||||
description: >
|
||||
Root volume (OSD and MON store) is dangerously full: {{ $value | humanize }}% free.
|
||||
Root volume is dangerously full: {{ $value | humanize }}% free.
|
||||
|
||||
# alert on nic packet errors and drops rates > 1% packets/s
|
||||
# alert on packet errors and drop rate
|
||||
- alert: CephNodeNetworkPacketDrops
|
||||
expr: |
|
||||
(
|
||||
@@ -660,7 +668,7 @@ groups:
|
||||
type: ceph_default
|
||||
oid: 1.3.6.1.4.1.50495.1.2.1.8.2
|
||||
annotations:
|
||||
summary: One or more Nics is seeing packet drops
|
||||
summary: One or more NICs reports packet drops
|
||||
description: >
|
||||
Node {{ $labels.instance }} experiences packet drop > 0.01% or >
|
||||
10 packets/s on interface {{ $labels.device }}.
|
||||
@@ -682,7 +690,7 @@ groups:
|
||||
type: ceph_default
|
||||
oid: 1.3.6.1.4.1.50495.1.2.1.8.3
|
||||
annotations:
|
||||
summary: One or more Nics is seeing packet errors
|
||||
summary: One or more NICs reports packet errors
|
||||
description: >
|
||||
Node {{ $labels.instance }} experiences packet errors > 0.01% or
|
||||
> 10 packets/s on interface {{ $labels.device }}.
|
||||
@@ -698,11 +706,11 @@ groups:
|
||||
type: ceph_default
|
||||
oid: 1.3.6.1.4.1.50495.1.2.1.8.4
|
||||
annotations:
|
||||
summary: Host filesystem freespace is getting low
|
||||
summary: Host filesystem free space is low
|
||||
description: >
|
||||
Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }}
|
||||
will be full in less than 5 days assuming the average fill-up
|
||||
rate of the past 48 hours.
|
||||
will be full in less than 5 days based on the 48 hour trailing
|
||||
fill rate.
|
||||
|
||||
- alert: CephNodeInconsistentMTU
|
||||
expr: node_network_mtu_bytes{device!="lo"} * (node_network_up{device!="lo"} > 0) != on() group_left() (quantile(0.5, node_network_mtu_bytes{device!="lo"}))
|
||||
@@ -710,25 +718,37 @@ groups:
|
||||
severity: warning
|
||||
type: ceph_default
|
||||
annotations:
|
||||
summary: MTU settings across Ceph hosts are inconsistent
|
||||
summary: MTU settings across hosts are inconsistent
|
||||
description: >
|
||||
Node {{ $labels.instance }} has a different MTU size ({{ $value }})
|
||||
than the median value on device {{ $labels.device }}.
|
||||
|
||||
- name: pools
|
||||
rules:
|
||||
- alert: CephPoolGrowthWarning
|
||||
expr: |
|
||||
(predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id)
|
||||
group_right ceph_pool_metadata) >= 95
|
||||
labels:
|
||||
severity: warning
|
||||
type: ceph_default
|
||||
oid: 1.3.6.1.4.1.50495.1.2.1.9.2
|
||||
annotations:
|
||||
summary: Pool growth rate may soon exceed capacity
|
||||
description: >
|
||||
Pool '{{ $labels.name }}' will be full in less than 5 days
|
||||
assuming the average fill-up rate of the past 48 hours.
|
||||
- alert: CephPoolBackfillFull
|
||||
expr: ceph_health_detail{name="POOL_BACKFILLFULL"} > 0
|
||||
labels:
|
||||
severity: warning
|
||||
type: ceph_default
|
||||
annotations:
|
||||
summary: Freespace in a pool is too low for recovery/rebalance
|
||||
summary: Free space in a pool is too low for recovery/backfill
|
||||
description: >
|
||||
A pool is approaching it's near full threshold, which will
|
||||
prevent rebalance operations from completing. You should
|
||||
consider adding more capacity to the pool.
|
||||
|
||||
A pool is approaching the near full threshold, which will
|
||||
prevent recovery/backfill from completing.
|
||||
Consider adding more capacity.
|
||||
- alert: CephPoolFull
|
||||
expr: ceph_health_detail{name="POOL_FULL"} > 0
|
||||
for: 1m
|
||||
@@ -740,15 +760,15 @@ groups:
|
||||
documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#pool-full
|
||||
summary: Pool is full - writes are blocked
|
||||
description: |
|
||||
A pool has reached it's MAX quota, or the OSDs supporting the pool
|
||||
have reached their FULL threshold. Until this is resolved, writes to
|
||||
A pool has reached its MAX quota, or OSDs supporting the pool
|
||||
have reached the FULL threshold. Until this is resolved, writes to
|
||||
the pool will be blocked.
|
||||
Pool Breakdown (top 5)
|
||||
{{- range query "topk(5, sort_desc(ceph_pool_percent_used * on(pool_id) group_right ceph_pool_metadata))" }}
|
||||
- {{ .Labels.name }} at {{ .Value }}%
|
||||
{{- end }}
|
||||
Either increase the pools quota, or add capacity to the cluster first
|
||||
then increase it's quota (e.g. ceph osd pool set quota <pool_name> max_bytes <bytes>)
|
||||
Increase the pool's quota, or add capacity to the cluster
|
||||
then increase the pool's quota (e.g. ceph osd pool set quota <pool_name> max_bytes <bytes>)
|
||||
- alert: CephPoolNearFull
|
||||
expr: ceph_health_detail{name="POOL_NEAR_FULL"} > 0
|
||||
for: 5m
|
||||
@@ -756,17 +776,18 @@ groups:
|
||||
severity: warning
|
||||
type: ceph_default
|
||||
annotations:
|
||||
summary: One or more Ceph pools are getting full
|
||||
summary: One or more Ceph pools are nearly full
|
||||
description: |
|
||||
A pool has exceeeded it warning (percent full) threshold, or the OSDs
|
||||
supporting the pool have reached their NEARFULL thresholds. Writes may
|
||||
continue, but you are at risk of the pool going read only if more capacity
|
||||
A pool has exceeded the warning (percent full) threshold, or OSDs
|
||||
supporting the pool have reached the NEARFULL threshold. Writes may
|
||||
continue, but you are at risk of the pool going read-only if more capacity
|
||||
isn't made available.
|
||||
|
||||
Determine the affected pool with 'ceph df detail', for example looking
|
||||
at QUOTA BYTES and STORED. Either increase the pools quota, or add
|
||||
capacity to the cluster first then increase it's quota
|
||||
(e.g. ceph osd pool set quota <pool_name> max_bytes <bytes>)
|
||||
Determine the affected pool with 'ceph df detail', looking
|
||||
at QUOTA BYTES and STORED. Increase the pool's quota, or add
|
||||
capacity to the cluster then increase the pool's quota
|
||||
(e.g. ceph osd pool set quota <pool_name> max_bytes <bytes>).
|
||||
Also ensure that the balancer is active.
|
||||
- name: healthchecks
|
||||
rules:
|
||||
- alert: CephSlowOps
|
||||
@@ -777,7 +798,7 @@ groups:
|
||||
type: ceph_default
|
||||
annotations:
|
||||
documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#slow-ops
|
||||
summary: MON/OSD operations are slow to complete
|
||||
summary: OSD operations are slow to complete
|
||||
description: >
|
||||
{{ $value }} OSD requests are taking too long to process (osd_op_complaint_time exceeded)
|
||||
|
||||
@@ -793,9 +814,9 @@ groups:
|
||||
oid: 1.3.6.1.4.1.50495.1.2.1.10.1
|
||||
annotations:
|
||||
documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#object-unfound
|
||||
summary: Object(s) has been marked UNFOUND
|
||||
summary: Object(s) marked UNFOUND
|
||||
description: |
|
||||
A version of a RADOS object can not be found, even though all OSDs are up. I/O
|
||||
The latest version of a RADOS object can not be found, even though all OSDs are up. I/O
|
||||
requests for this object from clients will block (hang). Resolving this issue may
|
||||
require the object to be rolled back to a prior version manually, and manually verified.
|
||||
# Generic
|
||||
@@ -813,5 +834,5 @@ groups:
|
||||
summary: One or more Ceph daemons have crashed, and are pending acknowledgement
|
||||
description: |
|
||||
One or more daemons have crashed recently, and need to be acknowledged. This notification
|
||||
ensures that software crashes don't go unseen. To acknowledge a crash, use the
|
||||
ensures that software crashes do not go unseen. To acknowledge a crash, use the
|
||||
'ceph crash archive <id>' command.
|
||||
@@ -19,9 +19,9 @@ spec:
|
||||
type: ceph_default
|
||||
oid: 1.3.6.1.4.1.50495.1.2.1.2.1
|
||||
annotations:
|
||||
summary: Cluster is in an ERROR state
|
||||
summary: Cluster is in the ERROR state
|
||||
description: >
|
||||
Ceph in HEALTH_ERROR state for more than 5 minutes.
|
||||
The cluster state has been HEALTH_ERROR for more than 5 minutes.
|
||||
Please check "ceph health detail" for more information.
|
||||
|
||||
- alert: CephHealthWarning
|
||||
@@ -31,9 +31,9 @@ spec:
|
||||
severity: warning
|
||||
type: ceph_default
|
||||
annotations:
|
||||
summary: Cluster is in a WARNING state
|
||||
summary: Cluster is in the WARNING state
|
||||
description: >
|
||||
Ceph has been in HEALTH_WARN for more than 15 minutes.
|
||||
The cluster state has been HEALTH_WARN for more than 15 minutes.
|
||||
Please check "ceph health detail" for more information.
|
||||
|
||||
- name: mon
|
||||
@@ -50,7 +50,7 @@ spec:
|
||||
summary: Monitor quorum is at risk
|
||||
description: |
|
||||
{{ $min := query "floor(count(ceph_mon_metadata) / 2) +1" | first | value }}Quorum requires a majority of monitors (x {{ $min }}) to be active
|
||||
Without quorum the cluster will become inoperable, affecting all connected clients and services.
|
||||
Without quorum the cluster will become inoperable, affecting all services and connected clients.
|
||||
|
||||
The following monitors are down:
|
||||
{{- range query "(ceph_mon_quorum_status == 0) + on(ceph_daemon) group_left(hostname) (ceph_mon_metadata * 0)" }}
|
||||
@@ -64,10 +64,10 @@ spec:
|
||||
type: ceph_default
|
||||
annotations:
|
||||
documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#mon-down
|
||||
summary: One of more ceph monitors are down
|
||||
summary: One or more monitors down
|
||||
description: |
|
||||
{{ $down := query "count(ceph_mon_quorum_status == 0)" | first | value }}{{ $s := "" }}{{ if gt $down 1.0 }}{{ $s = "s" }}{{ end }}You have {{ $down }} monitor{{ $s }} down.
|
||||
Quorum is still intact, but the loss of further monitors will make your cluster inoperable.
|
||||
{{ $down := query "count(ceph_mon_quorum_status == 0)" | first | value }}{{ $s := "" }}{{ if gt $down 1.0 }}{{ $s = "s" }}{{ end }}There are {{ $down }} monitor{{ $s }} down.
|
||||
Quorum is still intact, but the loss of an additional monitor will make your cluster inoperable.
|
||||
|
||||
The following monitors are down:
|
||||
{{- range query "(ceph_mon_quorum_status == 0) + on(ceph_daemon) group_left(hostname) (ceph_mon_metadata * 0)" }}
|
||||
@@ -82,11 +82,15 @@ spec:
|
||||
oid: 1.3.6.1.4.1.50495.1.2.1.3.2
|
||||
annotations:
|
||||
documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#mon-disk-crit
|
||||
summary: Disk space on at least one monitor is critically low
|
||||
summary: Filesystem space on at least one monitor is critically low
|
||||
description: |
|
||||
The free space available to a monitor's store is critically low (<5% by default).
|
||||
You should increase the space available to the monitor(s). The
|
||||
default location for the store sits under /var/lib/ceph. Your monitor hosts are;
|
||||
The free space available to a monitor's store is critically low.
|
||||
You should increase the space available to the monitor(s). The default directory
|
||||
is /var/lib/ceph/mon-*/data/store.db on traditional deployments, and under
|
||||
/var/lib/rook/mon-*/data/store.db on the mon pod's worker node for Rook.
|
||||
Look for old, rotated versions of *.log and MANIFEST*. Do NOT touch any *.sst files.
|
||||
Also check any other directories under /var/lib/rook and other directories on the
|
||||
same filesystem, often /var/log and /var/tmp are culprits. Your monitor hosts are;
|
||||
{{- range query "ceph_mon_metadata"}}
|
||||
- {{ .Labels.hostname }}
|
||||
{{- end }}
|
||||
@@ -102,8 +106,12 @@ spec:
|
||||
summary: Disk space on at least one monitor is approaching full
|
||||
description: |
|
||||
The space available to a monitor's store is approaching full (>70% is the default).
|
||||
You should increase the space available to the monitor store. The
|
||||
default location for the store sits under /var/lib/ceph. Your monitor hosts are;
|
||||
You should increase the space available to the monitor(s). The default directory
|
||||
is /var/lib/ceph/mon-*/data/store.db on traditional deployments, and under
|
||||
/var/lib/rook/mon-*/data/store.db on the mon pod's worker node for Rook.
|
||||
Look for old, rotated versions of *.log and MANIFEST*. Do NOT touch any *.sst files.
|
||||
Also check any other directories under /var/lib/rook and other directories on the
|
||||
same filesystem, often /var/log and /var/tmp are culprits. Your monitor hosts are;
|
||||
{{- range query "ceph_mon_metadata"}}
|
||||
- {{ .Labels.hostname }}
|
||||
{{- end }}
|
||||
@@ -116,14 +124,15 @@ spec:
|
||||
type: ceph_default
|
||||
annotations:
|
||||
documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#mon-clock-skew
|
||||
summary: Clock skew across the Monitor hosts detected
|
||||
summary: Clock skew detected among monitors
|
||||
description: |
|
||||
The ceph monitors rely on a consistent time reference to maintain
|
||||
quorum and cluster consistency. This event indicates that at least
|
||||
one of your mons is not sync'd correctly.
|
||||
Ceph monitors rely on closely synchronized time to maintain
|
||||
quorum and cluster consistency. This event indicates that time on at least
|
||||
one mon has drifted too far from the lead mon.
|
||||
|
||||
Review the cluster status with ceph -s. This will show which monitors
|
||||
are affected. Check the time sync status on each monitor host.
|
||||
Review cluster status with ceph -s. This will show which monitors
|
||||
are affected. Check the time sync status on each monitor host with
|
||||
"ceph time-sync-status" and the state and peers of your ntpd or chrony daemon.
|
||||
|
||||
- name: osd
|
||||
rules:
|
||||
@@ -165,7 +174,7 @@ spec:
|
||||
oid: 1.3.6.1.4.1.50495.1.2.1.4.2
|
||||
annotations:
|
||||
documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#osd-down
|
||||
summary: An OSD has been marked down/unavailable
|
||||
summary: An OSD has been marked down
|
||||
description: |
|
||||
{{ $num := query "count(ceph_osd_up == 0)" | first | value }}{{ $s := "" }}{{ if gt $num 1.0 }}{{ $s = "s" }}{{ end }}{{ $num }} OSD{{ $s }} down for over 5mins.
|
||||
|
||||
@@ -185,10 +194,10 @@ spec:
|
||||
documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#osd-nearfull
|
||||
summary: OSD(s) running low on free space (NEARFULL)
|
||||
description: |
|
||||
One or more OSDs have reached their NEARFULL threshold
|
||||
One or more OSDs have reached the NEARFULL threshold
|
||||
|
||||
Use 'ceph health detail' to identify which OSDs have reached this threshold.
|
||||
To resolve, either add capacity to the cluster, or delete unwanted data
|
||||
Use 'ceph health detail' and 'ceph osd df' to identify the problem.
|
||||
To resolve, add capacity to the affected OSD's failure domain, restore down/out OSDs, or delete unwanted data.
|
||||
- alert: CephOSDFull
|
||||
expr: ceph_health_detail{name="OSD_FULL"} > 0
|
||||
for: 1m
|
||||
@@ -198,12 +207,13 @@ spec:
|
||||
oid: 1.3.6.1.4.1.50495.1.2.1.4.6
|
||||
annotations:
|
||||
documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#osd-full
|
||||
summary: OSD(s) is full, writes blocked
|
||||
summary: OSD full, writes blocked
|
||||
description: |
|
||||
An OSD has reached it's full threshold. Writes from all pools that share the
|
||||
An OSD has reached the FULL threshold. Writes to pools that share the
|
||||
affected OSD will be blocked.
|
||||
|
||||
To resolve, either add capacity to the cluster, or delete unwanted data
|
||||
Use 'ceph health detail' and 'ceph osd df' to identify the problem.
|
||||
To resolve, add capacity to the affected OSD's failure domain, restore down/out OSDs, or delete unwanted data.
|
||||
- alert: CephOSDBackfillFull
|
||||
expr: ceph_health_detail{name="OSD_BACKFILLFULL"} > 0
|
||||
for: 1m
|
||||
@@ -214,10 +224,11 @@ spec:
|
||||
documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#osd-backfillfull
|
||||
summary: OSD(s) too full for backfill operations
|
||||
description: |
|
||||
An OSD has reached it's BACKFILL FULL threshold. This will prevent rebalance operations
|
||||
completing for some pools. Check the current capacity utilisation with 'ceph df'
|
||||
An OSD has reached the BACKFILL FULL threshold. This will prevent rebalance operations
|
||||
from completing.
|
||||
Use 'ceph health detail' and 'ceph osd df' to identify the problem.
|
||||
|
||||
To resolve, either add capacity to the cluster, or delete unwanted data
|
||||
To resolve, add capacity to the affected OSD's failure domain, restore down/out OSDs, or delete unwanted data.
|
||||
- alert: CephOSDTooManyRepairs
|
||||
expr: ceph_health_detail{name="OSD_TOO_MANY_REPAIRS"} == 1
|
||||
for: 30s
|
||||
@@ -226,7 +237,7 @@ spec:
|
||||
type: ceph_default
|
||||
annotations:
|
||||
documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#osd-too-many-repairs
|
||||
summary: OSD has hit a high number of read errors
|
||||
summary: OSD reports a high number of read errors
|
||||
description: |
|
||||
Reads from an OSD have used a secondary PG to return data to the client, indicating
|
||||
a potential failing disk.
|
||||
@@ -240,7 +251,7 @@ spec:
|
||||
summary: Network issues delaying OSD heartbeats (public network)
|
||||
description: |
|
||||
OSD heartbeats on the cluster's 'public' network (frontend) are running slow. Investigate the network
|
||||
for any latency issues on this subnet. Use 'ceph health detail' to show the affected OSDs.
|
||||
for latency or loss issues. Use 'ceph health detail' to show the affected OSDs.
|
||||
- alert: CephOSDTimeoutsClusterNetwork
|
||||
expr: ceph_health_detail{name="OSD_SLOW_PING_TIME_BACK"} == 1
|
||||
for: 1m
|
||||
@@ -251,7 +262,7 @@ spec:
|
||||
summary: Network issues delaying OSD heartbeats (cluster network)
|
||||
description: |
|
||||
OSD heartbeats on the cluster's 'cluster' network (backend) are running slow. Investigate the network
|
||||
for any latency issues on this subnet. Use 'ceph health detail' to show the affected OSDs.
|
||||
for latency or loss issues. Use 'ceph health detail' to show the affected OSDs.
|
||||
- alert: CephOSDInternalDiskSizeMismatch
|
||||
expr: ceph_health_detail{name="BLUESTORE_DISK_SIZE_MISMATCH"} == 1
|
||||
for: 1m
|
||||
@@ -262,8 +273,8 @@ spec:
|
||||
documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#bluestore-disk-size-mismatch
|
||||
summary: OSD size inconsistency error
|
||||
description: |
|
||||
One or more OSDs have an internal inconsistency between the size of the physical device and it's metadata.
|
||||
This could lead to the OSD(s) crashing in future. You should redeploy the effected OSDs.
|
||||
One or more OSDs have an internal inconsistency between metadata and the size of the device.
|
||||
This could lead to the OSD(s) crashing in future. You should redeploy the affected OSDs.
|
||||
- alert: CephDeviceFailurePredicted
|
||||
expr: ceph_health_detail{name="DEVICE_HEALTH"} == 1
|
||||
for: 1m
|
||||
@@ -272,14 +283,14 @@ spec:
|
||||
type: ceph_default
|
||||
annotations:
|
||||
documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#id2
|
||||
summary: Device(s) have been predicted to fail soon
|
||||
summary: Device(s) predicted to fail soon
|
||||
description: |
|
||||
The device health module has determined that one or more devices will fail
|
||||
soon. To review the device states use 'ceph device ls'. To show a specific
|
||||
soon. To review device status use 'ceph device ls'. To show a specific
|
||||
device use 'ceph device info <dev id>'.
|
||||
|
||||
Mark the OSD as out (so data may migrate to other OSDs in the cluster). Once
|
||||
the osd is empty remove and replace the OSD.
|
||||
Mark the OSD out so that data may migrate to other OSDs. Once
|
||||
the OSD has drained, destroy the OSD, replace the device, and redeploy the OSD.
|
||||
- alert: CephDeviceFailurePredictionTooHigh
|
||||
expr: ceph_health_detail{name="DEVICE_HEALTH_TOOMANY"} == 1
|
||||
for: 1m
|
||||
@@ -289,12 +300,12 @@ spec:
|
||||
oid: 1.3.6.1.4.1.50495.1.2.1.4.7
|
||||
annotations:
|
||||
documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#device-health-toomany
|
||||
summary: Too many devices have been predicted to fail, unable to resolve
|
||||
summary: Too many devices are predicted to fail, unable to resolve
|
||||
description: |
|
||||
The device health module has determined that the number of devices predicted to
|
||||
fail can not be remediated automatically, since it would take too many osd's out of
|
||||
the cluster, impacting performance and potentially availability. You should add new
|
||||
OSDs to the cluster to allow data to be relocated to avoid the data integrity issues.
|
||||
The device health module has determined that devices predicted to
|
||||
fail can not be remediated automatically, since too many OSDs would be removed from the
|
||||
cluster to ensure performance and availabililty. Prevent data
|
||||
integrity issues by adding new OSDs so that data may be relocated.
|
||||
- alert: CephDeviceFailureRelocationIncomplete
|
||||
expr: ceph_health_detail{name="DEVICE_HEALTH_IN_USE"} == 1
|
||||
for: 1m
|
||||
@@ -303,15 +314,15 @@ spec:
|
||||
type: ceph_default
|
||||
annotations:
|
||||
documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#device-health-in-use
|
||||
summary: A device failure is predicted, but unable to relocate data
|
||||
summary: Device failure is predicted, but unable to relocate data
|
||||
description: |
|
||||
The device health module has determined that one or more devices will fail
|
||||
soon, but the normal process of relocating the data on the device to other
|
||||
OSDs in the cluster is blocked.
|
||||
|
||||
Check the cluster has available freespace. It may be necessary to add
|
||||
more disks to the cluster to allow the data from the failing device to
|
||||
successfully migrate.
|
||||
Ensure that the cluster has available free space. It may be necessary to add
|
||||
capacity to the cluster to allow the data from the failing device to
|
||||
successfully migrate, or to enable the balancer.
|
||||
|
||||
- alert: CephOSDFlapping
|
||||
expr: |
|
||||
@@ -325,13 +336,13 @@ spec:
|
||||
oid: 1.3.6.1.4.1.50495.1.2.1.4.4
|
||||
annotations:
|
||||
documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds
|
||||
summary: Network issues are causing OSD's to flap (mark each other out)
|
||||
summary: Network issues are causing OSDs to flap (mark each other down)
|
||||
description: >
|
||||
OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was
|
||||
marked down and back up at {{ $value | humanize }} times once a
|
||||
minute for 5 minutes. This could indicate a network issue (latency,
|
||||
packet drop, disruption) on the clusters "cluster network". Check the
|
||||
network environment on the listed host(s).
|
||||
marked down and back up {{ $value | humanize }} times once a
|
||||
minute for 5 minutes. This may indicate a network issue (latency,
|
||||
packet loss, MTU mismatch) on the cluster network, or the public network if no cluster network
|
||||
is deployed. Check network stats on the listed host(s).
|
||||
|
||||
- alert: CephOSDReadErrors
|
||||
expr: ceph_health_detail{name="BLUESTORE_SPURIOUS_READ_ERRORS"} == 1
|
||||
@@ -344,7 +355,7 @@ spec:
|
||||
summary: Device read errors detected
|
||||
description: >
|
||||
An OSD has encountered read errors, but the OSD has recovered by retrying
|
||||
the reads. This may indicate an issue with the Hardware or Kernel.
|
||||
the reads. This may indicate an issue with hardware or the kernel.
|
||||
# alert on high deviation from average PG count
|
||||
- alert: CephPGImbalance
|
||||
expr: |
|
||||
@@ -352,14 +363,14 @@ spec:
|
||||
(
|
||||
(ceph_osd_numpg > 0) - on (job) group_left avg(ceph_osd_numpg > 0) by (job)
|
||||
) / on (job) group_left avg(ceph_osd_numpg > 0) by (job)
|
||||
) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata > 0.30
|
||||
) * on (ceph_daemon) group_left(hostname) ceph_osd_metadata > 0.30
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
type: ceph_default
|
||||
oid: 1.3.6.1.4.1.50495.1.2.1.4.5
|
||||
annotations:
|
||||
summary: PG allocations are not balanced across devices
|
||||
summary: PGs are not balanced across OSDs
|
||||
description: >
|
||||
OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} deviates
|
||||
by more than 30% from average PG count.
|
||||
@@ -376,13 +387,11 @@ spec:
|
||||
oid: 1.3.6.1.4.1.50495.1.2.1.5.1
|
||||
annotations:
|
||||
documentation: https://docs.ceph.com/en/latest/cephfs/health-messages#cephfs-health-messages
|
||||
summary: Ceph filesystem is damaged.
|
||||
summary: CephFS filesystem is damaged.
|
||||
description: >
|
||||
The filesystems metadata has been corrupted. Data access
|
||||
may be blocked.
|
||||
|
||||
Either analyse the output from the mds daemon admin socket, or
|
||||
escalate to support
|
||||
Filesystem metadata has been corrupted. Data may be inaccessible.
|
||||
Analyze metrics from the MDS daemon admin socket, or
|
||||
escalate to support.
|
||||
- alert: CephFilesystemOffline
|
||||
expr: ceph_health_detail{name="MDS_ALL_DOWN"} > 0
|
||||
for: 1m
|
||||
@@ -392,10 +401,10 @@ spec:
|
||||
oid: 1.3.6.1.4.1.50495.1.2.1.5.3
|
||||
annotations:
|
||||
documentation: https://docs.ceph.com/en/latest/cephfs/health-messages/#mds-all-down
|
||||
summary: Ceph filesystem is offline
|
||||
summary: CephFS filesystem is offline
|
||||
description: >
|
||||
All MDS ranks are unavailable. The ceph daemons providing the metadata
|
||||
for the Ceph filesystem are all down, rendering the filesystem offline.
|
||||
All MDS ranks are unavailable. The MDS daemons managing metadata
|
||||
are down, rendering the filesystem offline.
|
||||
- alert: CephFilesystemDegraded
|
||||
expr: ceph_health_detail{name="FS_DEGRADED"} > 0
|
||||
for: 1m
|
||||
@@ -405,11 +414,11 @@ spec:
|
||||
oid: 1.3.6.1.4.1.50495.1.2.1.5.4
|
||||
annotations:
|
||||
documentation: https://docs.ceph.com/en/latest/cephfs/health-messages/#fs-degraded
|
||||
summary: Ceph filesystem is degraded
|
||||
summary: CephFS filesystem is degraded
|
||||
description: >
|
||||
One or more metadata daemons (MDS ranks) are failed or in a
|
||||
damaged state. At best the filesystem is partially available,
|
||||
worst case is the filesystem is completely unusable.
|
||||
at worst the filesystem is completely unusable.
|
||||
- alert: CephFilesystemMDSRanksLow
|
||||
expr: ceph_health_detail{name="MDS_UP_LESS_THAN_MAX"} > 0
|
||||
for: 1m
|
||||
@@ -418,11 +427,11 @@ spec:
|
||||
type: ceph_default
|
||||
annotations:
|
||||
documentation: https://docs.ceph.com/en/latest/cephfs/health-messages/#mds-up-less-than-max
|
||||
summary: Ceph MDS daemon count is lower than configured
|
||||
summary: MDS daemon count is lower than configured
|
||||
description: >
|
||||
The filesystem's "max_mds" setting defined the number of MDS ranks in
|
||||
The filesystem's "max_mds" setting defines the number of MDS ranks in
|
||||
the filesystem. The current number of active MDS daemons is less than
|
||||
this setting.
|
||||
this value.
|
||||
- alert: CephFilesystemInsufficientStandby
|
||||
expr: ceph_health_detail{name="MDS_INSUFFICIENT_STANDBY"} > 0
|
||||
for: 1m
|
||||
@@ -431,11 +440,11 @@ spec:
|
||||
type: ceph_default
|
||||
annotations:
|
||||
documentation: https://docs.ceph.com/en/latest/cephfs/health-messages/#mds-insufficient-standby
|
||||
summary: Ceph filesystem standby daemons too low
|
||||
summary: Ceph filesystem standby daemons too few
|
||||
description: >
|
||||
The minimum number of standby daemons determined by standby_count_wanted
|
||||
is less than the actual number of standby daemons. Adjust the standby count
|
||||
or increase the number of mds daemons within the filesystem.
|
||||
The minimum number of standby daemons required by standby_count_wanted
|
||||
is less than the current number of standby daemons. Adjust the standby count
|
||||
or increase the number of MDS daemons.
|
||||
- alert: CephFilesystemFailureNoStandby
|
||||
expr: ceph_health_detail{name="FS_WITH_FAILED_MDS"} > 0
|
||||
for: 1m
|
||||
@@ -445,11 +454,11 @@ spec:
|
||||
oid: 1.3.6.1.4.1.50495.1.2.1.5.5
|
||||
annotations:
|
||||
documentation: https://docs.ceph.com/en/latest/cephfs/health-messages/#fs-with-failed-mds
|
||||
summary: Ceph MDS daemon failed, no further standby available
|
||||
summary: MDS daemon failed, no further standby available
|
||||
description: >
|
||||
An MDS daemon has failed, leaving only one active rank without
|
||||
further standby. Investigate the cause of the failure or add a
|
||||
standby daemon
|
||||
An MDS daemon has failed, leaving only one active rank and no
|
||||
available standby. Investigate the cause of the failure or add a
|
||||
standby MDS.
|
||||
- alert: CephFilesystemReadOnly
|
||||
expr: ceph_health_detail{name="MDS_HEALTH_READ_ONLY"} > 0
|
||||
for: 1m
|
||||
@@ -459,13 +468,13 @@ spec:
|
||||
oid: 1.3.6.1.4.1.50495.1.2.1.5.2
|
||||
annotations:
|
||||
documentation: https://docs.ceph.com/en/latest/cephfs/health-messages#cephfs-health-messages
|
||||
summary: Ceph filesystem in read only mode, due to write error(s)
|
||||
summary: CephFS filesystem in read only mode due to write error(s)
|
||||
description: >
|
||||
The filesystem has switched to READ ONLY due to an unexpected
|
||||
write error, when writing to the metadata pool
|
||||
error when writing to the metadata pool.
|
||||
|
||||
Either analyse the output from the mds daemon admin socket, or
|
||||
escalate to support
|
||||
Analyze the output from the MDS daemon admin socket, or
|
||||
escalate to support.
|
||||
|
||||
- name: mgr
|
||||
rules:
|
||||
@@ -478,11 +487,11 @@ spec:
|
||||
oid: 1.3.6.1.4.1.50495.1.2.1.6.1
|
||||
annotations:
|
||||
documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#recent-mgr-module-crash
|
||||
summary: A mgr module has recently crashed
|
||||
summary: A manager module has recently crashed
|
||||
description: >
|
||||
One or more mgr modules have crashed and are yet to be acknowledged by the administrator. A
|
||||
crashed module may impact functionality within the cluster. Use the 'ceph crash' commands to
|
||||
investigate which module has failed, and archive it to acknowledge the failure.
|
||||
One or more mgr modules have crashed and have yet to be acknowledged by an administrator. A
|
||||
crashed module may impact functionality within the cluster. Use the 'ceph crash' command to
|
||||
determine which module has failed, and archive it to acknowledge the failure.
|
||||
- alert: CephMgrPrometheusModuleInactive
|
||||
expr: up{job="ceph"} == 0
|
||||
for: 1m
|
||||
@@ -491,16 +500,16 @@ spec:
|
||||
type: ceph_default
|
||||
oid: 1.3.6.1.4.1.50495.1.2.1.6.2
|
||||
annotations:
|
||||
summary: Ceph's mgr/prometheus module is not available
|
||||
summary: The mgr/prometheus module is not available
|
||||
description: >
|
||||
The mgr/prometheus module at {{ $labels.instance }} is unreachable. This
|
||||
could mean that the module has been disabled or the mgr itself is down.
|
||||
could mean that the module has been disabled or the mgr daemon itself is down.
|
||||
|
||||
Without the mgr/prometheus module metrics and alerts will no longer
|
||||
function. Open a shell to ceph and use 'ceph -s' to to determine whether the
|
||||
mgr is active. If the mgr is not active, restart it, otherwise you can check
|
||||
the mgr/prometheus module is loaded with 'ceph mgr module ls' and if it's
|
||||
not listed as enabled, enable it with 'ceph mgr module enable prometheus'
|
||||
function. Open a shell to an admin node or toolbox pod and use 'ceph -s' to to determine whether the
|
||||
mgr is active. If the mgr is not active, restart it, otherwise you can determine
|
||||
the mgr/prometheus module status with 'ceph mgr module ls'. If it is
|
||||
not listed as enabled, enable it with 'ceph mgr module enable prometheus'.
|
||||
|
||||
- name: pgs
|
||||
rules:
|
||||
@@ -512,11 +521,10 @@ spec:
|
||||
type: ceph_default
|
||||
oid: 1.3.6.1.4.1.50495.1.2.1.7.1
|
||||
annotations:
|
||||
summary: One or more Placement Groups are inactive
|
||||
summary: One or more placement groups are inactive
|
||||
description: >
|
||||
{{ $value }} PGs have been inactive for more than 5 minutes in pool {{ $labels.name }}.
|
||||
Inactive placement groups aren't able to serve read/write
|
||||
requests.
|
||||
Inactive placement groups are not able to serve read/write requests.
|
||||
- alert: CephPGsUnclean
|
||||
expr: ceph_pool_metadata * on(pool_id,instance) group_left() (ceph_pg_total - ceph_pg_clean) > 0
|
||||
for: 15m
|
||||
@@ -527,8 +535,8 @@ spec:
|
||||
annotations:
|
||||
summary: One or more placement groups are marked unclean
|
||||
description: >
|
||||
{{ $value }} PGs haven't been clean for more than 15 minutes in pool {{ $labels.name }}.
|
||||
Unclean PGs haven't been able to completely recover from a previous failure.
|
||||
{{ $value }} PGs have been unclean for more than 15 minutes in pool {{ $labels.name }}.
|
||||
Unclean PGs have not recovered from a previous failure.
|
||||
- alert: CephPGsDamaged
|
||||
expr: ceph_health_detail{name=~"PG_DAMAGED|OSD_SCRUB_ERRORS"} == 1
|
||||
for: 5m
|
||||
@@ -538,13 +546,12 @@ spec:
|
||||
oid: 1.3.6.1.4.1.50495.1.2.1.7.4
|
||||
annotations:
|
||||
documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#pg-damaged
|
||||
summary: Placement group damaged, manual intervention needed
|
||||
summary: Placement group damaged; manual intervention needed
|
||||
description: >
|
||||
During data consistency checks (scrub), at least one PG has been flagged as being
|
||||
damaged or inconsistent.
|
||||
Scrubs have flagged at least one PG as damaged or inconsistent.
|
||||
|
||||
Check to see which PG is affected, and attempt a manual repair if necessary. To list
|
||||
problematic placement groups, use 'rados list-inconsistent-pg <pool>'. To repair PGs use
|
||||
problematic placement groups, use 'ceph health detail' or 'rados list-inconsistent-pg <pool>'. To repair PGs use
|
||||
the 'ceph pg repair <pg_num>' command.
|
||||
- alert: CephPGRecoveryAtRisk
|
||||
expr: ceph_health_detail{name="PG_RECOVERY_FULL"} == 1
|
||||
@@ -555,10 +562,10 @@ spec:
|
||||
oid: 1.3.6.1.4.1.50495.1.2.1.7.5
|
||||
annotations:
|
||||
documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#pg-recovery-full
|
||||
summary: OSDs are too full for automatic recovery
|
||||
summary: OSDs are too full for recovery
|
||||
description: >
|
||||
Data redundancy may be reduced, or is at risk, since one or more OSDs are at or above their
|
||||
'full' threshold. Add more capacity to the cluster, or delete unwanted data.
|
||||
Data redundancy is at risk since one or more OSDs are at or above the
|
||||
'full' threshold. Add capacity to the cluster, restore down/out OSDs, or delete unwanted data.
|
||||
- alert: CephPGUnavailableBlockingIO
|
||||
# PG_AVAILABILITY, but an OSD is not in a DOWN state
|
||||
expr: ((ceph_health_detail{name="PG_AVAILABILITY"} == 1) - scalar(ceph_health_detail{name="OSD_DOWN"})) == 1
|
||||
@@ -569,10 +576,10 @@ spec:
|
||||
oid: 1.3.6.1.4.1.50495.1.2.1.7.3
|
||||
annotations:
|
||||
documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#pg-availability
|
||||
summary: Placement group is unavailable, blocking some I/O
|
||||
summary: PG is unavailable, blocking I/O
|
||||
description: >
|
||||
Data availability is reduced impacting the clusters ability to service I/O to some data. One or
|
||||
more placement groups (PGs) are in a state that blocks IO.
|
||||
Data availability is reduced, impacting the cluster's ability to service I/O. One or
|
||||
more placement groups (PGs) are in a state that blocks I/O.
|
||||
- alert: CephPGBackfillAtRisk
|
||||
expr: ceph_health_detail{name="PG_BACKFILL_FULL"} == 1
|
||||
for: 1m
|
||||
@@ -582,7 +589,7 @@ spec:
|
||||
oid: 1.3.6.1.4.1.50495.1.2.1.7.6
|
||||
annotations:
|
||||
documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#pg-backfill-full
|
||||
summary: Backfill operations are blocked, due to lack of freespace
|
||||
summary: Backfill operations are blocked due to lack of free space
|
||||
description: >
|
||||
Data redundancy may be at risk due to lack of free space within the cluster. One or more OSDs
|
||||
have breached their 'backfillfull' threshold. Add more capacity, or delete unwanted data.
|
||||
@@ -596,10 +603,10 @@ spec:
|
||||
documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#pg-not-scrubbed
|
||||
summary: Placement group(s) have not been scrubbed
|
||||
description: |
|
||||
One or more PGs have not been scrubbed recently. The scrub process is a data integrity
|
||||
feature, protecting against bit-rot. It checks that objects and their metadata (size and
|
||||
attributes) match across object replicas. When PGs miss their scrub window, it may
|
||||
indicate the scrub window is too small, or PGs were not in a 'clean' state during the
|
||||
One or more PGs have not been scrubbed recently. Scrubs check metadata integrity,
|
||||
protecting against bit-rot. They check that metadata
|
||||
is consistent across data replicas. When PGs miss their scrub interval, it may
|
||||
indicate that the scrub window is too small, or PGs were not in a 'clean' state during the
|
||||
scrub window.
|
||||
|
||||
You can manually initiate a scrub with: ceph pg scrub <pgid>
|
||||
@@ -615,10 +622,11 @@ spec:
|
||||
description: |
|
||||
The number of placement groups per OSD is too high (exceeds the mon_max_pg_per_osd setting).
|
||||
|
||||
Check that the pg_autoscaler hasn't been disabled for any of the pools, with 'ceph osd pool autoscale-status'
|
||||
Check that the pg_autoscaler has not been disabled for any pools with 'ceph osd pool autoscale-status',
|
||||
and that the profile selected is appropriate. You may also adjust the target_size_ratio of a pool to guide
|
||||
the autoscaler based on the expected relative size of the pool
|
||||
(i.e. 'ceph osd pool set cephfs.cephfs.meta target_size_ratio .1')
|
||||
('ceph osd pool set cephfs.cephfs.meta target_size_ratio .1') or set the pg_autoscaler
|
||||
mode to "warn" and adjust pg_num appropriately for one or more pools.
|
||||
- alert: CephPGNotDeepScrubbed
|
||||
expr: ceph_health_detail{name="PG_NOT_DEEP_SCRUBBED"} == 1
|
||||
for: 5m
|
||||
@@ -629,9 +637,9 @@ spec:
|
||||
documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#pg-not-deep-scrubbed
|
||||
summary: Placement group(s) have not been deep scrubbed
|
||||
description: |
|
||||
One or more PGs have not been deep scrubbed recently. Deep scrub is a data integrity
|
||||
feature, protecting against bit-rot. It compares the contents of objects and their
|
||||
replicas for inconsistency. When PGs miss their deep scrub window, it may indicate
|
||||
One or more PGs have not been deep scrubbed recently. Deep scrubs
|
||||
protect against bit-rot. They compare data
|
||||
replicas to ensure consistency. When PGs miss their deep scrub interval, it may indicate
|
||||
that the window is too small or PGs were not in a 'clean' state during the deep-scrub
|
||||
window.
|
||||
|
||||
@@ -649,9 +657,9 @@ spec:
|
||||
annotations:
|
||||
summary: Root filesystem is dangerously full
|
||||
description: >
|
||||
Root volume (OSD and MON store) is dangerously full: {{ $value | humanize }}% free.
|
||||
Root volume is dangerously full: {{ $value | humanize }}% free.
|
||||
|
||||
# alert on nic packet errors and drops rates > 1% packets/s
|
||||
# alert on packet errors and drop rate
|
||||
- alert: CephNodeNetworkPacketDrops
|
||||
expr: |
|
||||
(
|
||||
@@ -669,7 +677,7 @@ spec:
|
||||
type: ceph_default
|
||||
oid: 1.3.6.1.4.1.50495.1.2.1.8.2
|
||||
annotations:
|
||||
summary: One or more Nics is seeing packet drops
|
||||
summary: One or more NICs reports packet drops
|
||||
description: >
|
||||
Node {{ $labels.instance }} experiences packet drop > 0.01% or >
|
||||
10 packets/s on interface {{ $labels.device }}.
|
||||
@@ -691,7 +699,7 @@ spec:
|
||||
type: ceph_default
|
||||
oid: 1.3.6.1.4.1.50495.1.2.1.8.3
|
||||
annotations:
|
||||
summary: One or more Nics is seeing packet errors
|
||||
summary: One or more NICs reports packet errors
|
||||
description: >
|
||||
Node {{ $labels.instance }} experiences packet errors > 0.01% or
|
||||
> 10 packets/s on interface {{ $labels.device }}.
|
||||
@@ -707,37 +715,48 @@ spec:
|
||||
type: ceph_default
|
||||
oid: 1.3.6.1.4.1.50495.1.2.1.8.4
|
||||
annotations:
|
||||
summary: Host filesystem freespace is getting low
|
||||
summary: Host filesystem free space is low
|
||||
description: >
|
||||
Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }}
|
||||
will be full in less than 5 days assuming the average fill-up
|
||||
rate of the past 48 hours.
|
||||
|
||||
will be full in less than 5 days based on the 48 hour trailing
|
||||
fill rate.
|
||||
- alert: CephNodeInconsistentMTU
|
||||
expr: node_network_mtu_bytes{device!="lo"} * (node_network_up{device!="lo"} > 0) != on() group_left() (quantile(0.5, node_network_mtu_bytes{device!="lo"}))
|
||||
labels:
|
||||
severity: warning
|
||||
type: ceph_default
|
||||
annotations:
|
||||
summary: MTU settings across Ceph hosts are inconsistent
|
||||
summary: MTU settings across hosts are inconsistent
|
||||
description: >
|
||||
Node {{ $labels.instance }} has a different MTU size ({{ $value }})
|
||||
than the median value on device {{ $labels.device }}.
|
||||
|
||||
- name: pools
|
||||
rules:
|
||||
- alert: CephPoolBackfillFull
|
||||
expr: ceph_health_detail{name="POOL_BACKFILLFULL"} > 0
|
||||
- alert: CephPoolGrowthWarning
|
||||
expr: |
|
||||
(predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id)
|
||||
group_right ceph_pool_metadata) >= 95
|
||||
labels:
|
||||
severity: warning
|
||||
type: ceph_default
|
||||
oid: 1.3.6.1.4.1.50495.1.2.1.9.2
|
||||
annotations:
|
||||
summary: Freespace in a pool is too low for recovery/rebalance
|
||||
summary: Pool growth rate may soon exceed capacity
|
||||
description: >
|
||||
A pool is approaching it's near full threshold, which will
|
||||
prevent rebalance operations from completing. You should
|
||||
consider adding more capacity to the pool.
|
||||
|
||||
Pool '{{ $labels.name }}' will be full in less than 5 days
|
||||
assuming the average fill-up rate of the past 48 hours.
|
||||
- alert: CephPoolBackfillFull
|
||||
expr: ceph_health_detail{name="POOL_BACKFILLFULL"} > 0
|
||||
labels:
|
||||
severity: warning
|
||||
type: ceph_default
|
||||
annotations:
|
||||
summary: Free space in a pool is too low for recovery/backfill
|
||||
description: >
|
||||
A pool is approaching the near full threshold, which will
|
||||
prevent recovery/backfill from completing.
|
||||
Consider adding more capacity.
|
||||
- alert: CephPoolFull
|
||||
expr: ceph_health_detail{name="POOL_FULL"} > 0
|
||||
for: 1m
|
||||
@@ -749,15 +768,15 @@ spec:
|
||||
documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#pool-full
|
||||
summary: Pool is full - writes are blocked
|
||||
description: |
|
||||
A pool has reached it's MAX quota, or the OSDs supporting the pool
|
||||
have reached their FULL threshold. Until this is resolved, writes to
|
||||
A pool has reached its MAX quota, or OSDs supporting the pool
|
||||
have reached the FULL threshold. Until this is resolved, writes to
|
||||
the pool will be blocked.
|
||||
Pool Breakdown (top 5)
|
||||
{{- range query "topk(5, sort_desc(ceph_pool_percent_used * on(pool_id) group_right ceph_pool_metadata))" }}
|
||||
- {{ .Labels.name }} at {{ .Value }}%
|
||||
{{- end }}
|
||||
Either increase the pools quota, or add capacity to the cluster first
|
||||
then increase it's quota (e.g. ceph osd pool set quota <pool_name> max_bytes <bytes>)
|
||||
Increase the pool's quota, or add capacity to the cluster
|
||||
then increase the pool's quota (e.g. ceph osd pool set quota <pool_name> max_bytes <bytes>)
|
||||
- alert: CephPoolNearFull
|
||||
expr: ceph_health_detail{name="POOL_NEAR_FULL"} > 0
|
||||
for: 5m
|
||||
@@ -765,17 +784,18 @@ spec:
|
||||
severity: warning
|
||||
type: ceph_default
|
||||
annotations:
|
||||
summary: One or more Ceph pools are getting full
|
||||
summary: One or more Ceph pools are nearly full
|
||||
description: |
|
||||
A pool has exceeded it warning (percent full) threshold, or the OSDs
|
||||
supporting the pool have reached their NEARFULL thresholds. Writes may
|
||||
continue, but you are at risk of the pool going read only if more capacity
|
||||
A pool has exceeded the warning (percent full) threshold, or OSDs
|
||||
supporting the pool have reached the NEARFULL threshold. Writes may
|
||||
continue, but you are at risk of the pool going read-only if more capacity
|
||||
isn't made available.
|
||||
|
||||
Determine the affected pool with 'ceph df detail', for example looking
|
||||
at QUOTA BYTES and STORED. Either increase the pools quota, or add
|
||||
capacity to the cluster first then increase it's quota
|
||||
(e.g. ceph osd pool set quota <pool_name> max_bytes <bytes>)
|
||||
Determine the affected pool with 'ceph df detail', looking
|
||||
at QUOTA BYTES and STORED. Increase the pool's quota, or add
|
||||
capacity to the cluster then increase the pool's quota
|
||||
(e.g. ceph osd pool set quota <pool_name> max_bytes <bytes>).
|
||||
Also ensure that the balancer is active.
|
||||
- name: healthchecks
|
||||
rules:
|
||||
- alert: CephSlowOps
|
||||
@@ -786,7 +806,7 @@ spec:
|
||||
type: ceph_default
|
||||
annotations:
|
||||
documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#slow-ops
|
||||
summary: MON/OSD operations are slow to complete
|
||||
summary: OSD operations are slow to complete
|
||||
description: >
|
||||
{{ $value }} OSD requests are taking too long to process (osd_op_complaint_time exceeded)
|
||||
|
||||
@@ -802,9 +822,9 @@ spec:
|
||||
oid: 1.3.6.1.4.1.50495.1.2.1.10.1
|
||||
annotations:
|
||||
documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#object-unfound
|
||||
summary: Object(s) has been marked UNFOUND
|
||||
summary: Object(s) marked UNFOUND
|
||||
description: |
|
||||
A version of a RADOS object can not be found, even though all OSDs are up. I/O
|
||||
The latest version of a RADOS object can not be found, even though all OSDs are up. I/O
|
||||
requests for this object from clients will block (hang). Resolving this issue may
|
||||
require the object to be rolled back to a prior version manually, and manually verified.
|
||||
# Generic
|
||||
@@ -822,5 +842,5 @@ spec:
|
||||
summary: One or more Ceph daemons have crashed, and are pending acknowledgement
|
||||
description: |
|
||||
One or more daemons have crashed recently, and need to be acknowledged. This notification
|
||||
ensures that software crashes don't go unseen. To acknowledge a crash, use the
|
||||
ensures that software crashes do not go unseen. To acknowledge a crash, use the
|
||||
'ceph crash archive <id>' command.
|
||||
Reference in new issue
Block a user