diff --git a/Documentation/ceph-advanced-configuration.md b/Documentation/ceph-advanced-configuration.md index f4a9f2fc8..40a371960 100644 --- a/Documentation/ceph-advanced-configuration.md +++ b/Documentation/ceph-advanced-configuration.md @@ -420,16 +420,37 @@ ceph osd pool set rbd pg_num 512 ## Custom ceph.conf Settings **WARNING:** The advised method for controlling Ceph configuration is to manually use the Ceph CLI -or the Ceph dashboard because this offers the most flexibility. If configs must be set as part of -Rook, the `configOverrides` section of the CephCluster resource is recommended. Read the -[CephCluster CRD docs](ceph-cluster-crd.md#ceph-config-overrides) first before considering this -advanced method. +or the Ceph dashboard because this offers the most flexibility. It is highly recommended that this +only be used when absolutely necessary and that the `config` be reset to an empty string if/when the +configurations are no longer necessary. Configurations in the config file will make the Ceph cluster +less configurable from the CLI and dashboard and may make future tuning or debugging difficult. -With Rook the full swath of -[Ceph settings](http://docs.ceph.com/docs/master/rados/configuration/) are available -to use on your storage cluster. When we supply Rook with a ceph.conf file those -settings will be propagated to all Mon, OSD, MDS, and RGW daemons to use. +Setting configs via Ceph's CLI requires that at least one mon be available for the configs to be +set, and setting configs via dashboard requires at least one mgr to be available. Ceph may also have +a small number of very advanced settings that aren't able to be modified easily via CLI or +dashboard. In order to set configurations before monitors are available or to set problematic +configuration settings, the `rook-config-override` ConfigMap exists, and the `config` field can be +set with the contents of a `ceph.conf` file. The contents will be propagated to all mon, mgr, OSD, +MDS, and RGW daemons as an `/etc/ceph/ceph.conf` file. +**WARNING:** Rook performs no validation on the config, so the validity of the settings is the +user's responsibility. + +Each daemon will need to be restarted where you want the settings applied: + +- mons: ensure all three mons are online and healthy before restarting each mon pod, one at a time. +- mgrs: the pods are stateless and can be restarted as needed, but note that this will disrupt the + Ceph dashboard during restart. +- OSDs: restart your the pods by deleting them, one at a time, and running `ceph -s` +between each restart to ensure the cluster goes back to "active/clean" state. +- RGW: the pods are stateless and can be restarted as needed. +- MDS: the pods are stateless and can be restarted as needed. + +After the pod restart, the new settings should be in effect. Note that if the ConfigMap in the Ceph +cluster's namespace is created before the cluster is created, the daemons will pick up the settings +at first launch. + +### Example In this example we will set the default pool `size` to two, and tell OSD daemons not to change the weight of OSDs on startup. @@ -455,9 +476,8 @@ data: config: "" ``` -To apply your desired configuration, you will need to update this ConfigMap. -The next time the daemon pod(s) start, the settings will be merged with the default -settings created by Rook. +To apply your desired configuration, you will need to update this ConfigMap. The next time the +daemon pod(s) start, they will use the updated configs. ```bash kubectl -n rook-ceph edit configmap rook-config-override @@ -478,22 +498,6 @@ data: osd pool default size = 2 ``` -Each daemon will need to be restarted where you want the settings applied: - -- Mons: ensure all three mons are online and healthy before restarting each mon pod, one at a time -- OSDs: restart your the pods by deleting them, one at a time, and running `ceph -s` -between each restart to ensure the cluster goes back to "active/clean" state. -- RGW: the pods are stateless and can be restarted as needed -- MDS: the pods are stateless and can be restarted as needed - -After the pod restart, your new settings should be in effect. Note that if you create -the ConfigMap in the `rook-ceph` namespace before the cluster is even created -the daemons will pick up the settings at first launch. - -The only validation of the settings done by Rook is whether the settings can be merged -using the ini file format with the default settings created by Rook. Beyond that, -the validity of the settings is your responsibility. - ## OSD CRUSH Settings A useful view of the [CRUSH Map](http://docs.ceph.com/docs/master/rados/operations/crush-map/) diff --git a/Documentation/ceph-cluster-crd.md b/Documentation/ceph-cluster-crd.md index 18e10d72b..d3f307325 100755 --- a/Documentation/ceph-cluster-crd.md +++ b/Documentation/ceph-cluster-crd.md @@ -289,62 +289,6 @@ For more information on resource requests/limits see the official Kubernetes doc - `cpu`: Limit for CPU (example: one CPU core `1`, 50% of one CPU core `500m`). - `memory`: Limit for Memory (example: one gigabyte of memory `1Gi`, half a gigabyte of memory `512Mi`). -### Ceph config overrides -Users are advised to use Ceph's CLI or dashboard to configure Ceph; however, some users may want to -specify some settings they want set in the `CephCluster` manifest. This is possible using the -`configOverrides` section. Config overrides follow the same syntax as [Ceph's documentation for -setting config values](https://docs.ceph.com/docs/master/rados/configuration/ceph-conf/#commands), -and configuration is stored in Ceph's centralized configuration database. - -Each item in `configOverrides` has `who`, `option`, and `value` properties. `who` can be `global` to -affect all Ceph daemons, a daemon type, an individual Ceph daemon, or a glob matching multiple -daemons. It may also use a -[mask](https://docs.ceph.com/docs/master/rados/configuration/ceph-conf/#sections-and-masks). -`option` is the configuration option to be overridden, and `value` is the value to set on the -configuration option. All properties are strings. - -Rook will only set configuration overrides once when an orchestration is run. For example, on node -addition or removal, when disks change on a node, or when the operator starts (or restarts). This -means that users can change Ceph's configuration from the CLI or dashboard as desired without Rook -constantly fighting to control the setting. This is particularly valuable in situations where values -need to be changed to debug errors or test changes to tuning parameters. - -**WARNING:** Remember that Rook will set these overrides any time the operator restarts, so user -changes via Ceph's CLI or dashboard to values set here will not persist forever. - -Some examples: -```yaml - configOverrides: - - who: global - option: log_file - value: /var/log/custom-log-dir - - who: mon - option: mon_compact_on_start - value: "true" - - who: mgr.a - option: mgr_stats_period - value: "10" - - who: osd.* - option: osd_memory_target - value: "2147483648" - - who: osd/rack:foo - option: debug_ms - value: "20" -``` - -#### Advanced config -Setting configs in the Ceph mons' centralized database this way requires that at least one mon be -available for the configs to be set. Ceph may also have a small number of very advanced settings -that aren't able to be modified easily in the configuration database. In order to set configurations -before monitors are available or to set problematic configuration settings, the -`rook-config-override` ConfigMap exists, and the `config` field can be set with the contents of a -`ceph.conf` file. It is highly recommended that this only be used when absolutely necessary and that -the `config` be reset to an empty string if/when the configurations are no longer necessary, as this -will make the Ceph cluster less configurable from the CLI and dashboard and may make future tuning -or debugging difficult. Read more about this in the -[advanced configuration docs](ceph-advanced-configuration.md#custom-cephconf-settings). - - ## Samples Here are several samples for configuring Ceph clusters. Each of the samples must also include the namespace and corresponding access granted for management by the Ceph operator. See the [common cluster resources](#common-cluster-resources) below. diff --git a/Documentation/ceph-configuration.md b/Documentation/ceph-configuration.md new file mode 100644 index 000000000..5840f1ac0 --- /dev/null +++ b/Documentation/ceph-configuration.md @@ -0,0 +1,68 @@ +--- +title: Configuration +weight: 3700 +indent: true +--- + +# Configuration + +For most any Ceph cluster, the user will want to--and may need to--change some Ceph +configurations. These changes often may be warranted in order to alter performance to meet SLAs or +to update default data resiliency settings. + +**WARNING:** Modify Ceph settings carefully, and review the +[Ceph configuration documentation](https://docs.ceph.com/docs/master/rados/configuration/) before +making any changes. Changing the settings could result in unhealthy daemons or even data loss if +used incorrectly. + + +## Required configurations +Rook and Ceph both strive to make configuration as easy as possible, but there are some +configuration options which users are well advised to consider for any production cluster. + +### Default PG and PGP counts +`osd_pool_default_pg_num` and `osd_pool_default_pgp_num`. The number of PGs and PGPs can be +configured on a per-pool basis, but it is highly advised to set default values that are appropriate +for your Ceph cluster. Appropriate values depend on the number of OSDs the user expects to have +backing each pool. The Ceph +[OSD and Pool config docs](https://docs.ceph.com/docs/master/rados/operations/placement-groups/#a-preselection-of-pg-num) +provide detailed information about how to tune these parameters. + +An easier option exists for Rook-Ceph clusters running Ceph Nautilus (v14.2.x) or newer. Nautilus +[introduced the PG auto-scaler mgr module](https://ceph.com/rados/new-in-nautilus-pg-merging-and-autotuning/) +capable of automatically managing PG and PGP values for pools. This module is not enabled by default +but can be enabled by running the following from the Ceph toolbox pod (more on the toolbox pod and +Ceph CLI later in this document). Using this module, the user does not need to configure +`osd_pool_default_pg_num` or `osd_pool_default_pgp_num`. +``` +ceph mgr module enable pg_autoscaler +``` + +The user must also decide whether they want the autoscaler to be used for all new pools by default. +Turning this on is recommended. The Ceph CLI command for turning this on is shown below, but this +can be configured via any of the configuration methods detailed later in this document. +``` +ceph config set global osd_pool_default_autoscale_mode on +``` + + +## Specifying configuration options + +### Toolbox + Ceph CLI +The most recommended way of configuring Ceph is to set Ceph's configuration directly. The first +method for doing so is to use Ceph's CLI from the Rook-Ceph toolbox pod. Using the toolbox pod is +detailed [here](ceph-toolbox.md). From the toolbox, the user can change Ceph configurations, enable +manager modules, create users and pools, and much more. + +### Ceph Dashboard +The Ceph Dashboard, examined in more detail [here](ceph-dashboard.md), is another way of setting +some of Ceph's configuration directly. Configuration by the Ceph dashboard is recommended with the +same priority as configuration via the Ceph CLI (above). + +### Advanced configuration via ceph.conf override ConfigMap +Setting configs via Ceph's CLI requires that at least one mon be available for the configs to be +set, and setting configs via dashboard requires at least one mgr to be available. Ceph may also have +a small number of very advanced settings that aren't able to be modified easily via CLI or +dashboard. The **least** recommended method for configuring Ceph is intended as a last-resort +fallback in situations like these. This is covered in detail +[here](ceph-advanced-configuration.md#custom-cephconf-settings). diff --git a/Documentation/ceph-upgrade.md b/Documentation/ceph-upgrade.md index 3664cc4fd..779c76473 100644 --- a/Documentation/ceph-upgrade.md +++ b/Documentation/ceph-upgrade.md @@ -277,54 +277,51 @@ At this point, your Rook operator should be running version `rook/ceph:v1.1.0` Verify the Ceph cluster's health using the [health verification section](#health-verification). -### 6. (Optional) Migrate config overrides from ConfigMap to the CephCluster resource +### 6. (Recommended) Migrate config overrides from ConfigMap to Ceph directly If there are Ceph configuration overrides set in the `config` field of the ConfigMap `rook-config-override`, it is now possible to migrate those configs manually from the ConfigMap to -the newly-added `configOverrides` section of the `CephCluster` custom resource. This is not required -but may be preferable, as the values set in `configOverrides` can be temporarily overridden by the -user from the toolbox pod as needed in debug/failure scenarios. Values are reset if the operator -restarts. - -Ceph config file headers correspond to -the `who` field of each config override as part of the section. For example, the `[global]` section -corresponds to `who: global`, and `[osd.0]` corresponds to `who: osd.0`. +Ceph directly as documented [here](ceph-configuration.md#specifying-configuration-options). This is +not required but is recommended because the values configured in Ceph directly can be temporarily +overridden by the user as needed in debug/failure scenarios. +#### Example +List the contents of the override ConfigMap. ```sh -# List the contents of the override ConfigMap -$> kubectl --namespace $ROOK_NAMESPACE describe configmap rook-ceph-override -o yaml -Name: rook-config-override -Namespace: rook-ceph -Labels: -Annotations: - -Data -==== -config: ----- -[global] -debug_ms = 1/5 - -[osd.0] -debug_osd = 10 -Events: - -# Edit the Ceph cluster resource, and add the overrides from the ConfigMap -$> kubectl --namespace $ROOK_NAMESPACE edit cephcluster $ROOK_NAMESPACE -# ... -spec: - # ... - configOverrides: - - who: global - option: debug_ms - value: 1/5 - - who: osd.0 - option: debug_osd - value: "10" - who: osd.0 -# .... +kubectl --namespace $ROOK_NAMESPACE describe configmap rook-config-override +# Name: rook-config-override +# Namespace: rook-ceph +# Labels: +# Annotations: +# +# Data +# ==== +# config: +# ---- +# [global] +# debug_ms = 1/5 +# +# [osd.0] +# debug_osd = 10 +# Events: ``` -### 7. Update Rook-Ceph custom resource definitions +Apply the configurations to Ceph directly using Ceph's CLI. +```sh +TOOLS_POD=$(kubectl -n $ROOK_NAMESPACE get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') +kubectl -n $ROOK_NAMESPACE exec -it $TOOLS_POD -- ceph config set global debug_ms 1/5 +kubectl -n $ROOK_NAMESPACE exec -it $TOOLS_POD -- ceph config set osd.0 debug_osd 10 +``` + +Empty the config override in the ConfigMap. +```sh +kubectl --namespace $ROOK_NAMESPACE patch configmap rook-config-override --type=merge -p '{"data": {"config": ""}}' +``` + +### 7. (Recommended) Consider required Ceph config settings +If the user determines that the [advised configuration options](ceph-configuration.md#default-pg-and-pgp-counts) +newly identified in Rook's 1.1 release apply to them, now is the time to set these configs. + +### 8. Update Rook-Ceph custom resource definitions **IMPORTANT: Do not perform this step until ALL existing Rook-Ceph clusters are updated** After all Rook-Ceph clusters have been updated following the steps above, update the Rook-Ceph @@ -369,8 +366,7 @@ Ceph image field in the cluster CRD (`spec:cephVersion:image`). ```sh NEW_CEPH_IMAGE='ceph/ceph:v14.2.2-20190830' CLUSTER_NAME="$ROOK_NAMESPACE" # change if your cluster name is not the Rook namespace -kubectl -n $ROOK_NAMESPACE patch CephCluster $CLUSTER_NAME --type=merge \ - -p "{\"spec\": {\"cephVersion\": {\"image\": \"$NEW_CEPH_IMAGE\"}}}" +kubectl -n $ROOK_NAMESPACE patch CephCluster $CLUSTER_NAME --type=merge -p "{\"spec\": {\"cephVersion\": {\"image\": \"$NEW_CEPH_IMAGE\"}}}" ``` ### 2. Wait for the daemon pod updates to complete diff --git a/PendingReleaseNotes.md b/PendingReleaseNotes.md index 54c15932b..4571cca8c 100644 --- a/PendingReleaseNotes.md +++ b/PendingReleaseNotes.md @@ -51,8 +51,6 @@ an example usage - Added a new property in `storageClassDeviceSets` named `portable`: - If `true`, the OSDs will be allowed to move between nodes during failover. This requires a storage class that supports portability (e.g. `aws-ebs`, but not the local storage provisioner). - If `false`, the OSDs will be assigned to a node permanently. Rook will configure Ceph's CRUSH map to support the portability. -- The Ceph cluster custom resource now contains a `configOverrides` section where users can specify - configuration changes to Ceph which Rook should apply. - Rook can now manage PodDisruptionBudgets for the following Daemons: OSD, Mon, RGW, MDS. OSD budgets are dynamically managed as documented in the [design](https://github.com/rook/rook/blob/master/design/ceph-managed-disruptionbudgets.md). This can be enabled with the `managePodBudgets` flag in the cluster CR. When this is enabled, drains on OSDs will be blocked by default and dynamically unblocked in a safe manner one failureDomain at a time. When a failure domain is draining, it will be marked as no out for a longer time than the default DOWN/OUT interval. ### YugabyteDB diff --git a/cluster/examples/kubernetes/ceph/cluster.yaml b/cluster/examples/kubernetes/ceph/cluster.yaml index 7b113d0f5..1e7b1581d 100644 --- a/cluster/examples/kubernetes/ceph/cluster.yaml +++ b/cluster/examples/kubernetes/ceph/cluster.yaml @@ -143,18 +143,6 @@ spec: # storeType: filestore # - name: "172.17.4.301" # deviceFilter: "^sd." - # Override Ceph configs. These can be overridden by the user at runtime using the Ceph CLI, and - # Rook will set them back to the values set here when the operator restarts. - configOverrides: - - who: global - option: debug ms - value: "1/5" - - who: osd.0 - option: debug_osd - value: "10" - - who: mon.a - option: debug-mon - value: "10" disruptionManagement: #The section for configuring management of daemon disruptions managePodBudgets: false #if `true`, the operator will create and manage PodDsruptionBudgets for OSD, Mon, RGW, and MDS daemons. OSD PDBs are managed dynamically via the strategy outlined in the [design](https://github.com/rook/rook/blob/master/design/ceph-managed-disruptionbudgets.md). The operator will block eviction of OSDs by default and unblock them safely when drains are detected. osdMaintenanceTimeout: 30 # is a duration in minutes that determines how long an entire failureDomain like `region/zone/host` will be held in `noout` (in addition to the default DOWN/OUT interval) when it is draining. This is only relevant when `managePodBudgets` is `true`. The default value is `30` minutes. diff --git a/cluster/examples/kubernetes/ceph/common.yaml b/cluster/examples/kubernetes/ceph/common.yaml index 26036a1cf..01010e91a 100644 --- a/cluster/examples/kubernetes/ceph/common.yaml +++ b/cluster/examples/kubernetes/ceph/common.yaml @@ -141,15 +141,6 @@ spec: type: integer placement: {} resources: {} - configOverrides: - items: - properties: - who: - type: string - option: - type: string - value: - type: string additionalPrinterColumns: - name: DataDirHostPath type: string diff --git a/cluster/examples/kubernetes/ceph/upgrade-from-v1.0-crds.yaml b/cluster/examples/kubernetes/ceph/upgrade-from-v1.0-crds.yaml index 38e61c8d5..5ca1fecae 100644 --- a/cluster/examples/kubernetes/ceph/upgrade-from-v1.0-crds.yaml +++ b/cluster/examples/kubernetes/ceph/upgrade-from-v1.0-crds.yaml @@ -124,15 +124,6 @@ spec: type: integer placement: {} resources: {} - configOverrides: - items: - properties: - who: - type: string - option: - type: string - value: - type: string additionalPrinterColumns: - name: DataDirHostPath type: string diff --git a/pkg/apis/ceph.rook.io/v1/types.go b/pkg/apis/ceph.rook.io/v1/types.go index aef413cd6..90b562656 100755 --- a/pkg/apis/ceph.rook.io/v1/types.go +++ b/pkg/apis/ceph.rook.io/v1/types.go @@ -71,9 +71,6 @@ type ClusterSpec struct { // The path on the host where config and data can be persisted. DataDirHostPath string `json:"dataDirHostPath,omitempty"` - // Ceph config overrides to apply. - ConfigOverrides ConfigOverridesSpec `json:"configOverrides,omitempty"` - // A spec for configuring disruption management. DisruptionManagement DisruptionManagementSpec `json:"disruptionManagement,omitempty"` @@ -158,18 +155,6 @@ const ( DefaultFailureDomain = "host" ) -// ConfigOverridesSpec defines how Ceph configurations can be overridden by Rook. -type ConfigOverridesSpec []ConfigOverride - -// ConfigOverride defines the syntax for overriding a Ceph config. This translates to a -// `ceph config set