diff --git a/Documentation/ceph-advanced-configuration.md b/Documentation/ceph-advanced-configuration.md index 5712872da..f4a9f2fc8 100644 --- a/Documentation/ceph-advanced-configuration.md +++ b/Documentation/ceph-advanced-configuration.md @@ -419,9 +419,14 @@ ceph osd pool set rbd pg_num 512 ``` ## Custom ceph.conf Settings +**WARNING:** The advised method for controlling Ceph configuration is to manually use the Ceph CLI +or the Ceph dashboard because this offers the most flexibility. If configs must be set as part of +Rook, the `configOverrides` section of the CephCluster resource is recommended. Read the +[CephCluster CRD docs](ceph-cluster-crd.md#ceph-config-overrides) first before considering this +advanced method. With Rook the full swath of -[Ceph settings](http://docs.ceph.com/docs/kraken/rados/configuration/) are available +[Ceph settings](http://docs.ceph.com/docs/master/rados/configuration/) are available to use on your storage cluster. When we supply Rook with a ceph.conf file those settings will be propagated to all Mon, OSD, MDS, and RGW daemons to use. @@ -431,7 +436,6 @@ daemons not to change the weight of OSDs on startup. **WARNING**: Modify Ceph settings carefully. You are leaving the sandbox tested by Rook. Changing the settings could result in unhealthy daemons or even data loss if used incorrectly. -### Kubernetes When the Rook Operator creates a cluster, a placeholder ConfigMap is created that will allow you to override Ceph configuration settings. When the daemon pods are started, the settings specified in this ConfigMap will be merged with the default settings @@ -492,7 +496,7 @@ the validity of the settings is your responsibility. ## OSD CRUSH Settings -A useful view of the [CRUSH Map](http://docs.ceph.com/docs/kraken/rados/operations/crush-map/) +A useful view of the [CRUSH Map](http://docs.ceph.com/docs/master/rados/operations/crush-map/) is generated with the following command: ```bash diff --git a/Documentation/ceph-cluster-crd.md b/Documentation/ceph-cluster-crd.md index 51d2920bd..f9a09ba67 100755 --- a/Documentation/ceph-cluster-crd.md +++ b/Documentation/ceph-cluster-crd.md @@ -286,6 +286,62 @@ For more information on resource requests/limits see the official Kubernetes doc - `cpu`: Limit for CPU (example: one CPU core `1`, 50% of one CPU core `500m`). - `memory`: Limit for Memory (example: one gigabyte of memory `1Gi`, half a gigabyte of memory `512Mi`). +### Ceph config overrides +Users are advised to use Ceph's CLI or dashboard to configure Ceph; however, some users may want to +specify some settings they want set in the `CephCluster` manifest. This is possible using the +`configOverrides` section. Config overrides follow the same syntax as [Ceph's documentation for +setting config values](https://docs.ceph.com/docs/master/rados/configuration/ceph-conf/#commands), +and configuration is stored in Ceph's centralized configuration database. + +Each item in `configOverrides` has `who`, `option`, and `value` properties. `who` can be `global` to +affect all Ceph daemons, a daemon type, an individual Ceph daemon, or a glob matching multiple +daemons. It may also use a +[mask](https://docs.ceph.com/docs/master/rados/configuration/ceph-conf/#sections-and-masks). +`option` is the configuration option to be overridden, and `value` is the value to set on the +configuration option. All properties are strings. + +Rook will only set configuration overrides once when an orchestration is run. For example, on node +addition or removal, when disks change on a node, or when the operator starts (or restarts). This +means that users can change Ceph's configuration from the CLI or dashboard as desired without Rook +constantly fighting to control the setting. This is particularly valuable in situations where values +need to be changed to debug errors or test changes to tuning parameters. + +**WARNING:** Remember that Rook will set these overrides any time the operator restarts, so user +changes via Ceph's CLI or dashboard to values set here will not persist forever. + +Some examples: +```yaml + configOverrides: + - who: global + option: log_file + value: /var/log/custom-log-dir + - who: mon + option: mon_compact_on_start + value: "true" + - who: mgr.a + option: mgr_stats_period + value: "10" + - who: osd.* + option: osd_memory_target + value: "2147483648" + - who: osd/rack:foo + option: debug_ms + value: "20" +``` + +#### Advanced config +Setting configs in the Ceph mons' centralized database this way requires that at least one mon be +available for the configs to be set. Ceph may also have a small number of very advanced settings +that aren't able to be modified easily in the configuration database. In order to set configurations +before monitors are available or to set problematic configuration settings, the +`rook-config-override` ConfigMap exists, and the `config` field can be set with the contents of a +`ceph.conf` file. It is highly recommended that this only be used when absolutely necessary and that +the `config` be reset to an empty string if/when the configurations are no longer necessary, as this +will make the Ceph cluster less configurable from the CLI and dashboard and may make future tuning +or debugging difficult. Read more about this in the +[advanced configuration docs](ceph-advanced-configuration.md#custom-cephconf-settings). + + ## Samples Here are several samples for configuring Ceph clusters. Each of the samples must also include the namespace and corresponding access granted for management by the Ceph operator. See the [common cluster resources](#common-cluster-resources) below. diff --git a/Documentation/ceph-upgrade.md b/Documentation/ceph-upgrade.md index 5b514fdfe..c21887808 100644 --- a/Documentation/ceph-upgrade.md +++ b/Documentation/ceph-upgrade.md @@ -272,6 +272,52 @@ At this point, your Rook operator should be running version `rook/ceph:v1.1.0` Verify the Ceph cluster's health using the [health verification section](#health-verification). +### 6. (Optional) Migrate config overrides from ConfigMap to the CephCluster resource +If there are Ceph configuration overrides set in the `config` field of the ConfigMap +`rook-config-override`, it is now possible to migrate those configs manually from the ConfigMap to +the newly-added `configOverrides` section of the `CephCluster` custom resource. This is not required +but may be preferable, as the values set in `configOverrides` can be temporarily overridden by the +user from the toolbox pod as needed in debug/failure scenarios. Values are reset if the operator +restarts. + +Ceph config file headers correspond to +the `who` field of each config override as part of the section. For example, the `[global]` section +corresponds to `who: global`, and `[osd.0]` corresponds to `who: osd.0`. + +```sh +# List the contents of the override ConfigMap +$> kubectl --namespace $ROOK_NAMESPACE describe configmap rook-ceph-override -o yaml +Name: rook-config-override +Namespace: rook-ceph +Labels: +Annotations: + +Data +==== +config: +---- +[global] +debug_ms = 1/5 + +[osd.0] +debug_osd = 10 +Events: + +# Edit the Ceph cluster resource, and add the overrides from the ConfigMap +$> kubectl --namespace $ROOK_NAMESPACE edit cephcluster $ROOK_NAMESPACE +# ... +spec: + # ... + configOverrides: + - who: global + option: debug_ms + value: 1/5 + - who: osd.0 + option: debug_osd + value: "10" + who: osd.0 +# .... +``` # Ceph Version Upgrades Rook 1.0 was the last Rook release which will support Ceph's Luminous (v12.x.x) version. Users are diff --git a/PendingReleaseNotes.md b/PendingReleaseNotes.md index 11a0c72f3..8b8ff3288 100644 --- a/PendingReleaseNotes.md +++ b/PendingReleaseNotes.md @@ -46,9 +46,11 @@ an example usage [monitor settings for more detail](Documentation/ceph-cluster-crd.md#mon-settings). - Ceph OSDs can be created by using StorageClassDeviceSet. See docs on [Storage Class Device Sets](Documentation/ceph-cluster-crd.md#storage-class-device-sets). - Rook can now connect to an external cluster, for more info about external cluster [read the design](https://github.com/rook/rook/blob/master/design/ceph-external-cluster.md) as well as the documentation [Ceph External cluster](Documentation/ceph-cluster-crd.md#external-cluster) -- Added a new property in `storageClassDeviceSets` named `portable`: +- Added a new property in `storageClassDeviceSets` named `portable`: - If `true`, the OSDs will be allowed to move between nodes during failover. This requires a storage class that supports portability (e.g. `aws-ebs`, but not the local storage provisioner). - If `false`, the OSDs will be assigned to a node permanently. Rook will configure Ceph's CRUSH map to support the portability. +- The Ceph cluster custom resource now contains a `configOverrides` section where users can specify + configuration changes to Ceph which Rook should apply. ### YugabyteDB diff --git a/cluster/examples/kubernetes/ceph/cluster.yaml b/cluster/examples/kubernetes/ceph/cluster.yaml index 221b71a3d..94ab05d1d 100644 --- a/cluster/examples/kubernetes/ceph/cluster.yaml +++ b/cluster/examples/kubernetes/ceph/cluster.yaml @@ -143,3 +143,15 @@ spec: # storeType: filestore # - name: "172.17.4.301" # deviceFilter: "^sd." + # Override Ceph configs. These can be overridden by the user at runtime using the Ceph CLI, and + # Rook will set them back to the values set here when the operator restarts. + configOverrides: + - who: global + option: debug ms + value: "1/5" + - who: osd.0 + option: debug_osd + value: "10" + - who: mon.a + option: debug-mon + value: "10" diff --git a/cluster/examples/kubernetes/ceph/common.yaml b/cluster/examples/kubernetes/ceph/common.yaml index 91caecfb3..36f3ba61e 100644 --- a/cluster/examples/kubernetes/ceph/common.yaml +++ b/cluster/examples/kubernetes/ceph/common.yaml @@ -141,6 +141,15 @@ spec: type: integer placement: {} resources: {} + configOverrides: + items: + properties: + who: + type: string + option: + type: string + value: + type: string additionalPrinterColumns: - name: DataDirHostPath type: string diff --git a/pkg/apis/ceph.rook.io/v1/types.go b/pkg/apis/ceph.rook.io/v1/types.go index f2cbfa000..5e66ad10f 100755 --- a/pkg/apis/ceph.rook.io/v1/types.go +++ b/pkg/apis/ceph.rook.io/v1/types.go @@ -69,6 +69,9 @@ type ClusterSpec struct { // The path on the host where config and data can be persisted. DataDirHostPath string `json:"dataDirHostPath,omitempty"` + // Ceph config overrides to apply. + ConfigOverrides ConfigOverridesSpec `json:"configOverrides,omitempty"` + // A spec for mon related options Mon MonSpec `json:"mon,omitempty"` @@ -148,6 +151,18 @@ const ( ClusterStateError ClusterState = "Error" ) +// ConfigOverridesSpec defines how Ceph configurations can be overridden by Rook. +type ConfigOverridesSpec []ConfigOverride + +// ConfigOverride defines the syntax for overriding a Ceph config. This translates to a +// `ceph config set