ceph: remove configOverride functionality from CRD

After further consideration and working with the feature for a few
weeks, the `configOverride` functionality seems to be more complexity
than is worth it for the expected benefit. Users still have the most
preferred option of setting configs in Ceph's config database using
`ceph cofig set` or the dashboard, and the override ConfigMap still
exists for fallback scenarios. The config database can still be used
by Rook for setting necessary configs, but the user has no way of
telling Rook to do so.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
This commit is contained in:
Blaine Gardner committed 2019-09-05 12:30:56 -06:00
1 parent 41fec6bedd
commit 3a243d2978
18 files changed
+175 -261

No files matched your search

+31 -27
View File
@@ -420,16 +420,37 @@ ceph osd pool set rbd pg_num 512
## Custom ceph.conf Settings
**WARNING:** The advised method for controlling Ceph configuration is to manually use the Ceph CLI
or the Ceph dashboard because this offers the most flexibility. If configs must be set as part of
Rook, the `configOverrides` section of the CephCluster resource is recommended. Read the
[CephCluster CRD docs](ceph-cluster-crd.md#ceph-config-overrides) first before considering this
advanced method.
or the Ceph dashboard because this offers the most flexibility. It is highly recommended that this
only be used when absolutely necessary and that the `config` be reset to an empty string if/when the
configurations are no longer necessary. Configurations in the config file will make the Ceph cluster
less configurable from the CLI and dashboard and may make future tuning or debugging difficult.
With Rook the full swath of
[Ceph settings](http://docs.ceph.com/docs/master/rados/configuration/) are available
to use on your storage cluster. When we supply Rook with a ceph.conf file those
settings will be propagated to all Mon, OSD, MDS, and RGW daemons to use.
Setting configs via Ceph's CLI requires that at least one mon be available for the configs to be
set, and setting configs via dashboard requires at least one mgr to be available. Ceph may also have
a small number of very advanced settings that aren't able to be modified easily via CLI or
dashboard. In order to set configurations before monitors are available or to set problematic
configuration settings, the `rook-config-override` ConfigMap exists, and the `config` field can be
set with the contents of a `ceph.conf` file. The contents will be propagated to all mon, mgr, OSD,
MDS, and RGW daemons as an `/etc/ceph/ceph.conf` file.
**WARNING:** Rook performs no validation on the config, so the validity of the settings is the
user's responsibility.
Each daemon will need to be restarted where you want the settings applied:
- mons: ensure all three mons are online and healthy before restarting each mon pod, one at a time.
- mgrs: the pods are stateless and can be restarted as needed, but note that this will disrupt the
Ceph dashboard during restart.
- OSDs: restart your the pods by deleting them, one at a time, and running `ceph -s`
between each restart to ensure the cluster goes back to "active/clean" state.
- RGW: the pods are stateless and can be restarted as needed.
- MDS: the pods are stateless and can be restarted as needed.
After the pod restart, the new settings should be in effect. Note that if the ConfigMap in the Ceph
cluster's namespace is created before the cluster is created, the daemons will pick up the settings
at first launch.
### Example
In this example we will set the default pool `size` to two, and tell OSD
daemons not to change the weight of OSDs on startup.
@@ -455,9 +476,8 @@ data:
config: ""
```
To apply your desired configuration, you will need to update this ConfigMap.
The next time the daemon pod(s) start, the settings will be merged with the default
settings created by Rook.
To apply your desired configuration, you will need to update this ConfigMap. The next time the
daemon pod(s) start, they will use the updated configs.
```bash
kubectl -n rook-ceph edit configmap rook-config-override
@@ -478,22 +498,6 @@ data:
osd pool default size = 2
```
Each daemon will need to be restarted where you want the settings applied:
- Mons: ensure all three mons are online and healthy before restarting each mon pod, one at a time
- OSDs: restart your the pods by deleting them, one at a time, and running `ceph -s`
between each restart to ensure the cluster goes back to "active/clean" state.
- RGW: the pods are stateless and can be restarted as needed
- MDS: the pods are stateless and can be restarted as needed
After the pod restart, your new settings should be in effect. Note that if you create
the ConfigMap in the `rook-ceph` namespace before the cluster is even created
the daemons will pick up the settings at first launch.
The only validation of the settings done by Rook is whether the settings can be merged
using the ini file format with the default settings created by Rook. Beyond that,
the validity of the settings is your responsibility.
## OSD CRUSH Settings
A useful view of the [CRUSH Map](http://docs.ceph.com/docs/master/rados/operations/crush-map/)
-56
View File
@@ -289,62 +289,6 @@ For more information on resource requests/limits see the official Kubernetes doc
- `cpu`: Limit for CPU (example: one CPU core `1`, 50% of one CPU core `500m`).
- `memory`: Limit for Memory (example: one gigabyte of memory `1Gi`, half a gigabyte of memory `512Mi`).
### Ceph config overrides
Users are advised to use Ceph's CLI or dashboard to configure Ceph; however, some users may want to
specify some settings they want set in the `CephCluster` manifest. This is possible using the
`configOverrides` section. Config overrides follow the same syntax as [Ceph's documentation for
setting config values](https://docs.ceph.com/docs/master/rados/configuration/ceph-conf/#commands),
and configuration is stored in Ceph's centralized configuration database.
Each item in `configOverrides` has `who`, `option`, and `value` properties. `who` can be `global` to
affect all Ceph daemons, a daemon type, an individual Ceph daemon, or a glob matching multiple
daemons. It may also use a
[mask](https://docs.ceph.com/docs/master/rados/configuration/ceph-conf/#sections-and-masks).
`option` is the configuration option to be overridden, and `value` is the value to set on the
configuration option. All properties are strings.
Rook will only set configuration overrides once when an orchestration is run. For example, on node
addition or removal, when disks change on a node, or when the operator starts (or restarts). This
means that users can change Ceph's configuration from the CLI or dashboard as desired without Rook
constantly fighting to control the setting. This is particularly valuable in situations where values
need to be changed to debug errors or test changes to tuning parameters.
**WARNING:** Remember that Rook will set these overrides any time the operator restarts, so user
changes via Ceph's CLI or dashboard to values set here will not persist forever.
Some examples:
```yaml
configOverrides:
- who: global
option: log_file
value: /var/log/custom-log-dir
- who: mon
option: mon_compact_on_start
value: "true"
- who: mgr.a
option: mgr_stats_period
value: "10"
- who: osd.*
option: osd_memory_target
value: "2147483648"
- who: osd/rack:foo
option: debug_ms
value: "20"
```
#### Advanced config
Setting configs in the Ceph mons' centralized database this way requires that at least one mon be
available for the configs to be set. Ceph may also have a small number of very advanced settings
that aren't able to be modified easily in the configuration database. In order to set configurations
before monitors are available or to set problematic configuration settings, the
`rook-config-override` ConfigMap exists, and the `config` field can be set with the contents of a
`ceph.conf` file. It is highly recommended that this only be used when absolutely necessary and that
the `config` be reset to an empty string if/when the configurations are no longer necessary, as this
will make the Ceph cluster less configurable from the CLI and dashboard and may make future tuning
or debugging difficult. Read more about this in the
[advanced configuration docs](ceph-advanced-configuration.md#custom-cephconf-settings).
## Samples
Here are several samples for configuring Ceph clusters. Each of the samples must also include the namespace and corresponding access granted for management by the Ceph operator. See the [common cluster resources](#common-cluster-resources) below.
+68
View File
@@ -0,0 +1,68 @@
---
title: Configuration
weight: 3700
indent: true
---
# Configuration
For most any Ceph cluster, the user will want to--and may need to--change some Ceph
configurations. These changes often may be warranted in order to alter performance to meet SLAs or
to update default data resiliency settings.
**WARNING:** Modify Ceph settings carefully, and review the
[Ceph configuration documentation](https://docs.ceph.com/docs/master/rados/configuration/) before
making any changes. Changing the settings could result in unhealthy daemons or even data loss if
used incorrectly.
## Required configurations
Rook and Ceph both strive to make configuration as easy as possible, but there are some
configuration options which users are well advised to consider for any production cluster.
### Default PG and PGP counts
`osd_pool_default_pg_num` and `osd_pool_default_pgp_num`. The number of PGs and PGPs can be
configured on a per-pool basis, but it is highly advised to set default values that are appropriate
for your Ceph cluster. Appropriate values depend on the number of OSDs the user expects to have
backing each pool. The Ceph
[OSD and Pool config docs](https://docs.ceph.com/docs/master/rados/operations/placement-groups/#a-preselection-of-pg-num)
provide detailed information about how to tune these parameters.
An easier option exists for Rook-Ceph clusters running Ceph Nautilus (v14.2.x) or newer. Nautilus
[introduced the PG auto-scaler mgr module](https://ceph.com/rados/new-in-nautilus-pg-merging-and-autotuning/)
capable of automatically managing PG and PGP values for pools. This module is not enabled by default
but can be enabled by running the following from the Ceph toolbox pod (more on the toolbox pod and
Ceph CLI later in this document). Using this module, the user does not need to configure
`osd_pool_default_pg_num` or `osd_pool_default_pgp_num`.
```
ceph mgr module enable pg_autoscaler
```
The user must also decide whether they want the autoscaler to be used for all new pools by default.
Turning this on is recommended. The Ceph CLI command for turning this on is shown below, but this
can be configured via any of the configuration methods detailed later in this document.
```
ceph config set global osd_pool_default_autoscale_mode on
```
## Specifying configuration options
### Toolbox + Ceph CLI
The most recommended way of configuring Ceph is to set Ceph's configuration directly. The first
method for doing so is to use Ceph's CLI from the Rook-Ceph toolbox pod. Using the toolbox pod is
detailed [here](ceph-toolbox.md). From the toolbox, the user can change Ceph configurations, enable
manager modules, create users and pools, and much more.
### Ceph Dashboard
The Ceph Dashboard, examined in more detail [here](ceph-dashboard.md), is another way of setting
some of Ceph's configuration directly. Configuration by the Ceph dashboard is recommended with the
same priority as configuration via the Ceph CLI (above).
### Advanced configuration via ceph.conf override ConfigMap
Setting configs via Ceph's CLI requires that at least one mon be available for the configs to be
set, and setting configs via dashboard requires at least one mgr to be available. Ceph may also have
a small number of very advanced settings that aren't able to be modified easily via CLI or
dashboard. The **least** recommended method for configuring Ceph is intended as a last-resort
fallback in situations like these. This is covered in detail
[here](ceph-advanced-configuration.md#custom-cephconf-settings).
+40 -44
View File
@@ -277,54 +277,51 @@ At this point, your Rook operator should be running version `rook/ceph:v1.1.0`
Verify the Ceph cluster's health using the [health verification section](#health-verification).
### 6. (Optional) Migrate config overrides from ConfigMap to the CephCluster resource
### 6. (Recommended) Migrate config overrides from ConfigMap to Ceph directly
If there are Ceph configuration overrides set in the `config` field of the ConfigMap
`rook-config-override`, it is now possible to migrate those configs manually from the ConfigMap to
the newly-added `configOverrides` section of the `CephCluster` custom resource. This is not required
but may be preferable, as the values set in `configOverrides` can be temporarily overridden by the
user from the toolbox pod as needed in debug/failure scenarios. Values are reset if the operator
restarts.
Ceph config file headers correspond to
the `who` field of each config override as part of the section. For example, the `[global]` section
corresponds to `who: global`, and `[osd.0]` corresponds to `who: osd.0`.
Ceph directly as documented [here](ceph-configuration.md#specifying-configuration-options). This is
not required but is recommended because the values configured in Ceph directly can be temporarily
overridden by the user as needed in debug/failure scenarios.
#### Example
List the contents of the override ConfigMap.
```sh
# List the contents of the override ConfigMap
$> kubectl --namespace $ROOK_NAMESPACE describe configmap rook-ceph-override -o yaml
Name: rook-config-override
Namespace: rook-ceph
Labels: <none>
Annotations: <none>
Data
====
config:
----
[global]
debug_ms = 1/5
[osd.0]
debug_osd = 10
Events: <none>
# Edit the Ceph cluster resource, and add the overrides from the ConfigMap
$> kubectl --namespace $ROOK_NAMESPACE edit cephcluster $ROOK_NAMESPACE
# ...
spec:
# ...
configOverrides:
- who: global
option: debug_ms
value: 1/5
- who: osd.0
option: debug_osd
value: "10"
who: osd.0
# ....
kubectl --namespace $ROOK_NAMESPACE describe configmap rook-config-override
# Name: rook-config-override
# Namespace: rook-ceph
# Labels: <none>
# Annotations: <none>
#
# Data
# ====
# config:
# ----
# [global]
# debug_ms = 1/5
#
# [osd.0]
# debug_osd = 10
# Events: <none>
```
### 7. Update Rook-Ceph custom resource definitions
Apply the configurations to Ceph directly using Ceph's CLI.
```sh
TOOLS_POD=$(kubectl -n $ROOK_NAMESPACE get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}')
kubectl -n $ROOK_NAMESPACE exec -it $TOOLS_POD -- ceph config set global debug_ms 1/5
kubectl -n $ROOK_NAMESPACE exec -it $TOOLS_POD -- ceph config set osd.0 debug_osd 10
```
Empty the config override in the ConfigMap.
```sh
kubectl --namespace $ROOK_NAMESPACE patch configmap rook-config-override --type=merge -p '{"data": {"config": ""}}'
```
### 7. (Recommended) Consider required Ceph config settings
If the user determines that the [advised configuration options](ceph-configuration.md#default-pg-and-pgp-counts)
newly identified in Rook's 1.1 release apply to them, now is the time to set these configs.
### 8. Update Rook-Ceph custom resource definitions
**IMPORTANT: Do not perform this step until ALL existing Rook-Ceph clusters are updated**
After all Rook-Ceph clusters have been updated following the steps above, update the Rook-Ceph
@@ -369,8 +366,7 @@ Ceph image field in the cluster CRD (`spec:cephVersion:image`).
```sh
NEW_CEPH_IMAGE='ceph/ceph:v14.2.2-20190830'
CLUSTER_NAME="$ROOK_NAMESPACE" # change if your cluster name is not the Rook namespace
kubectl -n $ROOK_NAMESPACE patch CephCluster $CLUSTER_NAME --type=merge \
-p "{\"spec\": {\"cephVersion\": {\"image\": \"$NEW_CEPH_IMAGE\"}}}"
kubectl -n $ROOK_NAMESPACE patch CephCluster $CLUSTER_NAME --type=merge -p "{\"spec\": {\"cephVersion\": {\"image\": \"$NEW_CEPH_IMAGE\"}}}"
```
### 2. Wait for the daemon pod updates to complete
-2
View File
@@ -51,8 +51,6 @@ an example usage
- Added a new property in `storageClassDeviceSets` named `portable`:
- If `true`, the OSDs will be allowed to move between nodes during failover. This requires a storage class that supports portability (e.g. `aws-ebs`, but not the local storage provisioner).
- If `false`, the OSDs will be assigned to a node permanently. Rook will configure Ceph's CRUSH map to support the portability.
- The Ceph cluster custom resource now contains a `configOverrides` section where users can specify
configuration changes to Ceph which Rook should apply.
- Rook can now manage PodDisruptionBudgets for the following Daemons: OSD, Mon, RGW, MDS. OSD budgets are dynamically managed as documented in the [design](https://github.com/rook/rook/blob/master/design/ceph-managed-disruptionbudgets.md). This can be enabled with the `managePodBudgets` flag in the cluster CR. When this is enabled, drains on OSDs will be blocked by default and dynamically unblocked in a safe manner one failureDomain at a time. When a failure domain is draining, it will be marked as no out for a longer time than the default DOWN/OUT interval.
### YugabyteDB
@@ -143,18 +143,6 @@ spec:
# storeType: filestore
# - name: "172.17.4.301"
# deviceFilter: "^sd."
# Override Ceph configs. These can be overridden by the user at runtime using the Ceph CLI, and
# Rook will set them back to the values set here when the operator restarts.
configOverrides:
- who: global
option: debug ms
value: "1/5"
- who: osd.0
option: debug_osd
value: "10"
- who: mon.a
option: debug-mon
value: "10"
disruptionManagement: #The section for configuring management of daemon disruptions
managePodBudgets: false #if `true`, the operator will create and manage PodDsruptionBudgets for OSD, Mon, RGW, and MDS daemons. OSD PDBs are managed dynamically via the strategy outlined in the [design](https://github.com/rook/rook/blob/master/design/ceph-managed-disruptionbudgets.md). The operator will block eviction of OSDs by default and unblock them safely when drains are detected.
osdMaintenanceTimeout: 30 # is a duration in minutes that determines how long an entire failureDomain like `region/zone/host` will be held in `noout` (in addition to the default DOWN/OUT interval) when it is draining. This is only relevant when `managePodBudgets` is `true`. The default value is `30` minutes.
@@ -141,15 +141,6 @@ spec:
type: integer
placement: {}
resources: {}
configOverrides:
items:
properties:
who:
type: string
option:
type: string
value:
type: string
additionalPrinterColumns:
- name: DataDirHostPath
type: string
@@ -124,15 +124,6 @@ spec:
type: integer
placement: {}
resources: {}
configOverrides:
items:
properties:
who:
type: string
option:
type: string
value:
type: string
additionalPrinterColumns:
- name: DataDirHostPath
type: string
-15
View File
@@ -71,9 +71,6 @@ type ClusterSpec struct {
// The path on the host where config and data can be persisted.
DataDirHostPath string `json:"dataDirHostPath,omitempty"`
// Ceph config overrides to apply.
ConfigOverrides ConfigOverridesSpec `json:"configOverrides,omitempty"`
// A spec for configuring disruption management.
DisruptionManagement DisruptionManagementSpec `json:"disruptionManagement,omitempty"`
@@ -158,18 +155,6 @@ const (
DefaultFailureDomain = "host"
)
// ConfigOverridesSpec defines how Ceph configurations can be overridden by Rook.
type ConfigOverridesSpec []ConfigOverride
// ConfigOverride defines the syntax for overriding a Ceph config. This translates to a
// `ceph config set <who> <option> <value>` (or similar) command.
// See Ceph docs: https://docs.ceph.com/docs/master/rados/configuration/ceph-conf/#monitor-configuration-database
type ConfigOverride struct {
Who string `json:"who"`
Option string `json:"option"`
Value string `json:"value"`
}
type MonSpec struct {
Count int `json:"count,omitempty"`
AllowMultiplePerNode bool `json:"allowMultiplePerNode,omitempty"`
@@ -479,11 +479,6 @@ func (in *ClusterSpec) DeepCopyInto(out *ClusterSpec) {
(*out)[key] = *val.DeepCopy()
}
}
if in.ConfigOverrides != nil {
in, out := &in.ConfigOverrides, &out.ConfigOverrides
*out = make(ConfigOverridesSpec, len(*in))
copy(*out, *in)
}
out.DisruptionManagement = in.DisruptionManagement
in.Mon.DeepCopyInto(&out.Mon)
out.RBDMirroring = in.RBDMirroring
@@ -524,42 +519,6 @@ func (in *ClusterStatus) DeepCopy() *ClusterStatus {
return out
}
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *ConfigOverride) DeepCopyInto(out *ConfigOverride) {
*out = *in
return
}
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ConfigOverride.
func (in *ConfigOverride) DeepCopy() *ConfigOverride {
if in == nil {
return nil
}
out := new(ConfigOverride)
in.DeepCopyInto(out)
return out
}
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in ConfigOverridesSpec) DeepCopyInto(out *ConfigOverridesSpec) {
{
in := &in
*out = make(ConfigOverridesSpec, len(*in))
copy(*out, *in)
return
}
}
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ConfigOverridesSpec.
func (in ConfigOverridesSpec) DeepCopy() ConfigOverridesSpec {
if in == nil {
return nil
}
out := new(ConfigOverridesSpec)
in.DeepCopyInto(out)
return *out
}
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *DashboardSpec) DeepCopyInto(out *DashboardSpec) {
*out = *in
+1 -1
View File
@@ -231,7 +231,7 @@ func (c *cluster) doOrchestration(rookImage string, cephVersion cephver.CephVers
if c.Spec.External.Enable {
// Apply CRD ConfigOverrideName to the external cluster
err = config.SetDefaultAndUserConfigs(c.context, c.Namespace, c.Info, c.Spec.ConfigOverrides)
err = config.SetDefaultConfigs(c.context, c.Namespace, c.Info)
if err != nil {
// Mons are up, so something else is wrong
return fmt.Errorf("failed to set Rook and/or user-defined Ceph config options on the external cluster monitors. %+v", err)
+1 -1
View File
@@ -278,7 +278,7 @@ func (c *ClusterController) configureExternalCephCluster(namespace, name string,
}
// Apply CRD ConfigOverrideName to the external cluster
err = config.SetDefaultAndUserConfigs(cluster.context, namespace, cluster.Info, cluster.Spec.ConfigOverrides)
err = config.SetDefaultConfigs(cluster.context, namespace, cluster.Info)
if err != nil {
// Mons are up, so something else is wrong
return fmt.Errorf("failed to set Rook and/or user-defined Ceph config options on the external cluster monitors. %+v", err)
+4 -4
View File
@@ -226,7 +226,7 @@ func (c *Cluster) startMons(targetCount int) error {
// only once and do it as early as possible in the mon orchestration.
setConfigsNeedsRetry := false
if existingCount > 0 {
err := config.SetDefaultAndUserConfigs(c.context, c.Namespace, c.ClusterInfo, c.spec.ConfigOverrides)
err := config.SetDefaultConfigs(c.context, c.Namespace, c.ClusterInfo)
if err != nil {
// If we fail here, it could be because the mons are not healthy, and this might be
// fixed by updating the mon deployments. Instead of returning error here, log a
@@ -247,7 +247,7 @@ func (c *Cluster) startMons(targetCount int) error {
// values in the config database. Do this only when the existing count is zero so that
// this is only done once when the cluster is created.
if existingCount == 0 {
err := config.SetDefaultAndUserConfigs(c.context, c.Namespace, c.ClusterInfo, c.spec.ConfigOverrides)
err := config.SetDefaultConfigs(c.context, c.Namespace, c.ClusterInfo)
if err != nil {
return fmt.Errorf("failed to set Rook and/or user-defined Ceph config options after creating the first mon. %+v", err)
}
@@ -255,7 +255,7 @@ func (c *Cluster) startMons(targetCount int) error {
// Or if we need to retry, only do this when we are on the first iteration of the
// loop. This could be in the same if statement as above, but separate it to get a
// different error message.
err := config.SetDefaultAndUserConfigs(c.context, c.Namespace, c.ClusterInfo, c.spec.ConfigOverrides)
err := config.SetDefaultConfigs(c.context, c.Namespace, c.ClusterInfo)
if err != nil {
return fmt.Errorf("failed to set Rook and/or user-defined Ceph config options after updating the existing mons. %+v", err)
}
@@ -269,7 +269,7 @@ func (c *Cluster) startMons(targetCount int) error {
}
if setConfigsNeedsRetry {
err := config.SetDefaultAndUserConfigs(c.context, c.Namespace, c.ClusterInfo, c.spec.ConfigOverrides)
err := config.SetDefaultConfigs(c.context, c.Namespace, c.ClusterInfo)
if err != nil {
return fmt.Errorf("failed to set Rook and/or user-defined Ceph config options after forcefully updating the existing mons. %+v", err)
}
+5 -13
View File
@@ -24,7 +24,6 @@ import (
"strings"
"github.com/coreos/pkg/capnslog"
rookceph "github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
"github.com/rook/rook/pkg/clusterd"
cephconfig "github.com/rook/rook/pkg/daemon/ceph/config"
)
@@ -100,31 +99,24 @@ func NewFlag(key, value string) string {
return fmt.Sprintf("--%s=%s", f, value)
}
// SetDefaultAndUserConfigs sets Rook's desired default configs and the user's override configs from
// the CephCluster CRD in the centralized monitor database. This cannot be called before at least
// one monitor is established.
func SetDefaultAndUserConfigs(
// SetDefaultConfigs sets Rook's desired default configs in the centralized monitor database. This
// cannot be called before at least one monitor is established.
func SetDefaultConfigs(
context *clusterd.Context,
namespace string,
clusterInfo *cephconfig.ClusterInfo,
configOverrides rookceph.ConfigOverridesSpec,
) error {
// ceph.conf is never used. All configurations are made in the centralized mon config database,
// or they are specified on the commandline when daemons are called.
monStore := GetMonStore(context, namespace)
if err := monStore.SetAll(DefaultCentralizedConfigs(clusterInfo.CephVersion)); err != nil {
if err := monStore.SetAll(DefaultCentralizedConfigs(clusterInfo.CephVersion)...); err != nil {
return fmt.Errorf("failed to apply default Ceph configurations. %+v", err)
}
if err := monStore.SetAll(DefaultLegacyConfigs()); err != nil {
if err := monStore.SetAll(DefaultLegacyConfigs()...); err != nil {
return fmt.Errorf("failed to apply legacy config overrides. %+v", err)
}
// user-specified config overrides from the CRD will go here
if err := monStore.SetAll(configOverrides); err != nil {
return fmt.Errorf("failed to apply one or more user-specified overrides. %+v", err)
}
return nil
}
+7 -8
View File
@@ -18,7 +18,6 @@ limitations under the License.
package config
import (
rookceph "github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
"github.com/rook/rook/pkg/operator/ceph/version"
)
@@ -55,14 +54,14 @@ func DefaultFlags(fsid, mountedKeyringPath string, cephVersion version.CephVersi
}
// makes it possible to be slightly less verbose to create a ConfigOverride here
func configOverride(who, option, value string) rookceph.ConfigOverride {
return rookceph.ConfigOverride{Who: who, Option: option, Value: value}
func configOverride(who, option, value string) Option {
return Option{Who: who, Option: option, Value: value}
}
// DefaultCentralizedConfigs returns the default configuration options Rook will set in Ceph's
// centralized config store.
func DefaultCentralizedConfigs(cephVersion version.CephVersion) rookceph.ConfigOverridesSpec {
overrides := []rookceph.ConfigOverride{
func DefaultCentralizedConfigs(cephVersion version.CephVersion) []Option {
overrides := []Option{
configOverride("global", "mon allow pool delete", "true"),
}
@@ -74,7 +73,7 @@ func DefaultCentralizedConfigs(cephVersion version.CephVersion) rookceph.ConfigO
if !cephVersion.IsAtLeast(version.CephVersion{Major: 14, Minor: 2, Extra: 1}) {
// Set the default log files to empty so they don't bloat containers. Can be changed in
// Mimic+ by users if needed.
overrides = append(overrides, []rookceph.ConfigOverride{
overrides = append(overrides, []Option{
configOverride("global", "log file", ""),
configOverride("global", "mon cluster log file", ""),
}...)
@@ -85,8 +84,8 @@ func DefaultCentralizedConfigs(cephVersion version.CephVersion) rookceph.ConfigO
// DefaultLegacyConfigs need to be added to the Ceph config file until the integration tests can be
// made to override these options for the Ceph clusters it creates.
func DefaultLegacyConfigs() rookceph.ConfigOverridesSpec {
overrides := []rookceph.ConfigOverride{
func DefaultLegacyConfigs() []Option {
overrides := []Option{
// TODO: drop this when FlexVolume is no longer supported
configOverride("global", "rbd_default_features", "3"),
}
+14 -3
View File
@@ -19,7 +19,6 @@ package config
import (
"fmt"
rookceph "github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
"github.com/rook/rook/pkg/clusterd"
"github.com/rook/rook/pkg/daemon/ceph/client"
)
@@ -39,6 +38,18 @@ func GetMonStore(context *clusterd.Context, namespace string) *MonStore {
}
}
// Option defines the pieces of information relevant to Ceph configuration options.
type Option struct {
// Who is the entity(-ies) the option should apply to.
Who string
// Option is the option key
Option string
// Value is the value for the option
Value string
}
// Set sets a config in the centralized mon configuration database.
// https://docs.ceph.com/docs/master/rados/configuration/ceph-conf/#monitor-configuration-database
func (m *MonStore) Set(who, option, value string) error {
@@ -54,9 +65,9 @@ func (m *MonStore) Set(who, option, value string) error {
// SetAll sets all configs from the overrides in the centralized mon configuration database.
// See MonStore.Set for more.
func (m *MonStore) SetAll(configOverrides rookceph.ConfigOverridesSpec) error {
func (m *MonStore) SetAll(options ...Option) error {
var errs []error
for _, override := range configOverrides {
for _, override := range options {
err := m.Set(override.Who, override.Option, override.Value)
if err != nil {
errs = append(errs, err)
+4 -7
View File
@@ -21,7 +21,6 @@ import (
"strings"
"testing"
rookceph "github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
"github.com/rook/rook/pkg/clusterd"
testop "github.com/rook/rook/pkg/operator/test"
exectest "github.com/rook/rook/pkg/util/exec/test"
@@ -96,16 +95,14 @@ func TestMonStore_SetAll(t *testing.T) {
monStore := GetMonStore(ctx, "ns")
var cfgOverrides rookceph.ConfigOverridesSpec
cfgOverrides = []rookceph.ConfigOverride{
cfgOverrides := []Option{
configOverride("global", "debug ms", "10"), // setting w/ spaces converts to underscores
configOverride("osd.0", "debug-osd", "20"), // setting w/ dashes converts to underscores
configOverride("mds.*", "debug_mds", "15"), // setting w/ underscores remains the same
}
// commands w/ no error
e := monStore.SetAll(cfgOverrides)
e := monStore.SetAll(cfgOverrides...)
assert.NoError(t, e)
assert.Len(t, execedCmds, 3)
assert.Contains(t, execedCmds[0], " global debug_ms 10 ")
@@ -116,7 +113,7 @@ func TestMonStore_SetAll(t *testing.T) {
// keep cfgOverrides from last test
execInjectErrOnKeyword = "debug_osd"
execedCmds = execedCmds[:0] // empty execedCmds slice
e = monStore.SetAll(cfgOverrides)
e = monStore.SetAll(cfgOverrides...)
assert.Error(t, e)
// Rook should not return error before trying to set all config overrides
assert.Len(t, execedCmds, 3)
@@ -125,7 +122,7 @@ func TestMonStore_SetAll(t *testing.T) {
// keep cfgOverrides
execInjectErrOnKeyword = "debug"
execedCmds = execedCmds[:0]
e = monStore.SetAll(cfgOverrides)
e = monStore.SetAll(cfgOverrides...)
assert.Error(t, e)
assert.Len(t, execedCmds, 3)
}
@@ -197,15 +197,6 @@ spec:
type: integer
placement: {}
resources: {}
configOverrides:
items:
properties:
who:
type: string
option:
type: string
value:
type: string
required:
- mon
additionalPrinterColumns: