ceph: remove configOverride functionality from CRD
After further consideration and working with the feature for a few weeks, the `configOverride` functionality seems to be more complexity than is worth it for the expected benefit. Users still have the most preferred option of setting configs in Ceph's config database using `ceph cofig set` or the dashboard, and the override ConfigMap still exists for fallback scenarios. The config database can still be used by Rook for setting necessary configs, but the user has no way of telling Rook to do so. Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
This commit is contained in:
18 files changed
+175
-261
No files matched your search
@@ -420,16 +420,37 @@ ceph osd pool set rbd pg_num 512
|
||||
|
||||
## Custom ceph.conf Settings
|
||||
**WARNING:** The advised method for controlling Ceph configuration is to manually use the Ceph CLI
|
||||
or the Ceph dashboard because this offers the most flexibility. If configs must be set as part of
|
||||
Rook, the `configOverrides` section of the CephCluster resource is recommended. Read the
|
||||
[CephCluster CRD docs](ceph-cluster-crd.md#ceph-config-overrides) first before considering this
|
||||
advanced method.
|
||||
or the Ceph dashboard because this offers the most flexibility. It is highly recommended that this
|
||||
only be used when absolutely necessary and that the `config` be reset to an empty string if/when the
|
||||
configurations are no longer necessary. Configurations in the config file will make the Ceph cluster
|
||||
less configurable from the CLI and dashboard and may make future tuning or debugging difficult.
|
||||
|
||||
With Rook the full swath of
|
||||
[Ceph settings](http://docs.ceph.com/docs/master/rados/configuration/) are available
|
||||
to use on your storage cluster. When we supply Rook with a ceph.conf file those
|
||||
settings will be propagated to all Mon, OSD, MDS, and RGW daemons to use.
|
||||
Setting configs via Ceph's CLI requires that at least one mon be available for the configs to be
|
||||
set, and setting configs via dashboard requires at least one mgr to be available. Ceph may also have
|
||||
a small number of very advanced settings that aren't able to be modified easily via CLI or
|
||||
dashboard. In order to set configurations before monitors are available or to set problematic
|
||||
configuration settings, the `rook-config-override` ConfigMap exists, and the `config` field can be
|
||||
set with the contents of a `ceph.conf` file. The contents will be propagated to all mon, mgr, OSD,
|
||||
MDS, and RGW daemons as an `/etc/ceph/ceph.conf` file.
|
||||
|
||||
**WARNING:** Rook performs no validation on the config, so the validity of the settings is the
|
||||
user's responsibility.
|
||||
|
||||
Each daemon will need to be restarted where you want the settings applied:
|
||||
|
||||
- mons: ensure all three mons are online and healthy before restarting each mon pod, one at a time.
|
||||
- mgrs: the pods are stateless and can be restarted as needed, but note that this will disrupt the
|
||||
Ceph dashboard during restart.
|
||||
- OSDs: restart your the pods by deleting them, one at a time, and running `ceph -s`
|
||||
between each restart to ensure the cluster goes back to "active/clean" state.
|
||||
- RGW: the pods are stateless and can be restarted as needed.
|
||||
- MDS: the pods are stateless and can be restarted as needed.
|
||||
|
||||
After the pod restart, the new settings should be in effect. Note that if the ConfigMap in the Ceph
|
||||
cluster's namespace is created before the cluster is created, the daemons will pick up the settings
|
||||
at first launch.
|
||||
|
||||
### Example
|
||||
In this example we will set the default pool `size` to two, and tell OSD
|
||||
daemons not to change the weight of OSDs on startup.
|
||||
|
||||
@@ -455,9 +476,8 @@ data:
|
||||
config: ""
|
||||
```
|
||||
|
||||
To apply your desired configuration, you will need to update this ConfigMap.
|
||||
The next time the daemon pod(s) start, the settings will be merged with the default
|
||||
settings created by Rook.
|
||||
To apply your desired configuration, you will need to update this ConfigMap. The next time the
|
||||
daemon pod(s) start, they will use the updated configs.
|
||||
|
||||
```bash
|
||||
kubectl -n rook-ceph edit configmap rook-config-override
|
||||
@@ -478,22 +498,6 @@ data:
|
||||
osd pool default size = 2
|
||||
```
|
||||
|
||||
Each daemon will need to be restarted where you want the settings applied:
|
||||
|
||||
- Mons: ensure all three mons are online and healthy before restarting each mon pod, one at a time
|
||||
- OSDs: restart your the pods by deleting them, one at a time, and running `ceph -s`
|
||||
between each restart to ensure the cluster goes back to "active/clean" state.
|
||||
- RGW: the pods are stateless and can be restarted as needed
|
||||
- MDS: the pods are stateless and can be restarted as needed
|
||||
|
||||
After the pod restart, your new settings should be in effect. Note that if you create
|
||||
the ConfigMap in the `rook-ceph` namespace before the cluster is even created
|
||||
the daemons will pick up the settings at first launch.
|
||||
|
||||
The only validation of the settings done by Rook is whether the settings can be merged
|
||||
using the ini file format with the default settings created by Rook. Beyond that,
|
||||
the validity of the settings is your responsibility.
|
||||
|
||||
## OSD CRUSH Settings
|
||||
|
||||
A useful view of the [CRUSH Map](http://docs.ceph.com/docs/master/rados/operations/crush-map/)
|
||||
|
||||
@@ -289,62 +289,6 @@ For more information on resource requests/limits see the official Kubernetes doc
|
||||
- `cpu`: Limit for CPU (example: one CPU core `1`, 50% of one CPU core `500m`).
|
||||
- `memory`: Limit for Memory (example: one gigabyte of memory `1Gi`, half a gigabyte of memory `512Mi`).
|
||||
|
||||
### Ceph config overrides
|
||||
Users are advised to use Ceph's CLI or dashboard to configure Ceph; however, some users may want to
|
||||
specify some settings they want set in the `CephCluster` manifest. This is possible using the
|
||||
`configOverrides` section. Config overrides follow the same syntax as [Ceph's documentation for
|
||||
setting config values](https://docs.ceph.com/docs/master/rados/configuration/ceph-conf/#commands),
|
||||
and configuration is stored in Ceph's centralized configuration database.
|
||||
|
||||
Each item in `configOverrides` has `who`, `option`, and `value` properties. `who` can be `global` to
|
||||
affect all Ceph daemons, a daemon type, an individual Ceph daemon, or a glob matching multiple
|
||||
daemons. It may also use a
|
||||
[mask](https://docs.ceph.com/docs/master/rados/configuration/ceph-conf/#sections-and-masks).
|
||||
`option` is the configuration option to be overridden, and `value` is the value to set on the
|
||||
configuration option. All properties are strings.
|
||||
|
||||
Rook will only set configuration overrides once when an orchestration is run. For example, on node
|
||||
addition or removal, when disks change on a node, or when the operator starts (or restarts). This
|
||||
means that users can change Ceph's configuration from the CLI or dashboard as desired without Rook
|
||||
constantly fighting to control the setting. This is particularly valuable in situations where values
|
||||
need to be changed to debug errors or test changes to tuning parameters.
|
||||
|
||||
**WARNING:** Remember that Rook will set these overrides any time the operator restarts, so user
|
||||
changes via Ceph's CLI or dashboard to values set here will not persist forever.
|
||||
|
||||
Some examples:
|
||||
```yaml
|
||||
configOverrides:
|
||||
- who: global
|
||||
option: log_file
|
||||
value: /var/log/custom-log-dir
|
||||
- who: mon
|
||||
option: mon_compact_on_start
|
||||
value: "true"
|
||||
- who: mgr.a
|
||||
option: mgr_stats_period
|
||||
value: "10"
|
||||
- who: osd.*
|
||||
option: osd_memory_target
|
||||
value: "2147483648"
|
||||
- who: osd/rack:foo
|
||||
option: debug_ms
|
||||
value: "20"
|
||||
```
|
||||
|
||||
#### Advanced config
|
||||
Setting configs in the Ceph mons' centralized database this way requires that at least one mon be
|
||||
available for the configs to be set. Ceph may also have a small number of very advanced settings
|
||||
that aren't able to be modified easily in the configuration database. In order to set configurations
|
||||
before monitors are available or to set problematic configuration settings, the
|
||||
`rook-config-override` ConfigMap exists, and the `config` field can be set with the contents of a
|
||||
`ceph.conf` file. It is highly recommended that this only be used when absolutely necessary and that
|
||||
the `config` be reset to an empty string if/when the configurations are no longer necessary, as this
|
||||
will make the Ceph cluster less configurable from the CLI and dashboard and may make future tuning
|
||||
or debugging difficult. Read more about this in the
|
||||
[advanced configuration docs](ceph-advanced-configuration.md#custom-cephconf-settings).
|
||||
|
||||
|
||||
## Samples
|
||||
Here are several samples for configuring Ceph clusters. Each of the samples must also include the namespace and corresponding access granted for management by the Ceph operator. See the [common cluster resources](#common-cluster-resources) below.
|
||||
|
||||
|
||||
@@ -0,0 +1,68 @@
|
||||
---
|
||||
title: Configuration
|
||||
weight: 3700
|
||||
indent: true
|
||||
---
|
||||
|
||||
# Configuration
|
||||
|
||||
For most any Ceph cluster, the user will want to--and may need to--change some Ceph
|
||||
configurations. These changes often may be warranted in order to alter performance to meet SLAs or
|
||||
to update default data resiliency settings.
|
||||
|
||||
**WARNING:** Modify Ceph settings carefully, and review the
|
||||
[Ceph configuration documentation](https://docs.ceph.com/docs/master/rados/configuration/) before
|
||||
making any changes. Changing the settings could result in unhealthy daemons or even data loss if
|
||||
used incorrectly.
|
||||
|
||||
|
||||
## Required configurations
|
||||
Rook and Ceph both strive to make configuration as easy as possible, but there are some
|
||||
configuration options which users are well advised to consider for any production cluster.
|
||||
|
||||
### Default PG and PGP counts
|
||||
`osd_pool_default_pg_num` and `osd_pool_default_pgp_num`. The number of PGs and PGPs can be
|
||||
configured on a per-pool basis, but it is highly advised to set default values that are appropriate
|
||||
for your Ceph cluster. Appropriate values depend on the number of OSDs the user expects to have
|
||||
backing each pool. The Ceph
|
||||
[OSD and Pool config docs](https://docs.ceph.com/docs/master/rados/operations/placement-groups/#a-preselection-of-pg-num)
|
||||
provide detailed information about how to tune these parameters.
|
||||
|
||||
An easier option exists for Rook-Ceph clusters running Ceph Nautilus (v14.2.x) or newer. Nautilus
|
||||
[introduced the PG auto-scaler mgr module](https://ceph.com/rados/new-in-nautilus-pg-merging-and-autotuning/)
|
||||
capable of automatically managing PG and PGP values for pools. This module is not enabled by default
|
||||
but can be enabled by running the following from the Ceph toolbox pod (more on the toolbox pod and
|
||||
Ceph CLI later in this document). Using this module, the user does not need to configure
|
||||
`osd_pool_default_pg_num` or `osd_pool_default_pgp_num`.
|
||||
```
|
||||
ceph mgr module enable pg_autoscaler
|
||||
```
|
||||
|
||||
The user must also decide whether they want the autoscaler to be used for all new pools by default.
|
||||
Turning this on is recommended. The Ceph CLI command for turning this on is shown below, but this
|
||||
can be configured via any of the configuration methods detailed later in this document.
|
||||
```
|
||||
ceph config set global osd_pool_default_autoscale_mode on
|
||||
```
|
||||
|
||||
|
||||
## Specifying configuration options
|
||||
|
||||
### Toolbox + Ceph CLI
|
||||
The most recommended way of configuring Ceph is to set Ceph's configuration directly. The first
|
||||
method for doing so is to use Ceph's CLI from the Rook-Ceph toolbox pod. Using the toolbox pod is
|
||||
detailed [here](ceph-toolbox.md). From the toolbox, the user can change Ceph configurations, enable
|
||||
manager modules, create users and pools, and much more.
|
||||
|
||||
### Ceph Dashboard
|
||||
The Ceph Dashboard, examined in more detail [here](ceph-dashboard.md), is another way of setting
|
||||
some of Ceph's configuration directly. Configuration by the Ceph dashboard is recommended with the
|
||||
same priority as configuration via the Ceph CLI (above).
|
||||
|
||||
### Advanced configuration via ceph.conf override ConfigMap
|
||||
Setting configs via Ceph's CLI requires that at least one mon be available for the configs to be
|
||||
set, and setting configs via dashboard requires at least one mgr to be available. Ceph may also have
|
||||
a small number of very advanced settings that aren't able to be modified easily via CLI or
|
||||
dashboard. The **least** recommended method for configuring Ceph is intended as a last-resort
|
||||
fallback in situations like these. This is covered in detail
|
||||
[here](ceph-advanced-configuration.md#custom-cephconf-settings).
|
||||
@@ -277,54 +277,51 @@ At this point, your Rook operator should be running version `rook/ceph:v1.1.0`
|
||||
|
||||
Verify the Ceph cluster's health using the [health verification section](#health-verification).
|
||||
|
||||
### 6. (Optional) Migrate config overrides from ConfigMap to the CephCluster resource
|
||||
### 6. (Recommended) Migrate config overrides from ConfigMap to Ceph directly
|
||||
If there are Ceph configuration overrides set in the `config` field of the ConfigMap
|
||||
`rook-config-override`, it is now possible to migrate those configs manually from the ConfigMap to
|
||||
the newly-added `configOverrides` section of the `CephCluster` custom resource. This is not required
|
||||
but may be preferable, as the values set in `configOverrides` can be temporarily overridden by the
|
||||
user from the toolbox pod as needed in debug/failure scenarios. Values are reset if the operator
|
||||
restarts.
|
||||
|
||||
Ceph config file headers correspond to
|
||||
the `who` field of each config override as part of the section. For example, the `[global]` section
|
||||
corresponds to `who: global`, and `[osd.0]` corresponds to `who: osd.0`.
|
||||
Ceph directly as documented [here](ceph-configuration.md#specifying-configuration-options). This is
|
||||
not required but is recommended because the values configured in Ceph directly can be temporarily
|
||||
overridden by the user as needed in debug/failure scenarios.
|
||||
|
||||
#### Example
|
||||
List the contents of the override ConfigMap.
|
||||
```sh
|
||||
# List the contents of the override ConfigMap
|
||||
$> kubectl --namespace $ROOK_NAMESPACE describe configmap rook-ceph-override -o yaml
|
||||
Name: rook-config-override
|
||||
Namespace: rook-ceph
|
||||
Labels: <none>
|
||||
Annotations: <none>
|
||||
|
||||
Data
|
||||
====
|
||||
config:
|
||||
----
|
||||
[global]
|
||||
debug_ms = 1/5
|
||||
|
||||
[osd.0]
|
||||
debug_osd = 10
|
||||
Events: <none>
|
||||
|
||||
# Edit the Ceph cluster resource, and add the overrides from the ConfigMap
|
||||
$> kubectl --namespace $ROOK_NAMESPACE edit cephcluster $ROOK_NAMESPACE
|
||||
# ...
|
||||
spec:
|
||||
# ...
|
||||
configOverrides:
|
||||
- who: global
|
||||
option: debug_ms
|
||||
value: 1/5
|
||||
- who: osd.0
|
||||
option: debug_osd
|
||||
value: "10"
|
||||
who: osd.0
|
||||
# ....
|
||||
kubectl --namespace $ROOK_NAMESPACE describe configmap rook-config-override
|
||||
# Name: rook-config-override
|
||||
# Namespace: rook-ceph
|
||||
# Labels: <none>
|
||||
# Annotations: <none>
|
||||
#
|
||||
# Data
|
||||
# ====
|
||||
# config:
|
||||
# ----
|
||||
# [global]
|
||||
# debug_ms = 1/5
|
||||
#
|
||||
# [osd.0]
|
||||
# debug_osd = 10
|
||||
# Events: <none>
|
||||
```
|
||||
|
||||
### 7. Update Rook-Ceph custom resource definitions
|
||||
Apply the configurations to Ceph directly using Ceph's CLI.
|
||||
```sh
|
||||
TOOLS_POD=$(kubectl -n $ROOK_NAMESPACE get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}')
|
||||
kubectl -n $ROOK_NAMESPACE exec -it $TOOLS_POD -- ceph config set global debug_ms 1/5
|
||||
kubectl -n $ROOK_NAMESPACE exec -it $TOOLS_POD -- ceph config set osd.0 debug_osd 10
|
||||
```
|
||||
|
||||
Empty the config override in the ConfigMap.
|
||||
```sh
|
||||
kubectl --namespace $ROOK_NAMESPACE patch configmap rook-config-override --type=merge -p '{"data": {"config": ""}}'
|
||||
```
|
||||
|
||||
### 7. (Recommended) Consider required Ceph config settings
|
||||
If the user determines that the [advised configuration options](ceph-configuration.md#default-pg-and-pgp-counts)
|
||||
newly identified in Rook's 1.1 release apply to them, now is the time to set these configs.
|
||||
|
||||
### 8. Update Rook-Ceph custom resource definitions
|
||||
**IMPORTANT: Do not perform this step until ALL existing Rook-Ceph clusters are updated**
|
||||
|
||||
After all Rook-Ceph clusters have been updated following the steps above, update the Rook-Ceph
|
||||
@@ -369,8 +366,7 @@ Ceph image field in the cluster CRD (`spec:cephVersion:image`).
|
||||
```sh
|
||||
NEW_CEPH_IMAGE='ceph/ceph:v14.2.2-20190830'
|
||||
CLUSTER_NAME="$ROOK_NAMESPACE" # change if your cluster name is not the Rook namespace
|
||||
kubectl -n $ROOK_NAMESPACE patch CephCluster $CLUSTER_NAME --type=merge \
|
||||
-p "{\"spec\": {\"cephVersion\": {\"image\": \"$NEW_CEPH_IMAGE\"}}}"
|
||||
kubectl -n $ROOK_NAMESPACE patch CephCluster $CLUSTER_NAME --type=merge -p "{\"spec\": {\"cephVersion\": {\"image\": \"$NEW_CEPH_IMAGE\"}}}"
|
||||
```
|
||||
|
||||
### 2. Wait for the daemon pod updates to complete
|
||||
|
||||
@@ -51,8 +51,6 @@ an example usage
|
||||
- Added a new property in `storageClassDeviceSets` named `portable`:
|
||||
- If `true`, the OSDs will be allowed to move between nodes during failover. This requires a storage class that supports portability (e.g. `aws-ebs`, but not the local storage provisioner).
|
||||
- If `false`, the OSDs will be assigned to a node permanently. Rook will configure Ceph's CRUSH map to support the portability.
|
||||
- The Ceph cluster custom resource now contains a `configOverrides` section where users can specify
|
||||
configuration changes to Ceph which Rook should apply.
|
||||
- Rook can now manage PodDisruptionBudgets for the following Daemons: OSD, Mon, RGW, MDS. OSD budgets are dynamically managed as documented in the [design](https://github.com/rook/rook/blob/master/design/ceph-managed-disruptionbudgets.md). This can be enabled with the `managePodBudgets` flag in the cluster CR. When this is enabled, drains on OSDs will be blocked by default and dynamically unblocked in a safe manner one failureDomain at a time. When a failure domain is draining, it will be marked as no out for a longer time than the default DOWN/OUT interval.
|
||||
|
||||
### YugabyteDB
|
||||
|
||||
@@ -143,18 +143,6 @@ spec:
|
||||
# storeType: filestore
|
||||
# - name: "172.17.4.301"
|
||||
# deviceFilter: "^sd."
|
||||
# Override Ceph configs. These can be overridden by the user at runtime using the Ceph CLI, and
|
||||
# Rook will set them back to the values set here when the operator restarts.
|
||||
configOverrides:
|
||||
- who: global
|
||||
option: debug ms
|
||||
value: "1/5"
|
||||
- who: osd.0
|
||||
option: debug_osd
|
||||
value: "10"
|
||||
- who: mon.a
|
||||
option: debug-mon
|
||||
value: "10"
|
||||
disruptionManagement: #The section for configuring management of daemon disruptions
|
||||
managePodBudgets: false #if `true`, the operator will create and manage PodDsruptionBudgets for OSD, Mon, RGW, and MDS daemons. OSD PDBs are managed dynamically via the strategy outlined in the [design](https://github.com/rook/rook/blob/master/design/ceph-managed-disruptionbudgets.md). The operator will block eviction of OSDs by default and unblock them safely when drains are detected.
|
||||
osdMaintenanceTimeout: 30 # is a duration in minutes that determines how long an entire failureDomain like `region/zone/host` will be held in `noout` (in addition to the default DOWN/OUT interval) when it is draining. This is only relevant when `managePodBudgets` is `true`. The default value is `30` minutes.
|
||||
@@ -141,15 +141,6 @@ spec:
|
||||
type: integer
|
||||
placement: {}
|
||||
resources: {}
|
||||
configOverrides:
|
||||
items:
|
||||
properties:
|
||||
who:
|
||||
type: string
|
||||
option:
|
||||
type: string
|
||||
value:
|
||||
type: string
|
||||
additionalPrinterColumns:
|
||||
- name: DataDirHostPath
|
||||
type: string
|
||||
|
||||
@@ -124,15 +124,6 @@ spec:
|
||||
type: integer
|
||||
placement: {}
|
||||
resources: {}
|
||||
configOverrides:
|
||||
items:
|
||||
properties:
|
||||
who:
|
||||
type: string
|
||||
option:
|
||||
type: string
|
||||
value:
|
||||
type: string
|
||||
additionalPrinterColumns:
|
||||
- name: DataDirHostPath
|
||||
type: string
|
||||
|
||||
@@ -71,9 +71,6 @@ type ClusterSpec struct {
|
||||
// The path on the host where config and data can be persisted.
|
||||
DataDirHostPath string `json:"dataDirHostPath,omitempty"`
|
||||
|
||||
// Ceph config overrides to apply.
|
||||
ConfigOverrides ConfigOverridesSpec `json:"configOverrides,omitempty"`
|
||||
|
||||
// A spec for configuring disruption management.
|
||||
DisruptionManagement DisruptionManagementSpec `json:"disruptionManagement,omitempty"`
|
||||
|
||||
@@ -158,18 +155,6 @@ const (
|
||||
DefaultFailureDomain = "host"
|
||||
)
|
||||
|
||||
// ConfigOverridesSpec defines how Ceph configurations can be overridden by Rook.
|
||||
type ConfigOverridesSpec []ConfigOverride
|
||||
|
||||
// ConfigOverride defines the syntax for overriding a Ceph config. This translates to a
|
||||
// `ceph config set <who> <option> <value>` (or similar) command.
|
||||
// See Ceph docs: https://docs.ceph.com/docs/master/rados/configuration/ceph-conf/#monitor-configuration-database
|
||||
type ConfigOverride struct {
|
||||
Who string `json:"who"`
|
||||
Option string `json:"option"`
|
||||
Value string `json:"value"`
|
||||
}
|
||||
|
||||
type MonSpec struct {
|
||||
Count int `json:"count,omitempty"`
|
||||
AllowMultiplePerNode bool `json:"allowMultiplePerNode,omitempty"`
|
||||
|
||||
@@ -479,11 +479,6 @@ func (in *ClusterSpec) DeepCopyInto(out *ClusterSpec) {
|
||||
(*out)[key] = *val.DeepCopy()
|
||||
}
|
||||
}
|
||||
if in.ConfigOverrides != nil {
|
||||
in, out := &in.ConfigOverrides, &out.ConfigOverrides
|
||||
*out = make(ConfigOverridesSpec, len(*in))
|
||||
copy(*out, *in)
|
||||
}
|
||||
out.DisruptionManagement = in.DisruptionManagement
|
||||
in.Mon.DeepCopyInto(&out.Mon)
|
||||
out.RBDMirroring = in.RBDMirroring
|
||||
@@ -524,42 +519,6 @@ func (in *ClusterStatus) DeepCopy() *ClusterStatus {
|
||||
return out
|
||||
}
|
||||
|
||||
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
|
||||
func (in *ConfigOverride) DeepCopyInto(out *ConfigOverride) {
|
||||
*out = *in
|
||||
return
|
||||
}
|
||||
|
||||
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ConfigOverride.
|
||||
func (in *ConfigOverride) DeepCopy() *ConfigOverride {
|
||||
if in == nil {
|
||||
return nil
|
||||
}
|
||||
out := new(ConfigOverride)
|
||||
in.DeepCopyInto(out)
|
||||
return out
|
||||
}
|
||||
|
||||
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
|
||||
func (in ConfigOverridesSpec) DeepCopyInto(out *ConfigOverridesSpec) {
|
||||
{
|
||||
in := &in
|
||||
*out = make(ConfigOverridesSpec, len(*in))
|
||||
copy(*out, *in)
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ConfigOverridesSpec.
|
||||
func (in ConfigOverridesSpec) DeepCopy() ConfigOverridesSpec {
|
||||
if in == nil {
|
||||
return nil
|
||||
}
|
||||
out := new(ConfigOverridesSpec)
|
||||
in.DeepCopyInto(out)
|
||||
return *out
|
||||
}
|
||||
|
||||
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
|
||||
func (in *DashboardSpec) DeepCopyInto(out *DashboardSpec) {
|
||||
*out = *in
|
||||
|
||||
@@ -231,7 +231,7 @@ func (c *cluster) doOrchestration(rookImage string, cephVersion cephver.CephVers
|
||||
|
||||
if c.Spec.External.Enable {
|
||||
// Apply CRD ConfigOverrideName to the external cluster
|
||||
err = config.SetDefaultAndUserConfigs(c.context, c.Namespace, c.Info, c.Spec.ConfigOverrides)
|
||||
err = config.SetDefaultConfigs(c.context, c.Namespace, c.Info)
|
||||
if err != nil {
|
||||
// Mons are up, so something else is wrong
|
||||
return fmt.Errorf("failed to set Rook and/or user-defined Ceph config options on the external cluster monitors. %+v", err)
|
||||
|
||||
@@ -278,7 +278,7 @@ func (c *ClusterController) configureExternalCephCluster(namespace, name string,
|
||||
}
|
||||
|
||||
// Apply CRD ConfigOverrideName to the external cluster
|
||||
err = config.SetDefaultAndUserConfigs(cluster.context, namespace, cluster.Info, cluster.Spec.ConfigOverrides)
|
||||
err = config.SetDefaultConfigs(cluster.context, namespace, cluster.Info)
|
||||
if err != nil {
|
||||
// Mons are up, so something else is wrong
|
||||
return fmt.Errorf("failed to set Rook and/or user-defined Ceph config options on the external cluster monitors. %+v", err)
|
||||
|
||||
@@ -226,7 +226,7 @@ func (c *Cluster) startMons(targetCount int) error {
|
||||
// only once and do it as early as possible in the mon orchestration.
|
||||
setConfigsNeedsRetry := false
|
||||
if existingCount > 0 {
|
||||
err := config.SetDefaultAndUserConfigs(c.context, c.Namespace, c.ClusterInfo, c.spec.ConfigOverrides)
|
||||
err := config.SetDefaultConfigs(c.context, c.Namespace, c.ClusterInfo)
|
||||
if err != nil {
|
||||
// If we fail here, it could be because the mons are not healthy, and this might be
|
||||
// fixed by updating the mon deployments. Instead of returning error here, log a
|
||||
@@ -247,7 +247,7 @@ func (c *Cluster) startMons(targetCount int) error {
|
||||
// values in the config database. Do this only when the existing count is zero so that
|
||||
// this is only done once when the cluster is created.
|
||||
if existingCount == 0 {
|
||||
err := config.SetDefaultAndUserConfigs(c.context, c.Namespace, c.ClusterInfo, c.spec.ConfigOverrides)
|
||||
err := config.SetDefaultConfigs(c.context, c.Namespace, c.ClusterInfo)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to set Rook and/or user-defined Ceph config options after creating the first mon. %+v", err)
|
||||
}
|
||||
@@ -255,7 +255,7 @@ func (c *Cluster) startMons(targetCount int) error {
|
||||
// Or if we need to retry, only do this when we are on the first iteration of the
|
||||
// loop. This could be in the same if statement as above, but separate it to get a
|
||||
// different error message.
|
||||
err := config.SetDefaultAndUserConfigs(c.context, c.Namespace, c.ClusterInfo, c.spec.ConfigOverrides)
|
||||
err := config.SetDefaultConfigs(c.context, c.Namespace, c.ClusterInfo)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to set Rook and/or user-defined Ceph config options after updating the existing mons. %+v", err)
|
||||
}
|
||||
@@ -269,7 +269,7 @@ func (c *Cluster) startMons(targetCount int) error {
|
||||
}
|
||||
|
||||
if setConfigsNeedsRetry {
|
||||
err := config.SetDefaultAndUserConfigs(c.context, c.Namespace, c.ClusterInfo, c.spec.ConfigOverrides)
|
||||
err := config.SetDefaultConfigs(c.context, c.Namespace, c.ClusterInfo)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to set Rook and/or user-defined Ceph config options after forcefully updating the existing mons. %+v", err)
|
||||
}
|
||||
|
||||
@@ -24,7 +24,6 @@ import (
|
||||
"strings"
|
||||
|
||||
"github.com/coreos/pkg/capnslog"
|
||||
rookceph "github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
|
||||
"github.com/rook/rook/pkg/clusterd"
|
||||
cephconfig "github.com/rook/rook/pkg/daemon/ceph/config"
|
||||
)
|
||||
@@ -100,31 +99,24 @@ func NewFlag(key, value string) string {
|
||||
return fmt.Sprintf("--%s=%s", f, value)
|
||||
}
|
||||
|
||||
// SetDefaultAndUserConfigs sets Rook's desired default configs and the user's override configs from
|
||||
// the CephCluster CRD in the centralized monitor database. This cannot be called before at least
|
||||
// one monitor is established.
|
||||
func SetDefaultAndUserConfigs(
|
||||
// SetDefaultConfigs sets Rook's desired default configs in the centralized monitor database. This
|
||||
// cannot be called before at least one monitor is established.
|
||||
func SetDefaultConfigs(
|
||||
context *clusterd.Context,
|
||||
namespace string,
|
||||
clusterInfo *cephconfig.ClusterInfo,
|
||||
configOverrides rookceph.ConfigOverridesSpec,
|
||||
) error {
|
||||
// ceph.conf is never used. All configurations are made in the centralized mon config database,
|
||||
// or they are specified on the commandline when daemons are called.
|
||||
monStore := GetMonStore(context, namespace)
|
||||
|
||||
if err := monStore.SetAll(DefaultCentralizedConfigs(clusterInfo.CephVersion)); err != nil {
|
||||
if err := monStore.SetAll(DefaultCentralizedConfigs(clusterInfo.CephVersion)...); err != nil {
|
||||
return fmt.Errorf("failed to apply default Ceph configurations. %+v", err)
|
||||
}
|
||||
|
||||
if err := monStore.SetAll(DefaultLegacyConfigs()); err != nil {
|
||||
if err := monStore.SetAll(DefaultLegacyConfigs()...); err != nil {
|
||||
return fmt.Errorf("failed to apply legacy config overrides. %+v", err)
|
||||
}
|
||||
|
||||
// user-specified config overrides from the CRD will go here
|
||||
if err := monStore.SetAll(configOverrides); err != nil {
|
||||
return fmt.Errorf("failed to apply one or more user-specified overrides. %+v", err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
@@ -18,7 +18,6 @@ limitations under the License.
|
||||
package config
|
||||
|
||||
import (
|
||||
rookceph "github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
|
||||
"github.com/rook/rook/pkg/operator/ceph/version"
|
||||
)
|
||||
|
||||
@@ -55,14 +54,14 @@ func DefaultFlags(fsid, mountedKeyringPath string, cephVersion version.CephVersi
|
||||
}
|
||||
|
||||
// makes it possible to be slightly less verbose to create a ConfigOverride here
|
||||
func configOverride(who, option, value string) rookceph.ConfigOverride {
|
||||
return rookceph.ConfigOverride{Who: who, Option: option, Value: value}
|
||||
func configOverride(who, option, value string) Option {
|
||||
return Option{Who: who, Option: option, Value: value}
|
||||
}
|
||||
|
||||
// DefaultCentralizedConfigs returns the default configuration options Rook will set in Ceph's
|
||||
// centralized config store.
|
||||
func DefaultCentralizedConfigs(cephVersion version.CephVersion) rookceph.ConfigOverridesSpec {
|
||||
overrides := []rookceph.ConfigOverride{
|
||||
func DefaultCentralizedConfigs(cephVersion version.CephVersion) []Option {
|
||||
overrides := []Option{
|
||||
configOverride("global", "mon allow pool delete", "true"),
|
||||
}
|
||||
|
||||
@@ -74,7 +73,7 @@ func DefaultCentralizedConfigs(cephVersion version.CephVersion) rookceph.ConfigO
|
||||
if !cephVersion.IsAtLeast(version.CephVersion{Major: 14, Minor: 2, Extra: 1}) {
|
||||
// Set the default log files to empty so they don't bloat containers. Can be changed in
|
||||
// Mimic+ by users if needed.
|
||||
overrides = append(overrides, []rookceph.ConfigOverride{
|
||||
overrides = append(overrides, []Option{
|
||||
configOverride("global", "log file", ""),
|
||||
configOverride("global", "mon cluster log file", ""),
|
||||
}...)
|
||||
@@ -85,8 +84,8 @@ func DefaultCentralizedConfigs(cephVersion version.CephVersion) rookceph.ConfigO
|
||||
|
||||
// DefaultLegacyConfigs need to be added to the Ceph config file until the integration tests can be
|
||||
// made to override these options for the Ceph clusters it creates.
|
||||
func DefaultLegacyConfigs() rookceph.ConfigOverridesSpec {
|
||||
overrides := []rookceph.ConfigOverride{
|
||||
func DefaultLegacyConfigs() []Option {
|
||||
overrides := []Option{
|
||||
// TODO: drop this when FlexVolume is no longer supported
|
||||
configOverride("global", "rbd_default_features", "3"),
|
||||
}
|
||||
|
||||
@@ -19,7 +19,6 @@ package config
|
||||
import (
|
||||
"fmt"
|
||||
|
||||
rookceph "github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
|
||||
"github.com/rook/rook/pkg/clusterd"
|
||||
"github.com/rook/rook/pkg/daemon/ceph/client"
|
||||
)
|
||||
@@ -39,6 +38,18 @@ func GetMonStore(context *clusterd.Context, namespace string) *MonStore {
|
||||
}
|
||||
}
|
||||
|
||||
// Option defines the pieces of information relevant to Ceph configuration options.
|
||||
type Option struct {
|
||||
// Who is the entity(-ies) the option should apply to.
|
||||
Who string
|
||||
|
||||
// Option is the option key
|
||||
Option string
|
||||
|
||||
// Value is the value for the option
|
||||
Value string
|
||||
}
|
||||
|
||||
// Set sets a config in the centralized mon configuration database.
|
||||
// https://docs.ceph.com/docs/master/rados/configuration/ceph-conf/#monitor-configuration-database
|
||||
func (m *MonStore) Set(who, option, value string) error {
|
||||
@@ -54,9 +65,9 @@ func (m *MonStore) Set(who, option, value string) error {
|
||||
|
||||
// SetAll sets all configs from the overrides in the centralized mon configuration database.
|
||||
// See MonStore.Set for more.
|
||||
func (m *MonStore) SetAll(configOverrides rookceph.ConfigOverridesSpec) error {
|
||||
func (m *MonStore) SetAll(options ...Option) error {
|
||||
var errs []error
|
||||
for _, override := range configOverrides {
|
||||
for _, override := range options {
|
||||
err := m.Set(override.Who, override.Option, override.Value)
|
||||
if err != nil {
|
||||
errs = append(errs, err)
|
||||
|
||||
@@ -21,7 +21,6 @@ import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
rookceph "github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
|
||||
"github.com/rook/rook/pkg/clusterd"
|
||||
testop "github.com/rook/rook/pkg/operator/test"
|
||||
exectest "github.com/rook/rook/pkg/util/exec/test"
|
||||
@@ -96,16 +95,14 @@ func TestMonStore_SetAll(t *testing.T) {
|
||||
|
||||
monStore := GetMonStore(ctx, "ns")
|
||||
|
||||
var cfgOverrides rookceph.ConfigOverridesSpec
|
||||
|
||||
cfgOverrides = []rookceph.ConfigOverride{
|
||||
cfgOverrides := []Option{
|
||||
configOverride("global", "debug ms", "10"), // setting w/ spaces converts to underscores
|
||||
configOverride("osd.0", "debug-osd", "20"), // setting w/ dashes converts to underscores
|
||||
configOverride("mds.*", "debug_mds", "15"), // setting w/ underscores remains the same
|
||||
}
|
||||
|
||||
// commands w/ no error
|
||||
e := monStore.SetAll(cfgOverrides)
|
||||
e := monStore.SetAll(cfgOverrides...)
|
||||
assert.NoError(t, e)
|
||||
assert.Len(t, execedCmds, 3)
|
||||
assert.Contains(t, execedCmds[0], " global debug_ms 10 ")
|
||||
@@ -116,7 +113,7 @@ func TestMonStore_SetAll(t *testing.T) {
|
||||
// keep cfgOverrides from last test
|
||||
execInjectErrOnKeyword = "debug_osd"
|
||||
execedCmds = execedCmds[:0] // empty execedCmds slice
|
||||
e = monStore.SetAll(cfgOverrides)
|
||||
e = monStore.SetAll(cfgOverrides...)
|
||||
assert.Error(t, e)
|
||||
// Rook should not return error before trying to set all config overrides
|
||||
assert.Len(t, execedCmds, 3)
|
||||
@@ -125,7 +122,7 @@ func TestMonStore_SetAll(t *testing.T) {
|
||||
// keep cfgOverrides
|
||||
execInjectErrOnKeyword = "debug"
|
||||
execedCmds = execedCmds[:0]
|
||||
e = monStore.SetAll(cfgOverrides)
|
||||
e = monStore.SetAll(cfgOverrides...)
|
||||
assert.Error(t, e)
|
||||
assert.Len(t, execedCmds, 3)
|
||||
}
|
||||
@@ -197,15 +197,6 @@ spec:
|
||||
type: integer
|
||||
placement: {}
|
||||
resources: {}
|
||||
configOverrides:
|
||||
items:
|
||||
properties:
|
||||
who:
|
||||
type: string
|
||||
option:
|
||||
type: string
|
||||
value:
|
||||
type: string
|
||||
required:
|
||||
- mon
|
||||
additionalPrinterColumns:
|
||||
|
||||
Reference in new issue
Block a user