Merge pull request #3573 from SUSE/mon-kv-store

Ceph: override configs in CRD
This commit is contained in:
Travis Nielsen authored and GitHub committed 2019-08-29 19:04:16 -06:00
commit 5eab7ca658
34 files changed
+811 -712

No files matched your search

+7 -3
View File
@@ -419,9 +419,14 @@ ceph osd pool set rbd pg_num 512
```
## Custom ceph.conf Settings
**WARNING:** The advised method for controlling Ceph configuration is to manually use the Ceph CLI
or the Ceph dashboard because this offers the most flexibility. If configs must be set as part of
Rook, the `configOverrides` section of the CephCluster resource is recommended. Read the
[CephCluster CRD docs](ceph-cluster-crd.md#ceph-config-overrides) first before considering this
advanced method.
With Rook the full swath of
[Ceph settings](http://docs.ceph.com/docs/kraken/rados/configuration/) are available
[Ceph settings](http://docs.ceph.com/docs/master/rados/configuration/) are available
to use on your storage cluster. When we supply Rook with a ceph.conf file those
settings will be propagated to all Mon, OSD, MDS, and RGW daemons to use.
@@ -431,7 +436,6 @@ daemons not to change the weight of OSDs on startup.
**WARNING**: Modify Ceph settings carefully. You are leaving the sandbox tested by Rook.
Changing the settings could result in unhealthy daemons or even data loss if used incorrectly.
### Kubernetes
When the Rook Operator creates a cluster, a placeholder ConfigMap is created that
will allow you to override Ceph configuration settings. When the daemon pods are started, the
settings specified in this ConfigMap will be merged with the default settings
@@ -492,7 +496,7 @@ the validity of the settings is your responsibility.
## OSD CRUSH Settings
A useful view of the [CRUSH Map](http://docs.ceph.com/docs/kraken/rados/operations/crush-map/)
A useful view of the [CRUSH Map](http://docs.ceph.com/docs/master/rados/operations/crush-map/)
is generated with the following command:
```bash
+56
View File
@@ -286,6 +286,62 @@ For more information on resource requests/limits see the official Kubernetes doc
- `cpu`: Limit for CPU (example: one CPU core `1`, 50% of one CPU core `500m`).
- `memory`: Limit for Memory (example: one gigabyte of memory `1Gi`, half a gigabyte of memory `512Mi`).
### Ceph config overrides
Users are advised to use Ceph's CLI or dashboard to configure Ceph; however, some users may want to
specify some settings they want set in the `CephCluster` manifest. This is possible using the
`configOverrides` section. Config overrides follow the same syntax as [Ceph's documentation for
setting config values](https://docs.ceph.com/docs/master/rados/configuration/ceph-conf/#commands),
and configuration is stored in Ceph's centralized configuration database.
Each item in `configOverrides` has `who`, `option`, and `value` properties. `who` can be `global` to
affect all Ceph daemons, a daemon type, an individual Ceph daemon, or a glob matching multiple
daemons. It may also use a
[mask](https://docs.ceph.com/docs/master/rados/configuration/ceph-conf/#sections-and-masks).
`option` is the configuration option to be overridden, and `value` is the value to set on the
configuration option. All properties are strings.
Rook will only set configuration overrides once when an orchestration is run. For example, on node
addition or removal, when disks change on a node, or when the operator starts (or restarts). This
means that users can change Ceph's configuration from the CLI or dashboard as desired without Rook
constantly fighting to control the setting. This is particularly valuable in situations where values
need to be changed to debug errors or test changes to tuning parameters.
**WARNING:** Remember that Rook will set these overrides any time the operator restarts, so user
changes via Ceph's CLI or dashboard to values set here will not persist forever.
Some examples:
```yaml
configOverrides:
- who: global
option: log_file
value: /var/log/custom-log-dir
- who: mon
option: mon_compact_on_start
value: "true"
- who: mgr.a
option: mgr_stats_period
value: "10"
- who: osd.*
option: osd_memory_target
value: "2147483648"
- who: osd/rack:foo
option: debug_ms
value: "20"
```
#### Advanced config
Setting configs in the Ceph mons' centralized database this way requires that at least one mon be
available for the configs to be set. Ceph may also have a small number of very advanced settings
that aren't able to be modified easily in the configuration database. In order to set configurations
before monitors are available or to set problematic configuration settings, the
`rook-config-override` ConfigMap exists, and the `config` field can be set with the contents of a
`ceph.conf` file. It is highly recommended that this only be used when absolutely necessary and that
the `config` be reset to an empty string if/when the configurations are no longer necessary, as this
will make the Ceph cluster less configurable from the CLI and dashboard and may make future tuning
or debugging difficult. Read more about this in the
[advanced configuration docs](ceph-advanced-configuration.md#custom-cephconf-settings).
## Samples
Here are several samples for configuring Ceph clusters. Each of the samples must also include the namespace and corresponding access granted for management by the Ceph operator. See the [common cluster resources](#common-cluster-resources) below.
+46
View File
@@ -272,6 +272,52 @@ At this point, your Rook operator should be running version `rook/ceph:v1.1.0`
Verify the Ceph cluster's health using the [health verification section](#health-verification).
### 6. (Optional) Migrate config overrides from ConfigMap to the CephCluster resource
If there are Ceph configuration overrides set in the `config` field of the ConfigMap
`rook-config-override`, it is now possible to migrate those configs manually from the ConfigMap to
the newly-added `configOverrides` section of the `CephCluster` custom resource. This is not required
but may be preferable, as the values set in `configOverrides` can be temporarily overridden by the
user from the toolbox pod as needed in debug/failure scenarios. Values are reset if the operator
restarts.
Ceph config file headers correspond to
the `who` field of each config override as part of the section. For example, the `[global]` section
corresponds to `who: global`, and `[osd.0]` corresponds to `who: osd.0`.
```sh
# List the contents of the override ConfigMap
$> kubectl --namespace $ROOK_NAMESPACE describe configmap rook-ceph-override -o yaml
Name: rook-config-override
Namespace: rook-ceph
Labels: <none>
Annotations: <none>
Data
====
config:
----
[global]
debug_ms = 1/5
[osd.0]
debug_osd = 10
Events: <none>
# Edit the Ceph cluster resource, and add the overrides from the ConfigMap
$> kubectl --namespace $ROOK_NAMESPACE edit cephcluster $ROOK_NAMESPACE
# ...
spec:
# ...
configOverrides:
- who: global
option: debug_ms
value: 1/5
- who: osd.0
option: debug_osd
value: "10"
who: osd.0
# ....
```
# Ceph Version Upgrades
Rook 1.0 was the last Rook release which will support Ceph's Luminous (v12.x.x) version. Users are
+3 -1
View File
@@ -46,9 +46,11 @@ an example usage
[monitor settings for more detail](Documentation/ceph-cluster-crd.md#mon-settings).
- Ceph OSDs can be created by using StorageClassDeviceSet. See docs on [Storage Class Device Sets](Documentation/ceph-cluster-crd.md#storage-class-device-sets).
- Rook can now connect to an external cluster, for more info about external cluster [read the design](https://github.com/rook/rook/blob/master/design/ceph-external-cluster.md) as well as the documentation [Ceph External cluster](Documentation/ceph-cluster-crd.md#external-cluster)
- Added a new property in `storageClassDeviceSets` named `portable`:
- Added a new property in `storageClassDeviceSets` named `portable`:
- If `true`, the OSDs will be allowed to move between nodes during failover. This requires a storage class that supports portability (e.g. `aws-ebs`, but not the local storage provisioner).
- If `false`, the OSDs will be assigned to a node permanently. Rook will configure Ceph's CRUSH map to support the portability.
- The Ceph cluster custom resource now contains a `configOverrides` section where users can specify
configuration changes to Ceph which Rook should apply.
### YugabyteDB
@@ -143,3 +143,15 @@ spec:
# storeType: filestore
# - name: "172.17.4.301"
# deviceFilter: "^sd."
# Override Ceph configs. These can be overridden by the user at runtime using the Ceph CLI, and
# Rook will set them back to the values set here when the operator restarts.
configOverrides:
- who: global
option: debug ms
value: "1/5"
- who: osd.0
option: debug_osd
value: "10"
- who: mon.a
option: debug-mon
value: "10"
@@ -141,6 +141,15 @@ spec:
type: integer
placement: {}
resources: {}
configOverrides:
items:
properties:
who:
type: string
option:
type: string
value:
type: string
additionalPrinterColumns:
- name: DataDirHostPath
type: string
+15
View File
@@ -69,6 +69,9 @@ type ClusterSpec struct {
// The path on the host where config and data can be persisted.
DataDirHostPath string `json:"dataDirHostPath,omitempty"`
// Ceph config overrides to apply.
ConfigOverrides ConfigOverridesSpec `json:"configOverrides,omitempty"`
// A spec for mon related options
Mon MonSpec `json:"mon,omitempty"`
@@ -148,6 +151,18 @@ const (
ClusterStateError ClusterState = "Error"
)
// ConfigOverridesSpec defines how Ceph configurations can be overridden by Rook.
type ConfigOverridesSpec []ConfigOverride
// ConfigOverride defines the syntax for overriding a Ceph config. This translates to a
// `ceph config set <who> <option> <value>` (or similar) command.
// See Ceph docs: https://docs.ceph.com/docs/master/rados/configuration/ceph-conf/#monitor-configuration-database
type ConfigOverride struct {
Who string `json:"who"`
Option string `json:"option"`
Value string `json:"value"`
}
type MonSpec struct {
Count int `json:"count,omitempty"`
AllowMultiplePerNode bool `json:"allowMultiplePerNode,omitempty"`
@@ -479,6 +479,11 @@ func (in *ClusterSpec) DeepCopyInto(out *ClusterSpec) {
(*out)[key] = *val.DeepCopy()
}
}
if in.ConfigOverrides != nil {
in, out := &in.ConfigOverrides, &out.ConfigOverrides
*out = make(ConfigOverridesSpec, len(*in))
copy(*out, *in)
}
in.Mon.DeepCopyInto(&out.Mon)
out.RBDMirroring = in.RBDMirroring
in.Dashboard.DeepCopyInto(&out.Dashboard)
@@ -518,6 +523,42 @@ func (in *ClusterStatus) DeepCopy() *ClusterStatus {
return out
}
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *ConfigOverride) DeepCopyInto(out *ConfigOverride) {
*out = *in
return
}
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ConfigOverride.
func (in *ConfigOverride) DeepCopy() *ConfigOverride {
if in == nil {
return nil
}
out := new(ConfigOverride)
in.DeepCopyInto(out)
return out
}
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in ConfigOverridesSpec) DeepCopyInto(out *ConfigOverridesSpec) {
{
in := &in
*out = make(ConfigOverridesSpec, len(*in))
copy(*out, *in)
return
}
}
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ConfigOverridesSpec.
func (in ConfigOverridesSpec) DeepCopy() ConfigOverridesSpec {
if in == nil {
return nil
}
out := new(ConfigOverridesSpec)
in.DeepCopyInto(out)
return *out
}
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *DashboardSpec) DeepCopyInto(out *DashboardSpec) {
*out = *in
+7 -2
View File
@@ -43,6 +43,12 @@ const (
cephConnectionTimeout = "15" // in seconds
)
// CephConfFilePath returns the location to the cluster's config file in the operator container.
func CephConfFilePath(configDir, clusterName string) string {
confFile := fmt.Sprintf("%s.config", clusterName)
return path.Join(configDir, clusterName, confFile)
}
// FinalizeCephCommandArgs builds the command line to be called
func FinalizeCephCommandArgs(command string, args []string, configDir, clusterName string) (string, []string) {
// the rbd client tool does not support the '--connect-timeout' option
@@ -68,11 +74,10 @@ func FinalizeCephCommandArgs(command string, args []string, configDir, clusterNa
}
// Append the args to find the config and keyring
confFile := fmt.Sprintf("%s.config", clusterName)
keyringFile := fmt.Sprintf("%s.keyring", AdminUsername)
configArgs := []string{
fmt.Sprintf("--cluster=%s", clusterName),
fmt.Sprintf("--conf=%s", path.Join(configDir, clusterName, confFile)),
fmt.Sprintf("--conf=%s", CephConfFilePath(configDir, clusterName)),
fmt.Sprintf("--keyring=%s", path.Join(configDir, clusterName, keyringFile)),
}
return command, append(args, configArgs...)
+17 -26
View File
@@ -36,10 +36,6 @@ import (
var logger = capnslog.NewPackageLogger("github.com/rook/rook", "cephconfig")
const (
// DefaultConfigDir is the default dir where Ceph stores its configs
DefaultConfigDir = "/etc/ceph"
// DefaultConfigFile is the default name of the file where Ceph stores its configs
DefaultConfigFile = "ceph.conf"
// DefaultKeyringFile is the default name of the file where Ceph stores its keyring info
DefaultKeyringFile = "keyring"
// Msgr2port is the listening port of the messenger v2 protocol
@@ -48,6 +44,16 @@ const (
msgr2Prefix = "v2:"
)
var (
// DefaultConfigDir is the default dir where Ceph stores its configs. Can be overridden for unit
// tests.
DefaultConfigDir = "/etc/ceph"
// DefaultConfigFile is the default name of the file where Ceph stores its configs. Can be
// overridden for unit tests.
DefaultConfigFile = "ceph.conf"
)
// GlobalConfig represents the [global] sections of Ceph's config file.
type GlobalConfig struct {
EnableExperimental string `ini:"enable experimental unrecoverable data corrupting features,omitempty"`
@@ -109,26 +115,27 @@ func GetConfFilePath(root, clusterName string) string {
// GenerateAdminConnectionConfig calls GenerateAdminConnectionConfigWithSettings with no settings
// overridden.
func GenerateAdminConnectionConfig(context *clusterd.Context, cluster *ClusterInfo) error {
func GenerateAdminConnectionConfig(context *clusterd.Context, cluster *ClusterInfo) (string, error) {
return GenerateAdminConnectionConfigWithSettings(context, cluster, nil)
}
// GenerateAdminConnectionConfigWithSettings generates a Ceph config and keyring which will allow
// the daemon to connect as an admin. Default config file settings can be overridden by specifying
// some subset of settings.
func GenerateAdminConnectionConfigWithSettings(context *clusterd.Context, cluster *ClusterInfo, settings *CephConfig) error {
func GenerateAdminConnectionConfigWithSettings(context *clusterd.Context, cluster *ClusterInfo, settings *CephConfig) (string, error) {
root := path.Join(context.ConfigDir, cluster.Name)
keyringPath := path.Join(root, fmt.Sprintf("%s.keyring", client.AdminUsername))
err := writeKeyring(AdminKeyring(cluster), keyringPath)
if err != nil {
return fmt.Errorf("failed to write keyring to %s. %+v", root, err)
return "", fmt.Errorf("failed to write keyring to %s. %+v", root, err)
}
if _, err = GenerateConfigFile(context, cluster, root, client.AdminUsername, keyringPath, settings, nil); err != nil {
return fmt.Errorf("failed to write config to %s. %+v", root, err)
filePath, err := GenerateConfigFile(context, cluster, root, client.AdminUsername, keyringPath, settings, nil)
if err != nil {
return "", fmt.Errorf("failed to write config to %s. %+v", root, err)
}
logger.Infof("generated admin config in %s", root)
return nil
return filePath, nil
}
// GenerateConfigFile generates and writes a config file to disk.
@@ -149,15 +156,6 @@ func GenerateConfigFile(context *clusterd.Context, cluster *ClusterInfo, pathRoo
return "", fmt.Errorf("failed to add admin client config section, %+v", err)
}
// if there's a config file override path given, process the given config file
if context.ConfigFileOverride != "" {
err := configFile.Append(context.ConfigFileOverride)
if err != nil {
// log the config file override failure as a warning, but proceed without it
logger.Warningf("failed to add config file override from '%s': %+v", context.ConfigFileOverride, err)
}
}
// write the entire config to disk
filePath := GetConfFilePath(pathRoot, cluster.Name)
logger.Infof("writing config file %s", filePath)
@@ -165,13 +163,6 @@ func GenerateConfigFile(context *clusterd.Context, cluster *ClusterInfo, pathRoo
return "", fmt.Errorf("failed to save config file %s. %+v", filePath, err)
}
// copy the config to /etc/ceph/ceph.conf
defaultPath := DefaultConfigFilePath()
logger.Infof("copying config to %s", defaultPath)
if err := configFile.SaveTo(defaultPath); err != nil {
logger.Warningf("failed to save config file %s. %+v", defaultPath, err)
}
return filePath, nil
}
+1 -14
View File
@@ -84,19 +84,9 @@ func TestGenerateConfigFile(t *testing.T) {
}
defer os.RemoveAll(configDir)
// set up a config file override (in the config dir so it also gets cleaned up)
configFileOverride := filepath.Join(configDir, "override.conf")
overrideContents := `[global]
debug bluestore = 1234`
err = ioutil.WriteFile(configFileOverride, []byte(overrideContents), 0644)
if err != nil {
t.Fatalf("failed to create config file override at '%s': %+v", configFileOverride, err)
}
// create mocked cluster context and info
context := &clusterd.Context{
ConfigDir: configDir,
ConfigFileOverride: configFileOverride,
ConfigDir: configDir,
}
clusterInfo := &ClusterInfo{
FSID: "myfsid",
@@ -118,9 +108,6 @@ debug bluestore = 1234`
actualConf, err := ini.Load(configFilePath)
assert.Nil(t, err)
verifyConfigValue(t, actualConf, "global", "fsid", clusterInfo.FSID)
// verify the content of the config file override successfully overwrote the default generated config
verifyConfigValue(t, actualConf, "global", "debug bluestore", "1234")
}
func verifyConfig(t *testing.T, cephConfig *CephConfig, cluster *ClusterInfo, loggingLevel int) {
-36
View File
@@ -1,36 +0,0 @@
/*
Copyright 2018 The Rook Authors. All rights reserved.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package config
import "k8s.io/api/core/v1"
const (
// DefaultConfigMountName is the name of the volume mount used to mount the default Ceph config
DefaultConfigMountName = "ceph-default-config-dir"
)
// DefaultConfigVolume returns an empty volume used to store Ceph's config at the default path
// in containers.
func DefaultConfigVolume() v1.Volume {
return v1.Volume{Name: DefaultConfigMountName,
VolumeSource: v1.VolumeSource{EmptyDir: &v1.EmptyDirVolumeSource{}}}
}
// DefaultConfigMount returns a volume mount to Ceph's default config path.
func DefaultConfigMount() v1.VolumeMount {
return v1.VolumeMount{Name: DefaultConfigMountName, MountPath: DefaultConfigDir}
}
-28
View File
@@ -1,28 +0,0 @@
/*
Copyright 2018 The Rook Authors. All rights reserved.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package config
import (
"testing"
"github.com/stretchr/testify/assert"
)
func TestDefaultVolumes(t *testing.T) {
// The name the volume uses must be the same as the one the mount uses
assert.Equal(t, DefaultConfigVolume().Name, DefaultConfigMount().Name)
}
+17 -1
View File
@@ -18,6 +18,7 @@ package osd
import (
"fmt"
"io/ioutil"
"os"
"path"
"regexp"
@@ -105,9 +106,24 @@ func Provision(context *clusterd.Context, agent *OsdAgent) error {
cephConfig.GlobalConfig.CrushLocation = agent.location
// write the latest config to the config dir
if err := cephconfig.GenerateAdminConnectionConfigWithSettings(context, agent.cluster, cephConfig); err != nil {
confFilePath, err := cephconfig.GenerateAdminConnectionConfigWithSettings(context, agent.cluster, cephConfig)
if err != nil {
return fmt.Errorf("failed to write connection config. %+v", err)
}
src, err := ioutil.ReadFile(confFilePath)
if err != nil {
return fmt.Errorf("failed to copy connection config to /etc/ceph. failed to read the connection config. %+v", err)
}
err = ioutil.WriteFile(cephconfig.DefaultConfigFilePath(), src, 0444)
if err != nil {
return fmt.Errorf("failed to copy connection config to /etc/ceph. failed to write %s. %+v", cephconfig.DefaultConfigFilePath(), err)
}
dst, err := ioutil.ReadFile(cephconfig.DefaultConfigFilePath())
if err == nil {
logger.Debugf("config file @ %s: %s", cephconfig.DefaultConfigFilePath(), dst)
} else {
logger.Warningf("wrote and copied config file but failed to read it back from %s for logging. %+v", cephconfig.DefaultConfigFilePath(), err)
}
logger.Infof("discovering hardware")
var rawDevices []*sys.LocalDisk
+5
View File
@@ -23,6 +23,7 @@ import (
"testing"
"github.com/rook/rook/pkg/clusterd"
cephconfig "github.com/rook/rook/pkg/daemon/ceph/config"
"github.com/rook/rook/pkg/operator/ceph/cluster/osd/config"
exectest "github.com/rook/rook/pkg/util/exec/test"
"github.com/rook/rook/pkg/util/sys"
@@ -67,6 +68,10 @@ func TestRunDaemon(t *testing.T) {
defer os.RemoveAll(configDir)
os.MkdirAll(configDir, 0755)
defBkp := cephconfig.DefaultConfigDir
cephconfig.DefaultConfigDir = configDir
defer func() { cephconfig.DefaultConfigDir = defBkp }()
agent, _, context := createTestAgent(t, "none", configDir, "node5375", &config.StoreConfig{StoreType: config.Bluestore})
agent.devices[0].IsFilter = true
+1 -1
View File
@@ -123,7 +123,7 @@ func CreateOrLoadClusterInfo(context *clusterd.Context, namespace string, ownerR
// WriteConnectionConfig save monitor connection config to disk
func WriteConnectionConfig(context *clusterd.Context, clusterInfo *cephconfig.ClusterInfo) error {
// write the latest config to the config dir
if err := cephconfig.GenerateAdminConnectionConfig(context, clusterInfo); err != nil {
if _, err := cephconfig.GenerateAdminConnectionConfig(context, clusterInfo); err != nil {
return fmt.Errorf("failed to write connection config. %+v", err)
}
+42 -1
View File
@@ -220,12 +220,46 @@ func (c *Cluster) startMons(targetCount int) error {
return fmt.Errorf("failed to assign pods to mons. %+v", err)
}
// The centralized mon config database can only be used if there is at least one mon
// operational. If we are starting mons, and one is already up, then there is a cluster already
// created, and we can immediately set values in the config database. The goal is to set configs
// only once and do it as early as possible in the mon orchestration.
setConfigsNeedsRetry := false
if existingCount > 0 {
err := config.SetDefaultAndUserConfigs(c.context, c.Namespace, c.ClusterInfo, c.spec.ConfigOverrides)
if err != nil {
// If we fail here, it could be because the mons are not healthy, and this might be
// fixed by updating the mon deployments. Instead of returning error here, log a
// warning, and retry setting this later.
setConfigsNeedsRetry = true
logger.Warningf("failed to set Rook and/or user-defined Ceph config options before starting mons; will retry after starting mons. %+v", err)
}
}
if existingCount < len(mons) {
// Start the new mons one at a time
for i := existingCount; i < targetCount; i++ {
if err := c.ensureMonsRunning(mons, i, targetCount, true); err != nil {
return err
}
// If this is the first mon being created, we have to wait until it is created to set
// values in the config database. Do this only when the existing count is zero so that
// this is only done once when the cluster is created.
if existingCount == 0 {
err := config.SetDefaultAndUserConfigs(c.context, c.Namespace, c.ClusterInfo, c.spec.ConfigOverrides)
if err != nil {
return fmt.Errorf("failed to set Rook and/or user-defined Ceph config options after creating the first mon. %+v", err)
}
} else if setConfigsNeedsRetry && i == existingCount {
// Or if we need to retry, only do this when we are on the first iteration of the
// loop. This could be in the same if statement as above, but separate it to get a
// different error message.
err := config.SetDefaultAndUserConfigs(c.context, c.Namespace, c.ClusterInfo, c.spec.ConfigOverrides)
if err != nil {
return fmt.Errorf("failed to set Rook and/or user-defined Ceph config options after updating the existing mons. %+v", err)
}
}
}
} else {
// Ensure all the expected mon deployments exist, but don't require full quorum to continue
@@ -233,6 +267,13 @@ func (c *Cluster) startMons(targetCount int) error {
if err := c.ensureMonsRunning(mons, lastMonIndex, targetCount, false); err != nil {
return err
}
if setConfigsNeedsRetry {
err := config.SetDefaultAndUserConfigs(c.context, c.Namespace, c.ClusterInfo, c.spec.ConfigOverrides)
if err != nil {
return fmt.Errorf("failed to set Rook and/or user-defined Ceph config options after forcefully updating the existing mons. %+v", err)
}
}
}
// Enable Ceph messenger 2 protocol on Nautilus
@@ -735,7 +776,7 @@ func (c *Cluster) updateMon(m *monConfig, d *apps.Deployment) error {
currentCephVersion, err := client.LeastUptodateDaemonVersion(c.context, c.ClusterInfo.Name, daemonType)
if err != nil {
logger.Warningf("failed to retrieve current ceph %s version. %+v", daemonType, err)
logger.Debug("could not detect ceph version during update, this is likely an initial bootstrap, proceeding with c.clusterInfo.CephVersion")
logger.Debug("could not detect ceph version during update, this is likely an initial bootstrap, proceeding with %+v", c.ClusterInfo.CephVersion)
cephVersionToUse = c.ClusterInfo.CephVersion
} else {
logger.Debugf("current cluster version for monitors before upgrading is: %+v", currentCephVersion)
+4 -2
View File
@@ -188,8 +188,10 @@ func (c *Cluster) makeMonPod(monConfig *monConfig) *v1.Pod {
c.makeMonDaemonContainer(monConfig),
},
RestartPolicy: v1.RestartPolicyAlways,
Volumes: opspec.DaemonVolumesBase(monConfig.DataPathMap, keyringStoreName),
HostNetwork: c.Network.IsHost(),
// we decide later whether to use a PVC volume or host volumes for mons, so only populate
// the base volumes at this point.
Volumes: opspec.DaemonVolumesBase(monConfig.DataPathMap, keyringStoreName),
HostNetwork: c.Network.IsHost(),
}
if c.Network.IsHost() {
podSpec.DNSPolicy = v1.DNSClusterFirstWithHostNet
+12 -6
View File
@@ -101,9 +101,9 @@ func (c *Cluster) makeJob(osdProps osdProperties) (*batch.Job, error) {
func (c *Cluster) makeDeployment(osdProps osdProperties, osd OSDInfo) (*apps.Deployment, error) {
replicaCount := int32(1)
volumeMounts := opspec.CephVolumeMounts()
configVolumeMounts := opspec.RookVolumeMounts()
volumes := opspec.PodVolumes(c.dataDirHostPath, c.Namespace)
volumeMounts := opspec.CephVolumeMounts(false)
configVolumeMounts := opspec.RookVolumeMounts(false)
volumes := opspec.PodVolumes(c.dataDirHostPath, c.Namespace, false)
failureDomainValue := osdProps.crushHostname
var dataDir string
@@ -387,7 +387,9 @@ func (c *Cluster) provisionPodTemplateSpec(osdProps osdProperties, restart v1.Re
copyBinariesVolume, copyBinariesContainer := c.getCopyBinariesContainer()
volumes := append(opspec.PodVolumes(c.dataDirHostPath, c.Namespace), copyBinariesVolume)
// ceph-volume is currently set up to use /etc/ceph/ceph.conf; this means no user config
// overrides will apply to ceph-volume, but this is unnecessary anyway
volumes := append(opspec.PodVolumes(c.dataDirHostPath, c.Namespace, true), copyBinariesVolume)
// by default, don't define any volume config unless it is required
if len(osdProps.devices) > 0 || osdProps.selection.DeviceFilter != "" || osdProps.selection.GetUseAllDevices() || osdProps.metadataDevice != "" {
@@ -422,8 +424,10 @@ func (c *Cluster) provisionPodTemplateSpec(osdProps osdProperties, restart v1.Re
podSpec := v1.PodSpec{
ServiceAccountName: serviceAccountName,
Containers: []v1.Container{
InitContainers: []v1.Container{
*copyBinariesContainer,
},
Containers: []v1.Container{
c.provisionOSDContainer(osdProps, copyBinariesContainer.VolumeMounts[0]),
},
RestartPolicy: restart,
@@ -574,7 +578,9 @@ func (c *Cluster) provisionOSDContainer(osdProps osdProperties, copyBinariesMoun
devMountNeeded = true
}
volumeMounts := append(opspec.CephVolumeMounts(), copyBinariesMount)
// ceph-volume is currently set up to use /etc/ceph/ceph.conf; this means no user config
// overrides will apply to ceph-volume, but this is unnecessary anyway
volumeMounts := append(opspec.CephVolumeMounts(true), copyBinariesMount)
if devMountNeeded {
devMount := v1.VolumeMount{Name: "devices", MountPath: "/dev"}
volumeMounts = append(volumeMounts, devMount)
+15 -17
View File
@@ -48,11 +48,12 @@ func TestPodContainer(t *testing.T) {
c, err := cluster.provisionPodTemplateSpec(osdProps, v1.RestartPolicyAlways)
assert.NotNil(t, c)
assert.Nil(t, err)
assert.Equal(t, 2, len(c.Spec.Containers))
container := c.Spec.Containers[0]
assert.Equal(t, 1, len(c.Spec.InitContainers))
assert.Equal(t, 1, len(c.Spec.Containers))
container := c.Spec.InitContainers[0]
logger.Infof("container: %+v", container)
assert.Equal(t, "copy-binaries", container.Args[0])
container = c.Spec.Containers[1]
container = c.Spec.Containers[0]
assert.Equal(t, "/rook/tini", container.Command[0])
assert.Equal(t, "--", container.Args[0])
assert.Equal(t, "/rook/rook", container.Args[1])
@@ -112,17 +113,16 @@ func testPodDevices(t *testing.T, dataDir, deviceName string, allDevices bool) {
assert.Equal(t, "node1", deployment.Spec.Template.Spec.NodeSelector[v1.LabelHostname])
assert.Equal(t, v1.RestartPolicyAlways, deployment.Spec.Template.Spec.RestartPolicy)
if devMountNeeded && len(dataDir) > 0 {
assert.Equal(t, 6, len(deployment.Spec.Template.Spec.Volumes))
assert.Equal(t, 5, len(deployment.Spec.Template.Spec.Volumes))
}
if devMountNeeded && len(dataDir) == 0 {
assert.Equal(t, 6, len(deployment.Spec.Template.Spec.Volumes))
assert.Equal(t, 5, len(deployment.Spec.Template.Spec.Volumes))
}
if !devMountNeeded && len(dataDir) > 0 {
assert.Equal(t, 2, len(deployment.Spec.Template.Spec.Volumes))
assert.Equal(t, 1, len(deployment.Spec.Template.Spec.Volumes))
}
assert.Equal(t, "rook-data", deployment.Spec.Template.Spec.Volumes[0].Name)
assert.Equal(t, "ceph-default-config-dir", deployment.Spec.Template.Spec.Volumes[1].Name)
assert.Equal(t, appName, deployment.Spec.Template.ObjectMeta.Name)
assert.Equal(t, appName, deployment.Spec.Template.ObjectMeta.Labels["app"])
@@ -133,7 +133,7 @@ func testPodDevices(t *testing.T, dataDir, deviceName string, allDevices bool) {
initCont := deployment.Spec.Template.Spec.InitContainers[0]
assert.Equal(t, "rook/rook:myversion", initCont.Image)
assert.Equal(t, "config-init", initCont.Name)
assert.Equal(t, 4, len(initCont.VolumeMounts))
assert.Equal(t, 3, len(initCont.VolumeMounts))
assert.Equal(t, 1, len(deployment.Spec.Template.Spec.Containers))
cont := deployment.Spec.Template.Spec.Containers[0]
@@ -198,7 +198,7 @@ func TestStorageSpecDevicesAndDirectories(t *testing.T) {
assert.Nil(t, err)
// pod spec should have a volume for the given dir in the main container and the init container
podSpec := deployment.Spec.Template.Spec
assert.Equal(t, 6, len(podSpec.Volumes))
assert.Equal(t, 5, len(podSpec.Volumes))
require.Equal(t, 1, len(podSpec.Containers))
cont := podSpec.Containers[0]
assert.Equal(t, 5, len(cont.VolumeMounts))
@@ -209,12 +209,11 @@ func TestStorageSpecDevicesAndDirectories(t *testing.T) {
require.Equal(t, 2, len(podSpec.InitContainers))
initCont := podSpec.InitContainers[0]
assert.Equal(t, 5, len(initCont.VolumeMounts))
assert.Equal(t, 4, len(initCont.VolumeMounts))
assert.Equal(t, "/var/lib/rook", initCont.VolumeMounts[0].MountPath)
assert.Equal(t, "/etc/ceph", initCont.VolumeMounts[1].MountPath)
assert.Equal(t, "/var/log/ceph", initCont.VolumeMounts[2].MountPath)
assert.Equal(t, "/etc/rook/config", initCont.VolumeMounts[3].MountPath)
assert.Equal(t, "/my/root/path", initCont.VolumeMounts[4].MountPath)
assert.Equal(t, "/my/root/path", initCont.VolumeMounts[3].MountPath)
// the default osd created on a node will be under /var/lib/rook, which won't need an extra mount
osd = OSDInfo{
@@ -227,7 +226,7 @@ func TestStorageSpecDevicesAndDirectories(t *testing.T) {
assert.Nil(t, err)
// pod spec should have a volume for the given dir in the main container and the init container
podSpec = deployment.Spec.Template.Spec
assert.Equal(t, 5, len(podSpec.Volumes))
assert.Equal(t, 4, len(podSpec.Volumes))
require.Equal(t, 1, len(podSpec.Containers))
cont = podSpec.Containers[0]
require.Equal(t, 4, len(cont.VolumeMounts))
@@ -238,11 +237,10 @@ func TestStorageSpecDevicesAndDirectories(t *testing.T) {
require.Equal(t, 2, len(podSpec.InitContainers))
initCont = podSpec.InitContainers[0]
require.Equal(t, 4, len(initCont.VolumeMounts))
require.Equal(t, 3, len(initCont.VolumeMounts))
assert.Equal(t, "/var/lib/rook", initCont.VolumeMounts[0].MountPath)
assert.Equal(t, "/etc/ceph", initCont.VolumeMounts[1].MountPath)
assert.Equal(t, "/var/log/ceph", initCont.VolumeMounts[2].MountPath)
assert.Equal(t, "/etc/rook/config", initCont.VolumeMounts[3].MountPath)
}
func TestStorageSpecConfig(t *testing.T) {
@@ -299,9 +297,9 @@ func TestStorageSpecConfig(t *testing.T) {
assert.NotNil(t, job)
assert.Nil(t, err)
assert.Equal(t, "rook-ceph-osd-prepare-node1", job.ObjectMeta.Name)
container := job.Spec.Template.Spec.Containers[0]
container := job.Spec.Template.Spec.InitContainers[0]
assert.NotNil(t, container)
container = job.Spec.Template.Spec.Containers[1]
container = job.Spec.Template.Spec.Containers[0]
assert.NotNil(t, container)
verifyEnvVar(t, container.Env, "ROOK_OSD_STORE", "bluestore", true)
verifyEnvVar(t, container.Env, "ROOK_OSD_DATABASE_SIZE", "10", true)
+104 -126
View File
@@ -25,7 +25,11 @@ import (
"github.com/coreos/pkg/capnslog"
"github.com/go-ini/ini"
rookceph "github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
"github.com/rook/rook/pkg/clusterd"
cephconfig "github.com/rook/rook/pkg/daemon/ceph/config"
"github.com/rook/rook/pkg/operator/k8sutil"
"k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
@@ -81,22 +85,6 @@ var (
ContainerPostStartCmd = []string{"chown", "--recursive", "--verbose", chownUserGroup, VarLogCephDir}
)
// Config is a Ceph config struct containing zero or more Ceph config sections.
type Config struct {
sections map[string]*Section
sectionOrder []string
context *clusterd.Context
namespace string
ownerRef *metav1.OwnerReference
dataDirHostPath string
}
// A Section is a collection of zero or more Ceph config key-value pairs.
type Section struct {
configs map[string]string
configOrder []string // allows us to make an ordered dict
}
// normalizeKey converts a key in any format to a key with underscores.
//
// The internal representation of Ceph config keys uses underscores only, where Ceph supports both
@@ -107,98 +95,6 @@ func normalizeKey(key string) string {
return strings.Replace(strings.Replace(key, " ", "_", -1), "-", "_", -1)
}
// NewConfig returns a new, blank Ceph Config.
func NewConfig() *Config {
return &Config{
sections: make(map[string]*Section),
sectionOrder: make([]string, 0, 1), // capacity=1 b/c usually we'll only create [global]
}
}
// Section returns a Ceph config Section with the given name from the Config. If the section does
// not exist, a new empty section is created and returned for the caller's use.
func (c *Config) Section(name string) *Section {
s, ok := c.sections[name]
if !ok {
s = &Section{
configs: make(map[string]string),
configOrder: []string{},
}
c.sections[name] = s
c.sectionOrder = append(c.sectionOrder, name)
}
return s
}
// Set sets the Ceph config key to the given value. Set accepts Ceph keys in both
// space-separated-words and underscore-separated-words formats. Set modifies the section in-place
// and also returns the same modified-in-place Section to give flexibility of usage.
//
// NOTE that if a config item in a map should have quotes around it, they will have to be added to
// the string explicitly. e.g., a Value of " test " will be interpreted as as 'Key = test', whereas
// a Value of "\" test \"" will be interpreted as 'Key = " test "'
func (s *Section) Set(key, value string) *Section {
key = normalizeKey(key)
_, ok := s.configs[key]
if !ok {
s.configOrder = append(s.configOrder, key)
}
s.configs[key] = value
return s
}
// Merge modifies the Section in-place adding values from the overrides Section and overriding
// existing values with the new value in overrides.
func (s *Section) Merge(overrides *Section) {
for _, k := range overrides.configOrder {
if v, ok := overrides.configs[k]; ok {
s.Set(k, v)
continue
}
// this condition should never exist except in case of some kind of mem corruption
logger.Errorf("override config key %s could not be applied. possible struct corruption of overrides: %+v", k, overrides)
}
}
// Merge modifies the Config in-place, adding Sections from the overrides config, and overriding
// existing Sections with Section values from overrides.
func (c *Config) Merge(overrides *Config) {
for _, hdr := range overrides.sectionOrder {
if s, ok := overrides.sections[hdr]; ok {
c.Section(hdr).Merge(s)
continue
}
// this condition should never exist except in case of some kind of mem corruption
logger.Errorf("override section %s could not be applied. possible struct corruption of overrides: %+v", hdr, overrides)
}
}
// IniFile converts the Ceph Config to an *ini.File.
func (c *Config) IniFile() (*ini.File, error) {
f := ini.Empty()
for _, hdr := range c.sectionOrder {
sec, ok := c.sections[hdr]
if !ok {
return nil, fmt.Errorf("Error: section header %s not found: %+v", hdr, c)
}
s, err := f.NewSection(hdr)
if err != nil {
return nil, fmt.Errorf("Error converting config to ini file text: %+v. %+v", c, err)
}
for _, k := range sec.configOrder {
v, ok := sec.configs[k]
if !ok {
return nil, fmt.Errorf("Error: config %s not found in section %s: %+v", k, hdr, sec)
}
if _, err := s.NewKey(k, v); err != nil {
return nil, fmt.Errorf("Error converting config to ini file text: %+v. %+v", c, err)
}
}
}
return f, nil
}
// NewFlag returns the key-value pair in the format of a Ceph command line-compatible flag.
func NewFlag(key, value string) string {
// A flag is a normalized key with underscores replaced by dashes.
@@ -208,23 +104,105 @@ func NewFlag(key, value string) string {
return fmt.Sprintf("--%s=%s", f, value)
}
// GlobalFlags converts the Config's 'global' Section to a list of command line flag strings which
// can be used with Kubernetes container specs for Ceph daemons.
//
// It should never be necessary to get flags for anything but the 'global' config section,
// otherwise, Rook will have to determine which sections apply for a given Ceph daemon. Does
// 'mon.a' apply to 'mon.b'? 'mon.aa'? 'osd.0'? The logic would be prone to errors, so only allow
// the 'global' section to be converted to flags.
func (c *Config) GlobalFlags() []string {
f := []string{}
g := c.Section("global")
for _, k := range g.configOrder {
if v, ok := g.configs[k]; ok {
f = append(f, NewFlag(k, v))
continue
}
// this condition should never exist except in case of some kind of mem corruption
logger.Errorf("override config key %s could not be applied. possible struct corruption of section: %+v", k, g)
// SetDefaultAndUserConfigs sets Rook's desired default configs and the user's override configs from
// the CephCluster CRD in the centralized monitor database. This cannot be called before at least
// one monitor is established.
func SetDefaultAndUserConfigs(
context *clusterd.Context,
namespace string,
clusterInfo *cephconfig.ClusterInfo,
configOverrides rookceph.ConfigOverridesSpec,
) error {
// ceph.conf is never used. All configurations are made in the centralized mon config database,
// or they are specified on the commandline when daemons are called.
monStore := GetMonStore(context, namespace)
if err := monStore.SetAll(DefaultCentralizedConfigs(clusterInfo.CephVersion)); err != nil {
return fmt.Errorf("failed to apply default Ceph configurations. %+v", err)
}
return f
if err := monStore.SetAll(DefaultLegacyConfigs()); err != nil {
return fmt.Errorf("failed to apply legacy config overrides. %+v", err)
}
// user-specified config overrides from the CRD will go here
if err := monStore.SetAll(configOverrides); err != nil {
return fmt.Errorf("failed to apply one or more user-specified overrides. %+v", err)
}
return nil
}
// LegacyConfigMapOverrideDefined checks to see if the legacy ConfigMap override exists with
// overrides defined. If it exists and has config overrides defined, this will return true. If it
// does not exist or exists without config overrides set, it will return false.
func LegacyConfigMapOverrideDefined(context *clusterd.Context, namespace string) (bool, error) {
cm, err := context.Clientset.CoreV1().ConfigMaps(namespace).
Get(k8sutil.ConfigOverrideName, metav1.GetOptions{})
if err != nil {
if errors.IsNotFound(err) {
logger.Debugf("override configmap does not exist")
return false, nil
}
// if we can't determine existence or not, assume it exists
return true, fmt.Errorf("failed to see if legacy override ConfigMap exists. %+v", err)
}
o, ok := cm.Data[k8sutil.ConfigOverrideVal]
if !ok || (ok && o == "") {
// if no override set or override is empty string
return false, nil
}
return true, nil
}
func configFileTextToOverrides(txt string) (rookceph.ConfigOverridesSpec, error) {
overrides := []rookceph.ConfigOverride{}
f, err := configFileTxtToINI(txt)
if err != nil {
return nil, err
}
sections := f.Sections()
for _, s := range sections {
who := s.Name()
keys := s.Keys()
for _, k := range keys {
option := k.Name()
value := k.String()
overrides = append(overrides, configOverride(who, option, value))
}
}
return overrides, nil
}
func configFileTxtToINI(txt string) (*ini.File, error) {
f := ini.Empty()
ovrTxt := []byte(txt)
if err := f.Append(ovrTxt); err != nil {
return nil, fmt.Errorf("failed to interpret text as INI file '''%s'''. %+v", ovrTxt, err)
}
return f, nil
}
func getOverrideConfigFromConfigMap(context *clusterd.Context, namespace string) string {
cm, err := context.Clientset.CoreV1().ConfigMaps(namespace).
Get(k8sutil.ConfigOverrideName, metav1.GetOptions{})
if err != nil {
if errors.IsNotFound(err) {
logger.Debugf("override configmap does not exist")
return "" // return empty string if cannot get override text
}
logger.Warningf("Error getting override configmap. %+v", err)
return "" // return empty string if cannot get override text
}
// the override config map exists
o, ok := cm.Data[k8sutil.ConfigOverrideVal]
if !ok {
logger.Warningf("The override config map does not have override text defined. %+v", o)
return "" // return empty string if cannot get override text
}
return o
}
-112
View File
@@ -17,108 +17,11 @@ limitations under the License.
package config
import (
"bytes"
"testing"
"github.com/stretchr/testify/assert"
)
// Everything here is for the ultimate purpose of generating ini config files, and this is the main
// output. Testing other functions would require this for testing output anyway, so just test all
// the functions here.
func TestConfig_FileText(t *testing.T) {
c := NewConfig()
iniText := func() string {
f, e := c.IniFile()
assert.NoError(t, e)
b := new(bytes.Buffer)
_, e = f.WriteTo(b)
assert.NoError(t, e)
return b.String()
}
// Empty config
f := iniText()
assert.Equal(t, "", f)
// single section, empty
c.Section("global")
f = iniText()
assert.Equal(t, "[global]\n\n", f)
// two sections, both empty
c.Section("osd")
f = iniText()
assert.Equal(t, "[global]\n\n[osd]\n\n", f)
// add configs to empty global section
c.Section("global").
Set("key", "val").
Set("key with spaces", "my-val").
Set("key_with_underscores", "my val").
Set("key-with-hyphens", "\"my_val \"")
f = iniText()
assert.Equal(t, `[global]
key = val
key_with_spaces = my-val
key_with_underscores = my val
key_with_hyphens = "my_val "
[osd]
`, f)
// change some of the global options to make sure the keys override properly
c.Section("global").
Set("key", "val"). // do not change this val to test this corner case
Set("key_with_spaces", "my-val2").
Set("key with underscores", "my val2")
// add new section with a config set
c.Section("mon").Set("debug mon", "100")
f = iniText()
assert.Equal(t, `[global]
key = val
key_with_spaces = my-val2
key_with_underscores = my val2
key_with_hyphens = "my_val "
[osd]
[mon]
debug_mon = 100
`, f)
// test merging by creating a new config and merging it with the previous one
// config merging uses section merging as part of it, so no need to test section merging
m := NewConfig()
m.Section("global"). // preexisting section
Set("key", "val"). // preexisting key overridden w/ same val
Set("key with spaces", "my-val"). // preexisting key overridden w/ diff val
Set("new key", "new") // new key
m.Section("mgr"). // new section
Set("debug mgr", "100") // new val
c.Merge(m)
f = iniText()
assert.Equal(t, `[global]
key = val
key_with_spaces = my-val
key_with_underscores = my val2
key_with_hyphens = "my_val "
new_key = new
[osd]
[mon]
debug_mon = 100
[mgr]
debug_mgr = 100
`, f)
}
func TestNewFlag(t *testing.T) {
assert.Equal(t, NewFlag("k", ""), "--k=")
assert.Equal(t, NewFlag("a-key", "a"), "--a-key=a")
@@ -126,18 +29,3 @@ func TestNewFlag(t *testing.T) {
assert.Equal(t, NewFlag("c key", "c"), "--c-key=c")
assert.Equal(t, NewFlag("quotes", "\"quoted\""), "--quotes=\"quoted\"")
}
func TestConfig_GlobalFlags(t *testing.T) {
c := NewConfig()
assert.Equal(t, c.GlobalFlags(), []string{})
c.Section("global").
Set("test key", "one").
Set("two_key", "2").
Set("3-key", "\"trois \"")
assert.ElementsMatch(t, []string{"--test-key=one", "--two-key=2", "--3-key=\"trois \""}, c.GlobalFlags())
c.Section("mon").
Set("mon key", "m")
assert.ElementsMatch(t, []string{"--test-key=one", "--two-key=2", "--3-key=\"trois \""}, c.GlobalFlags())
}
+47 -56
View File
@@ -17,51 +17,54 @@ limitations under the License.
// Package config provides default configurations which Rook will set in Ceph clusters.
package config
import "github.com/rook/rook/pkg/operator/ceph/version"
import (
rookceph "github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
"github.com/rook/rook/pkg/operator/ceph/version"
)
// DefaultFlags returns the default configuration flags Rook will set on the command line for all
// calls to Ceph daemons and tools. Values specified here will not be able to be overridden using
// the mon's central KV store, and that is (and should be) by intent.
func DefaultFlags(fsid, mountedKeyringPath string, cephVersion version.CephVersion) []string {
return defaultFlagConfigs(fsid, mountedKeyringPath, cephVersion).GlobalFlags()
}
func defaultFlagConfigs(fsid, mountedKeyringPath string, cephVersion version.CephVersion) *Config {
c := NewConfig()
c.Section("global").
flags := []string{
// fsid unnecessary but is a safety to make sure daemons can only connect to their cluster
Set("fsid", fsid).
Set("keyring", mountedKeyringPath).
NewFlag("fsid", fsid),
NewFlag("keyring", mountedKeyringPath),
// For containers, we're expected to log everything to stderr
Set("log-to-stderr", "true").
Set("err-to-stderr", "true").
Set("mon-cluster-log-to-stderr", "true").
Set("log-stderr-prefix", "debug ")
// ^ differentiate debug text from audit text, and the space after 'debug' is critical
// As of Nautilus 14.2.1 at least
// These new flags control Ceph's daemon logging behaviour to files
// By default we set them to False so no logs get written on file
// However they can be actived at any time via the centralized config store
if cephVersion.IsAtLeast(version.CephVersion{Major: 14, Minor: 2, Extra: 1}) {
c.Section("global").
Set("default-log-to-file", "false").
Set("default-mon-cluster-log-to-file", "false")
NewFlag("log-to-stderr", "true"),
NewFlag("err-to-stderr", "true"),
NewFlag("mon-cluster-log-to-stderr", "true"),
// differentiate debug text from audit text, and the space after 'debug' is critical
NewFlag("log-stderr-prefix", "debug "),
}
m := StoredMonHostEnvVarReferences()
c.Merge(m)
return c
// As of Nautilus 14.2.1 at least
// These new flags control Ceph's daemon logging behavior to files
// By default we set them to False so no logs get written on file
// However they can be activated at any time via the centralized config store
if cephVersion.IsAtLeast(version.CephVersion{Major: 14, Minor: 2, Extra: 1}) {
flags = append(flags, []string{
NewFlag("default-log-to-file", "false"),
NewFlag("default-mon-cluster-log-to-file", "false"),
}...)
}
flags = append(flags, StoredMonHostEnvVarFlags()...)
return flags
}
// makes it possible to be slightly less verbose to create a ConfigOverride here
func configOverride(who, option, value string) rookceph.ConfigOverride {
return rookceph.ConfigOverride{Who: who, Option: option, Value: value}
}
// DefaultCentralizedConfigs returns the default configuration options Rook will set in Ceph's
// centralized config store. If the version of Ceph does not support the centralized config store,
// these will be set in a shared config file instead.
func DefaultCentralizedConfigs(cephVersion version.CephVersion) *Config {
c := NewConfig()
c.Section("global").
Set("mon allow pool delete", "true")
// centralized config store.
func DefaultCentralizedConfigs(cephVersion version.CephVersion) rookceph.ConfigOverridesSpec {
overrides := []rookceph.ConfigOverride{
configOverride("global", "mon allow pool delete", "true"),
}
// Everything before Nautilus 14.2.1
// Prior to Nautilus 14.2.1 certain log flags were not present
@@ -71,33 +74,21 @@ func DefaultCentralizedConfigs(cephVersion version.CephVersion) *Config {
if !cephVersion.IsAtLeast(version.CephVersion{Major: 14, Minor: 2, Extra: 1}) {
// Set the default log files to empty so they don't bloat containers. Can be changed in
// Mimic+ by users if needed.
c.Section("global").
Set("log file", "").
Set("mon cluster log file", "")
overrides = append(overrides, []rookceph.ConfigOverride{
configOverride("global", "log file", ""),
configOverride("global", "mon cluster log file", ""),
}...)
}
return c
return overrides
}
// DefaultLegacyConfigs need to be added to the Ceph config file until the integration tests can be
// made to override these options for the Ceph clusters it creates.
func DefaultLegacyConfigs() *Config {
c := NewConfig()
c.Section("global").
Set("mon max pg per osd", "1000").
//
Set("osd pg bits", "11").
Set("osd pgp bits", "11").
Set("osd pool default size", "1").
Set("osd pool default min size", "1").
Set("osd pool default pg num", "100").
Set("osd pool default pgp num", "100").
//
Set("rbd_default_features", "3"). // TODO: still needed?
// Setting fatal signal handlers to true (the default) will print a lot of extra information
// from daemons when they encounter a failures, but it is VERY verbose. When the mon kv
// store is available, it will probably be best to set this to false by default if it is
// unset, but leave it set to true if the user has specified it as true during runtime.
Set("fatal signal handlers", "false")
return c
func DefaultLegacyConfigs() rookceph.ConfigOverridesSpec {
overrides := []rookceph.ConfigOverride{
// TODO: drop this when FlexVolume is no longer supported
configOverride("global", "rbd_default_features", "3"),
}
return overrides
}
+73
View File
@@ -0,0 +1,73 @@
/*
Copyright 2019 The Rook Authors. All rights reserved.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package config
import (
"fmt"
rookceph "github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
"github.com/rook/rook/pkg/clusterd"
"github.com/rook/rook/pkg/daemon/ceph/client"
)
// MonStore provides methods for setting Ceph configurations in the centralized mon
// configuration database.
type MonStore struct {
context *clusterd.Context
namespace string
}
// GetMonStore returns a new MonStore for the cluster.
func GetMonStore(context *clusterd.Context, namespace string) *MonStore {
return &MonStore{
context: context,
namespace: namespace,
}
}
// Set sets a config in the centralized mon configuration database.
// https://docs.ceph.com/docs/master/rados/configuration/ceph-conf/#monitor-configuration-database
func (m *MonStore) Set(who, option, value string) error {
args := []string{"config", "set", who, normalizeKey(option), value}
cephCmd := client.NewCephCommand(m.context, m.namespace, args)
out, err := cephCmd.Run()
if err != nil {
return fmt.Errorf("failed to set Ceph config in the centralized mon configuration database; "+
"you may need to use the rook-config-override ConfigMap. output: %s. %+v", string(out), err)
}
return nil
}
// SetAll sets all configs from the overrides in the centralized mon configuration database.
// See MonStore.Set for more.
func (m *MonStore) SetAll(configOverrides rookceph.ConfigOverridesSpec) error {
var errs []error
for _, override := range configOverrides {
err := m.Set(override.Who, override.Option, override.Value)
if err != nil {
errs = append(errs, err)
}
}
if len(errs) > 0 {
retErr := fmt.Errorf("failed to set one or more Ceph configs")
for _, err := range errs {
retErr = fmt.Errorf("%+v. %+v", retErr, err)
}
return retErr
}
return nil
}
+131
View File
@@ -0,0 +1,131 @@
/*
Copyright 2019 The Rook Authors. All rights reserved.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package config
import (
"fmt"
"strings"
"testing"
rookceph "github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
"github.com/rook/rook/pkg/clusterd"
testop "github.com/rook/rook/pkg/operator/test"
exectest "github.com/rook/rook/pkg/util/exec/test"
"github.com/stretchr/testify/assert"
)
func TestMonStore_Set(t *testing.T) {
executor := &exectest.MockExecutor{}
ctx := &clusterd.Context{
Clientset: testop.New(1),
Executor: executor,
}
// create a mock command runner which creates a simple string of the command it ran, and allow
// us to cause it to return an error when it detects a keyword.
execedCmd := ""
execInjectErr := false
executor.MockExecuteCommandWithOutputFile =
func(debug bool, actionName string, command string, outfile string, args ...string) (string, error) {
execedCmd = command + " " + strings.Join(args, " ")
if execInjectErr {
return "output from cmd with error", fmt.Errorf("mocked error")
}
return "", nil
}
monStore := GetMonStore(ctx, "ns")
// setting with spaces converts to underscores
e := monStore.Set("global", "debug ms", "10")
assert.NoError(t, e)
assert.Contains(t, execedCmd, "config set global debug_ms 10")
// setting with dashes converts to underscores
e = monStore.Set("osd.0", "debug-osd", "20")
assert.NoError(t, e)
assert.Contains(t, execedCmd, " config set osd.0 debug_osd 20 ")
// setting with underscores stays the same
e = monStore.Set("mds.*", "debug_mds", "15")
assert.NoError(t, e)
assert.Contains(t, execedCmd, " config set mds.* debug_mds 15 ")
// errors returned as expected
execInjectErr = true
e = monStore.Set("mon.*", "unknown_setting", "10")
assert.Error(t, e)
assert.Contains(t, execedCmd, " config set mon.* unknown_setting 10 ")
}
func TestMonStore_SetAll(t *testing.T) {
executor := &exectest.MockExecutor{}
ctx := &clusterd.Context{
Clientset: testop.New(1),
Executor: executor,
}
// create a mock command runner which creates a simple string of the command it ran, and allow
// us to cause it to return an error when it detects a keyword.
execedCmds := []string{}
execInjectErrOnKeyword := "donotinjectanerror"
executor.MockExecuteCommandWithOutputFile =
func(debug bool, actionName string, command string, outfile string, args ...string) (string, error) {
execedCmd := command + " " + strings.Join(args, " ")
execedCmds = append(execedCmds, execedCmd)
k := execInjectErrOnKeyword
if strings.Contains(execedCmd, k) {
return "output from cmd with error on keyword: " + k, fmt.Errorf("mocked error on keyword: " + k)
}
return "", nil
}
monStore := GetMonStore(ctx, "ns")
var cfgOverrides rookceph.ConfigOverridesSpec
cfgOverrides = []rookceph.ConfigOverride{
configOverride("global", "debug ms", "10"), // setting w/ spaces converts to underscores
configOverride("osd.0", "debug-osd", "20"), // setting w/ dashes converts to underscores
configOverride("mds.*", "debug_mds", "15"), // setting w/ underscores remains the same
}
// commands w/ no error
e := monStore.SetAll(cfgOverrides)
assert.NoError(t, e)
assert.Len(t, execedCmds, 3)
assert.Contains(t, execedCmds[0], " global debug_ms 10 ")
assert.Contains(t, execedCmds[1], " osd.0 debug_osd 20 ")
assert.Contains(t, execedCmds[2], " mds.* debug_mds 15 ")
// commands w/ one error
// keep cfgOverrides from last test
execInjectErrOnKeyword = "debug_osd"
execedCmds = execedCmds[:0] // empty execedCmds slice
e = monStore.SetAll(cfgOverrides)
assert.Error(t, e)
// Rook should not return error before trying to set all config overrides
assert.Len(t, execedCmds, 3)
// all commands return error
// keep cfgOverrides
execInjectErrOnKeyword = "debug"
execedCmds = execedCmds[:0]
e = monStore.SetAll(cfgOverrides)
assert.Error(t, e)
assert.Len(t, execedCmds, 3)
}
+7 -115
View File
@@ -19,13 +19,11 @@ limitations under the License.
package config
import (
"bytes"
"fmt"
"net"
"strconv"
"strings"
"github.com/go-ini/ini"
"github.com/rook/rook/pkg/clusterd"
cephconfig "github.com/rook/rook/pkg/daemon/ceph/config"
cephutil "github.com/rook/rook/pkg/daemon/ceph/util"
@@ -70,84 +68,14 @@ func GetStore(context *clusterd.Context, namespace string, ownerRef *metav1.Owne
// CreateOrUpdate creates or updates the stored Ceph config based on the cluster info.
func (s *Store) CreateOrUpdate(clusterInfo *cephconfig.ClusterInfo) error {
c := DefaultCentralizedConfigs(clusterInfo.CephVersion)
// DefaultLegacyConfigs need to be added to the Ceph config file until the integration tests can be
// made to override these options for the Ceph clusters it creates.
c.Merge(DefaultLegacyConfigs())
/* TODO: config overrides from the CRD will go here */
f, err := c.IniFile()
if err != nil {
return fmt.Errorf("failed to store the Ceph config file. could not generate default ini. %+v", err)
}
// merge config overrides from the legacy override configmap
b := new(bytes.Buffer)
if err := s.applyLegacyOverrides(f); err != nil {
logger.Warningf("failed to merge config override with default Ceph config file; using only the default config. %+v", err)
f, err = c.IniFile() // regenerate the orig ini in case override attempt altered the struct
if err != nil {
return fmt.Errorf("failed to store the Ceph config file. could not generate default ini after merge failure. %+v", err)
}
}
if _, err := f.WriteTo(b); err != nil {
return fmt.Errorf("failed to convert merged Ceph config to text; continuing with original config. %+v", err)
}
txt := b.String()
// Store the config in a configmap
if err := s.configMapStore.SetValue(StoreName, confFileName, txt); err != nil {
return fmt.Errorf("failed to store the Ceph config file. failed to store config to configmap. %+v", err)
}
logger.Debugf("Generated and stored config file:\n%s", txt)
// these are used for all ceph daemons on the commandline and must *always* be stored
if err := s.createOrUpdateMonHostSecrets(clusterInfo); err != nil {
return fmt.Errorf("failed to store mon host configs. %+v", err)
}
/*
TODO:
set config values with `ceph config assimilage-conf t` after at least one mon is active
*/
return nil
}
func (s *Store) applyLegacyOverrides(toFile *ini.File) error {
ovrTxt := []byte(s.overrideConfig())
if err := toFile.Append(ovrTxt); err != nil {
return fmt.Errorf("failed to apply config file override from '''%s'''. %+v", ovrTxt, err)
}
b := new(bytes.Buffer)
if _, err := toFile.WriteTo(b); err != nil {
return fmt.Errorf("failed to write out toFile config file '''%+v'''. %+v", toFile, err)
}
return nil
}
func (s *Store) overrideConfig() string {
cm, err := s.context.Clientset.CoreV1().ConfigMaps(s.namespace).
Get(k8sutil.ConfigOverrideName, metav1.GetOptions{})
if err != nil {
if errors.IsNotFound(err) {
logger.Debugf("override configmap does not exist")
return "" // return empty string if cannot get override text
}
logger.Warningf("Error getting override configmap. %+v", err)
return "" // return empty string if cannot get override text
}
// the override config map exists
o, ok := cm.Data[k8sutil.ConfigOverrideVal]
if !ok {
logger.Warningf("The override config map does not have override text defined. %+v", o)
return "" // return empty string if cannot get override text
}
return o
}
// update "mon_host" and "mon_initial_members" in the stored config
func (s *Store) createOrUpdateMonHostSecrets(clusterInfo *cephconfig.ClusterInfo) error {
hosts := make([]string, len(clusterInfo.Monitors))
@@ -216,40 +144,6 @@ func (s *Store) createOrUpdateMonHostSecrets(clusterInfo *cephconfig.ClusterInfo
return nil
}
// StoredFileVolume returns a pod volume sourced from the stored config file.
func StoredFileVolume() v1.Volume {
// TL;DR: mount the configmap's "ceph.conf" to a file called "ceph.conf" with 0444 permissions
// security: allow to be read by everyone since now ceph processes run as 'ceph' and not 'root' user
// Further investigation needs to be done to copy the ceph.conf and change its ownership
// since configuring a owner of a ConfigMap secret is currently impossible
// This also works around the following issue: https://tracker.ceph.com/issues/38606
//
// This design choice avoids the crash/restart situation in Rook
// If we don't set 0444 to the ceph.conf configuration file during its respawn (with exec) the ceph-mgr
// won't be able to read the ceph.conf and the container will die, the "restart" count will increase in k8s
// This will mislead users thinking something won't wrong but that a false positive
mode := int32(0444)
return v1.Volume{
Name: configVolumeName,
VolumeSource: v1.VolumeSource{
ConfigMap: &v1.ConfigMapVolumeSource{LocalObjectReference: v1.LocalObjectReference{
Name: StoreName},
Items: []v1.KeyToPath{
{Key: confFileName, Path: confFileName, Mode: &mode},
}}}}
}
// StoredFileVolumeMount returns a container volume mount that mounts the stored config from
// StoredFileVolume into the container at `/etc/ceph/ceph.conf`.
func StoredFileVolumeMount() v1.VolumeMount {
// configmap's "ceph.conf" to "/etc/ceph/ceph.conf"
return v1.VolumeMount{
Name: StoreName,
ReadOnly: true, // should be no reason to write to the config in pods, so enforce this
MountPath: "/etc/ceph",
}
}
// StoredMonHostEnvVars returns a container environment variable defined by the most updated stored
// "mon_host" and "mon_initial_members" information.
func StoredMonHostEnvVars() []v1.EnvVar {
@@ -267,13 +161,11 @@ func StoredMonHostEnvVars() []v1.EnvVar {
}
}
// StoredMonHostEnvVarReferences returns a small Ceph Config which references "mon_host" and
// "mon_initial_members" information from the StoredMonHostEnvVars. This config can be used to
// generate flags referencing the env vars or to generate the string representation of a Config.
func StoredMonHostEnvVarReferences() *Config {
c := NewConfig()
c.Section("global").
Set(monHostKey, "$(ROOK_CEPH_MON_HOST)").
Set(monInitialMembersKey, "$(ROOK_CEPH_MON_INITIAL_MEMBERS)")
return c
// StoredMonHostEnvVarFlags returns Ceph commandline flag references to "mon_host" and
// "mon_initial_members" sourced from the StoredMonHostEnvVars.
func StoredMonHostEnvVarFlags() []string {
return []string{
NewFlag(monHostKey, "$(ROOK_CEPH_MON_HOST)"),
NewFlag(monInitialMembersKey, "$(ROOK_CEPH_MON_INITIAL_MEMBERS)"),
}
}
+1 -131
View File
@@ -17,19 +17,14 @@ limitations under the License.
package config
import (
"bufio"
"fmt"
"path"
"regexp"
"strings"
"testing"
"github.com/rook/rook/pkg/clusterd"
cephconfig "github.com/rook/rook/pkg/daemon/ceph/config"
"github.com/rook/rook/pkg/operator/k8sutil"
testop "github.com/rook/rook/pkg/operator/test"
"github.com/stretchr/testify/assert"
v1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
@@ -43,16 +38,7 @@ func TestStore(t *testing.T) {
s := GetStore(ctx, ns, &owner)
var previousConfigText = ""
var recentConfigText = ""
assertConfigStore := func(ci *cephconfig.ClusterInfo) {
c, e := clientset.CoreV1().ConfigMaps(ns).Get(StoreName, metav1.GetOptions{})
assert.NoError(t, e)
previousConfigText = recentConfigText
recentConfigText = c.Data[confFileName]
assert.NotEqual(t, "", recentConfigText)
sec, e := clientset.CoreV1().Secrets(ns).Get(StoreName, metav1.GetOptions{})
assert.NoError(t, e)
mh := strings.Split(sec.StringData["mon_host"], ",") // list of mon ip:port pairs in cluster
@@ -66,20 +52,6 @@ func TestStore(t *testing.T) {
}
}
findLine := func(lineRegex, inText string) int {
scr := bufio.NewScanner(strings.NewReader(inText))
i := 1
for scr.Scan() {
m, e := regexp.Match(lineRegex, scr.Bytes())
assert.NoError(t, e)
if m {
return i
}
i++
}
return 0 // ret 0 if not found
}
i1 := testop.CreateConfigDir(1) // cluster w/ one mon
i3 := testop.CreateConfigDir(3) // same cluster w/ 3 mons
@@ -88,108 +60,6 @@ func TestStore(t *testing.T) {
s.CreateOrUpdate(i3)
assertConfigStore(i3)
// the config should be the same regardless of how many mons there are
assert.Equal(t, previousConfigText, recentConfigText)
//
// test overrides
//
createOverrideMap(t, ctx, ns, &owner)
s.CreateOrUpdate(i3)
assertConfigStore(i3)
// configs should still be equal since the created map doesn't have data in it
assert.Equal(t, previousConfigText, recentConfigText)
updateOverrideMap(t, "", ctx, ns, &owner)
s.CreateOrUpdate(i3)
assertConfigStore(i3)
// configs should still be equal since the created map's data is blank
assert.Equal(t, previousConfigText, recentConfigText)
updateOverrideMap(t, "this is not valid ini file text", ctx, ns, &owner)
s.CreateOrUpdate(i3)
assertConfigStore(i3)
// configs should still be equal since the created map's data is invalid ini
assert.Equal(t, previousConfigText, recentConfigText)
// complicated override with overrides, new keys, new section
updateOverrideMap(t, `[global]
log_file = notreal
new_key = fake
log stderr prefix = '"debug "'
[mon]
debug_mon = makehaste
`, ctx, ns, &owner)
s.CreateOrUpdate(i3)
assertConfigStore(i3)
// Verify some simple truths about the overridden config vs the original
assert.NotEqual(t, previousConfigText, recentConfigText) // the new config has changed (finally)
l1 := findLine("log_file[[:space:]]+= ", previousConfigText) //
assert.NotZero(t, l1) // orig 'log_file = <num>' exists
l2 := findLine("log_file[[:space:]]+= notreal", recentConfigText) //
assert.Equal(t, l1, l2) // overridden debug_default still has same line num
l3 := findLine("new_key[[:space:]]+= fake", recentConfigText) //
assert.True(t, l3 > l2) // new_key should come after debug_default
l4 := findLine("log stderr prefix[[:space:]]+= \"debug \"", recentConfigText) // becomes "debug " w/o single quotes
assert.Equal(t, l3+1, l4) // ... and is 1 line after new_key
l5 := findLine("[[]mon[]]", recentConfigText) //
assert.Equal(t, l4+2, l5) // [mon] hdr comes 2 after
l6 := findLine("debug_mon[[:space:]]+= makehaste", recentConfigText) //
assert.Equal(t, l5+1, l6) // debug_mon comes 1 after
//
lg := findLine("[[]global[]]", recentConfigText) //
assert.Equal(t, 1, lg) // [global] hdr is first line
ln := findLine("[[].*[]]", strings.TrimLeft(recentConfigText, "[global]")) //
assert.Equal(t, l5, ln) // next hdr after [global] is [mon]
}
func createOverrideMap(t *testing.T,
context *clusterd.Context, namespace string, ownerRef *metav1.OwnerReference,
) {
cm := &v1.ConfigMap{
ObjectMeta: metav1.ObjectMeta{
Name: k8sutil.ConfigOverrideName,
},
}
k8sutil.SetOwnerRef(&cm.ObjectMeta, ownerRef)
_, err := context.Clientset.CoreV1().ConfigMaps(namespace).Create(cm)
assert.NoError(t, err)
}
func updateOverrideMap(t *testing.T, overrideText string,
context *clusterd.Context, namespace string, ownerRef *metav1.OwnerReference,
) {
cm := &v1.ConfigMap{
ObjectMeta: metav1.ObjectMeta{
Name: k8sutil.ConfigOverrideName,
},
Data: map[string]string{k8sutil.ConfigOverrideVal: overrideText},
}
k8sutil.SetOwnerRef(&cm.ObjectMeta, ownerRef)
_, err := context.Clientset.CoreV1().ConfigMaps(namespace).Update(cm)
assert.NoError(t, err)
}
func TestFileVolumeAndMount(t *testing.T) {
clientset := testop.New(1)
ctx := &clusterd.Context{
Clientset: clientset,
}
ns := "rook-ceph"
owner := metav1.OwnerReference{}
s := GetStore(ctx, ns, &owner)
s.CreateOrUpdate(testop.CreateConfigDir(3))
v := StoredFileVolume()
m := StoredFileVolumeMount()
// Test that the configmapped file will make it into containers with the appropriate filename at
// the location where it is expected.
assert.Equal(t, v.Name, m.Name)
assert.Equal(t, StoreName, v.VolumeSource.ConfigMap.LocalObjectReference.Name)
assert.Equal(t, "/etc/ceph/ceph.conf", path.Join(m.MountPath, confFileName))
}
func TestEnvVarsAndFlags(t *testing.T) {
@@ -204,7 +74,7 @@ func TestEnvVarsAndFlags(t *testing.T) {
s.CreateOrUpdate(testop.CreateConfigDir(3))
v := StoredMonHostEnvVars()
f := StoredMonHostEnvVarReferences().GlobalFlags()
f := StoredMonHostEnvVarFlags()
// make sure the env var names and flags are matching pairs
mh := v[0].Name
+32 -3
View File
@@ -18,10 +18,14 @@ limitations under the License.
package nfs
import (
"bytes"
"fmt"
"os/exec"
"strings"
cephv1 "github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
"github.com/rook/rook/pkg/clusterd"
cephclient "github.com/rook/rook/pkg/daemon/ceph/client"
cephconfig "github.com/rook/rook/pkg/daemon/ceph/config"
opmon "github.com/rook/rook/pkg/operator/ceph/cluster/mon"
"github.com/rook/rook/pkg/operator/ceph/config"
@@ -107,13 +111,21 @@ func (c *CephNFSController) upCephNFS(n cephv1.CephNFS, oldActive int) error {
func (c *CephNFSController) addRADOSConfigFile(n cephv1.CephNFS, name string) error {
nodeID := getNFSNodeID(n, name)
config := getGaneshaConfigObject(nodeID)
err := c.context.Executor.ExecuteCommand(false, "", "rados", "--pool", n.Spec.RADOS.Pool, "--namespace", n.Spec.RADOS.Namespace, "stat", config)
cmd := "rados"
args := []string{
"--pool", n.Spec.RADOS.Pool,
"--namespace", n.Spec.RADOS.Namespace,
"--conf", cephclient.CephConfFilePath(c.context.ConfigDir, c.namespace),
}
moniker := "rados stat " + config
err := c.context.Executor.ExecuteCommand(false, moniker, cmd, append(args, "stat", config)...)
if err == nil {
// If stat works then we assume it's present already
return nil
}
// try to create it
return c.context.Executor.ExecuteCommand(false, "", "rados", "--pool", n.Spec.RADOS.Pool, "--namespace", n.Spec.RADOS.Namespace, "create", config)
moniker = "rados create " + config
return c.context.Executor.ExecuteCommand(false, moniker, cmd, append(args, "create", config)...)
}
func (c *CephNFSController) addServerToDatabase(nfs cephv1.CephNFS, name string) {
@@ -134,8 +146,25 @@ func (c *CephNFSController) removeServerFromDatabase(nfs cephv1.CephNFS, name st
func (c *CephNFSController) runGaneshaRadosGrace(nfs cephv1.CephNFS, name, action string) error {
nodeID := getNFSNodeID(nfs, name)
cmd := ganeshaRadosGraceCmd
args := []string{"--pool", nfs.Spec.RADOS.Pool, "--ns", nfs.Spec.RADOS.Namespace, action, nodeID}
return c.context.Executor.ExecuteCommand(false, "", ganeshaRadosGraceCmd, args...)
moniker := ganeshaRadosGraceCmd + " " + action + " " + nodeID
// Need to run a command with CEPH_CONF env var set, so don't use c.context.Executor
x := exec.Command(cmd, args...)
x.Env = []string{fmt.Sprintf("CEPH_CONF=%s", cephclient.CephConfFilePath(c.context.ConfigDir, c.namespace))}
var b bytes.Buffer
x.Stdout = &b
x.Stderr = &b
logger.Infof("Running command: %s %s %s", x.Env[0], cmd, strings.Join(args, " "))
if err := x.Run(); err != nil {
return fmt.Errorf(`failed to execute '%s'
stdout: %s
stderr: %s
error: %+v`, moniker, x.Stdout, x.Stderr, err)
}
return nil
// The below can be used when/if `ganesha-rados-grace` supports a `--cephconf` or similar option
// return c.context.Executor.ExecuteCommand(false, moniker, cmd, args...)
}
func (c *CephNFSController) generateConfig(n cephv1.CephNFS, name string) (string, error) {
+9 -10
View File
@@ -55,16 +55,15 @@ func rgwFrontend(v cephver.CephVersion) string {
}
// TODO: these should be set in the mon's central kv store for mimic+
func (c *clusterConfig) defaultSettings() *cephconfig.Config {
s := cephconfig.NewConfig()
s.Section("global").
Set("rgw log nonexistent bucket", "true").
Set("rgw intent log object name utc", "true").
Set("rgw enable usage log", "true").
Set("rgw frontends", fmt.Sprintf("%s %s", rgwFrontend(c.clusterInfo.CephVersion), c.portString(c.clusterInfo.CephVersion))).
Set("rgw zone", c.store.Name).
Set("rgw zonegroup", c.store.Name)
return s
func (c *clusterConfig) defaultFlags() []string {
return []string{
cephconfig.NewFlag("rgw log nonexistent bucket", "true"),
cephconfig.NewFlag("rgw intent log object name utc", "true"),
cephconfig.NewFlag("rgw enable usage log", "true"),
cephconfig.NewFlag("rgw frontends", fmt.Sprintf("%s %s", rgwFrontend(c.clusterInfo.CephVersion), c.portString(c.clusterInfo.CephVersion))),
cephconfig.NewFlag("rgw zone", c.store.Name),
cephconfig.NewFlag("rgw zonegroup", c.store.Name),
}
}
func (c *clusterConfig) portString(v cephver.CephVersion) string {
+1 -1
View File
@@ -117,7 +117,7 @@ func (c *clusterConfig) makeDaemonContainer(rgwConfig *rgwConfig) v1.Container {
cephconfig.NewFlag("name", generateCephXUser(rgwConfig.ResourceName)),
cephconfig.NewFlag("host", opspec.ContainerEnvVarReference("POD_NAME")),
cephconfig.NewFlag("rgw-mime-types-file", mimeTypesMountPath()),
), c.defaultSettings().GlobalFlags()..., // use default settings as flags until mon kv store supported
), c.defaultFlags()..., // use default settings as flags until mon kv store supported
),
VolumeMounts: append(
opspec.DaemonVolumeMounts(c.DataPathMap, rgwConfig.ResourceName),
+69 -11
View File
@@ -40,42 +40,99 @@ const (
var logger = capnslog.NewPackageLogger("github.com/rook/rook", "ceph-spec")
// return the volume and matching volume mount for mounting the config override ConfigMap into
// containers as "/rook/ceph/ceph.conf".
func configOverrideConfigMapVolumeAndMount() (v1.Volume, v1.VolumeMount) {
name := k8sutil.ConfigOverrideName // configmap name and name of volume
dir := config.EtcCephDir
file := "ceph.conf"
// TL;DR: mount the configmap's "config" to a file called "ceph.conf" with 0444 permissions
// security: allow to be read by everyone since now ceph processes run as 'ceph' and not 'root' user
// Further investigation needs to be done to copy the ceph.conf and change its ownership
// since configuring a owner of a ConfigMap secret is currently impossible
// This also works around the following issue: https://tracker.ceph.com/issues/38606
//
// This design choice avoids the crash/restart situation in Rook
// If we don't set 0444 to the ceph.conf configuration file during its respawn (with exec) the ceph-mgr
// won't be able to read the ceph.conf and the container will die, the "restart" count will increase in k8s
// This will mislead users thinking something won't wrong but that a false positive
mode := int32(0444)
v := v1.Volume{Name: name, VolumeSource: v1.VolumeSource{
ConfigMap: &v1.ConfigMapVolumeSource{LocalObjectReference: v1.LocalObjectReference{
Name: name},
Items: []v1.KeyToPath{
{Key: k8sutil.ConfigOverrideVal, Path: file, Mode: &mode},
},
}}}
// configmap's "config" to "/etc/ceph/ceph.conf"
m := v1.VolumeMount{
Name: name,
ReadOnly: true, // should be no reason to write to the config in pods, so enforce this
MountPath: dir,
}
return v, m
}
func confGeneratedInPodVolumeAndMount() (v1.Volume, v1.VolumeMount) {
name := "ceph-conf-emptydir"
dir := config.EtcCephDir
v := v1.Volume{Name: name, VolumeSource: v1.VolumeSource{
EmptyDir: &v1.EmptyDirVolumeSource{}}}
// configmap's "config" to "/etc/ceph/ceph.conf"
m := v1.VolumeMount{
Name: name,
MountPath: dir,
}
return v, m
}
// PodVolumes fills in the volumes parameter with the common list of Kubernetes volumes for use in Ceph pods.
func PodVolumes(dataDirHostPath, namespace string) []v1.Volume {
// This function is only used for OSDs.
func PodVolumes(dataDirHostPath, namespace string, confGeneratedInPod bool) []v1.Volume {
dataDirSource := v1.VolumeSource{EmptyDir: &v1.EmptyDirVolumeSource{}}
if dataDirHostPath != "" {
dataDirSource = v1.VolumeSource{HostPath: &v1.HostPathVolumeSource{Path: dataDirHostPath}}
}
configVolume, _ := configOverrideConfigMapVolumeAndMount()
if confGeneratedInPod {
configVolume, _ = confGeneratedInPodVolumeAndMount()
}
return []v1.Volume{
{Name: k8sutil.DataDirVolume, VolumeSource: dataDirSource},
cephconfig.DefaultConfigVolume(),
k8sutil.ConfigOverrideVolume(),
configVolume,
StoredLogVolume(path.Join(dataDirHostPath, "log", namespace)),
}
}
// CephVolumeMounts returns the common list of Kubernetes volume mounts for Ceph containers.
func CephVolumeMounts() []v1.VolumeMount {
// This function is only used for OSDs.
func CephVolumeMounts(confGeneratedInPod bool) []v1.VolumeMount {
_, configMount := configOverrideConfigMapVolumeAndMount()
if confGeneratedInPod {
_, configMount = confGeneratedInPodVolumeAndMount()
}
return []v1.VolumeMount{
{Name: k8sutil.DataDirVolume, MountPath: k8sutil.DataDir},
cephconfig.DefaultConfigMount(),
configMount,
StoredLogVolumeMount(),
// Rook doesn't run in ceph containers, so it doesn't need the config override mounted
}
}
// RookVolumeMounts returns the common list of Kubernetes volume mounts for Rook containers.
func RookVolumeMounts() []v1.VolumeMount {
// This function is only used by OSDs.
func RookVolumeMounts(confGeneratedInPod bool) []v1.VolumeMount {
return append(
CephVolumeMounts(),
k8sutil.ConfigOverrideMount(),
CephVolumeMounts(confGeneratedInPod),
)
}
// DaemonVolumesBase returns the common / static set of volumes.
func DaemonVolumesBase(dataPaths *config.DataPathMap, keyringResourceName string) []v1.Volume {
configOverrideVolume, _ := configOverrideConfigMapVolumeAndMount()
vols := []v1.Volume{
config.StoredFileVolume(),
configOverrideVolume,
}
if keyringResourceName != "" {
vols = append(vols, keyring.Volume().Resource(keyringResourceName))
@@ -139,8 +196,9 @@ func DaemonVolumes(dataPaths *config.DataPathMap, keyringResourceName string) []
// volume mounts are shared by most all Ceph daemon containers, both init and standard. If keyring
// resource name is empty, there will be no keyring mounted in the container.
func DaemonVolumeMounts(dataPaths *config.DataPathMap, keyringResourceName string) []v1.VolumeMount {
_, configOverrideMount := configOverrideConfigMapVolumeAndMount()
mounts := []v1.VolumeMount{
config.StoredFileVolumeMount(),
configOverrideMount,
}
if keyringResourceName != "" {
mounts = append(mounts, keyring.VolumeMount().Resource(keyringResourceName))
+14 -5
View File
@@ -29,10 +29,10 @@ import (
)
func TestPodVolumes(t *testing.T) {
if err := test.VolumeIsEmptyDir(k8sutil.DataDirVolume, PodVolumes("", "")); err != nil {
if err := test.VolumeIsEmptyDir(k8sutil.DataDirVolume, PodVolumes("", "", false)); err != nil {
t.Errorf("PodVolumes(\"\") - data dir source is not EmptyDir: %s", err.Error())
}
if err := test.VolumeIsHostPath(k8sutil.DataDirVolume, "/dev/sdb", PodVolumes("/dev/sdb", "rook-ceph")); err != nil {
if err := test.VolumeIsHostPath(k8sutil.DataDirVolume, "/dev/sdb", PodVolumes("/dev/sdb", "rook-ceph", false)); err != nil {
t.Errorf("PodVolumes(\"/dev/sdb\") - data dir source is not HostPath: %s", err.Error())
}
}
@@ -40,10 +40,19 @@ func TestPodVolumes(t *testing.T) {
func TestMountsMatchVolumes(t *testing.T) {
volsMountsTestDef := test.VolumesAndMountsTestDefinition{
VolumesSpec: &test.VolumesSpec{
Moniker: "PodVolumes(\"/dev/sdc\")", Volumes: PodVolumes("/dev/sdc", "rook-ceph")},
Moniker: "PodVolumes(\"/dev/sdc\")", Volumes: PodVolumes("/dev/sdc", "rook-ceph", false)},
MountsSpecItems: []*test.MountsSpec{
{Moniker: "CephVolumeMounts()", Mounts: CephVolumeMounts()},
{Moniker: "RookVolumeMounts()", Mounts: RookVolumeMounts()}},
{Moniker: "CephVolumeMounts(true)", Mounts: CephVolumeMounts(false)},
{Moniker: "RookVolumeMounts(true)", Mounts: RookVolumeMounts(false)}},
}
volsMountsTestDef.TestMountsMatchVolumes(t)
volsMountsTestDef = test.VolumesAndMountsTestDefinition{
VolumesSpec: &test.VolumesSpec{
Moniker: "PodVolumes(\"/dev/sdc\")", Volumes: PodVolumes("/dev/sdc", "rook-ceph", true)},
MountsSpecItems: []*test.MountsSpec{
{Moniker: "CephVolumeMounts(false)", Mounts: CephVolumeMounts(true)},
{Moniker: "RookVolumeMounts(false)", Mounts: RookVolumeMounts(true)}},
}
volsMountsTestDef.TestMountsMatchVolumes(t)
}
+1 -1
View File
@@ -71,7 +71,7 @@ func (ps *PodSpecTester) AssertVolumesMeetCephRequirements(
if daemonType == config.MonType {
keyringSecretName = "rook-ceph-mons-keyring" // mons share a keyring
}
requiredVols := []string{"rook-ceph-config", keyringSecretName}
requiredVols := []string{"rook-config-override", keyringSecretName}
if daemonType != config.RbdMirrorType {
requiredVols = append(requiredVols, "ceph-daemon-data")
}
+12 -3
View File
@@ -153,7 +153,7 @@ spec:
encryptedDevice:
type: string
pattern: ^(true|false)$
useAllDevices:
useAllDevices:
type: boolean
deviceFilter: {}
directories:
@@ -172,7 +172,7 @@ spec:
location: {}
resources: {}
type: array
useAllDevices:
useAllDevices:
type: boolean
deviceFilter: {}
location: {}
@@ -183,7 +183,7 @@ spec:
path:
type: string
config: {}
topologyAware:
topologyAware:
type: boolean
monitoring:
properties:
@@ -197,6 +197,15 @@ spec:
type: integer
placement: {}
resources: {}
configOverrides:
items:
properties:
who:
type: string
option:
type: string
value:
type: string
required:
- mon
additionalPrinterColumns: