Merge pull request #3573 from SUSE/mon-kv-store
Ceph: override configs in CRD
This commit is contained in:
34 files changed
+811
-712
No files matched your search
@@ -419,9 +419,14 @@ ceph osd pool set rbd pg_num 512
|
||||
```
|
||||
|
||||
## Custom ceph.conf Settings
|
||||
**WARNING:** The advised method for controlling Ceph configuration is to manually use the Ceph CLI
|
||||
or the Ceph dashboard because this offers the most flexibility. If configs must be set as part of
|
||||
Rook, the `configOverrides` section of the CephCluster resource is recommended. Read the
|
||||
[CephCluster CRD docs](ceph-cluster-crd.md#ceph-config-overrides) first before considering this
|
||||
advanced method.
|
||||
|
||||
With Rook the full swath of
|
||||
[Ceph settings](http://docs.ceph.com/docs/kraken/rados/configuration/) are available
|
||||
[Ceph settings](http://docs.ceph.com/docs/master/rados/configuration/) are available
|
||||
to use on your storage cluster. When we supply Rook with a ceph.conf file those
|
||||
settings will be propagated to all Mon, OSD, MDS, and RGW daemons to use.
|
||||
|
||||
@@ -431,7 +436,6 @@ daemons not to change the weight of OSDs on startup.
|
||||
**WARNING**: Modify Ceph settings carefully. You are leaving the sandbox tested by Rook.
|
||||
Changing the settings could result in unhealthy daemons or even data loss if used incorrectly.
|
||||
|
||||
### Kubernetes
|
||||
When the Rook Operator creates a cluster, a placeholder ConfigMap is created that
|
||||
will allow you to override Ceph configuration settings. When the daemon pods are started, the
|
||||
settings specified in this ConfigMap will be merged with the default settings
|
||||
@@ -492,7 +496,7 @@ the validity of the settings is your responsibility.
|
||||
|
||||
## OSD CRUSH Settings
|
||||
|
||||
A useful view of the [CRUSH Map](http://docs.ceph.com/docs/kraken/rados/operations/crush-map/)
|
||||
A useful view of the [CRUSH Map](http://docs.ceph.com/docs/master/rados/operations/crush-map/)
|
||||
is generated with the following command:
|
||||
|
||||
```bash
|
||||
|
||||
@@ -286,6 +286,62 @@ For more information on resource requests/limits see the official Kubernetes doc
|
||||
- `cpu`: Limit for CPU (example: one CPU core `1`, 50% of one CPU core `500m`).
|
||||
- `memory`: Limit for Memory (example: one gigabyte of memory `1Gi`, half a gigabyte of memory `512Mi`).
|
||||
|
||||
### Ceph config overrides
|
||||
Users are advised to use Ceph's CLI or dashboard to configure Ceph; however, some users may want to
|
||||
specify some settings they want set in the `CephCluster` manifest. This is possible using the
|
||||
`configOverrides` section. Config overrides follow the same syntax as [Ceph's documentation for
|
||||
setting config values](https://docs.ceph.com/docs/master/rados/configuration/ceph-conf/#commands),
|
||||
and configuration is stored in Ceph's centralized configuration database.
|
||||
|
||||
Each item in `configOverrides` has `who`, `option`, and `value` properties. `who` can be `global` to
|
||||
affect all Ceph daemons, a daemon type, an individual Ceph daemon, or a glob matching multiple
|
||||
daemons. It may also use a
|
||||
[mask](https://docs.ceph.com/docs/master/rados/configuration/ceph-conf/#sections-and-masks).
|
||||
`option` is the configuration option to be overridden, and `value` is the value to set on the
|
||||
configuration option. All properties are strings.
|
||||
|
||||
Rook will only set configuration overrides once when an orchestration is run. For example, on node
|
||||
addition or removal, when disks change on a node, or when the operator starts (or restarts). This
|
||||
means that users can change Ceph's configuration from the CLI or dashboard as desired without Rook
|
||||
constantly fighting to control the setting. This is particularly valuable in situations where values
|
||||
need to be changed to debug errors or test changes to tuning parameters.
|
||||
|
||||
**WARNING:** Remember that Rook will set these overrides any time the operator restarts, so user
|
||||
changes via Ceph's CLI or dashboard to values set here will not persist forever.
|
||||
|
||||
Some examples:
|
||||
```yaml
|
||||
configOverrides:
|
||||
- who: global
|
||||
option: log_file
|
||||
value: /var/log/custom-log-dir
|
||||
- who: mon
|
||||
option: mon_compact_on_start
|
||||
value: "true"
|
||||
- who: mgr.a
|
||||
option: mgr_stats_period
|
||||
value: "10"
|
||||
- who: osd.*
|
||||
option: osd_memory_target
|
||||
value: "2147483648"
|
||||
- who: osd/rack:foo
|
||||
option: debug_ms
|
||||
value: "20"
|
||||
```
|
||||
|
||||
#### Advanced config
|
||||
Setting configs in the Ceph mons' centralized database this way requires that at least one mon be
|
||||
available for the configs to be set. Ceph may also have a small number of very advanced settings
|
||||
that aren't able to be modified easily in the configuration database. In order to set configurations
|
||||
before monitors are available or to set problematic configuration settings, the
|
||||
`rook-config-override` ConfigMap exists, and the `config` field can be set with the contents of a
|
||||
`ceph.conf` file. It is highly recommended that this only be used when absolutely necessary and that
|
||||
the `config` be reset to an empty string if/when the configurations are no longer necessary, as this
|
||||
will make the Ceph cluster less configurable from the CLI and dashboard and may make future tuning
|
||||
or debugging difficult. Read more about this in the
|
||||
[advanced configuration docs](ceph-advanced-configuration.md#custom-cephconf-settings).
|
||||
|
||||
|
||||
## Samples
|
||||
Here are several samples for configuring Ceph clusters. Each of the samples must also include the namespace and corresponding access granted for management by the Ceph operator. See the [common cluster resources](#common-cluster-resources) below.
|
||||
|
||||
|
||||
@@ -272,6 +272,52 @@ At this point, your Rook operator should be running version `rook/ceph:v1.1.0`
|
||||
|
||||
Verify the Ceph cluster's health using the [health verification section](#health-verification).
|
||||
|
||||
### 6. (Optional) Migrate config overrides from ConfigMap to the CephCluster resource
|
||||
If there are Ceph configuration overrides set in the `config` field of the ConfigMap
|
||||
`rook-config-override`, it is now possible to migrate those configs manually from the ConfigMap to
|
||||
the newly-added `configOverrides` section of the `CephCluster` custom resource. This is not required
|
||||
but may be preferable, as the values set in `configOverrides` can be temporarily overridden by the
|
||||
user from the toolbox pod as needed in debug/failure scenarios. Values are reset if the operator
|
||||
restarts.
|
||||
|
||||
Ceph config file headers correspond to
|
||||
the `who` field of each config override as part of the section. For example, the `[global]` section
|
||||
corresponds to `who: global`, and `[osd.0]` corresponds to `who: osd.0`.
|
||||
|
||||
```sh
|
||||
# List the contents of the override ConfigMap
|
||||
$> kubectl --namespace $ROOK_NAMESPACE describe configmap rook-ceph-override -o yaml
|
||||
Name: rook-config-override
|
||||
Namespace: rook-ceph
|
||||
Labels: <none>
|
||||
Annotations: <none>
|
||||
|
||||
Data
|
||||
====
|
||||
config:
|
||||
----
|
||||
[global]
|
||||
debug_ms = 1/5
|
||||
|
||||
[osd.0]
|
||||
debug_osd = 10
|
||||
Events: <none>
|
||||
|
||||
# Edit the Ceph cluster resource, and add the overrides from the ConfigMap
|
||||
$> kubectl --namespace $ROOK_NAMESPACE edit cephcluster $ROOK_NAMESPACE
|
||||
# ...
|
||||
spec:
|
||||
# ...
|
||||
configOverrides:
|
||||
- who: global
|
||||
option: debug_ms
|
||||
value: 1/5
|
||||
- who: osd.0
|
||||
option: debug_osd
|
||||
value: "10"
|
||||
who: osd.0
|
||||
# ....
|
||||
```
|
||||
|
||||
# Ceph Version Upgrades
|
||||
Rook 1.0 was the last Rook release which will support Ceph's Luminous (v12.x.x) version. Users are
|
||||
|
||||
@@ -46,9 +46,11 @@ an example usage
|
||||
[monitor settings for more detail](Documentation/ceph-cluster-crd.md#mon-settings).
|
||||
- Ceph OSDs can be created by using StorageClassDeviceSet. See docs on [Storage Class Device Sets](Documentation/ceph-cluster-crd.md#storage-class-device-sets).
|
||||
- Rook can now connect to an external cluster, for more info about external cluster [read the design](https://github.com/rook/rook/blob/master/design/ceph-external-cluster.md) as well as the documentation [Ceph External cluster](Documentation/ceph-cluster-crd.md#external-cluster)
|
||||
- Added a new property in `storageClassDeviceSets` named `portable`:
|
||||
- Added a new property in `storageClassDeviceSets` named `portable`:
|
||||
- If `true`, the OSDs will be allowed to move between nodes during failover. This requires a storage class that supports portability (e.g. `aws-ebs`, but not the local storage provisioner).
|
||||
- If `false`, the OSDs will be assigned to a node permanently. Rook will configure Ceph's CRUSH map to support the portability.
|
||||
- The Ceph cluster custom resource now contains a `configOverrides` section where users can specify
|
||||
configuration changes to Ceph which Rook should apply.
|
||||
|
||||
### YugabyteDB
|
||||
|
||||
|
||||
@@ -143,3 +143,15 @@ spec:
|
||||
# storeType: filestore
|
||||
# - name: "172.17.4.301"
|
||||
# deviceFilter: "^sd."
|
||||
# Override Ceph configs. These can be overridden by the user at runtime using the Ceph CLI, and
|
||||
# Rook will set them back to the values set here when the operator restarts.
|
||||
configOverrides:
|
||||
- who: global
|
||||
option: debug ms
|
||||
value: "1/5"
|
||||
- who: osd.0
|
||||
option: debug_osd
|
||||
value: "10"
|
||||
- who: mon.a
|
||||
option: debug-mon
|
||||
value: "10"
|
||||
@@ -141,6 +141,15 @@ spec:
|
||||
type: integer
|
||||
placement: {}
|
||||
resources: {}
|
||||
configOverrides:
|
||||
items:
|
||||
properties:
|
||||
who:
|
||||
type: string
|
||||
option:
|
||||
type: string
|
||||
value:
|
||||
type: string
|
||||
additionalPrinterColumns:
|
||||
- name: DataDirHostPath
|
||||
type: string
|
||||
|
||||
@@ -69,6 +69,9 @@ type ClusterSpec struct {
|
||||
// The path on the host where config and data can be persisted.
|
||||
DataDirHostPath string `json:"dataDirHostPath,omitempty"`
|
||||
|
||||
// Ceph config overrides to apply.
|
||||
ConfigOverrides ConfigOverridesSpec `json:"configOverrides,omitempty"`
|
||||
|
||||
// A spec for mon related options
|
||||
Mon MonSpec `json:"mon,omitempty"`
|
||||
|
||||
@@ -148,6 +151,18 @@ const (
|
||||
ClusterStateError ClusterState = "Error"
|
||||
)
|
||||
|
||||
// ConfigOverridesSpec defines how Ceph configurations can be overridden by Rook.
|
||||
type ConfigOverridesSpec []ConfigOverride
|
||||
|
||||
// ConfigOverride defines the syntax for overriding a Ceph config. This translates to a
|
||||
// `ceph config set <who> <option> <value>` (or similar) command.
|
||||
// See Ceph docs: https://docs.ceph.com/docs/master/rados/configuration/ceph-conf/#monitor-configuration-database
|
||||
type ConfigOverride struct {
|
||||
Who string `json:"who"`
|
||||
Option string `json:"option"`
|
||||
Value string `json:"value"`
|
||||
}
|
||||
|
||||
type MonSpec struct {
|
||||
Count int `json:"count,omitempty"`
|
||||
AllowMultiplePerNode bool `json:"allowMultiplePerNode,omitempty"`
|
||||
|
||||
@@ -479,6 +479,11 @@ func (in *ClusterSpec) DeepCopyInto(out *ClusterSpec) {
|
||||
(*out)[key] = *val.DeepCopy()
|
||||
}
|
||||
}
|
||||
if in.ConfigOverrides != nil {
|
||||
in, out := &in.ConfigOverrides, &out.ConfigOverrides
|
||||
*out = make(ConfigOverridesSpec, len(*in))
|
||||
copy(*out, *in)
|
||||
}
|
||||
in.Mon.DeepCopyInto(&out.Mon)
|
||||
out.RBDMirroring = in.RBDMirroring
|
||||
in.Dashboard.DeepCopyInto(&out.Dashboard)
|
||||
@@ -518,6 +523,42 @@ func (in *ClusterStatus) DeepCopy() *ClusterStatus {
|
||||
return out
|
||||
}
|
||||
|
||||
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
|
||||
func (in *ConfigOverride) DeepCopyInto(out *ConfigOverride) {
|
||||
*out = *in
|
||||
return
|
||||
}
|
||||
|
||||
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ConfigOverride.
|
||||
func (in *ConfigOverride) DeepCopy() *ConfigOverride {
|
||||
if in == nil {
|
||||
return nil
|
||||
}
|
||||
out := new(ConfigOverride)
|
||||
in.DeepCopyInto(out)
|
||||
return out
|
||||
}
|
||||
|
||||
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
|
||||
func (in ConfigOverridesSpec) DeepCopyInto(out *ConfigOverridesSpec) {
|
||||
{
|
||||
in := &in
|
||||
*out = make(ConfigOverridesSpec, len(*in))
|
||||
copy(*out, *in)
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ConfigOverridesSpec.
|
||||
func (in ConfigOverridesSpec) DeepCopy() ConfigOverridesSpec {
|
||||
if in == nil {
|
||||
return nil
|
||||
}
|
||||
out := new(ConfigOverridesSpec)
|
||||
in.DeepCopyInto(out)
|
||||
return *out
|
||||
}
|
||||
|
||||
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
|
||||
func (in *DashboardSpec) DeepCopyInto(out *DashboardSpec) {
|
||||
*out = *in
|
||||
|
||||
@@ -43,6 +43,12 @@ const (
|
||||
cephConnectionTimeout = "15" // in seconds
|
||||
)
|
||||
|
||||
// CephConfFilePath returns the location to the cluster's config file in the operator container.
|
||||
func CephConfFilePath(configDir, clusterName string) string {
|
||||
confFile := fmt.Sprintf("%s.config", clusterName)
|
||||
return path.Join(configDir, clusterName, confFile)
|
||||
}
|
||||
|
||||
// FinalizeCephCommandArgs builds the command line to be called
|
||||
func FinalizeCephCommandArgs(command string, args []string, configDir, clusterName string) (string, []string) {
|
||||
// the rbd client tool does not support the '--connect-timeout' option
|
||||
@@ -68,11 +74,10 @@ func FinalizeCephCommandArgs(command string, args []string, configDir, clusterNa
|
||||
}
|
||||
|
||||
// Append the args to find the config and keyring
|
||||
confFile := fmt.Sprintf("%s.config", clusterName)
|
||||
keyringFile := fmt.Sprintf("%s.keyring", AdminUsername)
|
||||
configArgs := []string{
|
||||
fmt.Sprintf("--cluster=%s", clusterName),
|
||||
fmt.Sprintf("--conf=%s", path.Join(configDir, clusterName, confFile)),
|
||||
fmt.Sprintf("--conf=%s", CephConfFilePath(configDir, clusterName)),
|
||||
fmt.Sprintf("--keyring=%s", path.Join(configDir, clusterName, keyringFile)),
|
||||
}
|
||||
return command, append(args, configArgs...)
|
||||
|
||||
@@ -36,10 +36,6 @@ import (
|
||||
var logger = capnslog.NewPackageLogger("github.com/rook/rook", "cephconfig")
|
||||
|
||||
const (
|
||||
// DefaultConfigDir is the default dir where Ceph stores its configs
|
||||
DefaultConfigDir = "/etc/ceph"
|
||||
// DefaultConfigFile is the default name of the file where Ceph stores its configs
|
||||
DefaultConfigFile = "ceph.conf"
|
||||
// DefaultKeyringFile is the default name of the file where Ceph stores its keyring info
|
||||
DefaultKeyringFile = "keyring"
|
||||
// Msgr2port is the listening port of the messenger v2 protocol
|
||||
@@ -48,6 +44,16 @@ const (
|
||||
msgr2Prefix = "v2:"
|
||||
)
|
||||
|
||||
var (
|
||||
// DefaultConfigDir is the default dir where Ceph stores its configs. Can be overridden for unit
|
||||
// tests.
|
||||
DefaultConfigDir = "/etc/ceph"
|
||||
|
||||
// DefaultConfigFile is the default name of the file where Ceph stores its configs. Can be
|
||||
// overridden for unit tests.
|
||||
DefaultConfigFile = "ceph.conf"
|
||||
)
|
||||
|
||||
// GlobalConfig represents the [global] sections of Ceph's config file.
|
||||
type GlobalConfig struct {
|
||||
EnableExperimental string `ini:"enable experimental unrecoverable data corrupting features,omitempty"`
|
||||
@@ -109,26 +115,27 @@ func GetConfFilePath(root, clusterName string) string {
|
||||
|
||||
// GenerateAdminConnectionConfig calls GenerateAdminConnectionConfigWithSettings with no settings
|
||||
// overridden.
|
||||
func GenerateAdminConnectionConfig(context *clusterd.Context, cluster *ClusterInfo) error {
|
||||
func GenerateAdminConnectionConfig(context *clusterd.Context, cluster *ClusterInfo) (string, error) {
|
||||
return GenerateAdminConnectionConfigWithSettings(context, cluster, nil)
|
||||
}
|
||||
|
||||
// GenerateAdminConnectionConfigWithSettings generates a Ceph config and keyring which will allow
|
||||
// the daemon to connect as an admin. Default config file settings can be overridden by specifying
|
||||
// some subset of settings.
|
||||
func GenerateAdminConnectionConfigWithSettings(context *clusterd.Context, cluster *ClusterInfo, settings *CephConfig) error {
|
||||
func GenerateAdminConnectionConfigWithSettings(context *clusterd.Context, cluster *ClusterInfo, settings *CephConfig) (string, error) {
|
||||
root := path.Join(context.ConfigDir, cluster.Name)
|
||||
keyringPath := path.Join(root, fmt.Sprintf("%s.keyring", client.AdminUsername))
|
||||
err := writeKeyring(AdminKeyring(cluster), keyringPath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to write keyring to %s. %+v", root, err)
|
||||
return "", fmt.Errorf("failed to write keyring to %s. %+v", root, err)
|
||||
}
|
||||
|
||||
if _, err = GenerateConfigFile(context, cluster, root, client.AdminUsername, keyringPath, settings, nil); err != nil {
|
||||
return fmt.Errorf("failed to write config to %s. %+v", root, err)
|
||||
filePath, err := GenerateConfigFile(context, cluster, root, client.AdminUsername, keyringPath, settings, nil)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("failed to write config to %s. %+v", root, err)
|
||||
}
|
||||
logger.Infof("generated admin config in %s", root)
|
||||
return nil
|
||||
return filePath, nil
|
||||
}
|
||||
|
||||
// GenerateConfigFile generates and writes a config file to disk.
|
||||
@@ -149,15 +156,6 @@ func GenerateConfigFile(context *clusterd.Context, cluster *ClusterInfo, pathRoo
|
||||
return "", fmt.Errorf("failed to add admin client config section, %+v", err)
|
||||
}
|
||||
|
||||
// if there's a config file override path given, process the given config file
|
||||
if context.ConfigFileOverride != "" {
|
||||
err := configFile.Append(context.ConfigFileOverride)
|
||||
if err != nil {
|
||||
// log the config file override failure as a warning, but proceed without it
|
||||
logger.Warningf("failed to add config file override from '%s': %+v", context.ConfigFileOverride, err)
|
||||
}
|
||||
}
|
||||
|
||||
// write the entire config to disk
|
||||
filePath := GetConfFilePath(pathRoot, cluster.Name)
|
||||
logger.Infof("writing config file %s", filePath)
|
||||
@@ -165,13 +163,6 @@ func GenerateConfigFile(context *clusterd.Context, cluster *ClusterInfo, pathRoo
|
||||
return "", fmt.Errorf("failed to save config file %s. %+v", filePath, err)
|
||||
}
|
||||
|
||||
// copy the config to /etc/ceph/ceph.conf
|
||||
defaultPath := DefaultConfigFilePath()
|
||||
logger.Infof("copying config to %s", defaultPath)
|
||||
if err := configFile.SaveTo(defaultPath); err != nil {
|
||||
logger.Warningf("failed to save config file %s. %+v", defaultPath, err)
|
||||
}
|
||||
|
||||
return filePath, nil
|
||||
}
|
||||
|
||||
|
||||
@@ -84,19 +84,9 @@ func TestGenerateConfigFile(t *testing.T) {
|
||||
}
|
||||
defer os.RemoveAll(configDir)
|
||||
|
||||
// set up a config file override (in the config dir so it also gets cleaned up)
|
||||
configFileOverride := filepath.Join(configDir, "override.conf")
|
||||
overrideContents := `[global]
|
||||
debug bluestore = 1234`
|
||||
err = ioutil.WriteFile(configFileOverride, []byte(overrideContents), 0644)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to create config file override at '%s': %+v", configFileOverride, err)
|
||||
}
|
||||
|
||||
// create mocked cluster context and info
|
||||
context := &clusterd.Context{
|
||||
ConfigDir: configDir,
|
||||
ConfigFileOverride: configFileOverride,
|
||||
ConfigDir: configDir,
|
||||
}
|
||||
clusterInfo := &ClusterInfo{
|
||||
FSID: "myfsid",
|
||||
@@ -118,9 +108,6 @@ debug bluestore = 1234`
|
||||
actualConf, err := ini.Load(configFilePath)
|
||||
assert.Nil(t, err)
|
||||
verifyConfigValue(t, actualConf, "global", "fsid", clusterInfo.FSID)
|
||||
|
||||
// verify the content of the config file override successfully overwrote the default generated config
|
||||
verifyConfigValue(t, actualConf, "global", "debug bluestore", "1234")
|
||||
}
|
||||
|
||||
func verifyConfig(t *testing.T, cephConfig *CephConfig, cluster *ClusterInfo, loggingLevel int) {
|
||||
|
||||
@@ -1,36 +0,0 @@
|
||||
/*
|
||||
Copyright 2018 The Rook Authors. All rights reserved.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
*/
|
||||
|
||||
package config
|
||||
|
||||
import "k8s.io/api/core/v1"
|
||||
|
||||
const (
|
||||
// DefaultConfigMountName is the name of the volume mount used to mount the default Ceph config
|
||||
DefaultConfigMountName = "ceph-default-config-dir"
|
||||
)
|
||||
|
||||
// DefaultConfigVolume returns an empty volume used to store Ceph's config at the default path
|
||||
// in containers.
|
||||
func DefaultConfigVolume() v1.Volume {
|
||||
return v1.Volume{Name: DefaultConfigMountName,
|
||||
VolumeSource: v1.VolumeSource{EmptyDir: &v1.EmptyDirVolumeSource{}}}
|
||||
}
|
||||
|
||||
// DefaultConfigMount returns a volume mount to Ceph's default config path.
|
||||
func DefaultConfigMount() v1.VolumeMount {
|
||||
return v1.VolumeMount{Name: DefaultConfigMountName, MountPath: DefaultConfigDir}
|
||||
}
|
||||
@@ -1,28 +0,0 @@
|
||||
/*
|
||||
Copyright 2018 The Rook Authors. All rights reserved.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
*/
|
||||
|
||||
package config
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
)
|
||||
|
||||
func TestDefaultVolumes(t *testing.T) {
|
||||
// The name the volume uses must be the same as the one the mount uses
|
||||
assert.Equal(t, DefaultConfigVolume().Name, DefaultConfigMount().Name)
|
||||
}
|
||||
@@ -18,6 +18,7 @@ package osd
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"io/ioutil"
|
||||
"os"
|
||||
"path"
|
||||
"regexp"
|
||||
@@ -105,9 +106,24 @@ func Provision(context *clusterd.Context, agent *OsdAgent) error {
|
||||
cephConfig.GlobalConfig.CrushLocation = agent.location
|
||||
|
||||
// write the latest config to the config dir
|
||||
if err := cephconfig.GenerateAdminConnectionConfigWithSettings(context, agent.cluster, cephConfig); err != nil {
|
||||
confFilePath, err := cephconfig.GenerateAdminConnectionConfigWithSettings(context, agent.cluster, cephConfig)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to write connection config. %+v", err)
|
||||
}
|
||||
src, err := ioutil.ReadFile(confFilePath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to copy connection config to /etc/ceph. failed to read the connection config. %+v", err)
|
||||
}
|
||||
err = ioutil.WriteFile(cephconfig.DefaultConfigFilePath(), src, 0444)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to copy connection config to /etc/ceph. failed to write %s. %+v", cephconfig.DefaultConfigFilePath(), err)
|
||||
}
|
||||
dst, err := ioutil.ReadFile(cephconfig.DefaultConfigFilePath())
|
||||
if err == nil {
|
||||
logger.Debugf("config file @ %s: %s", cephconfig.DefaultConfigFilePath(), dst)
|
||||
} else {
|
||||
logger.Warningf("wrote and copied config file but failed to read it back from %s for logging. %+v", cephconfig.DefaultConfigFilePath(), err)
|
||||
}
|
||||
|
||||
logger.Infof("discovering hardware")
|
||||
var rawDevices []*sys.LocalDisk
|
||||
|
||||
@@ -23,6 +23,7 @@ import (
|
||||
"testing"
|
||||
|
||||
"github.com/rook/rook/pkg/clusterd"
|
||||
cephconfig "github.com/rook/rook/pkg/daemon/ceph/config"
|
||||
"github.com/rook/rook/pkg/operator/ceph/cluster/osd/config"
|
||||
exectest "github.com/rook/rook/pkg/util/exec/test"
|
||||
"github.com/rook/rook/pkg/util/sys"
|
||||
@@ -67,6 +68,10 @@ func TestRunDaemon(t *testing.T) {
|
||||
defer os.RemoveAll(configDir)
|
||||
os.MkdirAll(configDir, 0755)
|
||||
|
||||
defBkp := cephconfig.DefaultConfigDir
|
||||
cephconfig.DefaultConfigDir = configDir
|
||||
defer func() { cephconfig.DefaultConfigDir = defBkp }()
|
||||
|
||||
agent, _, context := createTestAgent(t, "none", configDir, "node5375", &config.StoreConfig{StoreType: config.Bluestore})
|
||||
agent.devices[0].IsFilter = true
|
||||
|
||||
|
||||
@@ -123,7 +123,7 @@ func CreateOrLoadClusterInfo(context *clusterd.Context, namespace string, ownerR
|
||||
// WriteConnectionConfig save monitor connection config to disk
|
||||
func WriteConnectionConfig(context *clusterd.Context, clusterInfo *cephconfig.ClusterInfo) error {
|
||||
// write the latest config to the config dir
|
||||
if err := cephconfig.GenerateAdminConnectionConfig(context, clusterInfo); err != nil {
|
||||
if _, err := cephconfig.GenerateAdminConnectionConfig(context, clusterInfo); err != nil {
|
||||
return fmt.Errorf("failed to write connection config. %+v", err)
|
||||
}
|
||||
|
||||
|
||||
@@ -220,12 +220,46 @@ func (c *Cluster) startMons(targetCount int) error {
|
||||
return fmt.Errorf("failed to assign pods to mons. %+v", err)
|
||||
}
|
||||
|
||||
// The centralized mon config database can only be used if there is at least one mon
|
||||
// operational. If we are starting mons, and one is already up, then there is a cluster already
|
||||
// created, and we can immediately set values in the config database. The goal is to set configs
|
||||
// only once and do it as early as possible in the mon orchestration.
|
||||
setConfigsNeedsRetry := false
|
||||
if existingCount > 0 {
|
||||
err := config.SetDefaultAndUserConfigs(c.context, c.Namespace, c.ClusterInfo, c.spec.ConfigOverrides)
|
||||
if err != nil {
|
||||
// If we fail here, it could be because the mons are not healthy, and this might be
|
||||
// fixed by updating the mon deployments. Instead of returning error here, log a
|
||||
// warning, and retry setting this later.
|
||||
setConfigsNeedsRetry = true
|
||||
logger.Warningf("failed to set Rook and/or user-defined Ceph config options before starting mons; will retry after starting mons. %+v", err)
|
||||
}
|
||||
}
|
||||
|
||||
if existingCount < len(mons) {
|
||||
// Start the new mons one at a time
|
||||
for i := existingCount; i < targetCount; i++ {
|
||||
if err := c.ensureMonsRunning(mons, i, targetCount, true); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// If this is the first mon being created, we have to wait until it is created to set
|
||||
// values in the config database. Do this only when the existing count is zero so that
|
||||
// this is only done once when the cluster is created.
|
||||
if existingCount == 0 {
|
||||
err := config.SetDefaultAndUserConfigs(c.context, c.Namespace, c.ClusterInfo, c.spec.ConfigOverrides)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to set Rook and/or user-defined Ceph config options after creating the first mon. %+v", err)
|
||||
}
|
||||
} else if setConfigsNeedsRetry && i == existingCount {
|
||||
// Or if we need to retry, only do this when we are on the first iteration of the
|
||||
// loop. This could be in the same if statement as above, but separate it to get a
|
||||
// different error message.
|
||||
err := config.SetDefaultAndUserConfigs(c.context, c.Namespace, c.ClusterInfo, c.spec.ConfigOverrides)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to set Rook and/or user-defined Ceph config options after updating the existing mons. %+v", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
} else {
|
||||
// Ensure all the expected mon deployments exist, but don't require full quorum to continue
|
||||
@@ -233,6 +267,13 @@ func (c *Cluster) startMons(targetCount int) error {
|
||||
if err := c.ensureMonsRunning(mons, lastMonIndex, targetCount, false); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if setConfigsNeedsRetry {
|
||||
err := config.SetDefaultAndUserConfigs(c.context, c.Namespace, c.ClusterInfo, c.spec.ConfigOverrides)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to set Rook and/or user-defined Ceph config options after forcefully updating the existing mons. %+v", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Enable Ceph messenger 2 protocol on Nautilus
|
||||
@@ -735,7 +776,7 @@ func (c *Cluster) updateMon(m *monConfig, d *apps.Deployment) error {
|
||||
currentCephVersion, err := client.LeastUptodateDaemonVersion(c.context, c.ClusterInfo.Name, daemonType)
|
||||
if err != nil {
|
||||
logger.Warningf("failed to retrieve current ceph %s version. %+v", daemonType, err)
|
||||
logger.Debug("could not detect ceph version during update, this is likely an initial bootstrap, proceeding with c.clusterInfo.CephVersion")
|
||||
logger.Debug("could not detect ceph version during update, this is likely an initial bootstrap, proceeding with %+v", c.ClusterInfo.CephVersion)
|
||||
cephVersionToUse = c.ClusterInfo.CephVersion
|
||||
} else {
|
||||
logger.Debugf("current cluster version for monitors before upgrading is: %+v", currentCephVersion)
|
||||
|
||||
@@ -188,8 +188,10 @@ func (c *Cluster) makeMonPod(monConfig *monConfig) *v1.Pod {
|
||||
c.makeMonDaemonContainer(monConfig),
|
||||
},
|
||||
RestartPolicy: v1.RestartPolicyAlways,
|
||||
Volumes: opspec.DaemonVolumesBase(monConfig.DataPathMap, keyringStoreName),
|
||||
HostNetwork: c.Network.IsHost(),
|
||||
// we decide later whether to use a PVC volume or host volumes for mons, so only populate
|
||||
// the base volumes at this point.
|
||||
Volumes: opspec.DaemonVolumesBase(monConfig.DataPathMap, keyringStoreName),
|
||||
HostNetwork: c.Network.IsHost(),
|
||||
}
|
||||
if c.Network.IsHost() {
|
||||
podSpec.DNSPolicy = v1.DNSClusterFirstWithHostNet
|
||||
|
||||
@@ -101,9 +101,9 @@ func (c *Cluster) makeJob(osdProps osdProperties) (*batch.Job, error) {
|
||||
func (c *Cluster) makeDeployment(osdProps osdProperties, osd OSDInfo) (*apps.Deployment, error) {
|
||||
|
||||
replicaCount := int32(1)
|
||||
volumeMounts := opspec.CephVolumeMounts()
|
||||
configVolumeMounts := opspec.RookVolumeMounts()
|
||||
volumes := opspec.PodVolumes(c.dataDirHostPath, c.Namespace)
|
||||
volumeMounts := opspec.CephVolumeMounts(false)
|
||||
configVolumeMounts := opspec.RookVolumeMounts(false)
|
||||
volumes := opspec.PodVolumes(c.dataDirHostPath, c.Namespace, false)
|
||||
failureDomainValue := osdProps.crushHostname
|
||||
|
||||
var dataDir string
|
||||
@@ -387,7 +387,9 @@ func (c *Cluster) provisionPodTemplateSpec(osdProps osdProperties, restart v1.Re
|
||||
|
||||
copyBinariesVolume, copyBinariesContainer := c.getCopyBinariesContainer()
|
||||
|
||||
volumes := append(opspec.PodVolumes(c.dataDirHostPath, c.Namespace), copyBinariesVolume)
|
||||
// ceph-volume is currently set up to use /etc/ceph/ceph.conf; this means no user config
|
||||
// overrides will apply to ceph-volume, but this is unnecessary anyway
|
||||
volumes := append(opspec.PodVolumes(c.dataDirHostPath, c.Namespace, true), copyBinariesVolume)
|
||||
|
||||
// by default, don't define any volume config unless it is required
|
||||
if len(osdProps.devices) > 0 || osdProps.selection.DeviceFilter != "" || osdProps.selection.GetUseAllDevices() || osdProps.metadataDevice != "" {
|
||||
@@ -422,8 +424,10 @@ func (c *Cluster) provisionPodTemplateSpec(osdProps osdProperties, restart v1.Re
|
||||
|
||||
podSpec := v1.PodSpec{
|
||||
ServiceAccountName: serviceAccountName,
|
||||
Containers: []v1.Container{
|
||||
InitContainers: []v1.Container{
|
||||
*copyBinariesContainer,
|
||||
},
|
||||
Containers: []v1.Container{
|
||||
c.provisionOSDContainer(osdProps, copyBinariesContainer.VolumeMounts[0]),
|
||||
},
|
||||
RestartPolicy: restart,
|
||||
@@ -574,7 +578,9 @@ func (c *Cluster) provisionOSDContainer(osdProps osdProperties, copyBinariesMoun
|
||||
devMountNeeded = true
|
||||
}
|
||||
|
||||
volumeMounts := append(opspec.CephVolumeMounts(), copyBinariesMount)
|
||||
// ceph-volume is currently set up to use /etc/ceph/ceph.conf; this means no user config
|
||||
// overrides will apply to ceph-volume, but this is unnecessary anyway
|
||||
volumeMounts := append(opspec.CephVolumeMounts(true), copyBinariesMount)
|
||||
if devMountNeeded {
|
||||
devMount := v1.VolumeMount{Name: "devices", MountPath: "/dev"}
|
||||
volumeMounts = append(volumeMounts, devMount)
|
||||
|
||||
@@ -48,11 +48,12 @@ func TestPodContainer(t *testing.T) {
|
||||
c, err := cluster.provisionPodTemplateSpec(osdProps, v1.RestartPolicyAlways)
|
||||
assert.NotNil(t, c)
|
||||
assert.Nil(t, err)
|
||||
assert.Equal(t, 2, len(c.Spec.Containers))
|
||||
container := c.Spec.Containers[0]
|
||||
assert.Equal(t, 1, len(c.Spec.InitContainers))
|
||||
assert.Equal(t, 1, len(c.Spec.Containers))
|
||||
container := c.Spec.InitContainers[0]
|
||||
logger.Infof("container: %+v", container)
|
||||
assert.Equal(t, "copy-binaries", container.Args[0])
|
||||
container = c.Spec.Containers[1]
|
||||
container = c.Spec.Containers[0]
|
||||
assert.Equal(t, "/rook/tini", container.Command[0])
|
||||
assert.Equal(t, "--", container.Args[0])
|
||||
assert.Equal(t, "/rook/rook", container.Args[1])
|
||||
@@ -112,17 +113,16 @@ func testPodDevices(t *testing.T, dataDir, deviceName string, allDevices bool) {
|
||||
assert.Equal(t, "node1", deployment.Spec.Template.Spec.NodeSelector[v1.LabelHostname])
|
||||
assert.Equal(t, v1.RestartPolicyAlways, deployment.Spec.Template.Spec.RestartPolicy)
|
||||
if devMountNeeded && len(dataDir) > 0 {
|
||||
assert.Equal(t, 6, len(deployment.Spec.Template.Spec.Volumes))
|
||||
assert.Equal(t, 5, len(deployment.Spec.Template.Spec.Volumes))
|
||||
}
|
||||
if devMountNeeded && len(dataDir) == 0 {
|
||||
assert.Equal(t, 6, len(deployment.Spec.Template.Spec.Volumes))
|
||||
assert.Equal(t, 5, len(deployment.Spec.Template.Spec.Volumes))
|
||||
}
|
||||
if !devMountNeeded && len(dataDir) > 0 {
|
||||
assert.Equal(t, 2, len(deployment.Spec.Template.Spec.Volumes))
|
||||
assert.Equal(t, 1, len(deployment.Spec.Template.Spec.Volumes))
|
||||
}
|
||||
|
||||
assert.Equal(t, "rook-data", deployment.Spec.Template.Spec.Volumes[0].Name)
|
||||
assert.Equal(t, "ceph-default-config-dir", deployment.Spec.Template.Spec.Volumes[1].Name)
|
||||
|
||||
assert.Equal(t, appName, deployment.Spec.Template.ObjectMeta.Name)
|
||||
assert.Equal(t, appName, deployment.Spec.Template.ObjectMeta.Labels["app"])
|
||||
@@ -133,7 +133,7 @@ func testPodDevices(t *testing.T, dataDir, deviceName string, allDevices bool) {
|
||||
initCont := deployment.Spec.Template.Spec.InitContainers[0]
|
||||
assert.Equal(t, "rook/rook:myversion", initCont.Image)
|
||||
assert.Equal(t, "config-init", initCont.Name)
|
||||
assert.Equal(t, 4, len(initCont.VolumeMounts))
|
||||
assert.Equal(t, 3, len(initCont.VolumeMounts))
|
||||
|
||||
assert.Equal(t, 1, len(deployment.Spec.Template.Spec.Containers))
|
||||
cont := deployment.Spec.Template.Spec.Containers[0]
|
||||
@@ -198,7 +198,7 @@ func TestStorageSpecDevicesAndDirectories(t *testing.T) {
|
||||
assert.Nil(t, err)
|
||||
// pod spec should have a volume for the given dir in the main container and the init container
|
||||
podSpec := deployment.Spec.Template.Spec
|
||||
assert.Equal(t, 6, len(podSpec.Volumes))
|
||||
assert.Equal(t, 5, len(podSpec.Volumes))
|
||||
require.Equal(t, 1, len(podSpec.Containers))
|
||||
cont := podSpec.Containers[0]
|
||||
assert.Equal(t, 5, len(cont.VolumeMounts))
|
||||
@@ -209,12 +209,11 @@ func TestStorageSpecDevicesAndDirectories(t *testing.T) {
|
||||
|
||||
require.Equal(t, 2, len(podSpec.InitContainers))
|
||||
initCont := podSpec.InitContainers[0]
|
||||
assert.Equal(t, 5, len(initCont.VolumeMounts))
|
||||
assert.Equal(t, 4, len(initCont.VolumeMounts))
|
||||
assert.Equal(t, "/var/lib/rook", initCont.VolumeMounts[0].MountPath)
|
||||
assert.Equal(t, "/etc/ceph", initCont.VolumeMounts[1].MountPath)
|
||||
assert.Equal(t, "/var/log/ceph", initCont.VolumeMounts[2].MountPath)
|
||||
assert.Equal(t, "/etc/rook/config", initCont.VolumeMounts[3].MountPath)
|
||||
assert.Equal(t, "/my/root/path", initCont.VolumeMounts[4].MountPath)
|
||||
assert.Equal(t, "/my/root/path", initCont.VolumeMounts[3].MountPath)
|
||||
|
||||
// the default osd created on a node will be under /var/lib/rook, which won't need an extra mount
|
||||
osd = OSDInfo{
|
||||
@@ -227,7 +226,7 @@ func TestStorageSpecDevicesAndDirectories(t *testing.T) {
|
||||
assert.Nil(t, err)
|
||||
// pod spec should have a volume for the given dir in the main container and the init container
|
||||
podSpec = deployment.Spec.Template.Spec
|
||||
assert.Equal(t, 5, len(podSpec.Volumes))
|
||||
assert.Equal(t, 4, len(podSpec.Volumes))
|
||||
require.Equal(t, 1, len(podSpec.Containers))
|
||||
cont = podSpec.Containers[0]
|
||||
require.Equal(t, 4, len(cont.VolumeMounts))
|
||||
@@ -238,11 +237,10 @@ func TestStorageSpecDevicesAndDirectories(t *testing.T) {
|
||||
|
||||
require.Equal(t, 2, len(podSpec.InitContainers))
|
||||
initCont = podSpec.InitContainers[0]
|
||||
require.Equal(t, 4, len(initCont.VolumeMounts))
|
||||
require.Equal(t, 3, len(initCont.VolumeMounts))
|
||||
assert.Equal(t, "/var/lib/rook", initCont.VolumeMounts[0].MountPath)
|
||||
assert.Equal(t, "/etc/ceph", initCont.VolumeMounts[1].MountPath)
|
||||
assert.Equal(t, "/var/log/ceph", initCont.VolumeMounts[2].MountPath)
|
||||
assert.Equal(t, "/etc/rook/config", initCont.VolumeMounts[3].MountPath)
|
||||
}
|
||||
|
||||
func TestStorageSpecConfig(t *testing.T) {
|
||||
@@ -299,9 +297,9 @@ func TestStorageSpecConfig(t *testing.T) {
|
||||
assert.NotNil(t, job)
|
||||
assert.Nil(t, err)
|
||||
assert.Equal(t, "rook-ceph-osd-prepare-node1", job.ObjectMeta.Name)
|
||||
container := job.Spec.Template.Spec.Containers[0]
|
||||
container := job.Spec.Template.Spec.InitContainers[0]
|
||||
assert.NotNil(t, container)
|
||||
container = job.Spec.Template.Spec.Containers[1]
|
||||
container = job.Spec.Template.Spec.Containers[0]
|
||||
assert.NotNil(t, container)
|
||||
verifyEnvVar(t, container.Env, "ROOK_OSD_STORE", "bluestore", true)
|
||||
verifyEnvVar(t, container.Env, "ROOK_OSD_DATABASE_SIZE", "10", true)
|
||||
|
||||
+104
-126
@@ -25,7 +25,11 @@ import (
|
||||
|
||||
"github.com/coreos/pkg/capnslog"
|
||||
"github.com/go-ini/ini"
|
||||
rookceph "github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
|
||||
"github.com/rook/rook/pkg/clusterd"
|
||||
cephconfig "github.com/rook/rook/pkg/daemon/ceph/config"
|
||||
"github.com/rook/rook/pkg/operator/k8sutil"
|
||||
"k8s.io/apimachinery/pkg/api/errors"
|
||||
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
|
||||
)
|
||||
|
||||
@@ -81,22 +85,6 @@ var (
|
||||
ContainerPostStartCmd = []string{"chown", "--recursive", "--verbose", chownUserGroup, VarLogCephDir}
|
||||
)
|
||||
|
||||
// Config is a Ceph config struct containing zero or more Ceph config sections.
|
||||
type Config struct {
|
||||
sections map[string]*Section
|
||||
sectionOrder []string
|
||||
context *clusterd.Context
|
||||
namespace string
|
||||
ownerRef *metav1.OwnerReference
|
||||
dataDirHostPath string
|
||||
}
|
||||
|
||||
// A Section is a collection of zero or more Ceph config key-value pairs.
|
||||
type Section struct {
|
||||
configs map[string]string
|
||||
configOrder []string // allows us to make an ordered dict
|
||||
}
|
||||
|
||||
// normalizeKey converts a key in any format to a key with underscores.
|
||||
//
|
||||
// The internal representation of Ceph config keys uses underscores only, where Ceph supports both
|
||||
@@ -107,98 +95,6 @@ func normalizeKey(key string) string {
|
||||
return strings.Replace(strings.Replace(key, " ", "_", -1), "-", "_", -1)
|
||||
}
|
||||
|
||||
// NewConfig returns a new, blank Ceph Config.
|
||||
func NewConfig() *Config {
|
||||
return &Config{
|
||||
sections: make(map[string]*Section),
|
||||
sectionOrder: make([]string, 0, 1), // capacity=1 b/c usually we'll only create [global]
|
||||
}
|
||||
}
|
||||
|
||||
// Section returns a Ceph config Section with the given name from the Config. If the section does
|
||||
// not exist, a new empty section is created and returned for the caller's use.
|
||||
func (c *Config) Section(name string) *Section {
|
||||
s, ok := c.sections[name]
|
||||
if !ok {
|
||||
s = &Section{
|
||||
configs: make(map[string]string),
|
||||
configOrder: []string{},
|
||||
}
|
||||
c.sections[name] = s
|
||||
c.sectionOrder = append(c.sectionOrder, name)
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// Set sets the Ceph config key to the given value. Set accepts Ceph keys in both
|
||||
// space-separated-words and underscore-separated-words formats. Set modifies the section in-place
|
||||
// and also returns the same modified-in-place Section to give flexibility of usage.
|
||||
//
|
||||
// NOTE that if a config item in a map should have quotes around it, they will have to be added to
|
||||
// the string explicitly. e.g., a Value of " test " will be interpreted as as 'Key = test', whereas
|
||||
// a Value of "\" test \"" will be interpreted as 'Key = " test "'
|
||||
func (s *Section) Set(key, value string) *Section {
|
||||
key = normalizeKey(key)
|
||||
_, ok := s.configs[key]
|
||||
if !ok {
|
||||
s.configOrder = append(s.configOrder, key)
|
||||
}
|
||||
s.configs[key] = value
|
||||
return s
|
||||
}
|
||||
|
||||
// Merge modifies the Section in-place adding values from the overrides Section and overriding
|
||||
// existing values with the new value in overrides.
|
||||
func (s *Section) Merge(overrides *Section) {
|
||||
for _, k := range overrides.configOrder {
|
||||
if v, ok := overrides.configs[k]; ok {
|
||||
s.Set(k, v)
|
||||
continue
|
||||
}
|
||||
// this condition should never exist except in case of some kind of mem corruption
|
||||
logger.Errorf("override config key %s could not be applied. possible struct corruption of overrides: %+v", k, overrides)
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
// Merge modifies the Config in-place, adding Sections from the overrides config, and overriding
|
||||
// existing Sections with Section values from overrides.
|
||||
func (c *Config) Merge(overrides *Config) {
|
||||
for _, hdr := range overrides.sectionOrder {
|
||||
if s, ok := overrides.sections[hdr]; ok {
|
||||
c.Section(hdr).Merge(s)
|
||||
continue
|
||||
}
|
||||
// this condition should never exist except in case of some kind of mem corruption
|
||||
logger.Errorf("override section %s could not be applied. possible struct corruption of overrides: %+v", hdr, overrides)
|
||||
}
|
||||
}
|
||||
|
||||
// IniFile converts the Ceph Config to an *ini.File.
|
||||
func (c *Config) IniFile() (*ini.File, error) {
|
||||
f := ini.Empty()
|
||||
for _, hdr := range c.sectionOrder {
|
||||
sec, ok := c.sections[hdr]
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("Error: section header %s not found: %+v", hdr, c)
|
||||
}
|
||||
s, err := f.NewSection(hdr)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("Error converting config to ini file text: %+v. %+v", c, err)
|
||||
}
|
||||
for _, k := range sec.configOrder {
|
||||
v, ok := sec.configs[k]
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("Error: config %s not found in section %s: %+v", k, hdr, sec)
|
||||
}
|
||||
if _, err := s.NewKey(k, v); err != nil {
|
||||
return nil, fmt.Errorf("Error converting config to ini file text: %+v. %+v", c, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
return f, nil
|
||||
}
|
||||
|
||||
// NewFlag returns the key-value pair in the format of a Ceph command line-compatible flag.
|
||||
func NewFlag(key, value string) string {
|
||||
// A flag is a normalized key with underscores replaced by dashes.
|
||||
@@ -208,23 +104,105 @@ func NewFlag(key, value string) string {
|
||||
return fmt.Sprintf("--%s=%s", f, value)
|
||||
}
|
||||
|
||||
// GlobalFlags converts the Config's 'global' Section to a list of command line flag strings which
|
||||
// can be used with Kubernetes container specs for Ceph daemons.
|
||||
//
|
||||
// It should never be necessary to get flags for anything but the 'global' config section,
|
||||
// otherwise, Rook will have to determine which sections apply for a given Ceph daemon. Does
|
||||
// 'mon.a' apply to 'mon.b'? 'mon.aa'? 'osd.0'? The logic would be prone to errors, so only allow
|
||||
// the 'global' section to be converted to flags.
|
||||
func (c *Config) GlobalFlags() []string {
|
||||
f := []string{}
|
||||
g := c.Section("global")
|
||||
for _, k := range g.configOrder {
|
||||
if v, ok := g.configs[k]; ok {
|
||||
f = append(f, NewFlag(k, v))
|
||||
continue
|
||||
}
|
||||
// this condition should never exist except in case of some kind of mem corruption
|
||||
logger.Errorf("override config key %s could not be applied. possible struct corruption of section: %+v", k, g)
|
||||
// SetDefaultAndUserConfigs sets Rook's desired default configs and the user's override configs from
|
||||
// the CephCluster CRD in the centralized monitor database. This cannot be called before at least
|
||||
// one monitor is established.
|
||||
func SetDefaultAndUserConfigs(
|
||||
context *clusterd.Context,
|
||||
namespace string,
|
||||
clusterInfo *cephconfig.ClusterInfo,
|
||||
configOverrides rookceph.ConfigOverridesSpec,
|
||||
) error {
|
||||
// ceph.conf is never used. All configurations are made in the centralized mon config database,
|
||||
// or they are specified on the commandline when daemons are called.
|
||||
monStore := GetMonStore(context, namespace)
|
||||
|
||||
if err := monStore.SetAll(DefaultCentralizedConfigs(clusterInfo.CephVersion)); err != nil {
|
||||
return fmt.Errorf("failed to apply default Ceph configurations. %+v", err)
|
||||
}
|
||||
return f
|
||||
|
||||
if err := monStore.SetAll(DefaultLegacyConfigs()); err != nil {
|
||||
return fmt.Errorf("failed to apply legacy config overrides. %+v", err)
|
||||
}
|
||||
|
||||
// user-specified config overrides from the CRD will go here
|
||||
if err := monStore.SetAll(configOverrides); err != nil {
|
||||
return fmt.Errorf("failed to apply one or more user-specified overrides. %+v", err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// LegacyConfigMapOverrideDefined checks to see if the legacy ConfigMap override exists with
|
||||
// overrides defined. If it exists and has config overrides defined, this will return true. If it
|
||||
// does not exist or exists without config overrides set, it will return false.
|
||||
func LegacyConfigMapOverrideDefined(context *clusterd.Context, namespace string) (bool, error) {
|
||||
cm, err := context.Clientset.CoreV1().ConfigMaps(namespace).
|
||||
Get(k8sutil.ConfigOverrideName, metav1.GetOptions{})
|
||||
if err != nil {
|
||||
if errors.IsNotFound(err) {
|
||||
logger.Debugf("override configmap does not exist")
|
||||
return false, nil
|
||||
}
|
||||
// if we can't determine existence or not, assume it exists
|
||||
return true, fmt.Errorf("failed to see if legacy override ConfigMap exists. %+v", err)
|
||||
}
|
||||
|
||||
o, ok := cm.Data[k8sutil.ConfigOverrideVal]
|
||||
if !ok || (ok && o == "") {
|
||||
// if no override set or override is empty string
|
||||
return false, nil
|
||||
}
|
||||
|
||||
return true, nil
|
||||
}
|
||||
|
||||
func configFileTextToOverrides(txt string) (rookceph.ConfigOverridesSpec, error) {
|
||||
overrides := []rookceph.ConfigOverride{}
|
||||
|
||||
f, err := configFileTxtToINI(txt)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
sections := f.Sections()
|
||||
for _, s := range sections {
|
||||
who := s.Name()
|
||||
keys := s.Keys()
|
||||
for _, k := range keys {
|
||||
option := k.Name()
|
||||
value := k.String()
|
||||
overrides = append(overrides, configOverride(who, option, value))
|
||||
}
|
||||
}
|
||||
|
||||
return overrides, nil
|
||||
}
|
||||
|
||||
func configFileTxtToINI(txt string) (*ini.File, error) {
|
||||
f := ini.Empty()
|
||||
ovrTxt := []byte(txt)
|
||||
if err := f.Append(ovrTxt); err != nil {
|
||||
return nil, fmt.Errorf("failed to interpret text as INI file '''%s'''. %+v", ovrTxt, err)
|
||||
}
|
||||
return f, nil
|
||||
}
|
||||
|
||||
func getOverrideConfigFromConfigMap(context *clusterd.Context, namespace string) string {
|
||||
cm, err := context.Clientset.CoreV1().ConfigMaps(namespace).
|
||||
Get(k8sutil.ConfigOverrideName, metav1.GetOptions{})
|
||||
if err != nil {
|
||||
if errors.IsNotFound(err) {
|
||||
logger.Debugf("override configmap does not exist")
|
||||
return "" // return empty string if cannot get override text
|
||||
}
|
||||
logger.Warningf("Error getting override configmap. %+v", err)
|
||||
return "" // return empty string if cannot get override text
|
||||
}
|
||||
// the override config map exists
|
||||
o, ok := cm.Data[k8sutil.ConfigOverrideVal]
|
||||
if !ok {
|
||||
logger.Warningf("The override config map does not have override text defined. %+v", o)
|
||||
return "" // return empty string if cannot get override text
|
||||
}
|
||||
return o
|
||||
}
|
||||
@@ -17,108 +17,11 @@ limitations under the License.
|
||||
package config
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
)
|
||||
|
||||
// Everything here is for the ultimate purpose of generating ini config files, and this is the main
|
||||
// output. Testing other functions would require this for testing output anyway, so just test all
|
||||
// the functions here.
|
||||
func TestConfig_FileText(t *testing.T) {
|
||||
c := NewConfig()
|
||||
|
||||
iniText := func() string {
|
||||
f, e := c.IniFile()
|
||||
assert.NoError(t, e)
|
||||
b := new(bytes.Buffer)
|
||||
_, e = f.WriteTo(b)
|
||||
assert.NoError(t, e)
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// Empty config
|
||||
f := iniText()
|
||||
assert.Equal(t, "", f)
|
||||
|
||||
// single section, empty
|
||||
c.Section("global")
|
||||
f = iniText()
|
||||
assert.Equal(t, "[global]\n\n", f)
|
||||
|
||||
// two sections, both empty
|
||||
c.Section("osd")
|
||||
f = iniText()
|
||||
assert.Equal(t, "[global]\n\n[osd]\n\n", f)
|
||||
|
||||
// add configs to empty global section
|
||||
c.Section("global").
|
||||
Set("key", "val").
|
||||
Set("key with spaces", "my-val").
|
||||
Set("key_with_underscores", "my val").
|
||||
Set("key-with-hyphens", "\"my_val \"")
|
||||
f = iniText()
|
||||
assert.Equal(t, `[global]
|
||||
key = val
|
||||
key_with_spaces = my-val
|
||||
key_with_underscores = my val
|
||||
key_with_hyphens = "my_val "
|
||||
|
||||
[osd]
|
||||
|
||||
`, f)
|
||||
|
||||
// change some of the global options to make sure the keys override properly
|
||||
c.Section("global").
|
||||
Set("key", "val"). // do not change this val to test this corner case
|
||||
Set("key_with_spaces", "my-val2").
|
||||
Set("key with underscores", "my val2")
|
||||
// add new section with a config set
|
||||
c.Section("mon").Set("debug mon", "100")
|
||||
f = iniText()
|
||||
assert.Equal(t, `[global]
|
||||
key = val
|
||||
key_with_spaces = my-val2
|
||||
key_with_underscores = my val2
|
||||
key_with_hyphens = "my_val "
|
||||
|
||||
[osd]
|
||||
|
||||
[mon]
|
||||
debug_mon = 100
|
||||
|
||||
`, f)
|
||||
|
||||
// test merging by creating a new config and merging it with the previous one
|
||||
// config merging uses section merging as part of it, so no need to test section merging
|
||||
m := NewConfig()
|
||||
m.Section("global"). // preexisting section
|
||||
Set("key", "val"). // preexisting key overridden w/ same val
|
||||
Set("key with spaces", "my-val"). // preexisting key overridden w/ diff val
|
||||
Set("new key", "new") // new key
|
||||
m.Section("mgr"). // new section
|
||||
Set("debug mgr", "100") // new val
|
||||
c.Merge(m)
|
||||
f = iniText()
|
||||
assert.Equal(t, `[global]
|
||||
key = val
|
||||
key_with_spaces = my-val
|
||||
key_with_underscores = my val2
|
||||
key_with_hyphens = "my_val "
|
||||
new_key = new
|
||||
|
||||
[osd]
|
||||
|
||||
[mon]
|
||||
debug_mon = 100
|
||||
|
||||
[mgr]
|
||||
debug_mgr = 100
|
||||
|
||||
`, f)
|
||||
}
|
||||
|
||||
func TestNewFlag(t *testing.T) {
|
||||
assert.Equal(t, NewFlag("k", ""), "--k=")
|
||||
assert.Equal(t, NewFlag("a-key", "a"), "--a-key=a")
|
||||
@@ -126,18 +29,3 @@ func TestNewFlag(t *testing.T) {
|
||||
assert.Equal(t, NewFlag("c key", "c"), "--c-key=c")
|
||||
assert.Equal(t, NewFlag("quotes", "\"quoted\""), "--quotes=\"quoted\"")
|
||||
}
|
||||
|
||||
func TestConfig_GlobalFlags(t *testing.T) {
|
||||
c := NewConfig()
|
||||
assert.Equal(t, c.GlobalFlags(), []string{})
|
||||
|
||||
c.Section("global").
|
||||
Set("test key", "one").
|
||||
Set("two_key", "2").
|
||||
Set("3-key", "\"trois \"")
|
||||
assert.ElementsMatch(t, []string{"--test-key=one", "--two-key=2", "--3-key=\"trois \""}, c.GlobalFlags())
|
||||
|
||||
c.Section("mon").
|
||||
Set("mon key", "m")
|
||||
assert.ElementsMatch(t, []string{"--test-key=one", "--two-key=2", "--3-key=\"trois \""}, c.GlobalFlags())
|
||||
}
|
||||
@@ -17,51 +17,54 @@ limitations under the License.
|
||||
// Package config provides default configurations which Rook will set in Ceph clusters.
|
||||
package config
|
||||
|
||||
import "github.com/rook/rook/pkg/operator/ceph/version"
|
||||
import (
|
||||
rookceph "github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
|
||||
"github.com/rook/rook/pkg/operator/ceph/version"
|
||||
)
|
||||
|
||||
// DefaultFlags returns the default configuration flags Rook will set on the command line for all
|
||||
// calls to Ceph daemons and tools. Values specified here will not be able to be overridden using
|
||||
// the mon's central KV store, and that is (and should be) by intent.
|
||||
func DefaultFlags(fsid, mountedKeyringPath string, cephVersion version.CephVersion) []string {
|
||||
return defaultFlagConfigs(fsid, mountedKeyringPath, cephVersion).GlobalFlags()
|
||||
}
|
||||
|
||||
func defaultFlagConfigs(fsid, mountedKeyringPath string, cephVersion version.CephVersion) *Config {
|
||||
c := NewConfig()
|
||||
|
||||
c.Section("global").
|
||||
flags := []string{
|
||||
// fsid unnecessary but is a safety to make sure daemons can only connect to their cluster
|
||||
Set("fsid", fsid).
|
||||
Set("keyring", mountedKeyringPath).
|
||||
NewFlag("fsid", fsid),
|
||||
NewFlag("keyring", mountedKeyringPath),
|
||||
// For containers, we're expected to log everything to stderr
|
||||
Set("log-to-stderr", "true").
|
||||
Set("err-to-stderr", "true").
|
||||
Set("mon-cluster-log-to-stderr", "true").
|
||||
Set("log-stderr-prefix", "debug ")
|
||||
// ^ differentiate debug text from audit text, and the space after 'debug' is critical
|
||||
|
||||
// As of Nautilus 14.2.1 at least
|
||||
// These new flags control Ceph's daemon logging behaviour to files
|
||||
// By default we set them to False so no logs get written on file
|
||||
// However they can be actived at any time via the centralized config store
|
||||
if cephVersion.IsAtLeast(version.CephVersion{Major: 14, Minor: 2, Extra: 1}) {
|
||||
c.Section("global").
|
||||
Set("default-log-to-file", "false").
|
||||
Set("default-mon-cluster-log-to-file", "false")
|
||||
NewFlag("log-to-stderr", "true"),
|
||||
NewFlag("err-to-stderr", "true"),
|
||||
NewFlag("mon-cluster-log-to-stderr", "true"),
|
||||
// differentiate debug text from audit text, and the space after 'debug' is critical
|
||||
NewFlag("log-stderr-prefix", "debug "),
|
||||
}
|
||||
|
||||
m := StoredMonHostEnvVarReferences()
|
||||
c.Merge(m)
|
||||
return c
|
||||
// As of Nautilus 14.2.1 at least
|
||||
// These new flags control Ceph's daemon logging behavior to files
|
||||
// By default we set them to False so no logs get written on file
|
||||
// However they can be activated at any time via the centralized config store
|
||||
if cephVersion.IsAtLeast(version.CephVersion{Major: 14, Minor: 2, Extra: 1}) {
|
||||
flags = append(flags, []string{
|
||||
NewFlag("default-log-to-file", "false"),
|
||||
NewFlag("default-mon-cluster-log-to-file", "false"),
|
||||
}...)
|
||||
}
|
||||
|
||||
flags = append(flags, StoredMonHostEnvVarFlags()...)
|
||||
|
||||
return flags
|
||||
}
|
||||
|
||||
// makes it possible to be slightly less verbose to create a ConfigOverride here
|
||||
func configOverride(who, option, value string) rookceph.ConfigOverride {
|
||||
return rookceph.ConfigOverride{Who: who, Option: option, Value: value}
|
||||
}
|
||||
|
||||
// DefaultCentralizedConfigs returns the default configuration options Rook will set in Ceph's
|
||||
// centralized config store. If the version of Ceph does not support the centralized config store,
|
||||
// these will be set in a shared config file instead.
|
||||
func DefaultCentralizedConfigs(cephVersion version.CephVersion) *Config {
|
||||
c := NewConfig()
|
||||
c.Section("global").
|
||||
Set("mon allow pool delete", "true")
|
||||
// centralized config store.
|
||||
func DefaultCentralizedConfigs(cephVersion version.CephVersion) rookceph.ConfigOverridesSpec {
|
||||
overrides := []rookceph.ConfigOverride{
|
||||
configOverride("global", "mon allow pool delete", "true"),
|
||||
}
|
||||
|
||||
// Everything before Nautilus 14.2.1
|
||||
// Prior to Nautilus 14.2.1 certain log flags were not present
|
||||
@@ -71,33 +74,21 @@ func DefaultCentralizedConfigs(cephVersion version.CephVersion) *Config {
|
||||
if !cephVersion.IsAtLeast(version.CephVersion{Major: 14, Minor: 2, Extra: 1}) {
|
||||
// Set the default log files to empty so they don't bloat containers. Can be changed in
|
||||
// Mimic+ by users if needed.
|
||||
c.Section("global").
|
||||
Set("log file", "").
|
||||
Set("mon cluster log file", "")
|
||||
overrides = append(overrides, []rookceph.ConfigOverride{
|
||||
configOverride("global", "log file", ""),
|
||||
configOverride("global", "mon cluster log file", ""),
|
||||
}...)
|
||||
}
|
||||
|
||||
return c
|
||||
return overrides
|
||||
}
|
||||
|
||||
// DefaultLegacyConfigs need to be added to the Ceph config file until the integration tests can be
|
||||
// made to override these options for the Ceph clusters it creates.
|
||||
func DefaultLegacyConfigs() *Config {
|
||||
c := NewConfig()
|
||||
c.Section("global").
|
||||
Set("mon max pg per osd", "1000").
|
||||
//
|
||||
Set("osd pg bits", "11").
|
||||
Set("osd pgp bits", "11").
|
||||
Set("osd pool default size", "1").
|
||||
Set("osd pool default min size", "1").
|
||||
Set("osd pool default pg num", "100").
|
||||
Set("osd pool default pgp num", "100").
|
||||
//
|
||||
Set("rbd_default_features", "3"). // TODO: still needed?
|
||||
// Setting fatal signal handlers to true (the default) will print a lot of extra information
|
||||
// from daemons when they encounter a failures, but it is VERY verbose. When the mon kv
|
||||
// store is available, it will probably be best to set this to false by default if it is
|
||||
// unset, but leave it set to true if the user has specified it as true during runtime.
|
||||
Set("fatal signal handlers", "false")
|
||||
return c
|
||||
func DefaultLegacyConfigs() rookceph.ConfigOverridesSpec {
|
||||
overrides := []rookceph.ConfigOverride{
|
||||
// TODO: drop this when FlexVolume is no longer supported
|
||||
configOverride("global", "rbd_default_features", "3"),
|
||||
}
|
||||
return overrides
|
||||
}
|
||||
@@ -0,0 +1,73 @@
|
||||
/*
|
||||
Copyright 2019 The Rook Authors. All rights reserved.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
*/
|
||||
|
||||
package config
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
|
||||
rookceph "github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
|
||||
"github.com/rook/rook/pkg/clusterd"
|
||||
"github.com/rook/rook/pkg/daemon/ceph/client"
|
||||
)
|
||||
|
||||
// MonStore provides methods for setting Ceph configurations in the centralized mon
|
||||
// configuration database.
|
||||
type MonStore struct {
|
||||
context *clusterd.Context
|
||||
namespace string
|
||||
}
|
||||
|
||||
// GetMonStore returns a new MonStore for the cluster.
|
||||
func GetMonStore(context *clusterd.Context, namespace string) *MonStore {
|
||||
return &MonStore{
|
||||
context: context,
|
||||
namespace: namespace,
|
||||
}
|
||||
}
|
||||
|
||||
// Set sets a config in the centralized mon configuration database.
|
||||
// https://docs.ceph.com/docs/master/rados/configuration/ceph-conf/#monitor-configuration-database
|
||||
func (m *MonStore) Set(who, option, value string) error {
|
||||
args := []string{"config", "set", who, normalizeKey(option), value}
|
||||
cephCmd := client.NewCephCommand(m.context, m.namespace, args)
|
||||
out, err := cephCmd.Run()
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to set Ceph config in the centralized mon configuration database; "+
|
||||
"you may need to use the rook-config-override ConfigMap. output: %s. %+v", string(out), err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// SetAll sets all configs from the overrides in the centralized mon configuration database.
|
||||
// See MonStore.Set for more.
|
||||
func (m *MonStore) SetAll(configOverrides rookceph.ConfigOverridesSpec) error {
|
||||
var errs []error
|
||||
for _, override := range configOverrides {
|
||||
err := m.Set(override.Who, override.Option, override.Value)
|
||||
if err != nil {
|
||||
errs = append(errs, err)
|
||||
}
|
||||
}
|
||||
if len(errs) > 0 {
|
||||
retErr := fmt.Errorf("failed to set one or more Ceph configs")
|
||||
for _, err := range errs {
|
||||
retErr = fmt.Errorf("%+v. %+v", retErr, err)
|
||||
}
|
||||
return retErr
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,131 @@
|
||||
/*
|
||||
Copyright 2019 The Rook Authors. All rights reserved.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
*/
|
||||
|
||||
package config
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
rookceph "github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
|
||||
"github.com/rook/rook/pkg/clusterd"
|
||||
testop "github.com/rook/rook/pkg/operator/test"
|
||||
exectest "github.com/rook/rook/pkg/util/exec/test"
|
||||
"github.com/stretchr/testify/assert"
|
||||
)
|
||||
|
||||
func TestMonStore_Set(t *testing.T) {
|
||||
executor := &exectest.MockExecutor{}
|
||||
ctx := &clusterd.Context{
|
||||
Clientset: testop.New(1),
|
||||
Executor: executor,
|
||||
}
|
||||
|
||||
// create a mock command runner which creates a simple string of the command it ran, and allow
|
||||
// us to cause it to return an error when it detects a keyword.
|
||||
execedCmd := ""
|
||||
execInjectErr := false
|
||||
executor.MockExecuteCommandWithOutputFile =
|
||||
func(debug bool, actionName string, command string, outfile string, args ...string) (string, error) {
|
||||
execedCmd = command + " " + strings.Join(args, " ")
|
||||
if execInjectErr {
|
||||
return "output from cmd with error", fmt.Errorf("mocked error")
|
||||
}
|
||||
return "", nil
|
||||
}
|
||||
|
||||
monStore := GetMonStore(ctx, "ns")
|
||||
|
||||
// setting with spaces converts to underscores
|
||||
e := monStore.Set("global", "debug ms", "10")
|
||||
assert.NoError(t, e)
|
||||
assert.Contains(t, execedCmd, "config set global debug_ms 10")
|
||||
|
||||
// setting with dashes converts to underscores
|
||||
e = monStore.Set("osd.0", "debug-osd", "20")
|
||||
assert.NoError(t, e)
|
||||
assert.Contains(t, execedCmd, " config set osd.0 debug_osd 20 ")
|
||||
|
||||
// setting with underscores stays the same
|
||||
e = monStore.Set("mds.*", "debug_mds", "15")
|
||||
assert.NoError(t, e)
|
||||
assert.Contains(t, execedCmd, " config set mds.* debug_mds 15 ")
|
||||
|
||||
// errors returned as expected
|
||||
execInjectErr = true
|
||||
e = monStore.Set("mon.*", "unknown_setting", "10")
|
||||
assert.Error(t, e)
|
||||
assert.Contains(t, execedCmd, " config set mon.* unknown_setting 10 ")
|
||||
}
|
||||
|
||||
func TestMonStore_SetAll(t *testing.T) {
|
||||
executor := &exectest.MockExecutor{}
|
||||
ctx := &clusterd.Context{
|
||||
Clientset: testop.New(1),
|
||||
Executor: executor,
|
||||
}
|
||||
|
||||
// create a mock command runner which creates a simple string of the command it ran, and allow
|
||||
// us to cause it to return an error when it detects a keyword.
|
||||
execedCmds := []string{}
|
||||
execInjectErrOnKeyword := "donotinjectanerror"
|
||||
executor.MockExecuteCommandWithOutputFile =
|
||||
func(debug bool, actionName string, command string, outfile string, args ...string) (string, error) {
|
||||
execedCmd := command + " " + strings.Join(args, " ")
|
||||
execedCmds = append(execedCmds, execedCmd)
|
||||
k := execInjectErrOnKeyword
|
||||
if strings.Contains(execedCmd, k) {
|
||||
return "output from cmd with error on keyword: " + k, fmt.Errorf("mocked error on keyword: " + k)
|
||||
}
|
||||
return "", nil
|
||||
}
|
||||
|
||||
monStore := GetMonStore(ctx, "ns")
|
||||
|
||||
var cfgOverrides rookceph.ConfigOverridesSpec
|
||||
|
||||
cfgOverrides = []rookceph.ConfigOverride{
|
||||
configOverride("global", "debug ms", "10"), // setting w/ spaces converts to underscores
|
||||
configOverride("osd.0", "debug-osd", "20"), // setting w/ dashes converts to underscores
|
||||
configOverride("mds.*", "debug_mds", "15"), // setting w/ underscores remains the same
|
||||
}
|
||||
|
||||
// commands w/ no error
|
||||
e := monStore.SetAll(cfgOverrides)
|
||||
assert.NoError(t, e)
|
||||
assert.Len(t, execedCmds, 3)
|
||||
assert.Contains(t, execedCmds[0], " global debug_ms 10 ")
|
||||
assert.Contains(t, execedCmds[1], " osd.0 debug_osd 20 ")
|
||||
assert.Contains(t, execedCmds[2], " mds.* debug_mds 15 ")
|
||||
|
||||
// commands w/ one error
|
||||
// keep cfgOverrides from last test
|
||||
execInjectErrOnKeyword = "debug_osd"
|
||||
execedCmds = execedCmds[:0] // empty execedCmds slice
|
||||
e = monStore.SetAll(cfgOverrides)
|
||||
assert.Error(t, e)
|
||||
// Rook should not return error before trying to set all config overrides
|
||||
assert.Len(t, execedCmds, 3)
|
||||
|
||||
// all commands return error
|
||||
// keep cfgOverrides
|
||||
execInjectErrOnKeyword = "debug"
|
||||
execedCmds = execedCmds[:0]
|
||||
e = monStore.SetAll(cfgOverrides)
|
||||
assert.Error(t, e)
|
||||
assert.Len(t, execedCmds, 3)
|
||||
}
|
||||
@@ -19,13 +19,11 @@ limitations under the License.
|
||||
package config
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"fmt"
|
||||
"net"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"github.com/go-ini/ini"
|
||||
"github.com/rook/rook/pkg/clusterd"
|
||||
cephconfig "github.com/rook/rook/pkg/daemon/ceph/config"
|
||||
cephutil "github.com/rook/rook/pkg/daemon/ceph/util"
|
||||
@@ -70,84 +68,14 @@ func GetStore(context *clusterd.Context, namespace string, ownerRef *metav1.Owne
|
||||
|
||||
// CreateOrUpdate creates or updates the stored Ceph config based on the cluster info.
|
||||
func (s *Store) CreateOrUpdate(clusterInfo *cephconfig.ClusterInfo) error {
|
||||
c := DefaultCentralizedConfigs(clusterInfo.CephVersion)
|
||||
|
||||
// DefaultLegacyConfigs need to be added to the Ceph config file until the integration tests can be
|
||||
// made to override these options for the Ceph clusters it creates.
|
||||
c.Merge(DefaultLegacyConfigs())
|
||||
|
||||
/* TODO: config overrides from the CRD will go here */
|
||||
|
||||
f, err := c.IniFile()
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to store the Ceph config file. could not generate default ini. %+v", err)
|
||||
}
|
||||
|
||||
// merge config overrides from the legacy override configmap
|
||||
b := new(bytes.Buffer)
|
||||
if err := s.applyLegacyOverrides(f); err != nil {
|
||||
logger.Warningf("failed to merge config override with default Ceph config file; using only the default config. %+v", err)
|
||||
f, err = c.IniFile() // regenerate the orig ini in case override attempt altered the struct
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to store the Ceph config file. could not generate default ini after merge failure. %+v", err)
|
||||
}
|
||||
}
|
||||
if _, err := f.WriteTo(b); err != nil {
|
||||
return fmt.Errorf("failed to convert merged Ceph config to text; continuing with original config. %+v", err)
|
||||
}
|
||||
txt := b.String()
|
||||
|
||||
// Store the config in a configmap
|
||||
if err := s.configMapStore.SetValue(StoreName, confFileName, txt); err != nil {
|
||||
return fmt.Errorf("failed to store the Ceph config file. failed to store config to configmap. %+v", err)
|
||||
}
|
||||
logger.Debugf("Generated and stored config file:\n%s", txt)
|
||||
|
||||
// these are used for all ceph daemons on the commandline and must *always* be stored
|
||||
if err := s.createOrUpdateMonHostSecrets(clusterInfo); err != nil {
|
||||
return fmt.Errorf("failed to store mon host configs. %+v", err)
|
||||
}
|
||||
|
||||
/*
|
||||
TODO:
|
||||
set config values with `ceph config assimilage-conf t` after at least one mon is active
|
||||
*/
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *Store) applyLegacyOverrides(toFile *ini.File) error {
|
||||
ovrTxt := []byte(s.overrideConfig())
|
||||
if err := toFile.Append(ovrTxt); err != nil {
|
||||
return fmt.Errorf("failed to apply config file override from '''%s'''. %+v", ovrTxt, err)
|
||||
}
|
||||
b := new(bytes.Buffer)
|
||||
if _, err := toFile.WriteTo(b); err != nil {
|
||||
return fmt.Errorf("failed to write out toFile config file '''%+v'''. %+v", toFile, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *Store) overrideConfig() string {
|
||||
cm, err := s.context.Clientset.CoreV1().ConfigMaps(s.namespace).
|
||||
Get(k8sutil.ConfigOverrideName, metav1.GetOptions{})
|
||||
if err != nil {
|
||||
if errors.IsNotFound(err) {
|
||||
logger.Debugf("override configmap does not exist")
|
||||
return "" // return empty string if cannot get override text
|
||||
}
|
||||
logger.Warningf("Error getting override configmap. %+v", err)
|
||||
return "" // return empty string if cannot get override text
|
||||
}
|
||||
// the override config map exists
|
||||
o, ok := cm.Data[k8sutil.ConfigOverrideVal]
|
||||
if !ok {
|
||||
logger.Warningf("The override config map does not have override text defined. %+v", o)
|
||||
return "" // return empty string if cannot get override text
|
||||
}
|
||||
return o
|
||||
}
|
||||
|
||||
// update "mon_host" and "mon_initial_members" in the stored config
|
||||
func (s *Store) createOrUpdateMonHostSecrets(clusterInfo *cephconfig.ClusterInfo) error {
|
||||
hosts := make([]string, len(clusterInfo.Monitors))
|
||||
@@ -216,40 +144,6 @@ func (s *Store) createOrUpdateMonHostSecrets(clusterInfo *cephconfig.ClusterInfo
|
||||
return nil
|
||||
}
|
||||
|
||||
// StoredFileVolume returns a pod volume sourced from the stored config file.
|
||||
func StoredFileVolume() v1.Volume {
|
||||
// TL;DR: mount the configmap's "ceph.conf" to a file called "ceph.conf" with 0444 permissions
|
||||
// security: allow to be read by everyone since now ceph processes run as 'ceph' and not 'root' user
|
||||
// Further investigation needs to be done to copy the ceph.conf and change its ownership
|
||||
// since configuring a owner of a ConfigMap secret is currently impossible
|
||||
// This also works around the following issue: https://tracker.ceph.com/issues/38606
|
||||
//
|
||||
// This design choice avoids the crash/restart situation in Rook
|
||||
// If we don't set 0444 to the ceph.conf configuration file during its respawn (with exec) the ceph-mgr
|
||||
// won't be able to read the ceph.conf and the container will die, the "restart" count will increase in k8s
|
||||
// This will mislead users thinking something won't wrong but that a false positive
|
||||
mode := int32(0444)
|
||||
return v1.Volume{
|
||||
Name: configVolumeName,
|
||||
VolumeSource: v1.VolumeSource{
|
||||
ConfigMap: &v1.ConfigMapVolumeSource{LocalObjectReference: v1.LocalObjectReference{
|
||||
Name: StoreName},
|
||||
Items: []v1.KeyToPath{
|
||||
{Key: confFileName, Path: confFileName, Mode: &mode},
|
||||
}}}}
|
||||
}
|
||||
|
||||
// StoredFileVolumeMount returns a container volume mount that mounts the stored config from
|
||||
// StoredFileVolume into the container at `/etc/ceph/ceph.conf`.
|
||||
func StoredFileVolumeMount() v1.VolumeMount {
|
||||
// configmap's "ceph.conf" to "/etc/ceph/ceph.conf"
|
||||
return v1.VolumeMount{
|
||||
Name: StoreName,
|
||||
ReadOnly: true, // should be no reason to write to the config in pods, so enforce this
|
||||
MountPath: "/etc/ceph",
|
||||
}
|
||||
}
|
||||
|
||||
// StoredMonHostEnvVars returns a container environment variable defined by the most updated stored
|
||||
// "mon_host" and "mon_initial_members" information.
|
||||
func StoredMonHostEnvVars() []v1.EnvVar {
|
||||
@@ -267,13 +161,11 @@ func StoredMonHostEnvVars() []v1.EnvVar {
|
||||
}
|
||||
}
|
||||
|
||||
// StoredMonHostEnvVarReferences returns a small Ceph Config which references "mon_host" and
|
||||
// "mon_initial_members" information from the StoredMonHostEnvVars. This config can be used to
|
||||
// generate flags referencing the env vars or to generate the string representation of a Config.
|
||||
func StoredMonHostEnvVarReferences() *Config {
|
||||
c := NewConfig()
|
||||
c.Section("global").
|
||||
Set(monHostKey, "$(ROOK_CEPH_MON_HOST)").
|
||||
Set(monInitialMembersKey, "$(ROOK_CEPH_MON_INITIAL_MEMBERS)")
|
||||
return c
|
||||
// StoredMonHostEnvVarFlags returns Ceph commandline flag references to "mon_host" and
|
||||
// "mon_initial_members" sourced from the StoredMonHostEnvVars.
|
||||
func StoredMonHostEnvVarFlags() []string {
|
||||
return []string{
|
||||
NewFlag(monHostKey, "$(ROOK_CEPH_MON_HOST)"),
|
||||
NewFlag(monInitialMembersKey, "$(ROOK_CEPH_MON_INITIAL_MEMBERS)"),
|
||||
}
|
||||
}
|
||||
@@ -17,19 +17,14 @@ limitations under the License.
|
||||
package config
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"fmt"
|
||||
"path"
|
||||
"regexp"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/rook/rook/pkg/clusterd"
|
||||
cephconfig "github.com/rook/rook/pkg/daemon/ceph/config"
|
||||
"github.com/rook/rook/pkg/operator/k8sutil"
|
||||
testop "github.com/rook/rook/pkg/operator/test"
|
||||
"github.com/stretchr/testify/assert"
|
||||
v1 "k8s.io/api/core/v1"
|
||||
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
|
||||
)
|
||||
|
||||
@@ -43,16 +38,7 @@ func TestStore(t *testing.T) {
|
||||
|
||||
s := GetStore(ctx, ns, &owner)
|
||||
|
||||
var previousConfigText = ""
|
||||
var recentConfigText = ""
|
||||
assertConfigStore := func(ci *cephconfig.ClusterInfo) {
|
||||
c, e := clientset.CoreV1().ConfigMaps(ns).Get(StoreName, metav1.GetOptions{})
|
||||
assert.NoError(t, e)
|
||||
|
||||
previousConfigText = recentConfigText
|
||||
recentConfigText = c.Data[confFileName]
|
||||
assert.NotEqual(t, "", recentConfigText)
|
||||
|
||||
sec, e := clientset.CoreV1().Secrets(ns).Get(StoreName, metav1.GetOptions{})
|
||||
assert.NoError(t, e)
|
||||
mh := strings.Split(sec.StringData["mon_host"], ",") // list of mon ip:port pairs in cluster
|
||||
@@ -66,20 +52,6 @@ func TestStore(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
findLine := func(lineRegex, inText string) int {
|
||||
scr := bufio.NewScanner(strings.NewReader(inText))
|
||||
i := 1
|
||||
for scr.Scan() {
|
||||
m, e := regexp.Match(lineRegex, scr.Bytes())
|
||||
assert.NoError(t, e)
|
||||
if m {
|
||||
return i
|
||||
}
|
||||
i++
|
||||
}
|
||||
return 0 // ret 0 if not found
|
||||
}
|
||||
|
||||
i1 := testop.CreateConfigDir(1) // cluster w/ one mon
|
||||
i3 := testop.CreateConfigDir(3) // same cluster w/ 3 mons
|
||||
|
||||
@@ -88,108 +60,6 @@ func TestStore(t *testing.T) {
|
||||
|
||||
s.CreateOrUpdate(i3)
|
||||
assertConfigStore(i3)
|
||||
// the config should be the same regardless of how many mons there are
|
||||
assert.Equal(t, previousConfigText, recentConfigText)
|
||||
|
||||
//
|
||||
// test overrides
|
||||
//
|
||||
createOverrideMap(t, ctx, ns, &owner)
|
||||
s.CreateOrUpdate(i3)
|
||||
assertConfigStore(i3)
|
||||
// configs should still be equal since the created map doesn't have data in it
|
||||
assert.Equal(t, previousConfigText, recentConfigText)
|
||||
|
||||
updateOverrideMap(t, "", ctx, ns, &owner)
|
||||
s.CreateOrUpdate(i3)
|
||||
assertConfigStore(i3)
|
||||
// configs should still be equal since the created map's data is blank
|
||||
assert.Equal(t, previousConfigText, recentConfigText)
|
||||
|
||||
updateOverrideMap(t, "this is not valid ini file text", ctx, ns, &owner)
|
||||
s.CreateOrUpdate(i3)
|
||||
assertConfigStore(i3)
|
||||
// configs should still be equal since the created map's data is invalid ini
|
||||
assert.Equal(t, previousConfigText, recentConfigText)
|
||||
|
||||
// complicated override with overrides, new keys, new section
|
||||
updateOverrideMap(t, `[global]
|
||||
log_file = notreal
|
||||
new_key = fake
|
||||
log stderr prefix = '"debug "'
|
||||
|
||||
[mon]
|
||||
debug_mon = makehaste
|
||||
`, ctx, ns, &owner)
|
||||
s.CreateOrUpdate(i3)
|
||||
assertConfigStore(i3)
|
||||
// Verify some simple truths about the overridden config vs the original
|
||||
assert.NotEqual(t, previousConfigText, recentConfigText) // the new config has changed (finally)
|
||||
l1 := findLine("log_file[[:space:]]+= ", previousConfigText) //
|
||||
assert.NotZero(t, l1) // orig 'log_file = <num>' exists
|
||||
l2 := findLine("log_file[[:space:]]+= notreal", recentConfigText) //
|
||||
assert.Equal(t, l1, l2) // overridden debug_default still has same line num
|
||||
l3 := findLine("new_key[[:space:]]+= fake", recentConfigText) //
|
||||
assert.True(t, l3 > l2) // new_key should come after debug_default
|
||||
l4 := findLine("log stderr prefix[[:space:]]+= \"debug \"", recentConfigText) // becomes "debug " w/o single quotes
|
||||
assert.Equal(t, l3+1, l4) // ... and is 1 line after new_key
|
||||
l5 := findLine("[[]mon[]]", recentConfigText) //
|
||||
assert.Equal(t, l4+2, l5) // [mon] hdr comes 2 after
|
||||
l6 := findLine("debug_mon[[:space:]]+= makehaste", recentConfigText) //
|
||||
assert.Equal(t, l5+1, l6) // debug_mon comes 1 after
|
||||
//
|
||||
lg := findLine("[[]global[]]", recentConfigText) //
|
||||
assert.Equal(t, 1, lg) // [global] hdr is first line
|
||||
ln := findLine("[[].*[]]", strings.TrimLeft(recentConfigText, "[global]")) //
|
||||
assert.Equal(t, l5, ln) // next hdr after [global] is [mon]
|
||||
|
||||
}
|
||||
|
||||
func createOverrideMap(t *testing.T,
|
||||
context *clusterd.Context, namespace string, ownerRef *metav1.OwnerReference,
|
||||
) {
|
||||
cm := &v1.ConfigMap{
|
||||
ObjectMeta: metav1.ObjectMeta{
|
||||
Name: k8sutil.ConfigOverrideName,
|
||||
},
|
||||
}
|
||||
k8sutil.SetOwnerRef(&cm.ObjectMeta, ownerRef)
|
||||
_, err := context.Clientset.CoreV1().ConfigMaps(namespace).Create(cm)
|
||||
assert.NoError(t, err)
|
||||
}
|
||||
|
||||
func updateOverrideMap(t *testing.T, overrideText string,
|
||||
context *clusterd.Context, namespace string, ownerRef *metav1.OwnerReference,
|
||||
) {
|
||||
cm := &v1.ConfigMap{
|
||||
ObjectMeta: metav1.ObjectMeta{
|
||||
Name: k8sutil.ConfigOverrideName,
|
||||
},
|
||||
Data: map[string]string{k8sutil.ConfigOverrideVal: overrideText},
|
||||
}
|
||||
k8sutil.SetOwnerRef(&cm.ObjectMeta, ownerRef)
|
||||
_, err := context.Clientset.CoreV1().ConfigMaps(namespace).Update(cm)
|
||||
assert.NoError(t, err)
|
||||
}
|
||||
|
||||
func TestFileVolumeAndMount(t *testing.T) {
|
||||
clientset := testop.New(1)
|
||||
ctx := &clusterd.Context{
|
||||
Clientset: clientset,
|
||||
}
|
||||
ns := "rook-ceph"
|
||||
owner := metav1.OwnerReference{}
|
||||
|
||||
s := GetStore(ctx, ns, &owner)
|
||||
s.CreateOrUpdate(testop.CreateConfigDir(3))
|
||||
|
||||
v := StoredFileVolume()
|
||||
m := StoredFileVolumeMount()
|
||||
// Test that the configmapped file will make it into containers with the appropriate filename at
|
||||
// the location where it is expected.
|
||||
assert.Equal(t, v.Name, m.Name)
|
||||
assert.Equal(t, StoreName, v.VolumeSource.ConfigMap.LocalObjectReference.Name)
|
||||
assert.Equal(t, "/etc/ceph/ceph.conf", path.Join(m.MountPath, confFileName))
|
||||
}
|
||||
|
||||
func TestEnvVarsAndFlags(t *testing.T) {
|
||||
@@ -204,7 +74,7 @@ func TestEnvVarsAndFlags(t *testing.T) {
|
||||
s.CreateOrUpdate(testop.CreateConfigDir(3))
|
||||
|
||||
v := StoredMonHostEnvVars()
|
||||
f := StoredMonHostEnvVarReferences().GlobalFlags()
|
||||
f := StoredMonHostEnvVarFlags()
|
||||
|
||||
// make sure the env var names and flags are matching pairs
|
||||
mh := v[0].Name
|
||||
|
||||
@@ -18,10 +18,14 @@ limitations under the License.
|
||||
package nfs
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"fmt"
|
||||
"os/exec"
|
||||
"strings"
|
||||
|
||||
cephv1 "github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
|
||||
"github.com/rook/rook/pkg/clusterd"
|
||||
cephclient "github.com/rook/rook/pkg/daemon/ceph/client"
|
||||
cephconfig "github.com/rook/rook/pkg/daemon/ceph/config"
|
||||
opmon "github.com/rook/rook/pkg/operator/ceph/cluster/mon"
|
||||
"github.com/rook/rook/pkg/operator/ceph/config"
|
||||
@@ -107,13 +111,21 @@ func (c *CephNFSController) upCephNFS(n cephv1.CephNFS, oldActive int) error {
|
||||
func (c *CephNFSController) addRADOSConfigFile(n cephv1.CephNFS, name string) error {
|
||||
nodeID := getNFSNodeID(n, name)
|
||||
config := getGaneshaConfigObject(nodeID)
|
||||
err := c.context.Executor.ExecuteCommand(false, "", "rados", "--pool", n.Spec.RADOS.Pool, "--namespace", n.Spec.RADOS.Namespace, "stat", config)
|
||||
cmd := "rados"
|
||||
args := []string{
|
||||
"--pool", n.Spec.RADOS.Pool,
|
||||
"--namespace", n.Spec.RADOS.Namespace,
|
||||
"--conf", cephclient.CephConfFilePath(c.context.ConfigDir, c.namespace),
|
||||
}
|
||||
moniker := "rados stat " + config
|
||||
err := c.context.Executor.ExecuteCommand(false, moniker, cmd, append(args, "stat", config)...)
|
||||
if err == nil {
|
||||
// If stat works then we assume it's present already
|
||||
return nil
|
||||
}
|
||||
// try to create it
|
||||
return c.context.Executor.ExecuteCommand(false, "", "rados", "--pool", n.Spec.RADOS.Pool, "--namespace", n.Spec.RADOS.Namespace, "create", config)
|
||||
moniker = "rados create " + config
|
||||
return c.context.Executor.ExecuteCommand(false, moniker, cmd, append(args, "create", config)...)
|
||||
}
|
||||
|
||||
func (c *CephNFSController) addServerToDatabase(nfs cephv1.CephNFS, name string) {
|
||||
@@ -134,8 +146,25 @@ func (c *CephNFSController) removeServerFromDatabase(nfs cephv1.CephNFS, name st
|
||||
|
||||
func (c *CephNFSController) runGaneshaRadosGrace(nfs cephv1.CephNFS, name, action string) error {
|
||||
nodeID := getNFSNodeID(nfs, name)
|
||||
cmd := ganeshaRadosGraceCmd
|
||||
args := []string{"--pool", nfs.Spec.RADOS.Pool, "--ns", nfs.Spec.RADOS.Namespace, action, nodeID}
|
||||
return c.context.Executor.ExecuteCommand(false, "", ganeshaRadosGraceCmd, args...)
|
||||
moniker := ganeshaRadosGraceCmd + " " + action + " " + nodeID
|
||||
// Need to run a command with CEPH_CONF env var set, so don't use c.context.Executor
|
||||
x := exec.Command(cmd, args...)
|
||||
x.Env = []string{fmt.Sprintf("CEPH_CONF=%s", cephclient.CephConfFilePath(c.context.ConfigDir, c.namespace))}
|
||||
var b bytes.Buffer
|
||||
x.Stdout = &b
|
||||
x.Stderr = &b
|
||||
logger.Infof("Running command: %s %s %s", x.Env[0], cmd, strings.Join(args, " "))
|
||||
if err := x.Run(); err != nil {
|
||||
return fmt.Errorf(`failed to execute '%s'
|
||||
stdout: %s
|
||||
stderr: %s
|
||||
error: %+v`, moniker, x.Stdout, x.Stderr, err)
|
||||
}
|
||||
return nil
|
||||
// The below can be used when/if `ganesha-rados-grace` supports a `--cephconf` or similar option
|
||||
// return c.context.Executor.ExecuteCommand(false, moniker, cmd, args...)
|
||||
}
|
||||
|
||||
func (c *CephNFSController) generateConfig(n cephv1.CephNFS, name string) (string, error) {
|
||||
|
||||
@@ -55,16 +55,15 @@ func rgwFrontend(v cephver.CephVersion) string {
|
||||
}
|
||||
|
||||
// TODO: these should be set in the mon's central kv store for mimic+
|
||||
func (c *clusterConfig) defaultSettings() *cephconfig.Config {
|
||||
s := cephconfig.NewConfig()
|
||||
s.Section("global").
|
||||
Set("rgw log nonexistent bucket", "true").
|
||||
Set("rgw intent log object name utc", "true").
|
||||
Set("rgw enable usage log", "true").
|
||||
Set("rgw frontends", fmt.Sprintf("%s %s", rgwFrontend(c.clusterInfo.CephVersion), c.portString(c.clusterInfo.CephVersion))).
|
||||
Set("rgw zone", c.store.Name).
|
||||
Set("rgw zonegroup", c.store.Name)
|
||||
return s
|
||||
func (c *clusterConfig) defaultFlags() []string {
|
||||
return []string{
|
||||
cephconfig.NewFlag("rgw log nonexistent bucket", "true"),
|
||||
cephconfig.NewFlag("rgw intent log object name utc", "true"),
|
||||
cephconfig.NewFlag("rgw enable usage log", "true"),
|
||||
cephconfig.NewFlag("rgw frontends", fmt.Sprintf("%s %s", rgwFrontend(c.clusterInfo.CephVersion), c.portString(c.clusterInfo.CephVersion))),
|
||||
cephconfig.NewFlag("rgw zone", c.store.Name),
|
||||
cephconfig.NewFlag("rgw zonegroup", c.store.Name),
|
||||
}
|
||||
}
|
||||
|
||||
func (c *clusterConfig) portString(v cephver.CephVersion) string {
|
||||
|
||||
@@ -117,7 +117,7 @@ func (c *clusterConfig) makeDaemonContainer(rgwConfig *rgwConfig) v1.Container {
|
||||
cephconfig.NewFlag("name", generateCephXUser(rgwConfig.ResourceName)),
|
||||
cephconfig.NewFlag("host", opspec.ContainerEnvVarReference("POD_NAME")),
|
||||
cephconfig.NewFlag("rgw-mime-types-file", mimeTypesMountPath()),
|
||||
), c.defaultSettings().GlobalFlags()..., // use default settings as flags until mon kv store supported
|
||||
), c.defaultFlags()..., // use default settings as flags until mon kv store supported
|
||||
),
|
||||
VolumeMounts: append(
|
||||
opspec.DaemonVolumeMounts(c.DataPathMap, rgwConfig.ResourceName),
|
||||
|
||||
@@ -40,42 +40,99 @@ const (
|
||||
|
||||
var logger = capnslog.NewPackageLogger("github.com/rook/rook", "ceph-spec")
|
||||
|
||||
// return the volume and matching volume mount for mounting the config override ConfigMap into
|
||||
// containers as "/rook/ceph/ceph.conf".
|
||||
func configOverrideConfigMapVolumeAndMount() (v1.Volume, v1.VolumeMount) {
|
||||
name := k8sutil.ConfigOverrideName // configmap name and name of volume
|
||||
dir := config.EtcCephDir
|
||||
file := "ceph.conf"
|
||||
// TL;DR: mount the configmap's "config" to a file called "ceph.conf" with 0444 permissions
|
||||
// security: allow to be read by everyone since now ceph processes run as 'ceph' and not 'root' user
|
||||
// Further investigation needs to be done to copy the ceph.conf and change its ownership
|
||||
// since configuring a owner of a ConfigMap secret is currently impossible
|
||||
// This also works around the following issue: https://tracker.ceph.com/issues/38606
|
||||
//
|
||||
// This design choice avoids the crash/restart situation in Rook
|
||||
// If we don't set 0444 to the ceph.conf configuration file during its respawn (with exec) the ceph-mgr
|
||||
// won't be able to read the ceph.conf and the container will die, the "restart" count will increase in k8s
|
||||
// This will mislead users thinking something won't wrong but that a false positive
|
||||
mode := int32(0444)
|
||||
v := v1.Volume{Name: name, VolumeSource: v1.VolumeSource{
|
||||
ConfigMap: &v1.ConfigMapVolumeSource{LocalObjectReference: v1.LocalObjectReference{
|
||||
Name: name},
|
||||
Items: []v1.KeyToPath{
|
||||
{Key: k8sutil.ConfigOverrideVal, Path: file, Mode: &mode},
|
||||
},
|
||||
}}}
|
||||
|
||||
// configmap's "config" to "/etc/ceph/ceph.conf"
|
||||
m := v1.VolumeMount{
|
||||
Name: name,
|
||||
ReadOnly: true, // should be no reason to write to the config in pods, so enforce this
|
||||
MountPath: dir,
|
||||
}
|
||||
|
||||
return v, m
|
||||
}
|
||||
|
||||
func confGeneratedInPodVolumeAndMount() (v1.Volume, v1.VolumeMount) {
|
||||
name := "ceph-conf-emptydir"
|
||||
dir := config.EtcCephDir
|
||||
v := v1.Volume{Name: name, VolumeSource: v1.VolumeSource{
|
||||
EmptyDir: &v1.EmptyDirVolumeSource{}}}
|
||||
// configmap's "config" to "/etc/ceph/ceph.conf"
|
||||
m := v1.VolumeMount{
|
||||
Name: name,
|
||||
MountPath: dir,
|
||||
}
|
||||
return v, m
|
||||
}
|
||||
|
||||
// PodVolumes fills in the volumes parameter with the common list of Kubernetes volumes for use in Ceph pods.
|
||||
func PodVolumes(dataDirHostPath, namespace string) []v1.Volume {
|
||||
// This function is only used for OSDs.
|
||||
func PodVolumes(dataDirHostPath, namespace string, confGeneratedInPod bool) []v1.Volume {
|
||||
dataDirSource := v1.VolumeSource{EmptyDir: &v1.EmptyDirVolumeSource{}}
|
||||
if dataDirHostPath != "" {
|
||||
dataDirSource = v1.VolumeSource{HostPath: &v1.HostPathVolumeSource{Path: dataDirHostPath}}
|
||||
}
|
||||
configVolume, _ := configOverrideConfigMapVolumeAndMount()
|
||||
if confGeneratedInPod {
|
||||
configVolume, _ = confGeneratedInPodVolumeAndMount()
|
||||
}
|
||||
return []v1.Volume{
|
||||
{Name: k8sutil.DataDirVolume, VolumeSource: dataDirSource},
|
||||
cephconfig.DefaultConfigVolume(),
|
||||
k8sutil.ConfigOverrideVolume(),
|
||||
configVolume,
|
||||
StoredLogVolume(path.Join(dataDirHostPath, "log", namespace)),
|
||||
}
|
||||
}
|
||||
|
||||
// CephVolumeMounts returns the common list of Kubernetes volume mounts for Ceph containers.
|
||||
func CephVolumeMounts() []v1.VolumeMount {
|
||||
// This function is only used for OSDs.
|
||||
func CephVolumeMounts(confGeneratedInPod bool) []v1.VolumeMount {
|
||||
_, configMount := configOverrideConfigMapVolumeAndMount()
|
||||
if confGeneratedInPod {
|
||||
_, configMount = confGeneratedInPodVolumeAndMount()
|
||||
}
|
||||
return []v1.VolumeMount{
|
||||
{Name: k8sutil.DataDirVolume, MountPath: k8sutil.DataDir},
|
||||
cephconfig.DefaultConfigMount(),
|
||||
configMount,
|
||||
StoredLogVolumeMount(),
|
||||
// Rook doesn't run in ceph containers, so it doesn't need the config override mounted
|
||||
}
|
||||
}
|
||||
|
||||
// RookVolumeMounts returns the common list of Kubernetes volume mounts for Rook containers.
|
||||
func RookVolumeMounts() []v1.VolumeMount {
|
||||
// This function is only used by OSDs.
|
||||
func RookVolumeMounts(confGeneratedInPod bool) []v1.VolumeMount {
|
||||
return append(
|
||||
CephVolumeMounts(),
|
||||
k8sutil.ConfigOverrideMount(),
|
||||
CephVolumeMounts(confGeneratedInPod),
|
||||
)
|
||||
}
|
||||
|
||||
// DaemonVolumesBase returns the common / static set of volumes.
|
||||
func DaemonVolumesBase(dataPaths *config.DataPathMap, keyringResourceName string) []v1.Volume {
|
||||
configOverrideVolume, _ := configOverrideConfigMapVolumeAndMount()
|
||||
vols := []v1.Volume{
|
||||
config.StoredFileVolume(),
|
||||
configOverrideVolume,
|
||||
}
|
||||
if keyringResourceName != "" {
|
||||
vols = append(vols, keyring.Volume().Resource(keyringResourceName))
|
||||
@@ -139,8 +196,9 @@ func DaemonVolumes(dataPaths *config.DataPathMap, keyringResourceName string) []
|
||||
// volume mounts are shared by most all Ceph daemon containers, both init and standard. If keyring
|
||||
// resource name is empty, there will be no keyring mounted in the container.
|
||||
func DaemonVolumeMounts(dataPaths *config.DataPathMap, keyringResourceName string) []v1.VolumeMount {
|
||||
_, configOverrideMount := configOverrideConfigMapVolumeAndMount()
|
||||
mounts := []v1.VolumeMount{
|
||||
config.StoredFileVolumeMount(),
|
||||
configOverrideMount,
|
||||
}
|
||||
if keyringResourceName != "" {
|
||||
mounts = append(mounts, keyring.VolumeMount().Resource(keyringResourceName))
|
||||
|
||||
@@ -29,10 +29,10 @@ import (
|
||||
)
|
||||
|
||||
func TestPodVolumes(t *testing.T) {
|
||||
if err := test.VolumeIsEmptyDir(k8sutil.DataDirVolume, PodVolumes("", "")); err != nil {
|
||||
if err := test.VolumeIsEmptyDir(k8sutil.DataDirVolume, PodVolumes("", "", false)); err != nil {
|
||||
t.Errorf("PodVolumes(\"\") - data dir source is not EmptyDir: %s", err.Error())
|
||||
}
|
||||
if err := test.VolumeIsHostPath(k8sutil.DataDirVolume, "/dev/sdb", PodVolumes("/dev/sdb", "rook-ceph")); err != nil {
|
||||
if err := test.VolumeIsHostPath(k8sutil.DataDirVolume, "/dev/sdb", PodVolumes("/dev/sdb", "rook-ceph", false)); err != nil {
|
||||
t.Errorf("PodVolumes(\"/dev/sdb\") - data dir source is not HostPath: %s", err.Error())
|
||||
}
|
||||
}
|
||||
@@ -40,10 +40,19 @@ func TestPodVolumes(t *testing.T) {
|
||||
func TestMountsMatchVolumes(t *testing.T) {
|
||||
volsMountsTestDef := test.VolumesAndMountsTestDefinition{
|
||||
VolumesSpec: &test.VolumesSpec{
|
||||
Moniker: "PodVolumes(\"/dev/sdc\")", Volumes: PodVolumes("/dev/sdc", "rook-ceph")},
|
||||
Moniker: "PodVolumes(\"/dev/sdc\")", Volumes: PodVolumes("/dev/sdc", "rook-ceph", false)},
|
||||
MountsSpecItems: []*test.MountsSpec{
|
||||
{Moniker: "CephVolumeMounts()", Mounts: CephVolumeMounts()},
|
||||
{Moniker: "RookVolumeMounts()", Mounts: RookVolumeMounts()}},
|
||||
{Moniker: "CephVolumeMounts(true)", Mounts: CephVolumeMounts(false)},
|
||||
{Moniker: "RookVolumeMounts(true)", Mounts: RookVolumeMounts(false)}},
|
||||
}
|
||||
volsMountsTestDef.TestMountsMatchVolumes(t)
|
||||
|
||||
volsMountsTestDef = test.VolumesAndMountsTestDefinition{
|
||||
VolumesSpec: &test.VolumesSpec{
|
||||
Moniker: "PodVolumes(\"/dev/sdc\")", Volumes: PodVolumes("/dev/sdc", "rook-ceph", true)},
|
||||
MountsSpecItems: []*test.MountsSpec{
|
||||
{Moniker: "CephVolumeMounts(false)", Mounts: CephVolumeMounts(true)},
|
||||
{Moniker: "RookVolumeMounts(false)", Mounts: RookVolumeMounts(true)}},
|
||||
}
|
||||
volsMountsTestDef.TestMountsMatchVolumes(t)
|
||||
}
|
||||
|
||||
@@ -71,7 +71,7 @@ func (ps *PodSpecTester) AssertVolumesMeetCephRequirements(
|
||||
if daemonType == config.MonType {
|
||||
keyringSecretName = "rook-ceph-mons-keyring" // mons share a keyring
|
||||
}
|
||||
requiredVols := []string{"rook-ceph-config", keyringSecretName}
|
||||
requiredVols := []string{"rook-config-override", keyringSecretName}
|
||||
if daemonType != config.RbdMirrorType {
|
||||
requiredVols = append(requiredVols, "ceph-daemon-data")
|
||||
}
|
||||
|
||||
@@ -153,7 +153,7 @@ spec:
|
||||
encryptedDevice:
|
||||
type: string
|
||||
pattern: ^(true|false)$
|
||||
useAllDevices:
|
||||
useAllDevices:
|
||||
type: boolean
|
||||
deviceFilter: {}
|
||||
directories:
|
||||
@@ -172,7 +172,7 @@ spec:
|
||||
location: {}
|
||||
resources: {}
|
||||
type: array
|
||||
useAllDevices:
|
||||
useAllDevices:
|
||||
type: boolean
|
||||
deviceFilter: {}
|
||||
location: {}
|
||||
@@ -183,7 +183,7 @@ spec:
|
||||
path:
|
||||
type: string
|
||||
config: {}
|
||||
topologyAware:
|
||||
topologyAware:
|
||||
type: boolean
|
||||
monitoring:
|
||||
properties:
|
||||
@@ -197,6 +197,15 @@ spec:
|
||||
type: integer
|
||||
placement: {}
|
||||
resources: {}
|
||||
configOverrides:
|
||||
items:
|
||||
properties:
|
||||
who:
|
||||
type: string
|
||||
option:
|
||||
type: string
|
||||
value:
|
||||
type: string
|
||||
required:
|
||||
- mon
|
||||
additionalPrinterColumns:
|
||||
|
||||
Reference in new issue
Block a user