ceph: ability to enable mgr module via CRD

We can now enable via the cluster CR any manager module with a
setting under the new mgr element

mgr:
  modules:
  - name: pg_autoscaler
    enabled: true

Co-authored-by: Sébastien Han <seb@redhat.com>
Co-authored-by: travisn <tnielsen@redhat.com>
Signed-off-by: travisn <tnielsen@redhat.com>
This commit is contained in:
travisnandSébastien Han committed 2019-09-06 11:32:51 -06:00
1 parent 5bcfb7205f
commit c30cc697c7
15 files changed
+290 -22

No files matched your search

+16
View File
@@ -130,6 +130,8 @@ For more details on the mons and when to choose a number other than `3`, see the
- `disruptionManagement`: The section for configuring management of daemon disruptions
- `managePodBudgets`: if `true`, the operator will create and manage PodDsruptionBudgets for OSD, Mon, RGW, and MDS daemons. OSD PDBs are managed dynamically via the strategy outlined in the [design](https://github.com/rook/rook/blob/master/design/ceph-managed-disruptionbudgets.md). The operator will block eviction of OSDs by default and unblock them safely when drains are detected.
- `osdMaintenanceTimeout`: is a duration in minutes that determines how long an entire failureDomain like `region/zone/host` will be held in `noout` (in addition to the default DOWN/OUT interval) when it is draining. This is only relevant when `managePodBudgets` is `true`. The default value is `30` minutes.
- `mgr`: manager top level section
- `modules`: is the list of ceph manager modules to enable
### Mon Settings
@@ -151,6 +153,20 @@ To change the defaults that the operator uses to determine the mon health and wh
- `ROOK_MON_HEALTHCHECK_INTERVAL`: The frequency with which to check if mons are in quorum (default is 45 seconds)
- `ROOK_MON_OUT_TIMEOUT`: The interval to wait before marking a mon as "out" and starting a new mon to replace it in the quorum (default is 600 seconds)
### Mgr Settings
You can use the cluster CR to enable or disable any manager module. This can be configured like so:
```yaml
mgr:
modules:
- name: <name of the module>
enabled: true
```
Some modules will have special configuration to ensure the module is fully functional after being enabled. Specifically:
- `pg_autoscaler`: Rook will configure all new pools with PG autoscaling by setting: `osd_pool_default_pg_autoscale_mode = on`
### Node Settings
In addition to the cluster level settings specified above, each individual node can also specify configuration to override the cluster level settings and defaults.
If a node does not specify any configuration then it will inherit the cluster level settings.
+10 -11
View File
@@ -31,20 +31,19 @@ provide detailed information about how to tune these parameters.
An easier option exists for Rook-Ceph clusters running Ceph Nautilus (v14.2.x) or newer. Nautilus
[introduced the PG auto-scaler mgr module](https://ceph.com/rados/new-in-nautilus-pg-merging-and-autotuning/)
capable of automatically managing PG and PGP values for pools. This module is not enabled by default
but can be enabled by running the following from the Ceph toolbox pod (more on the toolbox pod and
Ceph CLI later in this document). Using this module, the user does not need to configure
`osd_pool_default_pg_num` or `osd_pool_default_pgp_num`.
```
ceph mgr module enable pg_autoscaler
```
but can be enabled by the following setting in the [CephCluster CR](ceph-cluster-crd.md#mgr-settings):
The user must also decide whether they want the autoscaler to be used for all new pools by default.
Turning this on is recommended. The Ceph CLI command for turning this on is shown below, but this
can be configured via any of the configuration methods detailed later in this document.
```
ceph config set global osd_pool_default_autoscale_mode on
```yaml
mgr:
modules:
- name: pg_autoscaler
enabled: true
```
With that setting, the autoscaler will be enabled for all new pools. If you do not desire to have
the autoscaler enabled for all pools, you will need to use the Rook toolbox to enable the module
and [enable the autoscaling](https://docs.ceph.com/docs/master/rados/operations/placement-groups/)
on individual pools.
## Specifying configuration options
+1
View File
@@ -52,6 +52,7 @@ an example usage
- If `true`, the OSDs will be allowed to move between nodes during failover. This requires a storage class that supports portability (e.g. `aws-ebs`, but not the local storage provisioner).
- If `false`, the OSDs will be assigned to a node permanently. Rook will configure Ceph's CRUSH map to support the portability.
- Rook can now manage PodDisruptionBudgets for the following Daemons: OSD, Mon, RGW, MDS. OSD budgets are dynamically managed as documented in the [design](https://github.com/rook/rook/blob/master/design/ceph-managed-disruptionbudgets.md). This can be enabled with the `managePodBudgets` flag in the cluster CR. When this is enabled, drains on OSDs will be blocked by default and dynamically unblocked in a safe manner one failureDomain at a time. When a failure domain is draining, it will be marked as no out for a longer time than the default DOWN/OUT interval.
- Rook now has a new config CRD `mgr` to enable ceph manager modules
- Flexvolume plugin now supports dynamic PVC expansion.
- The Rook-enforced minimum memory for OSD pods has been reduced from 4096M to 2048M
@@ -47,6 +47,15 @@ spec:
maximum: 9
minimum: 0
type: integer
mgr:
properties:
modules:
items:
properties:
name:
type: string
enabled:
type: boolean
network:
properties:
hostNetwork:
@@ -29,6 +29,11 @@ spec:
hostNetwork: false
rbdMirroring:
workers: 0
mgr:
modules:
# the pg_autoscaler is only available on nautilus or newer. remove this if testing mimic.
- name: pg_autoscaler
enabled: true
storage:
useAllNodes: true
useAllDevices: false
@@ -33,6 +33,12 @@ spec:
mon:
count: 3
allowMultiplePerNode: false
mgr:
modules:
# Several modules should not need to be included in this list. The "dashboard" and "monitoring" modules
# are already enabled by other settings in the cluster CR and the "rook" module is always enabled.
# - name: pg_autoscaler
# enabled: true
# enable the ceph dashboard for viewing cluster status
dashboard:
enabled: true
@@ -66,6 +66,15 @@ spec:
maximum: 9
minimum: 0
type: integer
mgr:
properties:
modules:
items:
properties:
name:
type: string
enabled:
type: boolean
network:
properties:
hostNetwork:
@@ -49,7 +49,16 @@ spec:
maximum: 9
minimum: 0
type: integer
network:
mgr:
properties:
modules:
items:
properties:
name:
type: string
enabled:
type: boolean
network:
properties:
hostNetwork:
type: boolean
+14
View File
@@ -89,6 +89,9 @@ type ClusterSpec struct {
// Whether the Ceph Cluster is running external to this Kubernetes cluster
// mon, mgr, osd, mds, and discover daemons will not be created for external clusters.
External ExternalSpec `json:"external"`
// A spec for mgr related options
Mgr MgrSpec `json:"mgr,omitempty"`
}
// VersionSpec represents the settings for the Ceph version that Rook is orchestrating.
@@ -161,6 +164,17 @@ type MonSpec struct {
VolumeClaimTemplate *v1.PersistentVolumeClaim `json:"volumeClaimTemplate,omitempty"`
}
// MgrSpec represents options to configure a ceph mgr
type MgrSpec struct {
Modules []Module `json:"modules,omitempty"`
}
// Module represents mgr modules that the user wants to enable or disable
type Module struct {
Name string `json:"name,omitempty"`
Enabled bool `json:"enabled"`
}
// ExternalSpec represents the options supported by an external cluster
type ExternalSpec struct {
Enable bool `json:"enable"`
@@ -485,6 +485,7 @@ func (in *ClusterSpec) DeepCopyInto(out *ClusterSpec) {
in.Dashboard.DeepCopyInto(&out.Dashboard)
out.Monitoring = in.Monitoring
out.External = in.External
in.Mgr.DeepCopyInto(&out.Mgr)
return
}
@@ -702,6 +703,43 @@ func (in *MetadataServerSpec) DeepCopy() *MetadataServerSpec {
return out
}
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *MgrSpec) DeepCopyInto(out *MgrSpec) {
*out = *in
if in.Modules != nil {
in, out := &in.Modules, &out.Modules
*out = make([]Module, len(*in))
copy(*out, *in)
}
return
}
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new MgrSpec.
func (in *MgrSpec) DeepCopy() *MgrSpec {
if in == nil {
return nil
}
out := new(MgrSpec)
in.DeepCopyInto(out)
return out
}
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *Module) DeepCopyInto(out *Module) {
*out = *in
return
}
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new Module.
func (in *Module) DeepCopy() *Module {
if in == nil {
return nil
}
out := new(Module)
in.DeepCopyInto(out)
return out
}
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *MonSpec) DeepCopyInto(out *MonSpec) {
*out = *in
+1 -1
View File
@@ -257,7 +257,7 @@ func (c *cluster) doOrchestration(rookImage string, cephVersion cephver.CephVers
mgrs := mgr.New(c.Info, c.context, c.Namespace, rookImage,
spec.CephVersion, cephv1.GetMgrPlacement(spec.Placement), cephv1.GetMgrAnnotations(c.Spec.Annotations),
spec.Network, spec.Dashboard, spec.Monitoring, cephv1.GetMgrResources(spec.Resources), c.ownerRef, c.Spec.DataDirHostPath, c.isUpgrade)
spec.Network, spec.Dashboard, spec.Monitoring, spec.Mgr, cephv1.GetMgrResources(spec.Resources), c.ownerRef, c.Spec.DataDirHostPath, c.isUpgrade)
err = mgrs.Start()
if err != nil {
return fmt.Errorf("failed to start the ceph mgr. %+v", err)
+75 -6
View File
@@ -44,12 +44,13 @@ var logger = capnslog.NewPackageLogger("github.com/rook/rook", "op-mgr")
var prometheusRuleName = "prometheus-ceph-vVERSION-rules"
const (
appName = "rook-ceph-mgr"
serviceAccountName = "rook-ceph-mgr"
prometheusModuleName = "prometheus"
metricsPort = 9283
monitoringPath = "/etc/ceph-monitoring/"
serviceMonitorFile = "service-monitor.yaml"
appName = "rook-ceph-mgr"
serviceAccountName = "rook-ceph-mgr"
prometheusModuleName = "prometheus"
pgautoscalerModuleName = "pg_autoscaler"
metricsPort = 9283
monitoringPath = "/etc/ceph-monitoring/"
serviceMonitorFile = "service-monitor.yaml"
// minimum amount of memory in MB to run the pod
cephMgrPodMinimumMemory uint64 = 512
)
@@ -68,6 +69,7 @@ type Cluster struct {
ownerRef metav1.OwnerReference
dashboard cephv1.DashboardSpec
monitoringSpec cephv1.MonitoringSpec
mgrSpec cephv1.MgrSpec
cephVersion cephv1.CephVersionSpec
rookVersion string
exitCode func(err error) (int, bool)
@@ -86,6 +88,7 @@ func New(
network cephv1.NetworkSpec,
dashboard cephv1.DashboardSpec,
monitoringSpec cephv1.MonitoringSpec,
mgrSpec cephv1.MgrSpec,
resources v1.ResourceRequirements,
ownerRef metav1.OwnerReference,
dataDirHostPath string,
@@ -102,6 +105,7 @@ func New(
dataDir: k8sutil.DataDir,
dashboard: dashboard,
monitoringSpec: monitoringSpec,
mgrSpec: mgrSpec,
Network: network,
resources: resources,
ownerRef: ownerRef,
@@ -190,6 +194,10 @@ func (c *Cluster) Start() error {
logger.Errorf("failed to enable mgr dashboard. %+v", err)
}
if err := c.configureMgrModules(); err != nil {
logger.Errorf("failed to enable mgr module(s) from the spec. %+v", err)
}
}
// create the metrics service
@@ -240,6 +248,67 @@ func (c *Cluster) enablePrometheusModule(clusterName string) error {
return nil
}
func (c *Cluster) configureMgrModules() error {
// Enable mgr modules from the spec
for _, module := range c.mgrSpec.Modules {
if module.Name == "" {
return fmt.Errorf("name not specified for the mgr module configuration")
}
if wellKnownModule(module.Name) {
return fmt.Errorf("cannot configure mgr module %s that is configured with other cluster settings", module.Name)
}
minVersion, versionOK := c.moduleMeetsMinVersion(module.Name)
if !versionOK {
return fmt.Errorf("module %s cannot be configured because it requires at least Ceph version %v", module.Name, minVersion)
}
if module.Enabled {
if err := client.MgrEnableModule(c.context, c.Namespace, module.Name, false); err != nil {
return fmt.Errorf("failed to enable mgr module %s. %+v", module.Name, err)
}
// Configure special settings for individual modules that are enabled
if module.Name == pgautoscalerModuleName && c.clusterInfo.CephVersion.IsAtLeastNautilus() {
monStore := config.GetMonStore(c.context, c.Namespace)
// Ceph Octopus will have that option enabled
err := monStore.Set("global", "osd_pool_default_pg_autoscale_mode", "on")
if err != nil {
return fmt.Errorf("failed to enable pg autoscale mode for newly created pools. %+v", err)
}
}
} else {
if err := client.MgrDisableModule(c.context, c.Namespace, module.Name); err != nil {
return fmt.Errorf("failed to disable mgr module %s. %+v", module.Name, err)
}
}
}
return nil
}
func (c *Cluster) moduleMeetsMinVersion(name string) (*cephver.CephVersion, bool) {
minVersions := map[string]cephver.CephVersion{
// The PG autoscaler module requires Nautilus
pgautoscalerModuleName: {Major: 14},
}
if ver, ok := minVersions[name]; ok {
// Check if the required min version is met
return &ver, c.clusterInfo.CephVersion.IsAtLeast(ver)
}
// no min version was required
return nil, true
}
func wellKnownModule(name string) bool {
knownModules := []string{rookModuleName, dashboardModuleName, prometheusModuleName}
for _, known := range knownModules {
if name == known {
return true
}
}
return false
}
// add a servicemonitor that allows prometheus to scrape from the monitoring endpoint of the cluster
func (c *Cluster) enableServiceMonitor(service *v1.Service) error {
name := service.GetName()
+78
View File
@@ -26,6 +26,7 @@ import (
rookalpha "github.com/rook/rook/pkg/apis/rook.io/v1alpha2"
"github.com/rook/rook/pkg/clusterd"
cephconfig "github.com/rook/rook/pkg/daemon/ceph/config"
cephver "github.com/rook/rook/pkg/operator/ceph/version"
testopk8s "github.com/rook/rook/pkg/operator/k8sutil/test"
testop "github.com/rook/rook/pkg/operator/test"
exectest "github.com/rook/rook/pkg/util/exec/test"
@@ -64,6 +65,7 @@ func TestStartMGR(t *testing.T) {
cephv1.NetworkSpec{},
cephv1.DashboardSpec{Enabled: true},
cephv1.MonitoringSpec{Enabled: true, RulesNamespace: ""},
cephv1.MgrSpec{},
v1.ResourceRequirements{},
metav1.OwnerReference{},
"/var/lib/rook/",
@@ -125,3 +127,79 @@ func validateStart(t *testing.T, c *Cluster) {
assert.True(t, errors.IsNotFound(err))
}
}
func TestConfigureModules(t *testing.T) {
modulesEnabled := 0
modulesDisabled := 0
configSettings := map[string]string{}
lastModuleConfigured := ""
executor := &exectest.MockExecutor{
MockExecuteCommandWithOutputFile: func(debug bool, actionName string, command string, outFileArg string, args ...string) (string, error) {
logger.Infof("Command: %s %v", command, args)
if command == "ceph" && len(args) > 3 {
if args[0] == "mgr" && args[1] == "module" {
if args[2] == "enable" {
modulesEnabled++
}
if args[2] == "disable" {
modulesDisabled++
}
lastModuleConfigured = args[3]
}
if args[0] == "config" && args[1] == "set" && args[2] == "global" {
configSettings[args[3]] = args[4]
}
}
return "", nil //return "{\"key\":\"mysecurekey\"}", nil
},
}
context := &clusterd.Context{Executor: executor, Clientset: testop.New(3)}
clusterInfo := &cephconfig.ClusterInfo{}
c := &Cluster{
clusterInfo: clusterInfo,
context: context,
Namespace: "ns",
}
// one module without any special configuration
c.mgrSpec.Modules = []cephv1.Module{
{Name: "mymodule", Enabled: true},
}
assert.NoError(t, c.configureMgrModules())
assert.Equal(t, 1, modulesEnabled)
assert.Equal(t, 0, modulesDisabled)
assert.Equal(t, "mymodule", lastModuleConfigured)
// one module that has a min version that is not met
c.mgrSpec.Modules = []cephv1.Module{
{Name: "pg_autoscaler", Enabled: true},
}
c.clusterInfo.CephVersion = cephver.CephVersion{Major: 13}
assert.Error(t, c.configureMgrModules())
assert.Equal(t, 0, len(configSettings))
// one module that has a min version that is met
c.mgrSpec.Modules = []cephv1.Module{
{Name: "pg_autoscaler", Enabled: true},
}
c.clusterInfo.CephVersion = cephver.CephVersion{Major: 14}
modulesEnabled = 0
assert.NoError(t, c.configureMgrModules())
assert.Equal(t, 1, modulesEnabled)
assert.Equal(t, 0, modulesDisabled)
assert.Equal(t, "pg_autoscaler", lastModuleConfigured)
assert.Equal(t, 1, len(configSettings))
assert.Equal(t, "on", configSettings["osd_pool_default_pg_autoscale_mode"])
// disable the module
modulesEnabled = 0
lastModuleConfigured = ""
configSettings = map[string]string{}
c.mgrSpec.Modules[0].Enabled = false
assert.NoError(t, c.configureMgrModules())
assert.Equal(t, 0, modulesEnabled)
assert.Equal(t, 1, modulesDisabled)
assert.Equal(t, "pg_autoscaler", lastModuleConfigured)
assert.Equal(t, 0, len(configSettings))
}
@@ -46,6 +46,7 @@ func TestPodSpec(t *testing.T) {
cephv1.NetworkSpec{},
cephv1.DashboardSpec{},
cephv1.MonitoringSpec{},
cephv1.MgrSpec{},
v1.ResourceRequirements{
Limits: v1.ResourceList{
v1.ResourceCPU: *resource.NewQuantity(200.0, resource.BinarySI),
@@ -97,6 +98,7 @@ func TestServiceSpec(t *testing.T) {
cephv1.NetworkSpec{},
cephv1.DashboardSpec{},
cephv1.MonitoringSpec{},
cephv1.MgrSpec{},
v1.ResourceRequirements{},
metav1.OwnerReference{},
"/var/lib/rook/",
@@ -122,6 +124,7 @@ func TestHostNetwork(t *testing.T) {
cephv1.NetworkSpec{HostNetwork: true},
cephv1.DashboardSpec{},
cephv1.MonitoringSpec{},
cephv1.MgrSpec{},
v1.ResourceRequirements{},
metav1.OwnerReference{},
"/var/lib/rook/",
@@ -155,6 +158,7 @@ func TestHttpBindFix(t *testing.T) {
cephv1.NetworkSpec{},
cephv1.DashboardSpec{},
cephv1.MonitoringSpec{},
cephv1.MgrSpec{},
v1.ResourceRequirements{},
metav1.OwnerReference{},
"/var/lib/rook/",
+14 -3
View File
@@ -120,8 +120,15 @@ spec:
maximum: 9
minimum: 1
type: integer
required:
- count
mgr:
properties:
modules:
items:
properties:
name:
type: string
enabled:
type: boolean
network:
properties:
hostNetwork:
@@ -1603,7 +1610,11 @@ spec:
config:
storeType: "` + settings.StoreType + `"
databaseSizeMB: "1024"
journalSizeMB: "1024"`
journalSizeMB: "1024"
mgr:
modules:
- name: pg_autoscaler
enabled: true`
}
// GetRookToolBox returns rook-toolbox manifest