osd: update the device class if desired state changes

Normally the device class of an OSD is determined at provisioning
and is not updated thereafter. In some scenarios the admin may
want to force update the device class to a new value. The device
classes can be updated by first setting allowDeviceClassUpdate
in the storage spec of the cephcluster, then updating the
device class specified on the deviceSets or other OSDs.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
This commit is contained in:
Travis Nielsen committed 2024-07-17 13:54:21 -06:00
1 parent 3ccdb53cb6
commit 88e952a3ac
19 files changed
+212 -41

No files matched your search

@@ -80,6 +80,8 @@ For more details on the mons and when to choose a number other than `3`, see the
`useAllNodes` must be set to `false` to use specific nodes and their config.
See [node settings](#node-settings) below.
* `config`: Config settings applied to all OSDs on the node unless overridden by `devices`. See the [config settings](#osd-configuration-settings) below.
* `allowDeviceClassUpdate`: Whether to allow changing the device class of an OSD after it is created. The default is false
to prevent unintentional data movement or CRUSH changes if the device class is changed accidentally.
* [storage selection settings](#storage-selection-settings)
* [Storage Class Device Sets](#storage-class-device-sets)
* `onlyApplyOSDPlacement`: Whether the placement specific for OSDs is merged with the `all` placement. If `false`, the OSD placement will be merged with the `all` placement. If true, the `OSD placement will be applied` and the `all` placement will be ignored. The placement for OSDs is computed from several different places depending on the type of OSD:
@@ -340,7 +342,7 @@ The following storage selection settings are specific to Ceph and do not apply t
* `metadataDevice`: Name of a device, [partition](#limitations-of-metadata-device) or lvm to use for the metadata of OSDs on each node. Performance can be improved by using a low latency device (such as SSD or NVMe) as the metadata device, while other spinning platter (HDD) devices on a node are used to store data. Provisioning will fail if the user specifies a `metadataDevice` but that device is not used as a metadata device by Ceph. Notably, `ceph-volume` will not use a device of the same device class (HDD, SSD, NVMe) as OSD devices for metadata, resulting in this failure.
* `databaseSizeMB`: The size in MB of a bluestore database. Include quotes around the size.
* `walSizeMB`: The size in MB of a bluestore write ahead log (WAL). Include quotes around the size.
* `deviceClass`: The [CRUSH device class](https://ceph.io/community/new-luminous-crush-device-classes/) to use for this selection of storage devices. (By default, if a device's class has not already been set, OSDs will automatically set a device's class to either `hdd`, `ssd`, or `nvme` based on the hardware properties exposed by the Linux kernel.) These storage classes can then be used to select the devices backing a storage pool by specifying them as the value of [the pool spec's `deviceClass` field](../Block-Storage/ceph-block-pool-crd.md#spec).
* `deviceClass`: The [CRUSH device class](https://ceph.io/community/new-luminous-crush-device-classes/) to use for this selection of storage devices. (By default, if a device's class has not already been set, OSDs will automatically set a device's class to either `hdd`, `ssd`, or `nvme` based on the hardware properties exposed by the Linux kernel.) These storage classes can then be used to select the devices backing a storage pool by specifying them as the value of [the pool spec's `deviceClass` field](../Block-Storage/ceph-block-pool-crd.md#spec). If updating the device class of an OSD after the OSD is already created, `allowDeviceClassUpdate: true` must be set. Otherwise updates to this `deviceClass` will be ignored.
* `initialWeight`: The initial OSD weight in TiB units. By default, this value is derived from OSD's capacity.
* `primaryAffinity`: The [primary-affinity](https://docs.ceph.com/en/latest/rados/operations/crush-map/#primary-affinity) value of an OSD, within range `[0, 1]` (default: `1`).
* `osdsPerDevice`**: The number of OSDs to create on each device. High performance devices such as NVMe can handle running multiple OSDs. If desired, this can be overridden for each node and each device.
+12
View File
@@ -12218,6 +12218,18 @@ float64
<p>BackfillFullRatio is the ratio at which the cluster is too full for backfill. Backfill will be disabled if above this threshold. Default is 0.90.</p>
</td>
</tr>
<tr>
<td>
<code>allowDeviceClassUpdate</code><br/>
<em>
bool
</em>
</td>
<td>
<em>(Optional)</em>
<p>Whether to allow updating the device class after the OSD is initially provisioned</p>
</td>
</tr>
</tbody>
</table>
<h3 id="ceph.rook.io/v1.StoreType">StoreType
+1
View File
@@ -9,3 +9,4 @@ please update the `BucketClass` and `BucketAccessClass` for resolving refer [her
## Features
- Added support for Ceph Squid (v19)
- Allow updating the device class of OSDs, if `allowDeviceClassUpdate: true` is set
@@ -3204,6 +3204,9 @@ spec:
description: A spec for available storage in the cluster and how it should be used
nullable: true
properties:
allowDeviceClassUpdate:
description: Whether to allow updating the device class after the OSD is initially provisioned
type: boolean
backfillFullRatio:
description: BackfillFullRatio is the ratio at which the cluster is too full for backfill. Backfill will be disabled if above this threshold. Default is 0.90.
maximum: 1
+2 -1
View File
@@ -53,6 +53,7 @@ spec:
periodicity: daily # one of: hourly, daily, weekly, monthly
maxLogSize: 500M # SUFFIX may be 'M' or 'G'. Must be at least 1M.
storage:
allowDeviceClassUpdate: false # whether to allow changing the device class of an OSD after it is created
storageClassDeviceSets:
- name: set1
# The number of OSDs to create from this device set
@@ -123,7 +124,7 @@ spec:
volumeClaimTemplates:
- metadata:
name: data
# if you are looking at giving your OSD a different CRUSH device class than the one detected by Ceph
# set a different CRUSH device class on the OSD than the one detected by Ceph
# annotations:
# crushDeviceClass: hybrid
spec:
+3
View File
@@ -33,7 +33,10 @@ spec:
storage:
useAllNodes: true
useAllDevices: true
allowDeviceClassUpdate: true
#deviceFilter:
#config:
# deviceClass: testclass
monitoring:
enabled: false
healthCheck:
+2
View File
@@ -255,6 +255,8 @@ spec:
# databaseSizeMB: "1024" # uncomment if the disks are smaller than 100 GB
# osdsPerDevice: "1" # this value can be overridden at the node or device level
# encryptedDevice: "true" # the default value for this option is "false"
# deviceClass: "myclass" # specify a device class for OSDs in the cluster
allowDeviceClassUpdate: false # whether to allow changing the device class of an OSD after it is created
# Individual nodes and their config can be specified as well, but 'useAllNodes' above must be set to false. Then, only the named
# nodes below will be used as storage resources. Each node's 'name' field should match their 'kubernetes.io/hostname' label.
# nodes:
+3
View File
@@ -3202,6 +3202,9 @@ spec:
description: A spec for available storage in the cluster and how it should be used
nullable: true
properties:
allowDeviceClassUpdate:
description: Whether to allow updating the device class after the OSD is initially provisioned
type: boolean
backfillFullRatio:
description: BackfillFullRatio is the ratio at which the cluster is too full for backfill. Backfill will be disabled if above this threshold. Default is 0.90.
maximum: 1
+3
View File
@@ -2861,6 +2861,9 @@ type StorageScopeSpec struct {
// +optional
// +nullable
BackfillFullRatio *float64 `json:"backfillFullRatio,omitempty"`
// Whether to allow updating the device class after the OSD is initially provisioned
// +optional
AllowDeviceClassUpdate bool `json:"allowDeviceClassUpdate,omitempty"`
}
// OSDStore is the backend storage type used for creating the OSDs
+19
View File
@@ -38,6 +38,7 @@ type OSDUsage struct {
type OSDNodeUsage struct {
ID int `json:"id"`
Name string `json:"name"`
DeviceClass string `json:"device_class"`
CrushWeight json.Number `json:"crush_weight"`
Depth json.Number `json:"depth"`
Reweight json.Number `json:"reweight"`
@@ -219,6 +220,24 @@ func GetOSDUsage(context *clusterd.Context, clusterInfo *ClusterInfo) (*OSDUsage
return &osdUsage, nil
}
func SetDeviceClass(context *clusterd.Context, clusterInfo *ClusterInfo, osdID int, deviceClass string) error {
// First remove the existing device class
args := []string{"osd", "crush", "rm-device-class", fmt.Sprintf("osd.%d", osdID)}
buf, err := NewCephCommand(context, clusterInfo, args).Run()
if err != nil {
return errors.Wrap(err, "failed to remove device class. "+string(buf))
}
// Second, apply the desired device class
args = []string{"osd", "crush", "set-device-class", deviceClass, fmt.Sprintf("osd.%d", osdID)}
buf, err = NewCephCommand(context, clusterInfo, args).Run()
if err != nil {
return errors.Wrap(err, "failed to set the device class. "+string(buf))
}
return nil
}
func GetOSDPerfStats(context *clusterd.Context, clusterInfo *ClusterInfo) (*OSDPerfStats, error) {
args := []string{"osd", "perf"}
buf, err := NewCephCommand(context, clusterInfo, args).Run()
+5 -5
View File
@@ -88,7 +88,7 @@ func (c *createConfig) createNewOSDsFromStatus(
return
}
for _, osd := range status.OSDs {
for i, osd := range status.OSDs {
if c.deployments.Exists(osd.ID) {
// This OSD will be handled by the updater
logger.Debugf("not creating deployment for OSD %d which already exists", osd.ID)
@@ -96,13 +96,13 @@ func (c *createConfig) createNewOSDsFromStatus(
}
if status.PvcBackedOSD {
logger.Infof("creating OSD %d on PVC %q", osd.ID, nodeOrPVCName)
err := createDaemonOnPVCFunc(c.cluster, osd, nodeOrPVCName, c.provisionConfig)
err := createDaemonOnPVCFunc(c.cluster, &status.OSDs[i], nodeOrPVCName, c.provisionConfig)
if err != nil {
errs.addError("%v", errors.Wrapf(err, "failed to create OSD %d on PVC %q", osd.ID, nodeOrPVCName))
}
} else {
logger.Infof("creating OSD %d on node %q", osd.ID, nodeOrPVCName)
err := createDaemonOnNodeFunc(c.cluster, osd, nodeOrPVCName, c.provisionConfig)
err := createDaemonOnNodeFunc(c.cluster, &status.OSDs[i], nodeOrPVCName, c.provisionConfig)
if err != nil {
errs.addError("%v", errors.Wrapf(err, "failed to create OSD %d on node %q", osd.ID, nodeOrPVCName))
}
@@ -372,7 +372,7 @@ func (c *Cluster) runPrepareJob(osdProps *osdProperties, config *provisionConfig
return nil
}
func createDaemonOnPVC(c *Cluster, osd OSDInfo, pvcName string, config *provisionConfig) error {
func createDaemonOnPVC(c *Cluster, osd *OSDInfo, pvcName string, config *provisionConfig) error {
d, err := deploymentOnPVC(c, osd, pvcName, config)
if err != nil {
return err
@@ -402,7 +402,7 @@ func createDaemonOnPVC(c *Cluster, osd OSDInfo, pvcName string, config *provisio
return nil
}
func createDaemonOnNode(c *Cluster, osd OSDInfo, nodeName string, config *provisionConfig) error {
func createDaemonOnNode(c *Cluster, osd *OSDInfo, nodeName string, config *provisionConfig) error {
d, err := deploymentOnNode(c, osd, nodeName, config)
if err != nil {
return err
+2 -2
View File
@@ -62,7 +62,7 @@ func Test_createNewOSDsFromStatus(t *testing.T) {
}()
createCallsOnNode := []int{}
induceFailureCreatingOSD := -1 // allow causing the create call to fail for a given OSD ID
createDaemonOnNodeFunc = func(c *Cluster, osd OSDInfo, nodeName string, config *provisionConfig) error {
createDaemonOnNodeFunc = func(c *Cluster, osd *OSDInfo, nodeName string, config *provisionConfig) error {
createCallsOnNode = append(createCallsOnNode, osd.ID)
if induceFailureCreatingOSD == osd.ID {
return errors.Errorf("createOSDDaemonOnNode: induced failure on OSD %d", osd.ID)
@@ -76,7 +76,7 @@ func Test_createNewOSDsFromStatus(t *testing.T) {
}()
createCallsOnPVC := []int{}
// reuse induceFailureCreatingOSD from above
createDaemonOnPVCFunc = func(c *Cluster, osd OSDInfo, pvcName string, config *provisionConfig) error {
createDaemonOnPVCFunc = func(c *Cluster, osd *OSDInfo, pvcName string, config *provisionConfig) error {
createCallsOnPVC = append(createCallsOnPVC, osd.ID)
if induceFailureCreatingOSD == osd.ID {
return errors.Errorf("createOSDDaemonOnNode: induced failure on OSD %d", osd.ID)
@@ -585,6 +585,9 @@ func osdIntegrationTestExecutor(t *testing.T, clientset *fake.Clientset, namespa
return `["ssd"]`, nil
}
}
if args[1] == "df" {
return osdDFResults, nil
}
}
if args[0] == "versions" {
// the update deploy code only cares about the mons from the ceph version command results
+43 -3
View File
@@ -283,6 +283,11 @@ func (c *Cluster) Start() error {
return errors.Wrapf(err, "failed to reconcile key rotation cron jobs")
}
err = c.postReconcileUpdateOSDProperties(updateConfig.osdDesiredState)
if err != nil {
return errors.Wrap(err, "failed post reconcile of osd properties")
}
err = c.updateCephStorageStatus()
if err != nil {
return errors.Wrapf(err, "failed to update ceph storage status")
@@ -314,6 +319,40 @@ func (c *Cluster) Start() error {
return nil
}
func (c *Cluster) postReconcileUpdateOSDProperties(desiredOSDs map[int]*OSDInfo) error {
osdUsage, err := cephclient.GetOSDUsage(c.context, c.clusterInfo)
if err != nil {
return errors.Wrap(err, "failed to get osd usage")
}
logger.Debugf("post processing osd properties with %d actual osds from ceph osd df and %d existing osds found during reconcile", len(osdUsage.OSDNodes), len(desiredOSDs))
for _, actualOSD := range osdUsage.OSDNodes {
if desiredOSD, ok := desiredOSDs[actualOSD.ID]; ok {
if err := c.updateDeviceClassIfChanged(actualOSD.ID, desiredOSD.DeviceClass, actualOSD.DeviceClass); err != nil {
// Log the error and allow other updates to continue
logger.Error(err)
}
}
}
return nil
}
func (c *Cluster) updateDeviceClassIfChanged(osdID int, desiredDeviceClass, actualDeviceClass string) error {
if !c.spec.Storage.AllowDeviceClassUpdate {
// device class updates are not allowed by default
return nil
}
if desiredDeviceClass != "" && desiredDeviceClass != actualDeviceClass {
logger.Infof("updating osd.%d device class from %q to %q", osdID, actualDeviceClass, desiredDeviceClass)
err := cephclient.SetDeviceClass(c.context, c.clusterInfo, osdID, desiredDeviceClass)
if err != nil {
return errors.Wrapf(err, "failed to set device class on osd %d", osdID)
}
return nil
}
logger.Debugf("no device class change needed for osd.%d. desired=%q, actual=%q", osdID, desiredDeviceClass, actualDeviceClass)
return nil
}
func (c *Cluster) getExistingOSDDeploymentsOnPVCs() (sets.Set[string], error) {
listOpts := metav1.ListOptions{LabelSelector: fmt.Sprintf("%s=%s,%s", k8sutil.AppAttr, AppName, OSDOverPVCLabelKey)}
@@ -332,7 +371,7 @@ func (c *Cluster) getExistingOSDDeploymentsOnPVCs() (sets.Set[string], error) {
return result, nil
}
func deploymentOnNode(c *Cluster, osd OSDInfo, nodeName string, config *provisionConfig) (*appsv1.Deployment, error) {
func deploymentOnNode(c *Cluster, osd *OSDInfo, nodeName string, config *provisionConfig) (*appsv1.Deployment, error) {
osdLongName := fmt.Sprintf("OSD %d on node %q", osd.ID, nodeName)
osdProps, err := c.getOSDPropsForNode(nodeName, osd.DeviceClass)
@@ -353,7 +392,7 @@ func deploymentOnNode(c *Cluster, osd OSDInfo, nodeName string, config *provisio
return d, nil
}
func deploymentOnPVC(c *Cluster, osd OSDInfo, pvcName string, config *provisionConfig) (*appsv1.Deployment, error) {
func deploymentOnPVC(c *Cluster, osd *OSDInfo, pvcName string, config *provisionConfig) (*appsv1.Deployment, error) {
osdLongName := fmt.Sprintf("OSD %d on PVC %q", osd.ID, pvcName)
osdProps, err := c.getOSDPropsForPVC(pvcName, osd.DeviceClass)
@@ -376,7 +415,7 @@ func deploymentOnPVC(c *Cluster, osd OSDInfo, pvcName string, config *provisionC
// setOSDProperties is used to configure an OSD with parameters which can not be set via explicit
// command-line arguments.
func setOSDProperties(c *Cluster, osdProps osdProperties, osd OSDInfo) error {
func setOSDProperties(c *Cluster, osdProps osdProperties, osd *OSDInfo) error {
// OSD's 'primary-affinity' has to be configured via command which goes through mons
if osdProps.storeConfig.PrimaryAffinity != "" {
return cephclient.SetPrimaryAffinity(c.context, c.clusterInfo, osd.ID, osdProps.storeConfig.PrimaryAffinity)
@@ -460,6 +499,7 @@ func (c *Cluster) getOSDPropsForPVC(pvcName, osdDeviceClass string) (osdProperti
}
osdProps.storeConfig.InitialWeight = deviceSet.CrushInitialWeight
osdProps.storeConfig.PrimaryAffinity = deviceSet.CrushPrimaryAffinity
osdProps.storeConfig.DeviceClass = deviceSet.CrushDeviceClass
// If OSD isn't portable, we're getting the host name either from the osd deployment that was already initialized
// or from the osd prepare job from initial creation.
+79 -8
View File
@@ -51,6 +51,12 @@ import (
const (
healthyCephStatus = `{"fsid":"877a47e0-7f6c-435e-891a-76983ab8c509","health":{"checks":{},"status":"HEALTH_OK"},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"monmap":{"epoch":3,"fsid":"877a47e0-7f6c-435e-891a-76983ab8c509","modified":"2020-11-02 09:58:23.015313","created":"2020-11-02 09:57:37.719235","min_mon_release":14,"min_mon_release_name":"nautilus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.30.74.42:3300","nonce":0},{"type":"v1","addr":"172.30.74.42:6789","nonce":0}]},"addr":"172.30.74.42:6789/0","public_addr":"172.30.74.42:6789/0"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.30.101.61:3300","nonce":0},{"type":"v1","addr":"172.30.101.61:6789","nonce":0}]},"addr":"172.30.101.61:6789/0","public_addr":"172.30.101.61:6789/0"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.30.250.55:3300","nonce":0},{"type":"v1","addr":"172.30.250.55:6789","nonce":0}]},"addr":"172.30.250.55:6789/0","public_addr":"172.30.250.55:6789/0"}]},"osdmap":{"osdmap":{"epoch":19,"num_osds":3,"num_up_osds":3,"num_in_osds":3,"num_remapped_pgs":0}},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":96}],"num_pgs":96,"num_pools":3,"num_objects":79,"data_bytes":81553681,"bytes_used":3255447552,"bytes_avail":1646011994112,"bytes_total":1649267441664,"read_bytes_sec":853,"write_bytes_sec":5118,"read_op_per_sec":1,"write_op_per_sec":0},"fsmap":{"epoch":9,"id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"ocs-storagecluster-cephfilesystem-b","status":"up:active","gid":14161},{"filesystem_id":1,"rank":0,"name":"ocs-storagecluster-cephfilesystem-a","status":"up:standby-replay","gid":24146}],"up:standby":0},"mgrmap":{"epoch":10,"active_gid":14122,"active_name":"a","active_addrs":{"addrvec":[{"type":"v2","addr":"10.131.0.28:6800","nonce":1},{"type":"v1","addr":"10.131.0.28:6801","nonce":1}]}}}`
unHealthyCephStatus = `{"fsid":"613975f3-3025-4802-9de1-a2280b950e75","health":{"checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down"}},"OSD_HOST_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 host (1 osds) down"}},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 101 pgs stale"}},"POOL_APP_NOT_ENABLED":{"severity":"HEALTH_WARN","summary":{"message":"application not enabled on 1 pool(s)"}}},"status":"HEALTH_WARN","overall_status":"HEALTH_WARN"},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["rook-ceph-mon0","rook-ceph-mon2","rook-ceph-mon1"],"monmap":{"epoch":3,"fsid":"613975f3-3025-4802-9de1-a2280b950e75","modified":"2017-08-11 20:13:02.075679","created":"2017-08-11 20:12:35.314510","features":{"persistent":["kraken","luminous"],"optional":[]},"mons":[{"rank":0,"name":"rook-ceph-mon0","addr":"10.3.0.45:6789/0","public_addr":"10.3.0.45:6789/0"},{"rank":1,"name":"rook-ceph-mon2","addr":"10.3.0.249:6789/0","public_addr":"10.3.0.249:6789/0"},{"rank":2,"name":"rook-ceph-mon1","addr":"10.3.0.252:6789/0","public_addr":"10.3.0.252:6789/0"}]},"osdmap":{"osdmap":{"epoch":17,"num_osds":2,"num_up_osds":1,"num_in_osds":2,"full":false,"nearfull":true,"num_remapped_pgs":0}},"pgmap":{"pgs_by_state":[{"state_name":"stale+active+clean","count":101},{"state_name":"active+clean","count":99}],"num_pgs":200,"num_pools":2,"num_objects":243,"data_bytes":976793635,"bytes_used":13611479040,"bytes_avail":19825307648,"bytes_total":33436786688},"fsmap":{"epoch":1,"by_rank":[]},"mgrmap":{"epoch":3,"active_gid":14111,"active_name":"rook-ceph-mgr0","active_addr":"10.2.73.6:6800/9","available":true,"standbys":[],"modules":["restful","status"],"available_modules":["dashboard","prometheus","restful","status","zabbix"]},"servicemap":{"epoch":1,"modified":"0.000000","services":{}}}`
osdDFResults = `
{"nodes":[
{"id":0,"device_class":"hdd","name":"osd.0","type":"osd","type_id":0,"crush_weight":0.039093017578125,"depth":2,"pool_weights":{},"reweight":1,"kb":41943040,"kb_used":27640,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27198,"kb_avail":41915400,"utilization":0.065898895263671875,"var":0.99448308946989694,"pgs":9,"status":"up"},
{"id":1,"device_class":"hdd","name":"osd.1","type":"osd","type_id":0,"crush_weight":0.039093017578125,"depth":2,"pool_weights":{},"reweight":1,"kb":41943040,"kb_used":27960,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":27198,"kb_avail":41915080,"utilization":0.066661834716796875,"var":1.005996641880547,"pgs":15,"status":"up"},
{"id":2,"device_class":"hdd","name":"osd.2","type":"osd","type_id":0,"crush_weight":0.039093017578125,"depth":2,"pool_weights":{},"reweight":1,"kb":41943040,"kb_used":27780,"kb_used_data":564,"kb_used_omap":1,"kb_used_meta":27198,"kb_avail":41915260,"utilization":0.066232681274414062,"var":0.99952026864955634,"pgs":8,"status":"up"}],
"stray":[],"summary":{"total_kb":125829120,"total_kb_used":83380,"total_kb_used_data":1748,"total_kb_used_omap":3,"total_kb_used_meta":81596,"total_kb_avail":125745740,"average_utilization":0.066264470418294266,"min_var":0.99448308946989694,"max_var":1.005996641880547,"dev":0.00031227879054369131}}`
)
func TestOSDProperties(t *testing.T) {
@@ -80,6 +86,9 @@ func TestStart(t *testing.T) {
// Mock executor for OSD crush class list command, returning ssd as available device class
return `["ssd"]`, nil
}
if args[0] == "osd" && args[1] == "df" {
return osdDFResults, nil
}
return "", nil
},
}
@@ -361,6 +370,68 @@ func TestAddRemoveNode(t *testing.T) {
assert.True(t, k8serrors.IsNotFound(err))
}
func TestUpdateDeviceClass(t *testing.T) {
namespace := "ns"
clientset := fake.NewSimpleClientset()
removedDeviceClassOSD := ""
setDeviceClassOSD := ""
setDeviceClass := ""
executor := &exectest.MockExecutor{
MockExecuteCommandWithOutput: func(command string, args ...string) (string, error) {
logger.Infof("ExecuteCommandWithOutput: %s %v", command, args)
if args[0] == "osd" {
if args[1] == "df" {
return osdDFResults, nil
}
if args[1] == "crush" {
if args[2] == "rm-device-class" {
removedDeviceClassOSD = args[3]
} else if args[2] == "set-device-class" {
setDeviceClass = args[3]
setDeviceClassOSD = args[4]
}
}
}
return "", nil
},
}
cephCluster := &cephv1.CephCluster{
ObjectMeta: metav1.ObjectMeta{
Name: "testing",
Namespace: namespace,
},
Spec: cephv1.ClusterSpec{Storage: cephv1.StorageScopeSpec{AllowDeviceClassUpdate: true}},
}
// Objects to track in the fake client.
object := []runtime.Object{
cephCluster,
}
s := scheme.Scheme
// Create a fake client to mock API calls.
client := clientfake.NewClientBuilder().WithScheme(s).WithRuntimeObjects(object...).Build()
clusterInfo := &cephclient.ClusterInfo{
Namespace: namespace,
CephVersion: cephver.Quincy,
Context: context.TODO(),
}
clusterInfo.SetName("rook-ceph-test")
context := &clusterd.Context{Clientset: clientset, Client: client, ConfigDir: "/var/lib/rook", Executor: executor}
c := New(context, clusterInfo, cephCluster.Spec, "myversion")
// Start the first time
desiredOSDs := map[int]*OSDInfo{
0: {ID: 0, DeviceClass: "hdd"},
1: {ID: 1, DeviceClass: "hdd"},
2: {ID: 2, DeviceClass: "newclass"},
}
err := c.postReconcileUpdateOSDProperties(desiredOSDs)
assert.Nil(t, err)
assert.Equal(t, "newclass", setDeviceClass)
assert.Equal(t, "osd.2", setDeviceClassOSD)
assert.Equal(t, "osd.2", removedDeviceClassOSD)
}
func TestAddNodeFailure(t *testing.T) {
// create a storage spec with the given nodes/devices/dirs
nodeName := "node1672"
@@ -481,9 +552,9 @@ func TestGetOSDInfo(t *testing.T) {
node := "n1"
location := "root=default host=myhost zone=myzone"
osd1 := OSDInfo{ID: 3, UUID: "osd-uuid", BlockPath: "dev/logical-volume-path", CVMode: "raw", Location: location, TopologyAffinity: "topology.rook.io/rack=rack0"}
osd2 := OSDInfo{ID: 3, UUID: "osd-uuid", BlockPath: "vg1/lv1", CVMode: "lvm", LVBackedPV: true}
osd3 := OSDInfo{ID: 3, UUID: "osd-uuid", BlockPath: "", CVMode: "raw"}
osd1 := &OSDInfo{ID: 3, UUID: "osd-uuid", BlockPath: "dev/logical-volume-path", CVMode: "raw", Location: location, TopologyAffinity: "topology.rook.io/rack=rack0"}
osd2 := &OSDInfo{ID: 3, UUID: "osd-uuid", BlockPath: "vg1/lv1", CVMode: "lvm", LVBackedPV: true}
osd3 := &OSDInfo{ID: 3, UUID: "osd-uuid", BlockPath: "", CVMode: "raw"}
osdProp := osdProperties{
crushHostname: node,
pvc: corev1.PersistentVolumeClaimVolumeSource{ClaimName: "pvc"},
@@ -533,8 +604,8 @@ func TestGetOSDInfo(t *testing.T) {
t.Run("get info from node-based OSDs", func(t *testing.T) {
useAllDevices := true
osd4 := OSDInfo{ID: 3, UUID: "osd-uuid", BlockPath: "", CVMode: "lvm", Location: location}
osd5 := OSDInfo{ID: 3, UUID: "osd-uuid", BlockPath: "vg1/lv1", CVMode: "lvm"}
osd4 := &OSDInfo{ID: 3, UUID: "osd-uuid", BlockPath: "", CVMode: "lvm", Location: location}
osd5 := &OSDInfo{ID: 3, UUID: "osd-uuid", BlockPath: "vg1/lv1", CVMode: "lvm"}
osdProp = osdProperties{
crushHostname: node,
devices: []cephv1.Device{},
@@ -668,9 +739,9 @@ func TestGetOSDInfoWithCustomRoot(t *testing.T) {
node := "n1"
location := "root=custom-root host=myhost zone=myzone"
osd1 := OSDInfo{ID: 3, UUID: "osd-uuid", BlockPath: "dev/logical-volume-path", CVMode: "raw", Location: location}
osd2 := OSDInfo{ID: 3, UUID: "osd-uuid", BlockPath: "vg1/lv1", CVMode: "lvm", LVBackedPV: true, Location: location}
osd3 := OSDInfo{ID: 3, UUID: "osd-uuid", BlockPath: "", CVMode: "lvm", Location: location}
osd1 := &OSDInfo{ID: 3, UUID: "osd-uuid", BlockPath: "dev/logical-volume-path", CVMode: "raw", Location: location}
osd2 := &OSDInfo{ID: 3, UUID: "osd-uuid", BlockPath: "vg1/lv1", CVMode: "lvm", LVBackedPV: true, Location: location}
osd3 := &OSDInfo{ID: 3, UUID: "osd-uuid", BlockPath: "", CVMode: "lvm", Location: location}
osdProp := osdProperties{
crushHostname: node,
pvc: corev1.PersistentVolumeClaimVolumeSource{ClaimName: "pvc"},
+11 -6
View File
@@ -319,7 +319,7 @@ func deploymentName(osdID int) string {
return fmt.Sprintf(osdAppNameFmt, osdID)
}
func (c *Cluster) makeDeployment(osdProps osdProperties, osd OSDInfo, provisionConfig *provisionConfig) (*apps.Deployment, error) {
func (c *Cluster) makeDeployment(osdProps osdProperties, osd *OSDInfo, provisionConfig *provisionConfig) (*apps.Deployment, error) {
deploymentName := deploymentName(osd.ID)
replicaCount := int32(1)
volumeMounts := controller.CephVolumeMounts(provisionConfig.DataPathMap, false)
@@ -339,6 +339,11 @@ func (c *Cluster) makeDeployment(osdProps osdProperties, osd OSDInfo, provisionC
return nil, errors.Errorf("failed to generate deployment for OSD %d. required CVMode is not specified for this OSD", osd.ID)
}
if c.spec.Storage.AllowDeviceClassUpdate && osdProps.storeConfig.DeviceClass != "" && osdProps.storeConfig.DeviceClass != osd.DeviceClass {
logger.Infof("The device class for osd %d is changing from %q to %q", osd.ID, osd.DeviceClass, osdProps.storeConfig.DeviceClass)
osd.DeviceClass = osdProps.storeConfig.DeviceClass
}
dataDir := k8sutil.DataDir
// Create volume config for /dev so the pod can access devices on the host
// Only valid when running OSD on device or OSD on LV-backed PVC
@@ -507,10 +512,10 @@ func (c *Cluster) makeDeployment(osdProps osdProperties, osd OSDInfo, provisionC
// needed for luksOpen synchronization when devices are encrypted and the osd is prepared with LVM
hostIPC := osdProps.storeConfig.EncryptedDevice || osdProps.encrypted
osdLabels := c.getOSDLabels(osd, failureDomainValue, osdProps.portable)
osdLabels := c.getOSDLabels(*osd, failureDomainValue, osdProps.portable)
if osd.ExportService {
osdService, err := c.createOSDService(osd, osdLabels)
osdService, err := c.createOSDService(*osd, osdLabels)
if err != nil {
return nil, errors.Wrapf(err, "failed to configure osd service for osd.%d", osd.ID)
}
@@ -598,11 +603,11 @@ func (c *Cluster) makeDeployment(osdProps osdProperties, osd OSDInfo, provisionC
// so that it can pick the already mounted/activated osd metadata path
// This container will activate the OSD and place the activated files into an empty dir
// The empty dir will be shared by the "activate-osd" pod and the "osd" main pod
activateOSDVolume, activateOSDContainer := c.getActivateOSDInitContainer(c.spec.DataDirHostPath, c.clusterInfo.Namespace, osdID, osd, osdProps)
activateOSDVolume, activateOSDContainer := c.getActivateOSDInitContainer(c.spec.DataDirHostPath, c.clusterInfo.Namespace, osdID, *osd, osdProps)
volumes = append(volumes, activateOSDVolume...)
volumeMounts = append(volumeMounts, activateOSDContainer.VolumeMounts[0])
initContainers = append(initContainers, *activateOSDContainer)
initContainers = append(initContainers, c.getExpandInitContainer(osdProps, c.spec.DataDirHostPath, c.clusterInfo.Namespace, osdID, osd))
initContainers = append(initContainers, c.getExpandInitContainer(osdProps, c.spec.DataDirHostPath, c.clusterInfo.Namespace, osdID, *osd))
}
// Doing a chown in a post start lifecycle hook does not reliably complete before the OSD
@@ -734,7 +739,7 @@ func (c *Cluster) makeDeployment(osdProps osdProperties, osd OSDInfo, provisionC
}
if osdProps.portable {
// portable OSDs must have affinity to the topology where the osd prepare job was executed
if err := applyTopologyAffinity(&deployment.Spec.Template.Spec, osd); err != nil {
if err := applyTopologyAffinity(&deployment.Spec.Template.Spec, *osd); err != nil {
return nil, err
}
} else {
+11 -11
View File
@@ -114,7 +114,7 @@ func testPodDevices(t *testing.T, dataDir, deviceName string, allDevices bool) {
if len(devices) == 0 && len(dataDir) == 0 {
return
}
osd := OSDInfo{
osd := &OSDInfo{
ID: 0,
CVMode: "raw",
}
@@ -190,7 +190,7 @@ func testPodDevices(t *testing.T, dataDir, deviceName string, allDevices bool) {
pvc: corev1.PersistentVolumeClaimVolumeSource{ClaimName: "mypvc"},
}
// Not needed when running on PVC
osd = OSDInfo{
osd = &OSDInfo{
ID: 0,
CVMode: "lvm",
}
@@ -212,7 +212,7 @@ func testPodDevices(t *testing.T, dataDir, deviceName string, allDevices bool) {
assert.Equal(t, 9, len(cont.VolumeMounts), cont.VolumeMounts)
// Test OSD on PVC with RAW
osd = OSDInfo{
osd = &OSDInfo{
ID: 0,
CVMode: "raw",
}
@@ -251,7 +251,7 @@ func testPodDevices(t *testing.T, dataDir, deviceName string, allDevices bool) {
assert.Equal(t, 10, len(deployment.Spec.Template.Spec.Volumes), deployment.Spec.Template.Spec.Volumes)
// // Test OSD on PVC with RAW and metadata device
osd = OSDInfo{
osd = &OSDInfo{
ID: 0,
CVMode: "raw",
}
@@ -275,7 +275,7 @@ func testPodDevices(t *testing.T, dataDir, deviceName string, allDevices bool) {
assert.Equal(t, 10, len(deployment.Spec.Template.Spec.Volumes), deployment.Spec.Template.Spec.Volumes)
// // Test encrypted OSD on PVC with RAW and metadata device
osd = OSDInfo{
osd = &OSDInfo{
ID: 0,
CVMode: "raw",
}
@@ -307,7 +307,7 @@ func testPodDevices(t *testing.T, dataDir, deviceName string, allDevices bool) {
assert.Equal(t, 12, len(deployment.Spec.Template.Spec.Volumes), deployment.Spec.Template.Spec.Volumes)
// // Test OSD on PVC with RAW / metadata and wal device
osd = OSDInfo{
osd = &OSDInfo{
ID: 0,
CVMode: "raw",
}
@@ -333,7 +333,7 @@ func testPodDevices(t *testing.T, dataDir, deviceName string, allDevices bool) {
assert.Equal(t, 12, len(deployment.Spec.Template.Spec.Volumes), deployment.Spec.Template.Spec.Volumes)
// // Test encrypted OSD on PVC with RAW / metadata and wal device
osd = OSDInfo{
osd = &OSDInfo{
ID: 0,
CVMode: "raw",
}
@@ -608,7 +608,7 @@ func TestHostNetwork(t *testing.T) {
c := New(context, clusterInfo, spec, "rook/rook:myversion")
n := c.spec.Storage.ResolveNode(storageSpec.Nodes[0].Name)
osd := OSDInfo{
osd := &OSDInfo{
ID: 0,
CVMode: "raw",
}
@@ -712,7 +712,7 @@ func TestClusterGetPVCEncryptionInitContainerActivate(t *testing.T) {
// WARNING! modifies c.deviceSets
func getDummyDeploymentOnPVC(clientset *fake.Clientset, c *Cluster, pvcName string, osdID int) *appsv1.Deployment {
osd := OSDInfo{
osd := &OSDInfo{
ID: osdID,
UUID: "some-uuid",
BlockPath: "/some/path",
@@ -736,7 +736,7 @@ func getDummyDeploymentOnPVC(clientset *fake.Clientset, c *Cluster, pvcName stri
// WARNING! modifies c.ValidStorage
func getDummyDeploymentOnNode(clientset *fake.Clientset, c *Cluster, nodeName string, osdID int) *appsv1.Deployment {
osd := OSDInfo{
osd := &OSDInfo{
ID: osdID,
UUID: "some-uuid",
BlockPath: "/dev/vda",
@@ -855,7 +855,7 @@ func TestOSDPlacement(t *testing.T) {
}
c := New(context, clusterInfo, spec, "rook/rook:myversion")
osd := OSDInfo{
osd := &OSDInfo{
ID: 0,
CVMode: "raw",
}
+5 -2
View File
@@ -49,6 +49,7 @@ type updateConfig struct {
numUpdatesNeeded int // the number of OSDs that needed updating
deployments *existenceList // these OSDs have existing deployments
osdsToSkipReconcile sets.Set[string] // these OSDs should not be updated during reconcile
osdDesiredState map[int]*OSDInfo // the desired state of the OSDs determined during the reconcile
}
func (c *Cluster) newUpdateConfig(
@@ -64,6 +65,7 @@ func (c *Cluster) newUpdateConfig(
queue.Len(),
deployments,
osdsToSkipReconcile,
map[int]*OSDInfo{},
}
}
@@ -142,6 +144,7 @@ func (c *updateConfig) updateExistingOSDs(errs *provisionErrors) {
errs.addError("failed to update OSD %d. failed to extract OSD info from existing deployment %q. %v", osdID, depName, err)
continue
}
c.osdDesiredState[osdID] = &osdInfo
if c.osdsToSkipReconcile.Has(strconv.Itoa(osdID)) {
logger.Warningf("Skipping update for OSD %d since labeled with %s", osdID, cephv1.SkipReconcileLabelKey)
@@ -174,7 +177,7 @@ func (c *updateConfig) updateExistingOSDs(errs *provisionErrors) {
if osdIsOnPVC(dep) {
logger.Infof("updating OSD %d on PVC %q", osdID, nodeOrPVCName)
updatedDep, err = deploymentOnPVCFunc(c.cluster, osdInfo, nodeOrPVCName, c.provisionConfig)
updatedDep, err = deploymentOnPVCFunc(c.cluster, &osdInfo, nodeOrPVCName, c.provisionConfig)
message := fmt.Sprintf("Processing OSD %d on PVC %q", osdID, nodeOrPVCName)
updateConditionFunc(c.cluster.clusterInfo.Context, c.cluster.context, c.cluster.clusterInfo.NamespacedName(), k8sutil.ObservedGenerationNotAvailable, cephv1.ConditionProgressing, v1.ConditionTrue, cephv1.ClusterProgressingReason, message)
} else {
@@ -189,7 +192,7 @@ func (c *updateConfig) updateExistingOSDs(errs *provisionErrors) {
}
logger.Infof("updating OSD %d on node %q", osdID, nodeOrPVCName)
updatedDep, err = deploymentOnNodeFunc(c.cluster, osdInfo, nodeOrPVCName, c.provisionConfig)
updatedDep, err = deploymentOnNodeFunc(c.cluster, &osdInfo, nodeOrPVCName, c.provisionConfig)
message := fmt.Sprintf("Processing OSD %d on node %q", osdID, nodeOrPVCName)
updateConditionFunc(c.cluster.clusterInfo.Context, c.cluster.context, c.cluster.clusterInfo.NamespacedName(), k8sutil.ObservedGenerationNotAvailable, cephv1.ConditionProgressing, v1.ConditionTrue, cephv1.ClusterProgressingReason, message)
}
+2 -2
View File
@@ -175,11 +175,11 @@ func Test_updateExistingOSDs(t *testing.T) {
}
// simple wrappers to allow us to count how many OSDs on nodes/PVCs are identified
deploymentOnNodeFunc = func(c *Cluster, osd OSDInfo, nodeName string, config *provisionConfig) (*appsv1.Deployment, error) {
deploymentOnNodeFunc = func(c *Cluster, osd *OSDInfo, nodeName string, config *provisionConfig) (*appsv1.Deployment, error) {
osdsOnNodes = append(osdsOnNodes, osd.ID)
return deploymentOnNode(c, osd, nodeName, config)
}
deploymentOnPVCFunc = func(c *Cluster, osd OSDInfo, pvcName string, config *provisionConfig) (*appsv1.Deployment, error) {
deploymentOnPVCFunc = func(c *Cluster, osd *OSDInfo, pvcName string, config *provisionConfig) (*appsv1.Deployment, error) {
osdsOnPVCs = append(osdsOnPVCs, osd.ID)
return deploymentOnPVC(c, osd, pvcName, config)
}