diff --git a/Documentation/ceph-common-issues.md b/Documentation/ceph-common-issues.md index 5aa39c729..7774ab9b2 100644 --- a/Documentation/ceph-common-issues.md +++ b/Documentation/ceph-common-issues.md @@ -29,6 +29,7 @@ If after trying the suggestions found on this page and the problem is not resolv * [Too few PGs per OSD warning is shown](#too-few-pgs-per-osd-warning-is-shown) * [LVM metadata can be corrupted with OSD on LV-backed PVC](#lvm-metadata-can-be-corrupted-with-osd-on-lv-backed-pvc) * [OSD prepare job fails due to low aio-max-nr setting](#osd-prepare-job-fails-due-to-low-aio-max-nr-setting) +* [Failed to create CRDs](#failed-to-create-crds) ## Troubleshooting Techniques @@ -833,3 +834,10 @@ To overcome this, you need to increase the value of `fs.aio-max-nr` of your sysc You can do this with your favorite configuration management system. Alternatively, you can have a [DaemonSet](https://github.com/rook/rook/issues/6279#issuecomment-694390514) to apply the configuration for you on all your nodes. + +## Failed to create CRDs +If you are using Kubernetes version is v1.15 or older, you will see an error like this: +``` +unable to recognize "STDIN": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1" +``` +You need to create the CRDs found in `cluster/examples/kubernetes/ceph/pre-k8s-1.16`. Note that these pre-1.16 `apiextensions.k8s.io/v1beta1` CRDs are deprecated in k8s v1.16 and will no longer be supported from k8s v1.22. diff --git a/Documentation/ceph-quickstart.md b/Documentation/ceph-quickstart.md index 0b028f120..e537fcff0 100644 --- a/Documentation/ceph-quickstart.md +++ b/Documentation/ceph-quickstart.md @@ -16,6 +16,8 @@ from other pods running in your cluster. Kubernetes **v1.11** or higher is supported by Rook. +**Important** If you are using K8s 1.15 or older, you will need to create a different version of the Rook CRDs. Create the `crd.yaml` found in the [pre-k8s-1.16](https://github.com/rook/rook/blob/{{ branchName }}/cluster/examples/kubernetes/ceph/pre-k8s-1.16) subfolder of the example manifests. + ## Prerequisites To make sure you have a Kubernetes cluster that is ready for `Rook`, you can [follow these instructions](k8s-pre-reqs.md). @@ -76,6 +78,11 @@ kubectl create -f operator.yaml kubectl -n rook-ceph get pod ``` +**NOTE** + +If you are using kubernetes v1.15 or older you need to create CRDs found here `/cluster/examples/kubernetes/ceph` as apiextension v1beta1 version of CustomResourceDefinition +is deprecated in Kubernetes v1.16 and will no longer be supported from k8s v1.22. + You can also deploy the operator with the [Rook Helm Chart](helm-operator.md). ## Create a Rook Ceph Cluster diff --git a/Documentation/k8s-pre-reqs.md b/Documentation/k8s-pre-reqs.md index 30afea8f9..e4a1f021f 100644 --- a/Documentation/k8s-pre-reqs.md +++ b/Documentation/k8s-pre-reqs.md @@ -18,7 +18,9 @@ you can quickly set one up using [Minikube](#minikube), [Kubeadm](#kubeadm) or [ ## Minimum Version -Kubernetes v1.11 or higher is supported by Rook. +Kubernetes **v1.11** or higher is supported by Rook. + +**Important** If you are using K8s 1.15 or older, you will need to create a different version of the Rook CRDs. Create the `crd.yaml` found in the [pre-k8s-1.16](https://github.com/rook/rook/blob/{{ branchName }}/cluster/examples/kubernetes/ceph/pre-k8s-1.16) subfolder of the example manifests. ## Ceph Prerequisites diff --git a/PendingReleaseNotes.md b/PendingReleaseNotes.md index 0e03c9a24..ea202503f 100644 --- a/PendingReleaseNotes.md +++ b/PendingReleaseNotes.md @@ -11,6 +11,7 @@ v1.5... ### Ceph - Ceph mons require an odd number for a healthy quorum. An even number of mons is now disallowed. +- Update deprecated CRD apiextensions.k8s.io/v1beta1 to v1 ([#6424](https://github.com/rook/rook/pull/6424)) ## Features diff --git a/cluster/charts/rook-ceph/templates/resources.yaml b/cluster/charts/rook-ceph/templates/resources.yaml index 295ab6f41..e7c88ff34 100644 --- a/cluster/charts/rook-ceph/templates/resources.yaml +++ b/cluster/charts/rook-ceph/templates/resources.yaml @@ -1,4 +1,1276 @@ +{{- if semverCompare ">=1.16.0" .Capabilities.KubeVersion.GitVersion }} --- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + name: cephclusters.ceph.rook.io +spec: + group: ceph.rook.io + names: + kind: CephCluster + plural: cephclusters + singular: cephcluster + scope: Namespaced + versions: + - name: v1 + served: true + storage: true + schema: + openAPIV3Schema: + type: object + properties: + spec: + type: object + properties: + cephVersion: + type: object + properties: + allowUnsupported: + type: boolean + image: + type: string + dashboard: + type: object + properties: + enabled: + type: boolean + urlPrefix: + type: string + port: + type: integer + minimum: 0 + maximum: 65535 + ssl: + type: boolean + dataDirHostPath: + pattern: ^/(\S+) + type: string + disruptionManagement: + type: object + properties: + machineDisruptionBudgetNamespace: + type: string + managePodBudgets: + type: boolean + osdMaintenanceTimeout: + type: integer + manageMachineDisruptionBudgets: + type: boolean + skipUpgradeChecks: + type: boolean + continueUpgradeAfterChecksEvenIfNotHealthy: + type: boolean + mon: + type: object + properties: + allowMultiplePerNode: + type: boolean + count: + maximum: 9 + minimum: 0 + type: integer + volumeClaimTemplate: + type: object + x-kubernetes-preserve-unknown-fields: true + mgr: + type: object + properties: + modules: + type: array + items: + type: object + properties: + name: + type: string + enabled: + type: boolean + crashCollector: + type: object + properties: + disable: + type: boolean + network: + type: object + nullable: true + properties: + hostNetwork: + type: boolean + provider: + type: string + selectors: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + ipFamily: + type: string + storage: + type: object + properties: + disruptionManagement: + type: object + nullable: true + properties: + machineDisruptionBudgetNamespace: + type: string + managePodBudgets: + type: boolean + osdMaintenanceTimeout: + type: integer + manageMachineDisruptionBudgets: + type: boolean + useAllNodes: + type: boolean + nodes: + type: array + nullable: true + items: + type: object + properties: + name: + type: string + config: + type: object + nullable: true + properties: + metadataDevice: + type: string + storeType: + type: string + pattern: ^(bluestore)$ + databaseSizeMB: + type: string + walSizeMB: + type: string + journalSizeMB: + type: string + osdsPerDevice: + type: string + encryptedDevice: + type: string + pattern: ^(true|false)$ + useAllDevices: + type: boolean + deviceFilter: + type: string + nullable: true + devicePathFilter: + type: string + devices: + type: array + items: + type: object + properties: + name: + type: string + config: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + resources: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + useAllDevices: + type: boolean + deviceFilter: + type: string + nullable: true + devicePathFilter: + type: string + config: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + storageClassDeviceSets: + type: array + nullable: true + items: + type: object + properties: + name: + type: string + count: + type: integer + format: int32 + portable: + type: boolean + tuneDeviceClass: + type: boolean + tuneFastDeviceClass: + type: boolean + encrypted: + type: boolean + schedulerName: + type: string + config: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + placement: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + preparePlacement: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + resources: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + volumeClaimTemplates: + type: array + items: + type: object + x-kubernetes-preserve-unknown-fields: true + driveGroups: + type: array + nullable: true + items: + type: object + properties: + name: + type: string + spec: + type: object + x-kubernetes-preserve-unknown-fields: true + placement: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + required: + - name + - spec + monitoring: + type: object + properties: + enabled: + type: boolean + rulesNamespace: + type: string + externalMgrEndpoints: + type: array + items: + type: object + properties: + ip: + type: string + removeOSDsIfOutAndSafeToRemove: + type: boolean + external: + type: object + properties: + enable: + type: boolean + cleanupPolicy: + type: object + properties: + allowUninstallWithVolumes: + type: boolean + confirmation: + type: string + pattern: ^$|^yes-really-destroy-data$ + sanitizeDisks: + type: object + properties: + method: + type: string + pattern: ^(complete|quick)$ + dataSource: + type: string + pattern: ^(zero|random)$ + iteration: + type: integer + format: int32 + annotations: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + placement: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + labels: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + resources: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + healthCheck: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + priorityClassNames: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + status: + type: object + x-kubernetes-preserve-unknown-fields: true + additionalPrinterColumns: + - name: DataDirHostPath + type: string + description: Directory used on the K8s nodes + jsonPath: .spec.dataDirHostPath + - name: MonCount + type: string + description: Number of MONs + jsonPath: .spec.mon.count + - name: Age + type: date + jsonPath: .metadata.creationTimestamp + - name: Phase + type: string + description: Phase + jsonPath: .status.phase + - name: Message + type: string + description: Message + jsonPath: .status.message + - name: Health + type: string + description: Ceph Health + jsonPath: .status.ceph.health + subresources: + status: {} +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + name: cephclients.ceph.rook.io +spec: + group: ceph.rook.io + names: + kind: CephClient + listKind: CephClientList + plural: cephclients + singular: cephclient + scope: Namespaced + versions: + - name: v1 + served: true + storage: true + schema: + openAPIV3Schema: + type: object + properties: + spec: + type: object + properties: + caps: + type: object + x-kubernetes-preserve-unknown-fields: true + status: + type: object + x-kubernetes-preserve-unknown-fields: true + subresources: + status: {} +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + name: cephrbdmirrors.ceph.rook.io +spec: + group: ceph.rook.io + names: + kind: CephRBDMirror + listKind: CephRBDMirrorList + plural: cephrbdmirrors + singular: cephrbdmirror + scope: Namespaced + versions: + - name: v1 + served: true + storage: true + schema: + openAPIV3Schema: + type: object + properties: + spec: + type: object + properties: + count: + type: integer + minimum: 1 + maximum: 100 + peers: + type: object + properties: + secretNames: + type: array + items: + type: string + resources: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + priorityClassName: + type: string + placement: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + annotations: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + status: + type: object + x-kubernetes-preserve-unknown-fields: true + subresources: + status: {} +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + name: cephfilesystems.ceph.rook.io +spec: + group: ceph.rook.io + names: + kind: CephFilesystem + listKind: CephFilesystemList + plural: cephfilesystems + singular: cephfilesystem + scope: Namespaced + versions: + - name: v1 + served: true + storage: true + schema: + openAPIV3Schema: + type: object + properties: + spec: + type: object + properties: + metadataServer: + type: object + properties: + activeCount: + minimum: 1 + maximum: 10 + type: integer + activeStandby: + type: boolean + annotations: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + placement: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + resources: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + priorityClassName: + type: string + labels: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + metadataPool: + type: object + nullable: true + properties: + failureDomain: + type: string + crushRoot: + type: string + replicated: + type: object + nullable: true + properties: + size: + minimum: 0 + maximum: 10 + type: integer + requireSafeReplicaSize: + type: boolean + replicasPerFailureDomain: + type: integer + subFailureDomain: + type: string + parameters: + type: object + x-kubernetes-preserve-unknown-fields: true + erasureCoded: + type: object + nullable: true + properties: + dataChunks: + minimum: 0 + maximum: 10 + type: integer + codingChunks: + minimum: 0 + maximum: 10 + type: integer + compressionMode: + type: string + enum: + - "" + - none + - passive + - aggressive + - force + dataPools: + type: array + nullable: true + items: + type: object + properties: + failureDomain: + type: string + crushRoot: + type: string + replicated: + type: object + nullable: true + properties: + size: + minimum: 0 + maximum: 10 + type: integer + requireSafeReplicaSize: + type: boolean + replicasPerFailureDomain: + type: integer + subFailureDomain: + type: string + erasureCoded: + type: object + nullable: true + properties: + dataChunks: + minimum: 0 + maximum: 10 + type: integer + codingChunks: + minimum: 0 + maximum: 10 + type: integer + compressionMode: + type: string + enum: + - "" + - none + - passive + - aggressive + - force + parameters: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + preservePoolsOnDelete: + type: boolean + status: + type: object + x-kubernetes-preserve-unknown-fields: true + additionalPrinterColumns: + - name: ActiveMDS + type: string + description: Number of desired active MDS daemons + jsonPath: .spec.metadataServer.activeCount + - name: Age + type: date + jsonPath: .metadata.creationTimestamp + subresources: + status: {} +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + name: cephnfses.ceph.rook.io +spec: + group: ceph.rook.io + names: + kind: CephNFS + listKind: CephNFSList + plural: cephnfses + singular: cephnfs + shortNames: + - nfs + scope: Namespaced + versions: + - name: v1 + served: true + storage: true + schema: + openAPIV3Schema: + type: object + properties: + spec: + type: object + properties: + rados: + type: object + properties: + pool: + type: string + namespace: + type: string + server: + type: object + properties: + active: + type: integer + annotations: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + placement: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + resources: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + priorityClassName: + type: string + status: + type: object + x-kubernetes-preserve-unknown-fields: true + subresources: + status: {} +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + name: cephobjectstores.ceph.rook.io +spec: + group: ceph.rook.io + names: + kind: CephObjectStore + listKind: CephObjectStoreList + plural: cephobjectstores + singular: cephobjectstore + scope: Namespaced + versions: + - name: v1 + served: true + storage: true + schema: + openAPIV3Schema: + type: object + properties: + spec: + type: object + properties: + gateway: + type: object + properties: + type: + type: string + sslCertificateRef: + type: string + nullable: true + port: + type: integer + minimum: 0 + maximum: 65535 + securePort: + type: integer + minimum: 0 + maximum: 65535 + instances: + type: integer + externalRgwEndpoints: + type: array + nullable: true + items: + type: object + properties: + ip: + type: string + annotations: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + placement: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + resources: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + labels: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + priorityClassName: + type: string + metadataPool: + type: object + nullable: true + properties: + failureDomain: + type: string + crushRoot: + type: string + replicated: + type: object + nullable: true + properties: + size: + type: integer + requireSafeReplicaSize: + type: boolean + replicasPerFailureDomain: + type: integer + subFailureDomain: + type: string + erasureCoded: + type: object + nullable: true + properties: + dataChunks: + type: integer + codingChunks: + type: integer + compressionMode: + type: string + enum: + - "" + - none + - passive + - aggressive + - force + parameters: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + zone: + type: object + properties: + name: + type: string + dataPool: + type: object + nullable: true + properties: + failureDomain: + type: string + crushRoot: + type: string + replicated: + type: object + nullable: true + properties: + size: + type: integer + requireSafeReplicaSize: + type: boolean + replicasPerFailureDomain: + type: integer + subFailureDomain: + type: string + erasureCoded: + type: object + nullable: true + properties: + dataChunks: + type: integer + codingChunks: + type: integer + compressionMode: + type: string + enum: + - "" + - none + - passive + - aggressive + - force + parameters: + type: object + x-kubernetes-preserve-unknown-fields: true + preservePoolsOnDelete: + type: boolean + healthCheck: + type: object + nullable: true + properties: + bucket: + type: object + nullable: true + properties: + enabled: + type: boolean + interval: + type: string + status: + type: object + x-kubernetes-preserve-unknown-fields: true + subresources: + status: {} +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + name: cephobjectstoreusers.ceph.rook.io +spec: + group: ceph.rook.io + names: + kind: CephObjectStoreUser + listKind: CephObjectStoreUserList + plural: cephobjectstoreusers + singular: cephobjectstoreuser + shortNames: + - rcou + - objectuser + scope: Namespaced + versions: + - name: v1 + served: true + storage: true + schema: + openAPIV3Schema: + type: object + properties: + spec: + type: object + properties: + store: + type: string + displayName: + type: string + status: + type: object + x-kubernetes-preserve-unknown-fields: true + subresources: + status: {} +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + name: cephobjectrealms.ceph.rook.io +spec: + group: ceph.rook.io + names: + kind: CephObjectRealm + listKind: CephObjectRealmList + plural: cephobjectrealms + singular: cephobjectrealm + scope: Namespaced + versions: + - name: v1 + served: true + storage: true + schema: + openAPIV3Schema: + type: object + properties: + spec: + type: object + properties: + pull: + type: object + properties: + endpoint: + type: string + status: + type: object + x-kubernetes-preserve-unknown-fields: true + subresources: + status: {} +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + name: cephobjectzonegroups.ceph.rook.io +spec: + group: ceph.rook.io + names: + kind: CephObjectZoneGroup + listKind: CephObjectZoneGroupList + plural: cephobjectzonegroups + singular: cephobjectzonegroup + scope: Namespaced + versions: + - name: v1 + served: true + storage: true + schema: + openAPIV3Schema: + type: object + properties: + spec: + type: object + properties: + realm: + type: string + status: + type: object + x-kubernetes-preserve-unknown-fields: true + subresources: + status: {} +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + name: cephobjectzones.ceph.rook.io +spec: + group: ceph.rook.io + names: + kind: CephObjectZone + listKind: CephObjectZoneList + plural: cephobjectzones + singular: cephobjectzone + scope: Namespaced + versions: + - name: v1 + served: true + storage: true + schema: + openAPIV3Schema: + type: object + properties: + spec: + type: object + properties: + zoneGroup: + type: string + metadataPool: + type: object + nullable: true + properties: + failureDomain: + type: string + crushRoot: + type: string + replicated: + type: object + nullable: true + properties: + size: + type: integer + requireSafeReplicaSize: + type: boolean + erasureCoded: + type: object + nullable: true + properties: + dataChunks: + type: integer + codingChunks: + type: integer + compressionMode: + type: string + enum: + - "" + - none + - passive + - aggressive + - force + parameters: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + dataPool: + type: object + properties: + failureDomain: + type: string + crushRoot: + type: string + replicated: + type: object + nullable: true + properties: + size: + type: integer + requireSafeReplicaSize: + type: boolean + erasureCoded: + type: object + nullable: true + properties: + dataChunks: + type: integer + codingChunks: + type: integer + compressionMode: + type: string + enum: + - "" + - none + - passive + - aggressive + - force + parameters: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + preservePoolsOnDelete: + type: boolean + status: + type: object + x-kubernetes-preserve-unknown-fields: true + subresources: + status: {} +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + name: cephblockpools.ceph.rook.io +spec: + group: ceph.rook.io + names: + kind: CephBlockPool + listKind: CephBlockPoolList + plural: cephblockpools + singular: cephblockpool + scope: Namespaced + versions: + - name: v1 + served: true + storage: true + schema: + openAPIV3Schema: + type: object + properties: + spec: + type: object + properties: + failureDomain: + type: string + deviceClass: + type: string + crushRoot: + type: string + replicated: + type: object + nullable: true + properties: + size: + type: integer + minimum: 0 + maximum: 9 + targetSizeRatio: + type: number + requireSafeReplicaSize: + type: boolean + replicasPerFailureDomain: + type: integer + subFailureDomain: + type: string + erasureCoded: + type: object + nullable: true + properties: + dataChunks: + type: integer + minimum: 0 + maximum: 9 + codingChunks: + type: integer + minimum: 0 + maximum: 9 + compressionMode: + type: string + enum: + - "" + - none + - passive + - aggressive + - force + enableRBDStats: + description: + EnableRBDStats is used to enable gathering of statistics + for all RBD images in the pool + type: boolean + parameters: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + mirroring: + type: object + nullable: true + properties: + enabled: + type: boolean + mode: + type: string + enum: + - image + - pool + statusCheck: + type: object + x-kubernetes-preserve-unknown-fields: true + annotations: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + status: + type: object + x-kubernetes-preserve-unknown-fields: true + subresources: + status: {} +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + name: volumes.rook.io +spec: + group: rook.io + names: + kind: Volume + listKind: VolumeList + plural: volumes + singular: volume + shortNames: + - rv + scope: Namespaced + versions: + - name: v1alpha2 + served: true + storage: true + schema: + openAPIV3Schema: + type: object + properties: + attachments: + type: array + items: + type: object + properties: + node: + type: string + podNamespace: + type: string + podName: + type: string + clusterName: + type: string + mountDir: + type: string + readOnly: + type: boolean + status: + type: object + x-kubernetes-preserve-unknown-fields: true + subresources: + status: {} +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + name: objectbuckets.objectbucket.io +spec: + group: objectbucket.io + names: + kind: ObjectBucket + listKind: ObjectBucketList + plural: objectbuckets + singular: objectbucket + shortNames: + - ob + - obs + scope: Cluster + versions: + - name: v1alpha1 + served: true + storage: true + schema: + openAPIV3Schema: + type: object + properties: + spec: + type: object + properties: + storageClassName: + type: string + endpoint: + type: object + nullable: true + properties: + bucketHost: + type: string + bucketPort: + type: integer + format: int32 + bucketName: + type: string + region: + type: string + subRegion: + type: string + additionalConfig: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + authentication: + type: object + nullable: true + items: + type: object + x-kubernetes-preserve-unknown-fields: true + additionalState: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + reclaimPolicy: + type: string + claimRef: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + items: + type: object + x-kubernetes-preserve-unknown-fields: true + status: + type: object + x-kubernetes-preserve-unknown-fields: true + subresources: + status: {} +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + name: objectbucketclaims.objectbucket.io +spec: + group: objectbucket.io + names: + kind: ObjectBucketClaim + listKind: ObjectBucketClaimList + plural: objectbucketclaims + singular: objectbucketclaim + shortNames: + - obc + - obcs + scope: Namespaced + versions: + - name: v1alpha1 + served: true + storage: true + schema: + openAPIV3Schema: + type: object + properties: + spec: + type: object + properties: + storageClassName: + type: string + bucketName: + type: string + generateBucketName: + type: string + additionalConfig: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + status: + type: object + x-kubernetes-preserve-unknown-fields: true + subresources: + status: {} +{{- else }} apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: @@ -733,3 +2005,4 @@ spec: type: array subresources: status: {} +{{- end }} diff --git a/cluster/examples/kubernetes/ceph/cluster-with-drive-groups.yaml b/cluster/examples/kubernetes/ceph/cluster-with-drive-groups.yaml index 6a785f8f0..0fffb3ea8 100644 --- a/cluster/examples/kubernetes/ceph/cluster-with-drive-groups.yaml +++ b/cluster/examples/kubernetes/ceph/cluster-with-drive-groups.yaml @@ -54,7 +54,8 @@ spec: # placement: # is ignored here in favor of Rook's placement below # host_pattern: # (deprecated in Ceph) is similarly ignored here in favor of Rook's placement # service_id: # is always set to the Drive Group name ("all_nodes" here) - placement: # example: omitting 'placement' or leaving it empty matches all untainted nodes + # omitting 'placement' or leaving it empty matches all untainted nodes + #placement: # omitting 'placement' or leaving it empty matches all untainted nodes # # Further examples # - name: nodes_by_hostname diff --git a/cluster/examples/kubernetes/ceph/common.yaml b/cluster/examples/kubernetes/ceph/common.yaml index 7881b31a8..07976af64 100644 --- a/cluster/examples/kubernetes/ceph/common.yaml +++ b/cluster/examples/kubernetes/ceph/common.yaml @@ -18,7 +18,7 @@ metadata: # OLM: BEGIN CEPH CRD # The CRD declarations --- -apiVersion: apiextensions.k8s.io/v1beta1 +apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: cephclusters.ceph.rook.io @@ -26,80 +26,45 @@ spec: group: ceph.rook.io names: kind: CephCluster - listKind: CephClusterList plural: cephclusters singular: cephcluster scope: Namespaced - version: v1 - validation: - openAPIV3Schema: - properties: - spec: + versions: + - name: v1 + served: true + storage: true + schema: + openAPIV3Schema: + type: object properties: - annotations: {} - cephVersion: + spec: + type: object properties: - allowUnsupported: - type: boolean - image: + cephVersion: + type: object + properties: + allowUnsupported: + type: boolean + image: + type: string + dashboard: + type: object + properties: + enabled: + type: boolean + urlPrefix: + type: string + port: + type: integer + minimum: 0 + maximum: 65535 + ssl: + type: boolean + dataDirHostPath: + pattern: ^/(\S+) type: string - dashboard: - properties: - enabled: - type: boolean - urlPrefix: - type: string - port: - type: integer - minimum: 0 - maximum: 65535 - ssl: - type: boolean - dataDirHostPath: - pattern: ^/(\S+) - type: string - disruptionManagement: - properties: - machineDisruptionBudgetNamespace: - type: string - managePodBudgets: - type: boolean - osdMaintenanceTimeout: - type: integer - manageMachineDisruptionBudgets: - type: boolean - skipUpgradeChecks: - type: boolean - continueUpgradeAfterChecksEvenIfNotHealthy: - type: boolean - mon: - properties: - allowMultiplePerNode: - type: boolean - count: - maximum: 9 - minimum: 0 - type: integer - volumeClaimTemplate: {} - mgr: - properties: - modules: - items: - properties: - name: - type: string - enabled: - type: boolean - network: - properties: - hostNetwork: - type: boolean - provider: - type: string - selectors: {} - storage: - properties: disruptionManagement: + type: object properties: machineDisruptionBudgetNamespace: type: string @@ -109,133 +74,289 @@ spec: type: integer manageMachineDisruptionBudgets: type: boolean - useAllNodes: + skipUpgradeChecks: type: boolean - nodes: + continueUpgradeAfterChecksEvenIfNotHealthy: + type: boolean + mon: + type: object + properties: + allowMultiplePerNode: + type: boolean + count: + maximum: 9 + minimum: 0 + type: integer + volumeClaimTemplate: + type: object + x-kubernetes-preserve-unknown-fields: true + mgr: + type: object + properties: + modules: + type: array + items: + type: object + properties: + name: + type: string + enabled: + type: boolean + crashCollector: + type: object + properties: + disable: + type: boolean + network: + type: object + nullable: true + properties: + hostNetwork: + type: boolean + provider: + type: string + selectors: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + ipFamily: + type: string + storage: + type: object + properties: + disruptionManagement: + type: object + nullable: true + properties: + machineDisruptionBudgetNamespace: + type: string + managePodBudgets: + type: boolean + osdMaintenanceTimeout: + type: integer + manageMachineDisruptionBudgets: + type: boolean + useAllNodes: + type: boolean + nodes: + type: array + nullable: true + items: + type: object + properties: + name: + type: string + config: + type: object + nullable: true + properties: + metadataDevice: + type: string + storeType: + type: string + pattern: ^(bluestore)$ + databaseSizeMB: + type: string + walSizeMB: + type: string + journalSizeMB: + type: string + osdsPerDevice: + type: string + encryptedDevice: + type: string + pattern: ^(true|false)$ + useAllDevices: + type: boolean + deviceFilter: + type: string + nullable: true + devicePathFilter: + type: string + devices: + type: array + items: + type: object + properties: + name: + type: string + config: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + resources: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + useAllDevices: + type: boolean + deviceFilter: + type: string + nullable: true + devicePathFilter: + type: string + config: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + storageClassDeviceSets: + type: array + nullable: true + items: + type: object + properties: + name: + type: string + count: + type: integer + format: int32 + portable: + type: boolean + tuneDeviceClass: + type: boolean + tuneFastDeviceClass: + type: boolean + encrypted: + type: boolean + schedulerName: + type: string + config: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + placement: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + preparePlacement: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + resources: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + volumeClaimTemplates: + type: array + items: + type: object + x-kubernetes-preserve-unknown-fields: true + driveGroups: + type: array + nullable: true items: + type: object properties: name: type: string - config: - properties: - metadataDevice: - type: string - storeType: - type: string - pattern: ^(bluestore)$ - databaseSizeMB: - type: string - walSizeMB: - type: string - journalSizeMB: - type: string - osdsPerDevice: - type: string - encryptedDevice: - type: string - pattern: ^(true|false)$ - useAllDevices: - type: boolean - deviceFilter: - type: string - devicePathFilter: - type: string - devices: - type: array - items: - properties: - name: - type: string - config: {} - resources: {} - type: array - useAllDevices: - type: boolean - deviceFilter: - type: string - devicePathFilter: - type: string - config: {} - storageClassDeviceSets: {} - driveGroups: - type: array - nullable: true - items: - properties: - name: - type: string - spec: {} - placement: {} - required: - - name - - spec - monitoring: - properties: - enabled: - type: boolean - rulesNamespace: - type: string - externalMgrEndpoints: - type: array - items: - properties: - ip: - type: string - removeOSDsIfOutAndSafeToRemove: - type: boolean - external: - properties: - enable: - type: boolean - cleanupPolicy: - properties: - confirmation: - type: string - pattern: ^$|^yes-really-destroy-data$ - sanitizeDisks: + spec: + type: object + x-kubernetes-preserve-unknown-fields: true + placement: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + required: + - name + - spec + monitoring: + type: object properties: - method: + enabled: + type: boolean + rulesNamespace: type: string - pattern: ^(complete|quick)$ - dataSource: + externalMgrEndpoints: + type: array + items: + type: object + properties: + ip: + type: string + removeOSDsIfOutAndSafeToRemove: + type: boolean + external: + type: object + properties: + enable: + type: boolean + cleanupPolicy: + type: object + properties: + allowUninstallWithVolumes: + type: boolean + confirmation: type: string - pattern: ^(zero|random)$ - iteration: - type: integer - format: int32 - placement: {} - resources: {} - healthCheck: {} - subresources: - status: {} - additionalPrinterColumns: - - name: DataDirHostPath - type: string - description: Directory used on the K8s nodes - JSONPath: .spec.dataDirHostPath - - name: MonCount - type: string - description: Number of MONs - JSONPath: .spec.mon.count - - name: Age - type: date - JSONPath: .metadata.creationTimestamp - - name: Phase - type: string - description: Phase - JSONPath: .status.phase - - name: Message - type: string - description: Message - JSONPath: .status.message - - name: Health - type: string - description: Ceph Health - JSONPath: .status.ceph.health + pattern: ^$|^yes-really-destroy-data$ + sanitizeDisks: + type: object + properties: + method: + type: string + pattern: ^(complete|quick)$ + dataSource: + type: string + pattern: ^(zero|random)$ + iteration: + type: integer + format: int32 + annotations: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + placement: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + labels: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + resources: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + healthCheck: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + priorityClassNames: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + status: + type: object + x-kubernetes-preserve-unknown-fields: true + additionalPrinterColumns: + - name: DataDirHostPath + type: string + description: Directory used on the K8s nodes + jsonPath: .spec.dataDirHostPath + - name: MonCount + type: string + description: Number of MONs + jsonPath: .spec.mon.count + - name: Age + type: date + jsonPath: .metadata.creationTimestamp + - name: Phase + type: string + description: Phase + jsonPath: .status.phase + - name: Message + type: string + description: Message + jsonPath: .status.message + - name: Health + type: string + description: Ceph Health + jsonPath: .status.ceph.health + subresources: + status: {} # OLM: END CEPH CRD # OLM: BEGIN CEPH CLIENT CRD --- -apiVersion: apiextensions.k8s.io/v1beta1 +apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: cephclients.ceph.rook.io @@ -247,20 +368,29 @@ spec: plural: cephclients singular: cephclient scope: Namespaced - version: v1 - validation: - openAPIV3Schema: - properties: - spec: + versions: + - name: v1 + served: true + storage: true + schema: + openAPIV3Schema: + type: object properties: - caps: + spec: type: object - subresources: - status: {} + properties: + caps: + type: object + x-kubernetes-preserve-unknown-fields: true + status: + type: object + x-kubernetes-preserve-unknown-fields: true + subresources: + status: {} # OLM: END CEPH CLIENT CRD # OLM: BEGIN CEPH RBD MIRROR CRD --- -apiVersion: apiextensions.k8s.io/v1beta1 +apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: cephrbdmirrors.ceph.rook.io @@ -272,26 +402,52 @@ spec: plural: cephrbdmirrors singular: cephrbdmirror scope: Namespaced - version: v1 - validation: - openAPIV3Schema: - properties: - spec: + versions: + - name: v1 + served: true + storage: true + schema: + openAPIV3Schema: + type: object properties: - count: - type: integer - minimum: 1 - maximum: 100 - peers: + spec: + type: object properties: - secretNames: - type: array - subresources: - status: {} + count: + type: integer + minimum: 1 + maximum: 100 + peers: + type: object + properties: + secretNames: + type: array + items: + type: string + resources: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + priorityClassName: + type: string + placement: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + annotations: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + status: + type: object + x-kubernetes-preserve-unknown-fields: true + subresources: + status: {} + # OLM: END CEPH RBD MIRROR CRD # OLM: BEGIN CEPH FS CRD --- -apiVersion: apiextensions.k8s.io/v1beta1 +apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: cephfilesystems.ceph.rook.io @@ -303,115 +459,156 @@ spec: plural: cephfilesystems singular: cephfilesystem scope: Namespaced - version: v1 - validation: - openAPIV3Schema: - properties: - spec: + versions: + - name: v1 + served: true + storage: true + schema: + openAPIV3Schema: + type: object properties: - metadataServer: + spec: + type: object properties: - activeCount: - minimum: 1 - maximum: 10 - type: integer - activeStandby: - type: boolean - annotations: {} - placement: {} - resources: {} - metadataPool: - properties: - failureDomain: - type: string - crushRoot: - type: string - replicated: + metadataServer: + type: object properties: - size: - minimum: 0 + activeCount: + minimum: 1 maximum: 10 type: integer - requireSafeReplicaSize: + activeStandby: type: boolean - replicasPerFailureDomain: - type: integer - subFailureDomain: + annotations: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + placement: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + resources: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + priorityClassName: type: string - erasureCoded: + labels: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + metadataPool: + type: object + nullable: true properties: - dataChunks: - minimum: 0 - maximum: 10 - type: integer - codingChunks: - minimum: 0 - maximum: 10 - type: integer - compressionMode: - type: string - enum: - - "" - - none - - passive - - aggressive - - force - dataPools: - type: array - items: - properties: - failureDomain: - type: string - crushRoot: - type: string - replicated: - properties: - size: - minimum: 0 - maximum: 10 - type: integer - requireSafeReplicaSize: - type: boolean - replicasPerFailureDomain: - type: integer - subFailureDomain: - type: string - erasureCoded: - properties: - dataChunks: - minimum: 0 - maximum: 10 - type: integer - codingChunks: - minimum: 0 - maximum: 10 - type: integer - compressionMode: - type: string - enum: - - "" - - none - - passive - - aggressive - - force - parameters: + failureDomain: + type: string + crushRoot: + type: string + replicated: + type: object + nullable: true + properties: + size: + minimum: 0 + maximum: 10 + type: integer + requireSafeReplicaSize: + type: boolean + replicasPerFailureDomain: + type: integer + subFailureDomain: + type: string + parameters: + type: object + x-kubernetes-preserve-unknown-fields: true + erasureCoded: + type: object + nullable: true + properties: + dataChunks: + minimum: 0 + maximum: 10 + type: integer + codingChunks: + minimum: 0 + maximum: 10 + type: integer + compressionMode: + type: string + enum: + - "" + - none + - passive + - aggressive + - force + dataPools: + type: array + nullable: true + items: type: object - preservePoolsOnDelete: - type: boolean - additionalPrinterColumns: - - name: ActiveMDS - type: string - description: Number of desired active MDS daemons - JSONPath: .spec.metadataServer.activeCount - - name: Age - type: date - JSONPath: .metadata.creationTimestamp - subresources: - status: {} + properties: + failureDomain: + type: string + crushRoot: + type: string + replicated: + type: object + nullable: true + properties: + size: + minimum: 0 + maximum: 10 + type: integer + requireSafeReplicaSize: + type: boolean + replicasPerFailureDomain: + type: integer + subFailureDomain: + type: string + erasureCoded: + type: object + nullable: true + properties: + dataChunks: + minimum: 0 + maximum: 10 + type: integer + codingChunks: + minimum: 0 + maximum: 10 + type: integer + compressionMode: + type: string + enum: + - "" + - none + - passive + - aggressive + - force + parameters: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + preservePoolsOnDelete: + type: boolean + status: + type: object + x-kubernetes-preserve-unknown-fields: true + additionalPrinterColumns: + - name: ActiveMDS + type: string + description: Number of desired active MDS daemons + jsonPath: .spec.metadataServer.activeCount + - name: Age + type: date + jsonPath: .metadata.creationTimestamp + subresources: + status: {} # OLM: END CEPH FS CRD # OLM: BEGIN CEPH NFS CRD --- -apiVersion: apiextensions.k8s.io/v1beta1 +apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: cephnfses.ceph.rook.io @@ -423,33 +620,54 @@ spec: plural: cephnfses singular: cephnfs shortNames: - - nfs + - nfs scope: Namespaced - version: v1 - validation: - openAPIV3Schema: - properties: - spec: + versions: + - name: v1 + served: true + storage: true + schema: + openAPIV3Schema: + type: object properties: - rados: + spec: + type: object properties: - pool: - type: string - namespace: - type: string - server: - properties: - active: - type: integer - annotations: {} - placement: {} - resources: {} - subresources: - status: {} + rados: + type: object + properties: + pool: + type: string + namespace: + type: string + server: + type: object + properties: + active: + type: integer + annotations: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + placement: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + resources: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + priorityClassName: + type: string + status: + type: object + x-kubernetes-preserve-unknown-fields: true + subresources: + status: {} # OLM: END CEPH NFS CRD # OLM: BEGIN CEPH OBJECT STORE CRD --- -apiVersion: apiextensions.k8s.io/v1beta1 +apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: cephobjectstores.ceph.rook.io @@ -461,116 +679,169 @@ spec: plural: cephobjectstores singular: cephobjectstore scope: Namespaced - version: v1 - validation: - openAPIV3Schema: - properties: - spec: + versions: + - name: v1 + served: true + storage: true + schema: + openAPIV3Schema: + type: object properties: - gateway: + spec: + type: object properties: - type: - type: string - sslCertificateRef: {} - port: - type: integer - minimum: 0 - maximum: 65535 - securePort: - type: integer - minimum: 0 - maximum: 65535 - instances: - type: integer - externalRgwEndpoints: - type: array - items: - properties: - ip: - type: string - annotations: {} - placement: {} - resources: {} - metadataPool: - properties: - failureDomain: - type: string - crushRoot: - type: string - replicated: - properties: - size: - type: integer - requireSafeReplicaSize: - type: boolean - replicasPerFailureDomain: - type: integer - subFailureDomain: - type: string - erasureCoded: - properties: - dataChunks: - type: integer - codingChunks: - type: integer - compressionMode: - type: string - enum: - - "" - - none - - passive - - aggressive - - force - parameters: + gateway: type: object - dataPool: - properties: - failureDomain: - type: string - crushRoot: - type: string - replicated: properties: - size: - type: integer - requireSafeReplicaSize: - type: boolean - replicasPerFailureDomain: - type: integer - subFailureDomain: + type: type: string - erasureCoded: - properties: - dataChunks: + sslCertificateRef: + type: string + nullable: true + port: type: integer - codingChunks: + minimum: 0 + maximum: 65535 + securePort: type: integer - compressionMode: - type: string - enum: - - "" - - none - - passive - - aggressive - - force - parameters: + minimum: 0 + maximum: 65535 + instances: + type: integer + externalRgwEndpoints: + type: array + nullable: true + items: + type: object + properties: + ip: + type: string + annotations: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + placement: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + resources: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + labels: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + priorityClassName: + type: string + metadataPool: type: object - preservePoolsOnDelete: - type: boolean - healthCheck: - properties: - bucket: + nullable: true properties: - enabled: - type: boolean - interval: + failureDomain: type: string - subresources: - status: {} + crushRoot: + type: string + replicated: + type: object + nullable: true + properties: + size: + type: integer + requireSafeReplicaSize: + type: boolean + replicasPerFailureDomain: + type: integer + subFailureDomain: + type: string + erasureCoded: + type: object + nullable: true + properties: + dataChunks: + type: integer + codingChunks: + type: integer + compressionMode: + type: string + enum: + - "" + - none + - passive + - aggressive + - force + parameters: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + zone: + type: object + properties: + name: + type: string + dataPool: + type: object + nullable: true + properties: + failureDomain: + type: string + crushRoot: + type: string + replicated: + type: object + nullable: true + properties: + size: + type: integer + requireSafeReplicaSize: + type: boolean + replicasPerFailureDomain: + type: integer + subFailureDomain: + type: string + erasureCoded: + type: object + nullable: true + properties: + dataChunks: + type: integer + codingChunks: + type: integer + compressionMode: + type: string + enum: + - "" + - none + - passive + - aggressive + - force + parameters: + type: object + x-kubernetes-preserve-unknown-fields: true + preservePoolsOnDelete: + type: boolean + healthCheck: + type: object + nullable: true + properties: + bucket: + type: object + nullable: true + properties: + enabled: + type: boolean + interval: + type: string + status: + type: object + x-kubernetes-preserve-unknown-fields: true + subresources: + status: {} + # OLM: END CEPH OBJECT STORE CRD # OLM: BEGIN CEPH OBJECT STORE USERS CRD --- -apiVersion: apiextensions.k8s.io/v1beta1 +apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: cephobjectstoreusers.ceph.rook.io @@ -582,16 +853,33 @@ spec: plural: cephobjectstoreusers singular: cephobjectstoreuser shortNames: - - rcou - - objectuser + - rcou + - objectuser scope: Namespaced - version: v1 - subresources: - status: {} + versions: + - name: v1 + served: true + storage: true + schema: + openAPIV3Schema: + type: object + properties: + spec: + type: object + properties: + store: + type: string + displayName: + type: string + status: + type: object + x-kubernetes-preserve-unknown-fields: true + subresources: + status: {} # OLM: END CEPH OBJECT STORE USERS CRD # OLM: BEGIN CEPH OBJECT REALM CRD --- -apiVersion: apiextensions.k8s.io/v1beta1 +apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: cephobjectrealms.ceph.rook.io @@ -603,13 +891,31 @@ spec: plural: cephobjectrealms singular: cephobjectrealm scope: Namespaced - version: v1 - subresources: - status: {} + versions: + - name: v1 + served: true + storage: true + schema: + openAPIV3Schema: + type: object + properties: + spec: + type: object + properties: + pull: + type: object + properties: + endpoint: + type: string + status: + type: object + x-kubernetes-preserve-unknown-fields: true + subresources: + status: {} # OLM: END CEPH OBJECT REALM CRD # OLM: BEGIN CEPH OBJECT ZONEGROUP CRD --- -apiVersion: apiextensions.k8s.io/v1beta1 +apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: cephobjectzonegroups.ceph.rook.io @@ -621,13 +927,28 @@ spec: plural: cephobjectzonegroups singular: cephobjectzonegroup scope: Namespaced - version: v1 - subresources: - status: {} + versions: + - name: v1 + served: true + storage: true + schema: + openAPIV3Schema: + type: object + properties: + spec: + type: object + properties: + realm: + type: string + status: + type: object + x-kubernetes-preserve-unknown-fields: true + subresources: + status: {} # OLM: END CEPH OBJECT ZONEGROUP CRD # OLM: BEGIN CEPH OBJECT ZONE CRD --- -apiVersion: apiextensions.k8s.io/v1beta1 +apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: cephobjectzones.ceph.rook.io @@ -639,13 +960,101 @@ spec: plural: cephobjectzones singular: cephobjectzone scope: Namespaced - version: v1 - subresources: - status: {} + versions: + - name: v1 + served: true + storage: true + schema: + openAPIV3Schema: + type: object + properties: + spec: + type: object + properties: + zoneGroup: + type: string + metadataPool: + type: object + nullable: true + properties: + failureDomain: + type: string + crushRoot: + type: string + replicated: + type: object + nullable: true + properties: + size: + type: integer + requireSafeReplicaSize: + type: boolean + erasureCoded: + type: object + nullable: true + properties: + dataChunks: + type: integer + codingChunks: + type: integer + compressionMode: + type: string + enum: + - "" + - none + - passive + - aggressive + - force + parameters: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + dataPool: + type: object + properties: + failureDomain: + type: string + crushRoot: + type: string + replicated: + type: object + nullable: true + properties: + size: + type: integer + requireSafeReplicaSize: + type: boolean + erasureCoded: + type: object + nullable: true + properties: + dataChunks: + type: integer + codingChunks: + type: integer + compressionMode: + type: string + enum: + - "" + - none + - passive + - aggressive + - force + parameters: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + preservePoolsOnDelete: + type: boolean + status: + type: object + x-kubernetes-preserve-unknown-fields: true + subresources: + status: {} # OLM: END CEPH OBJECT ZONE CRD # OLM: BEGIN CEPH BLOCK POOL CRD --- -apiVersion: apiextensions.k8s.io/v1beta1 +apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: cephblockpools.ceph.rook.io @@ -657,69 +1066,96 @@ spec: plural: cephblockpools singular: cephblockpool scope: Namespaced - version: v1 - validation: - openAPIV3Schema: - properties: - spec: + versions: + - name: v1 + served: true + storage: true + schema: + openAPIV3Schema: + type: object properties: - failureDomain: - type: string - crushRoot: - type: string - replicated: - properties: - size: - type: integer - minimum: 0 - maximum: 9 - targetSizeRatio: - type: number - requireSafeReplicaSize: - type: boolean - replicasPerFailureDomain: - type: integer - subFailureDomain: - type: string - erasureCoded: - properties: - dataChunks: - type: integer - minimum: 0 - maximum: 9 - codingChunks: - type: integer - minimum: 0 - maximum: 9 - compressionMode: - type: string - enum: - - "" - - none - - passive - - aggressive - - force - enableRBDStats: - description: EnableRBDStats is used to enable gathering of statistics - for all RBD images in the pool - type: boolean - parameters: + spec: type: object - mirroring: properties: - enabled: - type: boolean - mode: + failureDomain: + type: string + deviceClass: + type: string + crushRoot: + type: string + replicated: + type: object + nullable: true + properties: + size: + type: integer + minimum: 0 + maximum: 9 + targetSizeRatio: + type: number + requireSafeReplicaSize: + type: boolean + replicasPerFailureDomain: + type: integer + subFailureDomain: + type: string + erasureCoded: + type: object + nullable: true + properties: + dataChunks: + type: integer + minimum: 0 + maximum: 9 + codingChunks: + type: integer + minimum: 0 + maximum: 9 + compressionMode: type: string enum: - - image - - pool - subresources: - status: {} + - "" + - none + - passive + - aggressive + - force + enableRBDStats: + description: + EnableRBDStats is used to enable gathering of statistics + for all RBD images in the pool + type: boolean + parameters: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + mirroring: + type: object + nullable: true + properties: + enabled: + type: boolean + mode: + type: string + enum: + - image + - pool + statusCheck: + type: object + x-kubernetes-preserve-unknown-fields: true + annotations: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + status: + type: object + x-kubernetes-preserve-unknown-fields: true + subresources: + status: {} + # OLM: END CEPH BLOCK POOL CRD # OLM: BEGIN CEPH VOLUME POOL CRD --- -apiVersion: apiextensions.k8s.io/v1beta1 +apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: volumes.rook.io @@ -731,24 +1167,47 @@ spec: plural: volumes singular: volume shortNames: - - rv + - rv scope: Namespaced - version: v1alpha2 - subresources: - status: {} + versions: + - name: v1alpha2 + served: true + storage: true + schema: + openAPIV3Schema: + type: object + properties: + attachments: + type: array + items: + type: object + properties: + node: + type: string + podNamespace: + type: string + podName: + type: string + clusterName: + type: string + mountDir: + type: string + readOnly: + type: boolean + status: + type: object + x-kubernetes-preserve-unknown-fields: true + subresources: + status: {} # OLM: END CEPH VOLUME POOL CRD # OLM: BEGIN OBJECTBUCKET CRD --- -apiVersion: apiextensions.k8s.io/v1beta1 +apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: objectbuckets.objectbucket.io spec: group: objectbucket.io - versions: - - name: v1alpha1 - served: true - storage: true names: kind: ObjectBucket listKind: ObjectBucketList @@ -758,20 +1217,69 @@ spec: - ob - obs scope: Cluster - subresources: - status: {} -# OLM: END OBJECTBUCKET CRD -# OLM: BEGIN OBJECTBUCKETCLAIM CRD ---- -apiVersion: apiextensions.k8s.io/v1beta1 -kind: CustomResourceDefinition -metadata: - name: objectbucketclaims.objectbucket.io -spec: versions: - name: v1alpha1 served: true storage: true + schema: + openAPIV3Schema: + type: object + properties: + spec: + type: object + properties: + storageClassName: + type: string + endpoint: + type: object + nullable: true + properties: + bucketHost: + type: string + bucketPort: + type: integer + format: int32 + bucketName: + type: string + region: + type: string + subRegion: + type: string + additionalConfig: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + authentication: + type: object + nullable: true + items: + type: object + x-kubernetes-preserve-unknown-fields: true + additionalState: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + reclaimPolicy: + type: string + claimRef: + type: object + nullable: true + items: + type: object + x-kubernetes-preserve-unknown-fields: true + status: + type: object + x-kubernetes-preserve-unknown-fields: true + subresources: + status: {} +# OLM: END OBJECTBUCKET CRD +# OLM: BEGIN OBJECTBUCKETCLAIM CRD +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + name: objectbucketclaims.objectbucket.io +spec: group: objectbucket.io names: kind: ObjectBucketClaim @@ -782,8 +1290,32 @@ spec: - obc - obcs scope: Namespaced - subresources: - status: {} + versions: + - name: v1alpha1 + served: true + storage: true + schema: + openAPIV3Schema: + type: object + properties: + spec: + type: object + properties: + storageClassName: + type: string + bucketName: + type: string + generateBucketName: + type: string + additionalConfig: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + status: + type: object + x-kubernetes-preserve-unknown-fields: true + subresources: + status: {} # OLM: END OBJECTBUCKETCLAIM CRD # OLM: BEGIN OBJECTBUCKET ROLEBINDING --- diff --git a/cluster/examples/kubernetes/ceph/nfs.yaml b/cluster/examples/kubernetes/ceph/nfs.yaml index 951b6e55f..eca03069b 100644 --- a/cluster/examples/kubernetes/ceph/nfs.yaml +++ b/cluster/examples/kubernetes/ceph/nfs.yaml @@ -43,4 +43,4 @@ spec: # cpu: "500m" # memory: "1024Mi" # the priority class to set to influence the scheduler's pod preemption - priorityClassName: + #priorityClassName: diff --git a/cluster/examples/kubernetes/ceph/pre-k8s-1.16/crd.yaml b/cluster/examples/kubernetes/ceph/pre-k8s-1.16/crd.yaml new file mode 100644 index 000000000..c7c8cf0d2 --- /dev/null +++ b/cluster/examples/kubernetes/ceph/pre-k8s-1.16/crd.yaml @@ -0,0 +1,752 @@ +################################################################################################################### +# Create the common resources that are necessary to start the operator and the ceph cluster. +# These resources *must* be created before the operator.yaml and cluster.yaml or their variants. +# The samples all assume that a single operator will manage a single cluster crd in the same "rook-ceph" namespace. +# +# If the operator needs to manage multiple clusters (in different namespaces), see the section below +# for "cluster-specific resources". The resources below that section will need to be created for each namespace +# where the operator needs to manage the cluster. The resources above that section do not be created again. +# +# Most of the sections are prefixed with a 'OLM' keyword which is used to build our CSV for an OLM (Operator Life Cycle manager) +################################################################################################################### + +# Namespace where the operator and other rook resources are created +apiVersion: v1 +kind: Namespace +metadata: + name: rook-ceph +# The CRD declarations +--- +apiVersion: apiextensions.k8s.io/v1beta1 +kind: CustomResourceDefinition +metadata: + name: cephclusters.ceph.rook.io +spec: + group: ceph.rook.io + names: + kind: CephCluster + listKind: CephClusterList + plural: cephclusters + singular: cephcluster + scope: Namespaced + version: v1 + validation: + openAPIV3Schema: + properties: + spec: + properties: + annotations: {} + cephVersion: + properties: + allowUnsupported: + type: boolean + image: + type: string + dashboard: + properties: + enabled: + type: boolean + urlPrefix: + type: string + port: + type: integer + minimum: 0 + maximum: 65535 + ssl: + type: boolean + dataDirHostPath: + pattern: ^/(\S+) + type: string + disruptionManagement: + properties: + machineDisruptionBudgetNamespace: + type: string + managePodBudgets: + type: boolean + osdMaintenanceTimeout: + type: integer + manageMachineDisruptionBudgets: + type: boolean + skipUpgradeChecks: + type: boolean + continueUpgradeAfterChecksEvenIfNotHealthy: + type: boolean + mon: + properties: + allowMultiplePerNode: + type: boolean + count: + maximum: 9 + minimum: 0 + type: integer + volumeClaimTemplate: {} + mgr: + properties: + modules: + items: + properties: + name: + type: string + enabled: + type: boolean + network: + properties: + hostNetwork: + type: boolean + provider: + type: string + selectors: {} + storage: + properties: + disruptionManagement: + properties: + machineDisruptionBudgetNamespace: + type: string + managePodBudgets: + type: boolean + osdMaintenanceTimeout: + type: integer + manageMachineDisruptionBudgets: + type: boolean + useAllNodes: + type: boolean + nodes: + items: + properties: + name: + type: string + config: + properties: + metadataDevice: + type: string + storeType: + type: string + pattern: ^(bluestore)$ + databaseSizeMB: + type: string + walSizeMB: + type: string + journalSizeMB: + type: string + osdsPerDevice: + type: string + encryptedDevice: + type: string + pattern: ^(true|false)$ + useAllDevices: + type: boolean + deviceFilter: + type: string + devicePathFilter: + type: string + devices: + type: array + items: + properties: + name: + type: string + config: {} + resources: {} + type: array + useAllDevices: + type: boolean + deviceFilter: + type: string + devicePathFilter: + type: string + config: {} + storageClassDeviceSets: {} + driveGroups: + type: array + nullable: true + items: + properties: + name: + type: string + spec: {} + placement: {} + required: + - name + - spec + monitoring: + properties: + enabled: + type: boolean + rulesNamespace: + type: string + externalMgrEndpoints: + type: array + items: + properties: + ip: + type: string + removeOSDsIfOutAndSafeToRemove: + type: boolean + external: + properties: + enable: + type: boolean + cleanupPolicy: + properties: + confirmation: + type: string + pattern: ^$|^yes-really-destroy-data$ + sanitizeDisks: + properties: + method: + type: string + pattern: ^(complete|quick)$ + dataSource: + type: string + pattern: ^(zero|random)$ + iteration: + type: integer + format: int32 + placement: {} + resources: {} + healthCheck: {} + subresources: + status: {} + additionalPrinterColumns: + - name: DataDirHostPath + type: string + description: Directory used on the K8s nodes + JSONPath: .spec.dataDirHostPath + - name: MonCount + type: string + description: Number of MONs + JSONPath: .spec.mon.count + - name: Age + type: date + JSONPath: .metadata.creationTimestamp + - name: Phase + type: string + description: Phase + JSONPath: .status.phase + - name: Message + type: string + description: Message + JSONPath: .status.message + - name: Health + type: string + description: Ceph Health + JSONPath: .status.ceph.health + +--- +apiVersion: apiextensions.k8s.io/v1beta1 +kind: CustomResourceDefinition +metadata: + name: cephclients.ceph.rook.io +spec: + group: ceph.rook.io + names: + kind: CephClient + listKind: CephClientList + plural: cephclients + singular: cephclient + scope: Namespaced + version: v1 + validation: + openAPIV3Schema: + properties: + spec: + properties: + caps: + type: object + subresources: + status: {} + +--- +apiVersion: apiextensions.k8s.io/v1beta1 +kind: CustomResourceDefinition +metadata: + name: cephrbdmirrors.ceph.rook.io +spec: + group: ceph.rook.io + names: + kind: CephRBDMirror + listKind: CephRBDMirrorList + plural: cephrbdmirrors + singular: cephrbdmirror + scope: Namespaced + version: v1 + validation: + openAPIV3Schema: + properties: + spec: + properties: + count: + type: integer + minimum: 1 + maximum: 100 + peers: + properties: + secretNames: + type: array + subresources: + status: {} + +--- +apiVersion: apiextensions.k8s.io/v1beta1 +kind: CustomResourceDefinition +metadata: + name: cephfilesystems.ceph.rook.io +spec: + group: ceph.rook.io + names: + kind: CephFilesystem + listKind: CephFilesystemList + plural: cephfilesystems + singular: cephfilesystem + scope: Namespaced + version: v1 + validation: + openAPIV3Schema: + properties: + spec: + properties: + metadataServer: + properties: + activeCount: + minimum: 1 + maximum: 10 + type: integer + activeStandby: + type: boolean + annotations: {} + placement: {} + resources: {} + metadataPool: + properties: + failureDomain: + type: string + crushRoot: + type: string + replicated: + properties: + size: + minimum: 0 + maximum: 10 + type: integer + requireSafeReplicaSize: + type: boolean + erasureCoded: + properties: + dataChunks: + minimum: 0 + maximum: 10 + type: integer + codingChunks: + minimum: 0 + maximum: 10 + type: integer + compressionMode: + type: string + enum: + - "" + - none + - passive + - aggressive + - force + dataPools: + type: array + items: + properties: + failureDomain: + type: string + crushRoot: + type: string + replicated: + properties: + size: + minimum: 0 + maximum: 10 + type: integer + requireSafeReplicaSize: + type: boolean + erasureCoded: + properties: + dataChunks: + minimum: 0 + maximum: 10 + type: integer + codingChunks: + minimum: 0 + maximum: 10 + type: integer + compressionMode: + type: string + enum: + - "" + - none + - passive + - aggressive + - force + parameters: + type: object + preservePoolsOnDelete: + type: boolean + additionalPrinterColumns: + - name: ActiveMDS + type: string + description: Number of desired active MDS daemons + JSONPath: .spec.metadataServer.activeCount + - name: Age + type: date + JSONPath: .metadata.creationTimestamp + subresources: + status: {} + +--- +apiVersion: apiextensions.k8s.io/v1beta1 +kind: CustomResourceDefinition +metadata: + name: cephnfses.ceph.rook.io +spec: + group: ceph.rook.io + names: + kind: CephNFS + listKind: CephNFSList + plural: cephnfses + singular: cephnfs + shortNames: + - nfs + scope: Namespaced + version: v1 + validation: + openAPIV3Schema: + properties: + spec: + properties: + rados: + properties: + pool: + type: string + namespace: + type: string + server: + properties: + active: + type: integer + annotations: {} + placement: {} + resources: {} + subresources: + status: {} + +--- +apiVersion: apiextensions.k8s.io/v1beta1 +kind: CustomResourceDefinition +metadata: + name: cephobjectstores.ceph.rook.io +spec: + group: ceph.rook.io + names: + kind: CephObjectStore + listKind: CephObjectStoreList + plural: cephobjectstores + singular: cephobjectstore + scope: Namespaced + version: v1 + validation: + openAPIV3Schema: + properties: + spec: + properties: + gateway: + properties: + type: + type: string + sslCertificateRef: {} + port: + type: integer + minimum: 0 + maximum: 65535 + securePort: + type: integer + minimum: 0 + maximum: 65535 + instances: + type: integer + externalRgwEndpoints: + type: array + items: + properties: + ip: + type: string + annotations: {} + placement: {} + resources: {} + metadataPool: + properties: + failureDomain: + type: string + crushRoot: + type: string + replicated: + properties: + size: + type: integer + requireSafeReplicaSize: + type: boolean + erasureCoded: + properties: + dataChunks: + type: integer + codingChunks: + type: integer + compressionMode: + type: string + enum: + - "" + - none + - passive + - aggressive + - force + parameters: + type: object + dataPool: + properties: + failureDomain: + type: string + crushRoot: + type: string + replicated: + properties: + size: + type: integer + requireSafeReplicaSize: + type: boolean + erasureCoded: + properties: + dataChunks: + type: integer + codingChunks: + type: integer + compressionMode: + type: string + enum: + - "" + - none + - passive + - aggressive + - force + parameters: + type: object + preservePoolsOnDelete: + type: boolean + healthCheck: + properties: + bucket: + properties: + enabled: + type: boolean + interval: + type: string + subresources: + status: {} + +--- +apiVersion: apiextensions.k8s.io/v1beta1 +kind: CustomResourceDefinition +metadata: + name: cephobjectstoreusers.ceph.rook.io +spec: + group: ceph.rook.io + names: + kind: CephObjectStoreUser + listKind: CephObjectStoreUserList + plural: cephobjectstoreusers + singular: cephobjectstoreuser + shortNames: + - rcou + - objectuser + scope: Namespaced + version: v1 + subresources: + status: {} + +--- +apiVersion: apiextensions.k8s.io/v1beta1 +kind: CustomResourceDefinition +metadata: + name: cephobjectrealms.ceph.rook.io +spec: + group: ceph.rook.io + names: + kind: CephObjectRealm + listKind: CephObjectRealmList + plural: cephobjectrealms + singular: cephobjectrealm + scope: Namespaced + version: v1 + subresources: + status: {} + +--- +apiVersion: apiextensions.k8s.io/v1beta1 +kind: CustomResourceDefinition +metadata: + name: cephobjectzonegroups.ceph.rook.io +spec: + group: ceph.rook.io + names: + kind: CephObjectZoneGroup + listKind: CephObjectZoneGroupList + plural: cephobjectzonegroups + singular: cephobjectzonegroup + scope: Namespaced + version: v1 + subresources: + status: {} + +--- +apiVersion: apiextensions.k8s.io/v1beta1 +kind: CustomResourceDefinition +metadata: + name: cephobjectzones.ceph.rook.io +spec: + group: ceph.rook.io + names: + kind: CephObjectZone + listKind: CephObjectZoneList + plural: cephobjectzones + singular: cephobjectzone + scope: Namespaced + version: v1 + subresources: + status: {} + +--- +apiVersion: apiextensions.k8s.io/v1beta1 +kind: CustomResourceDefinition +metadata: + name: cephblockpools.ceph.rook.io +spec: + group: ceph.rook.io + names: + kind: CephBlockPool + listKind: CephBlockPoolList + plural: cephblockpools + singular: cephblockpool + scope: Namespaced + version: v1 + validation: + openAPIV3Schema: + properties: + spec: + properties: + failureDomain: + type: string + crushRoot: + type: string + replicated: + properties: + size: + type: integer + minimum: 0 + maximum: 9 + targetSizeRatio: + type: number + requireSafeReplicaSize: + type: boolean + erasureCoded: + properties: + dataChunks: + type: integer + minimum: 0 + maximum: 9 + codingChunks: + type: integer + minimum: 0 + maximum: 9 + compressionMode: + type: string + enum: + - "" + - none + - passive + - aggressive + - force + enableRBDStats: + description: EnableRBDStats is used to enable gathering of statistics + for all RBD images in the pool + type: boolean + parameters: + type: object + mirroring: + properties: + enabled: + type: boolean + mode: + type: string + enum: + - image + - pool + subresources: + status: {} + +--- +apiVersion: apiextensions.k8s.io/v1beta1 +kind: CustomResourceDefinition +metadata: + name: volumes.rook.io +spec: + group: rook.io + names: + kind: Volume + listKind: VolumeList + plural: volumes + singular: volume + shortNames: + - rv + scope: Namespaced + version: v1alpha2 + subresources: + status: {} + +--- +apiVersion: apiextensions.k8s.io/v1beta1 +kind: CustomResourceDefinition +metadata: + name: objectbuckets.objectbucket.io +spec: + group: objectbucket.io + versions: + - name: v1alpha1 + served: true + storage: true + names: + kind: ObjectBucket + listKind: ObjectBucketList + plural: objectbuckets + singular: objectbucket + shortNames: + - ob + - obs + scope: Cluster + subresources: + status: {} + +--- +apiVersion: apiextensions.k8s.io/v1beta1 +kind: CustomResourceDefinition +metadata: + name: objectbucketclaims.objectbucket.io +spec: + versions: + - name: v1alpha1 + served: true + storage: true + group: objectbucket.io + names: + kind: ObjectBucketClaim + listKind: ObjectBucketClaimList + plural: objectbucketclaims + singular: objectbucketclaim + shortNames: + - obc + - obcs + scope: Namespaced + subresources: + status: {} diff --git a/design/common/object-bucket.md b/design/common/object-bucket.md index a323c388e..2d2c5521e 100644 --- a/design/common/object-bucket.md +++ b/design/common/object-bucket.md @@ -53,7 +53,6 @@ spec: port: 80 securePort: 443 instances: 3 - allNodes: false ``` Now create the object store. diff --git a/tests/framework/installer/ceph_installer.go b/tests/framework/installer/ceph_installer.go index 164138f11..b8afa1668 100644 --- a/tests/framework/installer/ceph_installer.go +++ b/tests/framework/installer/ceph_installer.go @@ -80,9 +80,14 @@ func (h *CephInstaller) CreateCephOperator(namespace string) (err error) { // creating clusterrolebinding for kubeadm env. h.k8shelper.CreateAnonSystemClusterBinding() + var v1ExtensionsSupported bool + minVersion := "v1.16.0" + if h.k8shelper.VersionAtLeast(minVersion) { + v1ExtensionsSupported = true + } // creating rook resources logger.Info("Creating Rook CRDs") - resources := h.Manifests.GetRookCRDs() + resources := h.Manifests.GetRookCRDs(v1ExtensionsSupported) if _, err = h.k8shelper.KubectlWithStdin(resources, createFromStdinArgs...); err != nil { return err } diff --git a/tests/framework/installer/ceph_manifests.go b/tests/framework/installer/ceph_manifests.go index c82144768..b45aaadf1 100644 --- a/tests/framework/installer/ceph_manifests.go +++ b/tests/framework/installer/ceph_manifests.go @@ -5,7 +5,7 @@ Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at - http://www.apache.org/licenses/LICENSE-2.0 + http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, @@ -33,7 +33,7 @@ const ( ) type CephManifests interface { - GetRookCRDs() string + GetRookCRDs(v1ExtensionsSupported bool) string GetRookOperator(namespace string) string GetClusterRoles(namespace, systemNamespace string) string GetClusterExternalRoles(namespace, systemNamespace string) string @@ -95,7 +95,1279 @@ func NewCephManifests(version string) CephManifests { panic(fmt.Errorf("unrecognized ceph manifest version: %s", version)) } -func (m *CephManifestsMaster) GetRookCRDs() string { +func (m *CephManifestsMaster) GetRookCRDs(v1ExtensionsSupported bool) string { + if v1ExtensionsSupported { + return ` +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + name: cephclusters.ceph.rook.io +spec: + group: ceph.rook.io + names: + kind: CephCluster + plural: cephclusters + singular: cephcluster + scope: Namespaced + versions: + - name: v1 + served: true + storage: true + schema: + openAPIV3Schema: + type: object + properties: + spec: + type: object + properties: + cephVersion: + type: object + properties: + allowUnsupported: + type: boolean + image: + type: string + dashboard: + type: object + properties: + enabled: + type: boolean + urlPrefix: + type: string + port: + type: integer + minimum: 0 + maximum: 65535 + ssl: + type: boolean + dataDirHostPath: + pattern: ^/(\S+) + type: string + disruptionManagement: + type: object + properties: + machineDisruptionBudgetNamespace: + type: string + managePodBudgets: + type: boolean + osdMaintenanceTimeout: + type: integer + manageMachineDisruptionBudgets: + type: boolean + skipUpgradeChecks: + type: boolean + continueUpgradeAfterChecksEvenIfNotHealthy: + type: boolean + mon: + type: object + properties: + allowMultiplePerNode: + type: boolean + count: + maximum: 9 + minimum: 0 + type: integer + volumeClaimTemplate: + type: object + x-kubernetes-preserve-unknown-fields: true + mgr: + type: object + properties: + modules: + type: array + items: + type: object + properties: + name: + type: string + enabled: + type: boolean + crashCollector: + type: object + properties: + disable: + type: boolean + network: + type: object + nullable: true + properties: + hostNetwork: + type: boolean + provider: + type: string + selectors: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + ipFamily: + type: string + storage: + type: object + properties: + disruptionManagement: + type: object + nullable: true + properties: + machineDisruptionBudgetNamespace: + type: string + managePodBudgets: + type: boolean + osdMaintenanceTimeout: + type: integer + manageMachineDisruptionBudgets: + type: boolean + useAllNodes: + type: boolean + nodes: + type: array + nullable: true + items: + type: object + properties: + name: + type: string + config: + type: object + nullable: true + properties: + metadataDevice: + type: string + storeType: + type: string + pattern: ^(bluestore)$ + databaseSizeMB: + type: string + walSizeMB: + type: string + journalSizeMB: + type: string + osdsPerDevice: + type: string + encryptedDevice: + type: string + pattern: ^(true|false)$ + useAllDevices: + type: boolean + deviceFilter: + type: string + nullable: true + devicePathFilter: + type: string + devices: + type: array + items: + type: object + properties: + name: + type: string + config: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + resources: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + useAllDevices: + type: boolean + deviceFilter: + type: string + nullable: true + devicePathFilter: + type: string + config: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + storageClassDeviceSets: + type: array + nullable: true + items: + type: object + properties: + name: + type: string + count: + type: integer + format: int32 + portable: + type: boolean + tuneDeviceClass: + type: boolean + tuneFastDeviceClass: + type: boolean + encrypted: + type: boolean + schedulerName: + type: string + config: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + placement: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + preparePlacement: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + resources: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + volumeClaimTemplates: + type: array + items: + type: object + x-kubernetes-preserve-unknown-fields: true + driveGroups: + type: array + nullable: true + items: + type: object + properties: + name: + type: string + spec: + type: object + x-kubernetes-preserve-unknown-fields: true + placement: + type: object + x-kubernetes-preserve-unknown-fields: true + required: + - name + - spec + monitoring: + type: object + properties: + enabled: + type: boolean + rulesNamespace: + type: string + externalMgrEndpoints: + type: array + items: + type: object + properties: + ip: + type: string + removeOSDsIfOutAndSafeToRemove: + type: boolean + external: + type: object + properties: + enable: + type: boolean + cleanupPolicy: + type: object + properties: + allowUninstallWithVolumes: + type: boolean + confirmation: + type: string + pattern: ^$|^yes-really-destroy-data$ + sanitizeDisks: + type: object + properties: + method: + type: string + pattern: ^(complete|quick)$ + dataSource: + type: string + pattern: ^(zero|random)$ + iteration: + type: integer + format: int32 + annotations: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + placement: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + labels: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + resources: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + healthCheck: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + priorityClassNames: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + status: + type: object + x-kubernetes-preserve-unknown-fields: true + additionalPrinterColumns: + - name: DataDirHostPath + type: string + description: Directory used on the K8s nodes + jsonPath: .spec.dataDirHostPath + - name: MonCount + type: string + description: Number of MONs + jsonPath: .spec.mon.count + - name: Age + type: date + jsonPath: .metadata.creationTimestamp + - name: Phase + type: string + description: Phase + jsonPath: .status.phase + - name: Message + type: string + description: Message + jsonPath: .status.message + - name: Health + type: string + description: Ceph Health + jsonPath: .status.ceph.health + subresources: + status: {} +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + name: cephclients.ceph.rook.io +spec: + group: ceph.rook.io + names: + kind: CephClient + listKind: CephClientList + plural: cephclients + singular: cephclient + scope: Namespaced + versions: + - name: v1 + served: true + storage: true + schema: + openAPIV3Schema: + type: object + properties: + spec: + type: object + properties: + caps: + type: object + x-kubernetes-preserve-unknown-fields: true + status: + type: object + x-kubernetes-preserve-unknown-fields: true + subresources: + status: {} +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + name: cephrbdmirrors.ceph.rook.io +spec: + group: ceph.rook.io + names: + kind: CephRBDMirror + listKind: CephRBDMirrorList + plural: cephrbdmirrors + singular: cephrbdmirror + scope: Namespaced + versions: + - name: v1 + served: true + storage: true + schema: + openAPIV3Schema: + type: object + properties: + spec: + type: object + properties: + count: + type: integer + minimum: 1 + maximum: 100 + peers: + type: object + properties: + secretNames: + type: array + items: + type: string + resources: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + priorityClassName: + type: string + placement: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + annotations: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + status: + type: object + x-kubernetes-preserve-unknown-fields: true + subresources: + status: {} +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + name: cephfilesystems.ceph.rook.io +spec: + group: ceph.rook.io + names: + kind: CephFilesystem + listKind: CephFilesystemList + plural: cephfilesystems + singular: cephfilesystem + scope: Namespaced + versions: + - name: v1 + served: true + storage: true + schema: + openAPIV3Schema: + type: object + properties: + spec: + type: object + properties: + metadataServer: + type: object + properties: + activeCount: + minimum: 1 + maximum: 10 + type: integer + activeStandby: + type: boolean + annotations: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + placement: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + resources: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + priorityClassName: + type: string + labels: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + metadataPool: + type: object + nullable: true + properties: + failureDomain: + type: string + crushRoot: + type: string + replicated: + type: object + nullable: true + properties: + size: + minimum: 0 + maximum: 10 + type: integer + requireSafeReplicaSize: + type: boolean + replicasPerFailureDomain: + type: integer + subFailureDomain: + type: string + parameters: + type: object + x-kubernetes-preserve-unknown-fields: true + erasureCoded: + type: object + nullable: true + properties: + dataChunks: + minimum: 0 + maximum: 10 + type: integer + codingChunks: + minimum: 0 + maximum: 10 + type: integer + compressionMode: + type: string + enum: + - "" + - none + - passive + - aggressive + - force + dataPools: + type: array + nullable: true + items: + type: object + properties: + failureDomain: + type: string + crushRoot: + type: string + replicated: + type: object + nullable: true + properties: + size: + minimum: 0 + maximum: 10 + type: integer + requireSafeReplicaSize: + type: boolean + replicasPerFailureDomain: + type: integer + subFailureDomain: + type: string + erasureCoded: + type: object + nullable: true + properties: + dataChunks: + minimum: 0 + maximum: 10 + type: integer + codingChunks: + minimum: 0 + maximum: 10 + type: integer + compressionMode: + type: string + enum: + - "" + - none + - passive + - aggressive + - force + parameters: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + preservePoolsOnDelete: + type: boolean + status: + type: object + x-kubernetes-preserve-unknown-fields: true + additionalPrinterColumns: + - name: ActiveMDS + type: string + description: Number of desired active MDS daemons + jsonPath: .spec.metadataServer.activeCount + - name: Age + type: date + jsonPath: .metadata.creationTimestamp + subresources: + status: {} +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + name: cephnfses.ceph.rook.io +spec: + group: ceph.rook.io + names: + kind: CephNFS + listKind: CephNFSList + plural: cephnfses + singular: cephnfs + shortNames: + - nfs + scope: Namespaced + versions: + - name: v1 + served: true + storage: true + schema: + openAPIV3Schema: + type: object + properties: + spec: + type: object + properties: + rados: + type: object + properties: + pool: + type: string + namespace: + type: string + server: + type: object + properties: + active: + type: integer + annotations: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + placement: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + resources: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + priorityClassName: + type: string + status: + type: object + x-kubernetes-preserve-unknown-fields: true + subresources: + status: {} +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + name: cephobjectstores.ceph.rook.io +spec: + group: ceph.rook.io + names: + kind: CephObjectStore + listKind: CephObjectStoreList + plural: cephobjectstores + singular: cephobjectstore + scope: Namespaced + versions: + - name: v1 + served: true + storage: true + schema: + openAPIV3Schema: + type: object + properties: + spec: + type: object + properties: + gateway: + type: object + properties: + type: + type: string + sslCertificateRef: + type: string + nullable: true + port: + type: integer + minimum: 0 + maximum: 65535 + securePort: + type: integer + minimum: 0 + maximum: 65535 + instances: + type: integer + externalRgwEndpoints: + type: array + nullable: true + items: + type: object + properties: + ip: + type: string + annotations: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + placement: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + resources: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + labels: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + priorityClassName: + type: string + metadataPool: + type: object + nullable: true + properties: + failureDomain: + type: string + crushRoot: + type: string + replicated: + type: object + nullable: true + properties: + size: + type: integer + requireSafeReplicaSize: + type: boolean + replicasPerFailureDomain: + type: integer + subFailureDomain: + type: string + erasureCoded: + type: object + nullable: true + properties: + dataChunks: + type: integer + codingChunks: + type: integer + compressionMode: + type: string + enum: + - "" + - none + - passive + - aggressive + - force + parameters: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + zone: + type: object + properties: + name: + type: string + dataPool: + type: object + nullable: true + properties: + failureDomain: + type: string + crushRoot: + type: string + replicated: + type: object + nullable: true + properties: + size: + type: integer + requireSafeReplicaSize: + type: boolean + replicasPerFailureDomain: + type: integer + subFailureDomain: + type: string + erasureCoded: + type: object + nullable: true + properties: + dataChunks: + type: integer + codingChunks: + type: integer + compressionMode: + type: string + enum: + - "" + - none + - passive + - aggressive + - force + parameters: + type: object + x-kubernetes-preserve-unknown-fields: true + preservePoolsOnDelete: + type: boolean + healthCheck: + type: object + nullable: true + properties: + bucket: + type: object + nullable: true + properties: + enabled: + type: boolean + interval: + type: string + status: + type: object + x-kubernetes-preserve-unknown-fields: true + subresources: + status: {} +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + name: cephobjectstoreusers.ceph.rook.io +spec: + group: ceph.rook.io + names: + kind: CephObjectStoreUser + listKind: CephObjectStoreUserList + plural: cephobjectstoreusers + singular: cephobjectstoreuser + shortNames: + - rcou + - objectuser + scope: Namespaced + versions: + - name: v1 + served: true + storage: true + schema: + openAPIV3Schema: + type: object + properties: + spec: + type: object + properties: + store: + type: string + displayName: + type: string + status: + type: object + x-kubernetes-preserve-unknown-fields: true + subresources: + status: {} +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + name: cephobjectrealms.ceph.rook.io +spec: + group: ceph.rook.io + names: + kind: CephObjectRealm + listKind: CephObjectRealmList + plural: cephobjectrealms + singular: cephobjectrealm + scope: Namespaced + versions: + - name: v1 + served: true + storage: true + schema: + openAPIV3Schema: + type: object + properties: + spec: + type: object + properties: + pull: + type: object + properties: + endpoint: + type: string + status: + type: object + x-kubernetes-preserve-unknown-fields: true + subresources: + status: {} +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + name: cephobjectzonegroups.ceph.rook.io +spec: + group: ceph.rook.io + names: + kind: CephObjectZoneGroup + listKind: CephObjectZoneGroupList + plural: cephobjectzonegroups + singular: cephobjectzonegroup + scope: Namespaced + versions: + - name: v1 + served: true + storage: true + schema: + openAPIV3Schema: + type: object + properties: + spec: + type: object + properties: + realm: + type: string + status: + type: object + x-kubernetes-preserve-unknown-fields: true + subresources: + status: {} +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + name: cephobjectzones.ceph.rook.io +spec: + group: ceph.rook.io + names: + kind: CephObjectZone + listKind: CephObjectZoneList + plural: cephobjectzones + singular: cephobjectzone + scope: Namespaced + versions: + - name: v1 + served: true + storage: true + schema: + openAPIV3Schema: + type: object + properties: + spec: + type: object + properties: + zoneGroup: + type: string + metadataPool: + type: object + nullable: true + properties: + failureDomain: + type: string + crushRoot: + type: string + replicated: + type: object + nullable: true + properties: + size: + type: integer + requireSafeReplicaSize: + type: boolean + erasureCoded: + type: object + nullable: true + properties: + dataChunks: + type: integer + codingChunks: + type: integer + compressionMode: + type: string + enum: + - "" + - none + - passive + - aggressive + - force + parameters: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + dataPool: + type: object + properties: + failureDomain: + type: string + crushRoot: + type: string + replicated: + type: object + nullable: true + properties: + size: + type: integer + requireSafeReplicaSize: + type: boolean + erasureCoded: + type: object + nullable: true + properties: + dataChunks: + type: integer + codingChunks: + type: integer + compressionMode: + type: string + enum: + - "" + - none + - passive + - aggressive + - force + parameters: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + preservePoolsOnDelete: + type: boolean + status: + type: object + x-kubernetes-preserve-unknown-fields: true + subresources: + status: {} +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + name: cephblockpools.ceph.rook.io +spec: + group: ceph.rook.io + names: + kind: CephBlockPool + listKind: CephBlockPoolList + plural: cephblockpools + singular: cephblockpool + scope: Namespaced + versions: + - name: v1 + served: true + storage: true + schema: + openAPIV3Schema: + type: object + properties: + spec: + type: object + properties: + failureDomain: + type: string + deviceClass: + type: string + crushRoot: + type: string + replicated: + type: object + nullable: true + properties: + size: + type: integer + minimum: 0 + maximum: 9 + targetSizeRatio: + type: number + requireSafeReplicaSize: + type: boolean + replicasPerFailureDomain: + type: integer + subFailureDomain: + type: string + erasureCoded: + type: object + nullable: true + properties: + dataChunks: + type: integer + minimum: 0 + maximum: 9 + codingChunks: + type: integer + minimum: 0 + maximum: 9 + compressionMode: + type: string + enum: + - "" + - none + - passive + - aggressive + - force + enableRBDStats: + description: + EnableRBDStats is used to enable gathering of statistics + for all RBD images in the pool + type: boolean + parameters: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + mirroring: + type: object + nullable: true + properties: + enabled: + type: boolean + mode: + type: string + enum: + - image + - pool + statusCheck: + type: object + x-kubernetes-preserve-unknown-fields: true + annotations: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + status: + type: object + x-kubernetes-preserve-unknown-fields: true + subresources: + status: {} +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + name: volumes.rook.io +spec: + group: rook.io + names: + kind: Volume + listKind: VolumeList + plural: volumes + singular: volume + shortNames: + - rv + scope: Namespaced + versions: + - name: v1alpha2 + served: true + storage: true + schema: + openAPIV3Schema: + type: object + properties: + attachments: + type: array + items: + type: object + properties: + node: + type: string + podNamespace: + type: string + podName: + type: string + clusterName: + type: string + mountDir: + type: string + readOnly: + type: boolean + status: + type: object + x-kubernetes-preserve-unknown-fields: true + subresources: + status: {} +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + name: objectbuckets.objectbucket.io +spec: + group: objectbucket.io + names: + kind: ObjectBucket + listKind: ObjectBucketList + plural: objectbuckets + singular: objectbucket + shortNames: + - ob + - obs + scope: Cluster + versions: + - name: v1alpha1 + served: true + storage: true + schema: + openAPIV3Schema: + type: object + properties: + spec: + type: object + properties: + storageClassName: + type: string + endpoint: + type: object + nullable: true + properties: + bucketHost: + type: string + bucketPort: + type: integer + format: int32 + bucketName: + type: string + region: + type: string + subRegion: + type: string + additionalConfig: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + authentication: + type: object + nullable: true + items: + type: object + x-kubernetes-preserve-unknown-fields: true + additionalState: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + reclaimPolicy: + type: string + claimRef: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + items: + type: object + x-kubernetes-preserve-unknown-fields: true + status: + type: object + x-kubernetes-preserve-unknown-fields: true + subresources: + status: {} +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + name: objectbucketclaims.objectbucket.io +spec: + group: objectbucket.io + names: + kind: ObjectBucketClaim + listKind: ObjectBucketClaimList + plural: objectbucketclaims + singular: objectbucketclaim + shortNames: + - obc + - obcs + scope: Namespaced + versions: + - name: v1alpha1 + served: true + storage: true + schema: + openAPIV3Schema: + type: object + properties: + spec: + type: object + properties: + storageClassName: + type: string + bucketName: + type: string + generateBucketName: + type: string + additionalConfig: + type: object + nullable: true + x-kubernetes-preserve-unknown-fields: true + status: + type: object + x-kubernetes-preserve-unknown-fields: true + subresources: + status: {}` + } return ` apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition @@ -2459,7 +3731,6 @@ spec: sslCertificateRef: port: ` + strconv.Itoa(port) + ` instances: ` + strconv.Itoa(replicaCount) + ` - allNodes: false healthCheck: bucket: disabled: false diff --git a/tests/framework/installer/ceph_manifests_v1.2.go b/tests/framework/installer/ceph_manifests_v1.2.go index 1ff97c42d..2850fcda1 100644 --- a/tests/framework/installer/ceph_manifests_v1.2.go +++ b/tests/framework/installer/ceph_manifests_v1.2.go @@ -26,7 +26,7 @@ type CephManifestsV1_2 struct { imageTag string } -func (m *CephManifestsV1_2) GetRookCRDs() string { +func (m *CephManifestsV1_2) GetRookCRDs(v1ExtensionsSupported bool) string { return ` apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition @@ -1874,7 +1874,6 @@ spec: sslCertificateRef: port: ` + strconv.Itoa(port) + ` instances: ` + strconv.Itoa(replicaCount) + ` - allNodes: false ` }