-
group
string
diff --git a/Documentation/Storage-Configuration/Block-Storage-RBD/nvme-of.md b/Documentation/Storage-Configuration/Block-Storage-RBD/nvme-of.md
index a11928cbf..f5e1822e1 100644
--- a/Documentation/Storage-Configuration/Block-Storage-RBD/nvme-of.md
+++ b/Documentation/Storage-Configuration/Block-Storage-RBD/nvme-of.md
@@ -28,59 +28,47 @@ This guide assumes a Rook cluster as explained in the [Quickstart Guide](../../G
- **Ceph Version**: Ceph v20 (Tentacle) or later
-## Step 1: Create a Ceph Block Pool
-
-Before creating the NVMe-oF gateway, you need to create a CephBlockPool that will be used by the gateway:
-
-```yaml
-apiVersion: ceph.rook.io/v1
-kind: CephBlockPool
-metadata:
- name: nvmeof
- namespace: rook-ceph
-spec:
- failureDomain: host
- replicated:
- size: 3
-```
-
-Create the pool:
-
-```console
-kubectl create -f deploy/examples/csi/nvmeof/nvmeof-pool.yaml
-```
-
-## Step 2: Create the NVMe-oF Gateway
+## Step 1: Create the NVMe-oF Gateway
The `CephNVMeOFGateway` CRD manages the NVMe-oF gateway infrastructure. The operator will automatically create the following resources:
- **Service**: One per gateway instance for service discovery
- **Deployment**: One per gateway instance running the NVMe-oF gateway daemon
-Create the gateway:
+The gateway also requires an internal `.nvmeof` pool for its state. This pool is defined as a
+CephBlockPool CR with `spec.name: .nvmeof`.
+
+Create the gateway and the `.nvmeof` pool:
```yaml
apiVersion: ceph.rook.io/v1
+kind: CephBlockPool
+metadata:
+ name: builtin-nvmeof
+ namespace: rook-ceph
+spec:
+ name: .nvmeof
+ failureDomain: host
+ replicated:
+ size: 3
+---
+apiVersion: ceph.rook.io/v1
kind: CephNVMeOFGateway
metadata:
name: nvmeof
namespace: rook-ceph
spec:
- # Container image for the NVMe-oF gateway daemon
- image: quay.io/ceph/nvmeof:1.5
- # Pool name that will be used by the NVMe-oF gateway
- pool: nvmeof
# ANA (Asymmetric Namespace Access) group name
group: group-a
- # Number of gateway instances to run
- instances: 1
+ # Number of gateway instances to run (2 recommended for HA)
+ instances: 2
hostNetwork: false
```
Apply the gateway configuration:
```console
-kubectl create -f deploy/examples/nvmeof-test.yaml
+kubectl create -f deploy/examples/nvmeof.yaml
```
Verify the gateway is running:
@@ -96,7 +84,7 @@ NAME READY STATUS RESTARTS AGE
rook-ceph-nvmeof-nvmeof-a-85844ff6b8-4r8gj 1/1 Running 0 91s
```
-## Step 3: Deploy the NVMe-oF CSI Driver via CSI Operator
+## Step 2: Deploy the NVMe-oF CSI Driver via CSI Operator
The NVMe-oF CSI driver is deployed via the ceph-csi operator.
@@ -120,6 +108,26 @@ rook-ceph.nvmeof.csi.ceph.com-ctrlplugin-d9d77fb7c-kkl28 5/5 Running 0
rook-ceph.nvmeof.csi.ceph.com-nodeplugin-xvt5g 2/2 Running 0 60s
```
+## Step 3: Create the Data Pool
+
+Create a CephBlockPool that will be used by the StorageClass to store data:
+
+```yaml
+apiVersion: ceph.rook.io/v1
+kind: CephBlockPool
+metadata:
+ name: nvmeof
+ namespace: rook-ceph
+spec:
+ failureDomain: host
+ replicated:
+ size: 3
+```
+
+```console
+kubectl create -f deploy/examples/csi/nvmeof/nvmeof-pool.yaml
+```
+
## Step 4: Create the StorageClass
Create a StorageClass that uses the NVMe-oF CSI driver.
@@ -292,22 +300,9 @@ sudo mount /dev/nvmeXnY /mnt/nvmeof
## High Availability
-For production deployments, configure multiple gateway instances for high availability:
-
-1. **Increase Gateway Instances**: Set `instances: 2` or higher in the `CephNVMeOFGateway` spec
-2. **Update StorageClass Listeners**: Add all gateway deployment hostnames to the `listeners` array
-3. **Load Balancing**: Each gateway instance has its own Service; list all of them to support multipath/HA
-
-Example with multiple instances:
-
-```yaml
-spec:
- instances: 2
- # ... other settings
-```
-
-Then update the StorageClass `listeners` to include all gateway
-hostnames:
+The example (`nvmeof.yaml`) configures `instances: 2` for high availability.
+When running multiple gateway instances, update the StorageClass `listeners` to include all gateway
+deployment hostnames for multipath/HA:
```yaml
listeners: |
@@ -375,10 +370,10 @@ kubectl delete storageclass ceph-nvmeof
# Delete the NVMe-oF CSI operator resources
kubectl delete -f deploy/examples/csi/nvmeof/csi-operator-nvmeof.yaml
-# Delete the NVMe-oF gateway
-kubectl delete -f deploy/examples/nvmeof-test.yaml
+# Delete the NVMe-oF gateway and its metadata pool
+kubectl delete -f deploy/examples/nvmeof.yaml
-# Delete the block pool (optional)
+# Delete the data pool (optional)
kubectl delete -f deploy/examples/csi/nvmeof/nvmeof-pool.yaml
```
diff --git a/deploy/charts/rook-ceph/templates/resources.yaml b/deploy/charts/rook-ceph/templates/resources.yaml
index be9a46a0f..54a3d4512 100644
--- a/deploy/charts/rook-ceph/templates/resources.yaml
+++ b/deploy/charts/rook-ceph/templates/resources.yaml
@@ -543,6 +543,7 @@ spec:
- .rgw.root
- .nfs
- .mgr
+ - .nvmeof
type: string
parameters:
additionalProperties:
@@ -12402,10 +12403,6 @@ spec:
type: object
type: array
type: object
- pool:
- description: Pool is the RADOS pool where NVMe-oF configuration is stored
- minLength: 1
- type: string
ports:
description: Ports configuration for the NVMe-oF gateway
properties:
@@ -12499,7 +12496,6 @@ spec:
required:
- group
- instances
- - pool
type: object
status:
description: NVMeOFGatewayStatus represents the status of Ceph NVMe-oF Gateway
diff --git a/deploy/examples/crds.yaml b/deploy/examples/crds.yaml
index f4e4ace32..340c71a7b 100644
--- a/deploy/examples/crds.yaml
+++ b/deploy/examples/crds.yaml
@@ -545,6 +545,7 @@ spec:
- .rgw.root
- .nfs
- .mgr
+ - .nvmeof
type: string
parameters:
additionalProperties:
@@ -12394,10 +12395,6 @@ spec:
type: object
type: array
type: object
- pool:
- description: Pool is the RADOS pool where NVMe-oF configuration is stored
- minLength: 1
- type: string
ports:
description: Ports configuration for the NVMe-oF gateway
properties:
@@ -12491,7 +12488,6 @@ spec:
required:
- group
- instances
- - pool
type: object
status:
description: NVMeOFGatewayStatus represents the status of Ceph NVMe-oF Gateway
diff --git a/deploy/examples/nvmeof-test.yaml b/deploy/examples/nvmeof-test.yaml
index 98a332fca..61399eee5 100644
--- a/deploy/examples/nvmeof-test.yaml
+++ b/deploy/examples/nvmeof-test.yaml
@@ -1,5 +1,19 @@
# This example is for Ceph v20 and above only
+# This is a test configuration for CI environments only.
+# For production deployments, use nvmeof.yaml instead.
+apiVersion: ceph.rook.io/v1
+kind: CephBlockPool
+metadata:
+ name: builtin-nvmeof
+ namespace: rook-ceph # namespace:cluster
+spec:
+ name: .nvmeof
+ failureDomain: osd
+ replicated:
+ size: 1
+ requireSafeReplicaSize: false
+---
apiVersion: ceph.rook.io/v1
kind: CephNVMeOFGateway
metadata:
@@ -10,7 +24,6 @@ spec:
# (ceph config get mgr mgr/cephadm/container_image_nvmeof).
# Set the image property to override the default:
# image: quay.io/ceph/nvmeof:
- pool: nvmeof
group: group-a
instances: 1
hostNetwork: false
diff --git a/deploy/examples/nvmeof.yaml b/deploy/examples/nvmeof.yaml
new file mode 100644
index 000000000..cdaf2e11d
--- /dev/null
+++ b/deploy/examples/nvmeof.yaml
@@ -0,0 +1,26 @@
+# This example is for Ceph v20 and above only
+
+apiVersion: ceph.rook.io/v1
+kind: CephBlockPool
+metadata:
+ name: builtin-nvmeof
+ namespace: rook-ceph # namespace:cluster
+spec:
+ name: .nvmeof
+ failureDomain: host
+ replicated:
+ size: 3
+---
+apiVersion: ceph.rook.io/v1
+kind: CephNVMeOFGateway
+metadata:
+ name: nvmeof
+ namespace: rook-ceph # namespace:cluster
+spec:
+ # image is optional. If not specified, the image is fetched from the Ceph config store
+ # (ceph config get mgr mgr/cephadm/container_image_nvmeof).
+ # Set the image property to override the default:
+ # image: quay.io/ceph/nvmeof:
+ group: group-a
+ instances: 2
+ hostNetwork: false
diff --git a/pkg/apis/ceph.rook.io/v1/types.go b/pkg/apis/ceph.rook.io/v1/types.go
index f30ad0a57..f105d90f0 100755
--- a/pkg/apis/ceph.rook.io/v1/types.go
+++ b/pkg/apis/ceph.rook.io/v1/types.go
@@ -1111,7 +1111,7 @@ type PoolSpec struct {
// allowed pool names that can be specified.
type NamedBlockPoolSpec struct {
// The desired name of the pool if different from the CephBlockPool CR name.
- // +kubebuilder:validation:Enum=.rgw.root;.nfs;.mgr
+ // +kubebuilder:validation:Enum=.rgw.root;.nfs;.mgr;.nvmeof
// +optional
Name string `json:"name,omitempty"`
// The core pool configuration
@@ -4238,10 +4238,6 @@ type NVMeOFGatewaySpec struct {
// +kubebuilder:validation:Minimum=1
Instances int `json:"instances"`
- // Pool is the RADOS pool where NVMe-oF configuration is stored
- // +kubebuilder:validation:MinLength=1
- Pool string `json:"pool"`
-
// Group is the gateway group name for high availability (ANA group)
// +kubebuilder:validation:MinLength=1
Group string `json:"group"`
diff --git a/pkg/operator/ceph/nvmeof/controller.go b/pkg/operator/ceph/nvmeof/controller.go
index 3368bb0e3..5599a365f 100644
--- a/pkg/operator/ceph/nvmeof/controller.go
+++ b/pkg/operator/ceph/nvmeof/controller.go
@@ -55,6 +55,10 @@ import (
const (
packageName = "ceph-nvmeof-gateway"
controllerName = packageName + "-controller"
+
+ // nvmeofPoolName is the required Ceph pool for NVMe-oF gateway state.
+ // This matches the pool that Ceph NVMe-oF creates automatically.
+ nvmeofPoolName = ".nvmeof"
)
var logger = capnslog.NewPackageLogger("github.com/rook/rook", packageName)
@@ -473,14 +477,13 @@ func getNVMeOFGatewayConfig(poolName, podName, podIP, anaGroup string, userConfi
}
func (r *ReconcileCephNVMeOFGateway) generateConfigMap(nvmeof *cephv1.CephNVMeOFGateway, daemonID string) (*v1.ConfigMap, error) {
- poolName := nvmeof.Spec.Pool
anaGroup := nvmeof.Spec.Group
podName := instanceName(nvmeof, daemonID)
// Use placeholder that will be replaced at runtime with actual pod IP
// The init container will replace @@POD_IP@@ with the actual pod IP
podIP := "@@POD_IP@@"
- configContent, err := getNVMeOFGatewayConfig(poolName, podName, podIP, anaGroup, nvmeof.Spec.NVMeOFConfig)
+ configContent, err := getNVMeOFGatewayConfig(nvmeofPoolName, podName, podIP, anaGroup, nvmeof.Spec.NVMeOFConfig)
if err != nil {
return nil, errors.Wrap(err, "failed to generate nvmeof config")
}
@@ -540,10 +543,6 @@ func validateGateway(g *cephv1.CephNVMeOFGateway) error {
return errors.New("gateway group name is required")
}
- if g.Spec.Pool == "" {
- return errors.New("pool name is required")
- }
-
return nil
}
diff --git a/pkg/operator/ceph/nvmeof/controller_test.go b/pkg/operator/ceph/nvmeof/controller_test.go
index 281fcdbf3..4b17e7d22 100644
--- a/pkg/operator/ceph/nvmeof/controller_test.go
+++ b/pkg/operator/ceph/nvmeof/controller_test.go
@@ -119,7 +119,6 @@ func TestCephNVMeOFGatewayController(t *testing.T) {
Finalizers: []string{"cephnvmeofgateway.ceph.rook.io"},
},
Spec: cephv1.NVMeOFGatewaySpec{
- Pool: "nvmeof",
Group: "group-a",
Instances: 1,
},
@@ -461,7 +460,6 @@ func TestNVMeOFKeyRotation(t *testing.T) {
Namespace: namespace,
},
Spec: cephv1.NVMeOFGatewaySpec{
- Pool: "nvmeof",
Group: "group-a",
Instances: 1,
},
@@ -696,7 +694,6 @@ func TestNVMeOFConfigMapGeneration(t *testing.T) {
Namespace: "rook-ceph",
},
Spec: cephv1.NVMeOFGatewaySpec{
- Pool: "pool-a",
Group: "group-a",
Instances: 1,
NVMeOFConfig: map[string]map[string]string{
@@ -721,5 +718,5 @@ func TestNVMeOFConfigMapGeneration(t *testing.T) {
assert.Equal(t, "@@POD_IP@@", cfg.Section("gateway").Key("addr").String())
assert.Equal(t, "group-a", cfg.Section("gateway").Key("group").String())
assert.Equal(t, "5511", cfg.Section("gateway").Key("port").String())
- assert.Equal(t, "pool-a", cfg.Section("ceph").Key("pool").String())
+ assert.Equal(t, nvmeofPoolName, cfg.Section("ceph").Key("pool").String())
}
diff --git a/pkg/operator/ceph/nvmeof/spec.go b/pkg/operator/ceph/nvmeof/spec.go
index 5f73904b0..47119931a 100644
--- a/pkg/operator/ceph/nvmeof/spec.go
+++ b/pkg/operator/ceph/nvmeof/spec.go
@@ -299,7 +299,6 @@ func (r *ReconcileCephNVMeOFGateway) createCephConfigInitContainer(nvmeof *cephv
script := connectionConfigScript
gatewayName := instanceName(nvmeof, daemonID)
- poolName := nvmeof.Spec.Pool
anaGroup := nvmeof.Spec.Group
// Build environment variables using Rook's helper functions
@@ -312,7 +311,7 @@ func (r *ReconcileCephNVMeOFGateway) createCephConfigInitContainer(nvmeof *cephv
},
v1.EnvVar{
Name: "POOL_NAME",
- Value: poolName,
+ Value: nvmeofPoolName,
},
v1.EnvVar{
Name: "ANA_GROUP",
diff --git a/pkg/operator/ceph/nvmeof/spec_test.go b/pkg/operator/ceph/nvmeof/spec_test.go
index 7ebb06c09..e3c749a5b 100644
--- a/pkg/operator/ceph/nvmeof/spec_test.go
+++ b/pkg/operator/ceph/nvmeof/spec_test.go
@@ -43,8 +43,16 @@ import (
func newDeploymentSpecTest(t *testing.T) (*ReconcileCephNVMeOFGateway, string) {
clientset := optest.New(t, 1)
+ executor := &exectest.MockExecutor{
+ MockExecuteCommandWithTimeout: func(timeout time.Duration, command string, arg ...string) (string, error) {
+ if command == "ceph" && arg[0] == "config" && arg[1] == "get" {
+ return "quay.io/ceph/nvmeof:1.5", nil
+ }
+ return "", nil
+ },
+ }
c := &clusterd.Context{
- Executor: &exectest.MockExecutor{},
+ Executor: executor,
RookClientset: rookclient.NewSimpleClientset(),
Clientset: clientset,
}
@@ -59,7 +67,6 @@ func newDeploymentSpecTest(t *testing.T) (*ReconcileCephNVMeOFGateway, string) {
TypeMeta: controllerTypeMeta,
Spec: cephv1.NVMeOFGatewaySpec{
Image: "quay.io/ceph/nvmeof:1.5",
- Pool: "nvmeof",
Group: "group-a",
Instances: 1,
},
@@ -97,7 +104,6 @@ func TestDeploymentSpec(t *testing.T) {
},
Spec: cephv1.NVMeOFGatewaySpec{
Image: "quay.io/ceph/nvmeof:1.5",
- Pool: "nvmeof",
Group: "group-a",
Instances: 3,
Resources: v1.ResourceRequirements{
@@ -149,8 +155,6 @@ func TestDeploymentSpec(t *testing.T) {
Namespace: "rook-ceph-test-ns",
},
Spec: cephv1.NVMeOFGatewaySpec{
- Image: "quay.io/ceph/nvmeof:1.5",
- Pool: "nvmeof",
Group: "group-a",
Instances: 3,
Resources: v1.ResourceRequirements{
@@ -198,7 +202,6 @@ func TestDeploymentSpec(t *testing.T) {
},
Spec: cephv1.NVMeOFGatewaySpec{
Image: "quay.io/ceph/nvmeof:1.5",
- Pool: "nvmeof",
Group: "group-a",
Instances: 3,
},
@@ -228,7 +231,6 @@ func TestDeploymentSpec(t *testing.T) {
},
Spec: cephv1.NVMeOFGatewaySpec{
Image: "quay.io/ceph/nvmeof:1.5",
- Pool: "nvmeof",
Group: "group-a",
Instances: 3,
Placement: cephv1.Placement{
@@ -271,7 +273,6 @@ func TestDeploymentSpec(t *testing.T) {
assert.Empty(t, d.Spec.Template.Spec.Affinity.PodAntiAffinity.PreferredDuringSchedulingIgnoredDuringExecution)
assert.Equal(t, "topology.kubernetes.io/zone", d.Spec.Template.Spec.Affinity.PodAntiAffinity.RequiredDuringSchedulingIgnoredDuringExecution[0].TopologyKey)
assert.Equal(t, map[string]string{"app": "custom-app"}, d.Spec.Template.Spec.Affinity.PodAntiAffinity.RequiredDuringSchedulingIgnoredDuringExecution[0].LabelSelector.MatchLabels)
-
assert.Len(t, d.Spec.Template.Spec.TopologySpreadConstraints, 1)
assert.Equal(t, int32(2), d.Spec.Template.Spec.TopologySpreadConstraints[0].MaxSkew)
assert.Equal(t, "topology.kubernetes.io/zone", d.Spec.Template.Spec.TopologySpreadConstraints[0].TopologyKey)
@@ -287,7 +288,6 @@ func TestDeploymentSpec(t *testing.T) {
},
Spec: cephv1.NVMeOFGatewaySpec{
Image: "quay.io/ceph/nvmeof:1.5",
- Pool: "nvmeof",
Group: "group-a",
Instances: 1,
},
@@ -316,7 +316,6 @@ func TestDeploymentSpec(t *testing.T) {
},
Spec: cephv1.NVMeOFGatewaySpec{
Image: "quay.io/ceph/nvmeof:1.5",
- Pool: "nvmeof",
Group: "group-a",
Instances: 1,
ConfigMapRef: "custom-configmap",
@@ -350,7 +349,6 @@ func TestDeploymentSpec(t *testing.T) {
},
Spec: cephv1.NVMeOFGatewaySpec{
Image: "quay.io/ceph/nvmeof:1.5",
- Pool: "nvmeof",
Group: "group-a",
Instances: 1,
},
@@ -370,7 +368,7 @@ func TestDeploymentSpec(t *testing.T) {
assert.Contains(t, initCont.Args, "--keyring=/etc/ceph/keyring")
assertEnvVar(t, initCont.Env, "GATEWAY_NAME", instanceName(nvmeof, "0"))
- assertEnvVar(t, initCont.Env, "POOL_NAME", nvmeof.Spec.Pool)
+ assertEnvVar(t, initCont.Env, "POOL_NAME", nvmeofPoolName)
assertEnvVar(t, initCont.Env, "ANA_GROUP", nvmeof.Spec.Group)
assertEnvVarPresent(t, initCont.Env, "POD_IP")
@@ -391,7 +389,6 @@ func TestDeploymentSpec(t *testing.T) {
},
Spec: cephv1.NVMeOFGatewaySpec{
Image: "quay.io/ceph/nvmeof:1.5",
- Pool: "nvmeof",
Group: "group-a",
Instances: 1,
},
@@ -433,19 +430,12 @@ func TestDeploymentSpec(t *testing.T) {
Namespace: "rook-ceph-test-ns",
},
Spec: cephv1.NVMeOFGatewaySpec{
- Pool: "nvmeof",
Group: "group-a",
Instances: 1,
},
}
r, configHash := newDeploymentSpecTest(t)
- executor := &exectest.MockExecutor{
- MockExecuteCommandWithTimeout: func(timeout time.Duration, command string, arg ...string) (string, error) {
- return "quay.io/ceph/nvmeof:1.5", nil
- },
- }
- r.context.Executor = executor
configMapName := fmt.Sprintf("rook-ceph-nvmeof-%s-config", nvmeof.Name)
d, err := r.makeDeployment(nvmeof, "0", configMapName, configHash)
@@ -464,7 +454,6 @@ func TestDeploymentSpec(t *testing.T) {
Namespace: "rook-ceph-test-ns",
},
Spec: cephv1.NVMeOFGatewaySpec{
- Pool: "nvmeof",
Group: "group-a",
Instances: 1,
},
@@ -492,7 +481,6 @@ func TestDeploymentSpec(t *testing.T) {
},
Spec: cephv1.NVMeOFGatewaySpec{
Image: "quay.io/ceph/nvmeof:1.5",
- Pool: "nvmeof",
Group: "group-a",
Instances: 1,
},
@@ -513,7 +501,6 @@ func TestDeploymentSpec(t *testing.T) {
},
Spec: cephv1.NVMeOFGatewaySpec{
Image: "quay.io/ceph/nvmeof:1.5",
- Pool: "nvmeof",
Group: "group-a",
Instances: 1,
},
@@ -533,8 +520,6 @@ func TestDeploymentSpec(t *testing.T) {
Namespace: "rook-ceph-test-ns",
},
Spec: cephv1.NVMeOFGatewaySpec{
- Image: "quay.io/ceph/nvmeof:1.5",
- Pool: "nvmeof",
Group: "group-a",
Instances: 1,
},
@@ -582,8 +567,6 @@ func TestDeploymentSpec(t *testing.T) {
Namespace: "rook-ceph-test-ns",
},
Spec: cephv1.NVMeOFGatewaySpec{
- Image: "quay.io/ceph/nvmeof:1.5",
- Pool: "nvmeof",
Group: "group-a",
Instances: 1,
Ports: &cephv1.NVMeOFGatewayPorts{
diff --git a/tests/scripts/github-action-helper.sh b/tests/scripts/github-action-helper.sh
index 989ffd659..a5fa16541 100755
--- a/tests/scripts/github-action-helper.sh
+++ b/tests/scripts/github-action-helper.sh
@@ -1056,6 +1056,7 @@ function test_csi_nvmeof_workload {
wait_for cephblockpool nvmeof rook-ceph 600
kubectl create -f "${REPO_DIR}/deploy/examples/nvmeof-test.yaml"
+ wait_for cephblockpool builtin-nvmeof rook-ceph 600
wait_for cephnvmeofgateway nvmeof rook-ceph 600
timeout 300 bash </dev/null | awk '/rook-ceph-nvmeof-nvmeof-a-/ && \$3 == "Running" {f=1} END {exit !f}'; do
|