object: create an object store based on shared pools

Until now, an object store would create all the necessary
metadata pools and the data pool that were exclusively
for its own object store. When isolation between object
stores is necessary, this would cause many pools and
PGs to be created in the cluster, which was not
manageable.

Now one set of pools can be created to be shared
by any number of object stores. The metadata and data
between each object store is isolated by
RADOS namespaces, which by design will keep the
data safe for multi-tenancy.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
This commit is contained in:
Travis Nielsen committed 2024-03-11 11:20:57 -06:00
1 parent b35b27e1d5
commit fdacfd51c5
20 files changed
+1233 -92

No files matched your search

@@ -10,17 +10,21 @@ This guide assumes a Rook cluster as explained in the [Quickstart](../../Getting
## Configure an Object Store
Rook has the ability to either deploy an object store in Kubernetes or to connect to an external RGW service.
Most commonly, the object store will be configured locally by Rook.
Alternatively, if you have an existing Ceph cluster with Rados Gateways, see the
[external section](#connect-to-an-external-object-store) to consume it from Rook.
Rook can configure the Ceph Object Store for several different scenarios. See each linked section for the configuration details.
1. Create a [local object store](#create-a-local-object-store) with dedicated Ceph pools. This option is recommended if a single object store is required, and is the simplest to get started.
2. Create [one or more object stores with shared Ceph pools](#create-local-object-stores-with-shared-pools). This option is recommended when multiple object stores are required.
3. Connect to an [RGW service in an external Ceph cluster](#connect-to-an-external-object-store), rather than create a local object store.
4. Configure [RGW Multisite](#object-multisite) to synchronize buckets between object stores in different clusters.
!!! note
Updating the configuration of an object store between these types is not supported.
### Create a Local Object Store
The below sample will create a `CephObjectStore` that starts the RGW service in the cluster with an S3 API.
!!! note
This sample requires *at least 3 bluestore OSDs*, with each OSD located on a *different node*.
This sample requires *at least 3 OSDs*, with each OSD located on a *different node*.
The OSDs must be located on different nodes, because the [`failureDomain`](../../CRDs/Block-Storage/ceph-block-pool-crd.md#spec) is set to `host` and the `erasureCoded` chunk settings require at least 3 different OSDs (2 `dataChunks` + 1 `codingChunks`).
@@ -39,6 +43,7 @@ spec:
size: 3
dataPool:
failureDomain: host
# For production it is recommended to use more chunks, such as 4+2 or 8+4
erasureCoded:
dataChunks: 2
codingChunks: 1
@@ -64,6 +69,120 @@ To confirm the object store is configured, wait for the RGW pod(s) to start:
kubectl -n rook-ceph get pod -l app=rook-ceph-rgw
```
To consume the object store, continue below in the section to [Create a bucket](#create-a-bucket).
### Create Local Object Store(s) with Shared Pools
The below sample will create one or more object stores. Shared Ceph pools will be created, which reduces the overhead of additional Ceph pools for each additional object store.
Data isolation is enforced between the object stores with the use of Ceph RADOS namespaces. The separate RADOS namespaces do not allow access of the data across object stores.
!!! note
This sample requires *at least 3 OSDs*, with each OSD located on a *different node*.
The OSDs must be located on different nodes, because the [`failureDomain`](../../CRDs/Block-Storage/ceph-block-pool-crd.md#spec) is set to `host` and the `erasureCoded` chunk settings require at least 3 different OSDs (2 `dataChunks` + 1 `codingChunks`).
#### Shared Pools
Create the shared pools that will be used by each of the object stores.
!!! note
If object stores have been previously created, the first pool below (`.rgw.root`)
does not need to be defined again as it would have already been created
with an existing object store. There is only one `.rgw.root` pool existing
to store metadata for all object stores.
```yaml
apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
name: rgw-root
namespace: rook-ceph # namespace:cluster
spec:
name: .rgw.root
failureDomain: host
replicated:
size: 3
requireSafeReplicaSize: false
parameters:
pg_num: "8"
application: rgw
---
apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
name: rgw-meta-pool
namespace: rook-ceph # namespace:cluster
spec:
failureDomain: host
replicated:
size: 3
requireSafeReplicaSize: false
parameters:
pg_num: "8"
application: rgw
---
apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
name: rgw-data-pool
namespace: rook-ceph # namespace:cluster
spec:
failureDomain: osd
erasureCoded:
# For production it is recommended to use more chunks, such as 4+2 or 8+4
dataChunks: 2
codingChunks: 1
application: rgw
```
Create the shared pools:
```console
kubectl create -f object-shared-pools.yaml
```
#### Create Each Object Store
After the pools have been created above, create each object store to consume the shared pools.
```yaml
apiVersion: ceph.rook.io/v1
kind: CephObjectStore
metadata:
name: store-a
namespace: rook-ceph # namespace:cluster
spec:
sharedPools:
metadataPoolName: rgw-meta-pool
dataPoolName: rgw-data-pool
preserveRadosNamespaceDataOnDelete: true
gateway:
sslCertificateRef:
port: 80
instances: 1
```
Create the object store:
```console
kubectl create -f object-a.yaml
```
To confirm the object store is configured, wait for the RGW pod(s) to start:
```console
kubectl -n rook-ceph get pod -l rgw=store-a
```
Additional object stores can be created based on the same shared pools by simply changing the
`name` of the CephObjectStore. In the example manifests folder, two object store examples are
provided: `object-a.yaml` and `object-b.yaml`.
To consume the object store, continue below in the section to [Create a bucket](#create-a-bucket).
Modify the default example object store name from `my-store` to the alternate name of the object store
such as `store-a` in this example.
### Connect to an External Object Store
Rook can connect to existing RGW gateways to work in conjunction with the external mode of the `CephCluster` CRD. First, create a `rgw-admin-ops-user` user in the Ceph cluster with the necessary caps: