feedc0de/my-rook-config
1
0 Fork 0
forked from rook/rook
Code Pull Requests Activity

docs: use mkdocs and restructure docs

Browse Source 
Signed-off-by: Alexander Trost <galexrt@googlemail.com>
This commit is contained in:
Alexander Trost committed 2022-05-18 16:06:22 +02:00
1 parent bb5812316d
commit 74431443e3
93 files changed
+2232 -2230
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

Documentation/Storage-Configuration/Object-Storage-RGW/.pages
+4
View File
@@ -0,0 +1,4 @@
title: Object Storage (RGW)
nav:
- object-storage.md
- ...
Documentation/Storage-Configuration/Object-Storage-RGW/ceph-object-bucket-claim.md
+124
View File
@@ -0,0 +1,124 @@
---
title: Bucket Claim
---
Rook supports the creation of new buckets and access to existing buckets via two custom resources:
* an `Object Bucket Claim (OBC)` is custom resource which requests a bucket (new or existing) and is described by a Custom Resource Definition (CRD) shown below.
* an `Object Bucket (OB)` is a custom resource automatically generated when a bucket is provisioned. It is a global resource, typically not visible to non-admin users, and contains information specific to the bucket. It is described by an OB CRD, also shown below.
An OBC references a storage class which is created by an administrator. The storage class defines whether the bucket requested is a new bucket or an existing bucket. It also defines the bucket retention policy.
Users request a new or existing bucket by creating an OBC which is shown below. The ceph provisioner detects the OBC and creates a new bucket or grants access to an existing bucket, depending the the storage class referenced in the OBC. It also generates a Secret which provides credentials to access the bucket, and a ConfigMap which contains the bucket's endpoint. Application pods consume the information in the Secret and ConfigMap to access the bucket. Please note that to make provisioner watch the cluster namespace only you need to set `ROOK_OBC_WATCH_OPERATOR_NAMESPACE` to `true` in the operator manifest, otherwise it watches all namespaces.
## Example
### OBC Custom Resource
```yaml
apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
name: ceph-bucket [1]
namespace: rook-ceph [2]
spec:
bucketName: [3]
generateBucketName: photo-booth [4]
storageClassName: rook-ceph-bucket [4]
additionalConfig: [5]
maxObjects: "1000"
maxSize: "2G"
```
1. `name` of the `ObjectBucketClaim`. This name becomes the name of the Secret and ConfigMap.
1. `namespace`(optional) of the `ObjectBucketClaim`, which is also the namespace of the ConfigMap and Secret.
1. `bucketName` name of the `bucket`.
**Not** recommended for new buckets since names must be unique within
an entire object store.
1. `generateBucketName` value becomes the prefix for a randomly generated name, if supplied then `bucketName` must be empty.
If both `bucketName` and `generateBucketName` are supplied then `BucketName` has precedence and `GenerateBucketName` is ignored.
If both `bucketName` and `generateBucketName` are blank or omitted then the storage class is expected to contain the name of an _existing_ bucket. It's an error if all three bucket related names are blank or omitted.
1. `storageClassName` which defines the StorageClass which contains the names of the bucket provisioner, the object-store and specifies the bucket retention policy.
1. `additionalConfig` is an optional list of key-value pairs used to define attributes specific to the bucket being provisioned by this OBC. This information is typically tuned to a particular bucket provisioner and may limit application portability. Options supported:
* `maxObjects`: The maximum number of objects in the bucket
* `maxSize`: The maximum size of the bucket, please note minimum recommended value is 4K.
### OBC Custom Resource after Bucket Provisioning
```yaml
apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
creationTimestamp: "2019-10-18T09:54:01Z"
generation: 2
name: ceph-bucket
namespace: default [1]
resourceVersion: "559491"
spec:
ObjectBucketName: obc-default-ceph-bucket [2]
additionalConfig: null
bucketName: photo-booth-c1178d61-1517-431f-8408-ec4c9fa50bee [3]
storageClassName: rook-ceph-bucket [4]
status:
phase: Bound [5]
```
1. `namespace` where OBC got created.
1. `ObjectBucketName` generated OB name created using name space and OBC name.
1. the generated (in this case), unique `bucket name` for the new bucket.
1. name of the storage class from OBC got created.
1. phases of bucket creation:
* _Pending_: the operator is processing the request.
* _Bound_: the operator finished processing the request and linked the OBC and OB
* _Released_: the OB has been deleted, leaving the OBC unclaimed but unavailable.
* _Failed_: not currently set.
### App Pod
```yaml
apiVersion: v1
kind: Pod
metadata:
name: app-pod
namespace: dev-user
spec:
containers:
- name: mycontainer
image: redis
envFrom: [1]
- configMapRef:
name: ceph-bucket [2]
- secretRef:
name: ceph-bucket [3]
```
1. use `env:` if mapping of the defined key names to the env var names used by the app is needed.
1. makes available to the pod as env variables: `BUCKET_HOST`, `BUCKET_PORT`, `BUCKET_NAME`
1. makes available to the pod as env variables: `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`
### StorageClass
```yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: rook-ceph-bucket
labels:
aws-s3/object [1]
provisioner: rook-ceph.ceph.rook.io/bucket [2]
parameters: [3]
objectStoreName: my-store
objectStoreNamespace: rook-ceph
bucketName: ceph-bucket [4]
reclaimPolicy: Delete [5]
```
1. `label`(optional) here associates this `StorageClass` to a specific provisioner.
1. `provisioner` responsible for handling `OBCs` referencing this `StorageClass`.
1. **all** `parameter` required.
1. `bucketName` is required for access to existing buckets but is omitted when provisioning new buckets.
Unlike greenfield provisioning, the brownfield bucket name appears in the `StorageClass`, not the `OBC`.
1. rook-ceph provisioner decides how to treat the `reclaimPolicy` when an `OBC` is deleted for the bucket. See explanation as [specified in Kubernetes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#retain)
* _Delete_ = physically delete the bucket.
* _Retain_ = do not physically delete the bucket.
Documentation/Storage-Configuration/Object-Storage-RGW/ceph-object-bucket-notifications.md
+185
View File
@@ -0,0 +1,185 @@
---
title: Object Bucket Notifications
---
!!! info
This feature is **experimental**
Rook supports the creation of bucket notifications via two custom resources:
* a `CephBucketNotification` is a custom resource the defines: topic, events and filters of a bucket notification, and is described by a Custom Resource Definition (CRD) shown below. Bucket notifications are associated with a bucket by setting labels on the Object Bucket claim (OBC).
See the Ceph documentation for detailed information: [Bucket Notifications - Ceph Object Gateway - Ceph Documentation](https://docs.ceph.com/en/latest/radosgw/notifications/).
* a `CephBucketTopic` is custom resource which represents a bucket notification topic and is described by a CRD shown below. A bucket notification topic represents an endpoint (or a "topic" inside this endpoint) to which bucket notifications could be sent.
## Notifications
A CephBucketNotification defines what bucket actions trigger the notification and which topic to send notifications to. A CephBucketNotification may also define a filter, based on the object's name and other object attributes. Notifications can be associated with buckets created via ObjectBucketClaims by adding labels to an ObjectBucketClaim with the following format:
```yaml
bucket-notification-<notification name>: <notification name>
```
The CephBucketTopic, CephBucketNotification and ObjectBucketClaim must all belong to the same namespace.
If a bucket was created manually (not via an ObjectBucketClaim), notifications on this bucket should also be created manually. However, topics in these notifications may reference topics that were created via CephBucketTopic resources.
## Topics
A CephBucketTopic represents an endpoint (of types: Kafka, AMQP0.9.1 or HTTP), or a specific resource inside this endpoint (e.g a Kafka or an AMQP topic, or a specific URI in an HTTP server). The CephBucketTopic also holds any additional info needed for a CephObjectStore's RADOS Gateways (RGW) to connect to the endpoint. Topics don't belong to a specific bucket or notification. Notifications from multiple buckets may be sent to the same topic, and one bucket (via multiple CephBucketNotifications) may send notifications to multiple topics.
## Notification Reliability and Delivery
Notifications may be sent synchronously, as part of the operation that triggered them. In this mode, the operation is acknowledged only after the notification is sent to the topic’s configured endpoint, which means that the round trip time of the notification is added to the latency of the operation itself.
The original triggering operation will still be considered as successful even if the notification fail with an error, cannot be delivered or times out.
Notifications may also be sent asynchronously. They will be committed into persistent storage and then asynchronously sent to the topic’s configured endpoint. In this case, the only latency added to the original operation is of committing the notification to persistent storage.
If the notification fail with an error, cannot be delivered or times out, it will be retried until successfully acknowledged.
## Example
### CephBucketTopic Custom Resource
```yaml
apiVersion: objectbucket.io/v1alpha1
kind: CephBucketTopic
metadata:
name: my-topic [1]
namespace: my-app-space [2]
spec:
objectStoreName: my-store [3]
objectStoreNamespace: rook-ceph [4]
opaqueData: my@email.com [5]
persistent: false [6]
endpoint: [7]
http: [8]
uri: http://my-notification-endpoint:8080
# uri: http://my-notification-endpoint:8080/my-topic
# uri: https://my-notification-endpoint:8443
disableVerifySSL: true [9]
sendCloudEvents: false [10]
# amqp: [11]
# uri: amqp://my-rabbitmq-service:5672
# uri: amqp://my-rabbitmq-service:5672/vhost1
# uri: amqps://user@password:my-rabbitmq-service:5672
# disableVerifySSL: true [12]
# ackLevel: broker [13]
# exchange: my-exchange [14]
# kafka: [15]
# uri: kafka://my-kafka-service:9092
# disableVerifySSL: true [16]
# ackLevel: broker [17]
# useSSL: false [18]
```
1. `name` of the `CephBucketTopic`
* In case of AMQP endpoint, the name is used for the AMQP topic (“routing key” for a topic exchange)
* In case of Kafka endpoint, the name is used as the Kafka topic
1. `namespace`(optional) of the `CephBucketTopic`. Should match the namespace of the CephBucketNotification associated with this CephBucketTopic, and the OBC with the label referencing the CephBucketNotification
1. `objectStoreName` is the name of the object store in which the topic should be created. This must be the same object store used for the buckets associated with the notifications referencing this topic.
1. `objectStoreNamespace` is the namespace of the object store in which the topic should be created
1. `opaqueData` (optional) is added to all notifications triggered by a notifications associated with the topic
1. `persistent` (optional) indicates whether notifications to this endpoint are persistent (=asynchronous) or sent synchronously (“false” by default)
1. `endpoint` to which to send the notifications to. Exactly one of the endpoints must be defined: `http`, `amqp`, `kafka`
1. `http` (optional) hold the spec for an HTTP endpoint. The format of the URI would be: `http[s]://<fqdn>[:<port>][/<resource>]`
* port defaults to: 80/443 for HTTP/S accordingly
1. `disableVerifySSL` indicates whether the RGW is going to verify the SSL certificate of the HTTP server in case HTTPS is used ("false" by default)
1. `amqp` (optional) hold the spec for an AMQP endpoint. The format of the URI would be: `amqp[s]://[<user>:<password>@]<fqdn>[:<port>][/<vhost>]`
* port defaults to: 5672/5671 for AMQP/S accordingly
* user/password defaults to: guest/guest
* user/password may only be provided if HTTPS is used with the RGW. If not, topic creation request will be rejected
* vhost defaults to: “/”
1. `disableVerifySSL` (optional) indicates whether the RGW is going to verify the SSL certificate of the AMQP server in case AMQPS is used ("false" by default)
1. `sendCloudEvents`: (optional) send the notifications with the [CloudEvents header](https://github.com/cloudevents/spec/blob/main/cloudevents/adapters/aws-s3.md). Supported for Ceph Quincy (v17) or newer ("false" by default)
1. `ackLevel` (optional) indicates what kind of ack the RGW is waiting for after sending the notifications:
* “none”: message is considered “delivered” if sent to broker
* “broker”: message is considered “delivered” if acked by broker (default)
* “routable”: message is considered “delivered” if broker can route to a consumer
1. `exchange` in the AMQP broker that would route the notifications. Different topics pointing to the same endpoint must use the same exchange
1. `kafka` (optional) hold the spec for a Kafka endpoint. The format of the URI would be: `kafka://[<user>:<password>@]<fqdn>[:<port]`
* port defaults to: 9092
* user/password may only be provided if HTTPS is used with the RGW. If not, topic creation request will be rejected
* user/password may only be provided together with `useSSL`, if not, the connection to the broker would fail
1. `disableVerifySSL` (optional) indicates whether the RGW is going to verify the SSL certificate of the Kafka server in case `useSSL` flag is used ("false" by default)
1. `ackLevel` (optional) indicates what kind of ack the RGW is waiting for after sending the notifications:
* “none”: message is considered “delivered” if sent to broker
* “broker”: message is considered “delivered” if acked by broker (default)
1. `useSSL` (optional) indicates that secure connection will be used for connecting with the broker (“false” by default)
!!! note
In case of Kafka and AMQP, the consumer of the notifications is not required to ack the notifications, since the broker persists the messages before delivering them to their final destinations.
### CephBucketNotification Custom Resource
```yaml
apiVersion: ceph.rook.io/v1
kind: CephBucketNotification
metadata:
name: my-notification [1]
namespace: my-app-space [2]
spec:
topic: my-topic [3]
filter: [4]
keyFilters: [5]
# match objects with keys that start with "hello"
- name: prefix
value: hello
# match objects with keys that end with ".png"
- name: suffix
value: .png
# match objects with keys with only lowercase characters
- name: regex
value: "[a-z]*\\.*"
metadataFilters: [6]
- name: x-amz-meta-color
value: blue
- name: x-amz-meta-user-type
value: free
tagFilters: [7]
- name: project
value: brown
# notification apply for any of the events
# full list of supported events is here:
# https://docs.ceph.com/en/latest/radosgw/s3-notification-compatibility/#event-types
events: [8]
- s3:ObjectCreated:Put
- s3:ObjectCreated:Copy
```
1. `name` of the `CephBucketNotification`
1. `namespace`(optional) of the `CephBucketNotification`. Should match the namespace of the CephBucketTopic referenced in [3], and the OBC with the label referencing the CephBucketNotification
1. `topic` to which the notifications should be sent
1. `filter` (optional) holds a list of filtering rules of different types. Only objects that match all the filters will trigger notification sending
1. `keyFilter` (optional) are filters based on the object key. There could be up to 3 key filters defined: `prefix`, `suffix` and `regex`
1. `metadataFilters` (optional) are filters based on the object metadata. All metadata fields defined as filters must exists in the object, with the values defined in the filter. Other metadata fields may exist in the object
1. `tagFilters` (optional) are filters based on object tags. All tags defined as filters must exists in the object, with the values defined in the filter. Other tags may exist in the object
1. `events` (optional) is a list of events that should trigger the notifications. By default all events should trigger notifications. Valid Events are:
* s3:ObjectCreated:*
* s3:ObjectCreated:Put
* s3:ObjectCreated:Post
* s3:ObjectCreated:Copy
* s3:ObjectCreated:CompleteMultipartUpload
* s3:ObjectRemoved:*
* s3:ObjectRemoved:Delete
* s3:ObjectRemoved:DeleteMarkerCreated
### OBC Custom Resource
For a notifications to be associated with a bucket, a labels must be added to the OBC, indicating the name of the notification.
To delete a notification from a bucket the matching label must be removed.
When an OBC is deleted, all of the notifications associated with the bucket will be deleted as well.
```yaml
apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
name: ceph-notification-bucket
labels:
# labels that don't have this structure: bucket-notification-<name> : <name>
# are ignored by the operator's bucket notifications provisioning mechanism
some-label: some-value
# the following label adds notifications to this bucket
bucket-notification-my-notification: my-notification
bucket-notification-another-notification: another-notification
spec:
generateBucketName: ceph-bkt
storageClassName: rook-ceph-delete-bucket
```
Documentation/Storage-Configuration/Object-Storage-RGW/ceph-object-multisite.md
+295
View File
@@ -0,0 +1,295 @@
---
title: Object Store Multisite
---
Multisite is a feature of Ceph that allows object stores to replicate their data over multiple Ceph clusters.
Multisite also allows object stores to be independent and isolated from other object stores in a cluster.
When a ceph-object-store is created without the `zone` section; a realm, zone group, and zone is created with the same name as the ceph-object-store.
Since it is the only ceph-object-store in the realm, the data in the ceph-object-store remain independent and isolated from others on the same cluster.
When a ceph-object-store is created with the `zone` section, the ceph-object-store will join a custom created zone, zone group, and realm each with a different names than its own.
This allows the ceph-object-store to replicate its data over multiple Ceph clusters.
To review core multisite concepts please read the
[ceph-multisite design overview](https://github.com/rook/rook/blob/master/design/ceph/object/ceph-multisite-overview.md).
### Prerequisites
This guide assumes a Rook cluster as explained in the [Quickstart](../../Getting-Started/quickstart.md).
## Creating Object Multisite
If an admin wants to set up multisite on a Rook Ceph cluster, the admin should create:
1. A [realm](../../CRDs/Object-Storage/ceph-object-multisite-crd.md#object-realm-settings)
1. A [zonegroup](../../CRDs/Object-Storage/ceph-object-multisite-crd.md#object-zone-group-settings)
1. A [zone](../../CRDs/Object-Storage/ceph-object-multisite-crd.md#object-zone-settings)
1. A ceph object store with the `zone` section
object-multisite.yaml in the [examples](https://github.com/rook/rook/blob/master/deploy/examples/) directory can be used to create the multisite CRDs.
```console
kubectl create -f object-multisite.yaml
```
The first zone group created in a realm is the master zone group. The first zone created in a zone group is the master zone.
When a non-master zone or non-master zone group is created, the zone group or zone is not in the Ceph Radosgw Multisite [Period](https://docs.ceph.com/docs/en/latest/radosgw/multisite/) until an object-store is created in that zone (and zone group).
The zone will create the pools for the object-store(s) that are in the zone to use.
When one of the multisite CRs (realm, zone group, zone) is deleted the underlying ceph realm/zone group/zone is not deleted, neither are the pools created by the zone. See the "Multisite Cleanup" section for more information.
For more information on the multisite CRDs please read [ceph-object-multisite-crd](../../CRDs/Object-Storage/ceph-object-multisite-crd.md).
## Pulling a Realm
If an admin wants to sync data from another cluster, the admin needs to pull a realm on a Rook Ceph cluster from another Rook Ceph (or Ceph) cluster.
To begin doing this, the admin needs 2 pieces of information:
1. An endpoint from the realm being pulled from
1. The access key and the system key of the system user from the realm being pulled from.
### Getting the Pull Endpoint
To pull a Ceph realm from a remote Ceph cluster, an `endpoint` must be added to the CephObjectRealm's `pull` section in the `spec`. This endpoint must be from the master zone in the master zone group of that realm.
If an admin does not know of an endpoint that fits this criteria, the admin can find such an endpoint on the remote Ceph cluster (via the tool box if it is a Rook Ceph Cluster) by running:
```console
$ radosgw-admin zonegroup get --rgw-realm=$REALM_NAME --rgw-zonegroup=$MASTER_ZONEGROUP_NAME
{
...
"endpoints": [http://10.17.159.77:80],
...
}
```
A list of endpoints in the master zone group in the master zone is in the `endpoints` section of the JSON output of the `zonegoup get` command.
This endpoint must also be resolvable from the new Rook Ceph cluster. To test this run the `curl` command on the endpoint:
```console
$ curl -L http://10.17.159.77:80
<?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>
```
Finally add the endpoint to the `pull` section of the CephObjectRealm's spec. The CephObjectRealm should have the same name as the CephObjectRealm/Ceph realm it is pulling from.
```yaml
apiVersion: ceph.rook.io/v1
kind: CephObjectRealm
metadata:
name: realm-a
namespace: rook-ceph
spec:
pull:
endpoint: http://10.17.159.77:80
```
### Getting Realm Access Key and Secret Key
The access key and secret key of the system user are keys that allow other Ceph clusters to pull the realm of the system user.
#### Getting the Realm Access Key and Secret Key from the Rook Ceph Cluster
When an admin creates a ceph-object-realm a system user automatically gets created for the realm with an access key and a secret key.
This system user has the name "$REALM_NAME-system-user". For the example realm, the uid for the system user is "realm-a-system-user".
These keys for the user are exported as a kubernetes [secret](https://kubernetes.io/docs/concepts/configuration/secret/) called "$REALM_NAME-keys" (ex: realm-a-keys).
To get these keys from the cluster the realm was originally created on, run:
```console
kubectl -n $ORIGINAL_CLUSTER_NAMESPACE get secrets realm-a-keys -o yaml > realm-a-keys.yaml
```
Edit the `realm-a-keys.yaml` file, and change the `namespace` with the namespace that the new Rook Ceph cluster exists in.
Then create a kubernetes secret on the pulling Rook Ceph cluster with the same secrets yaml file.
```console
kubectl create -f realm-a-keys.yaml
```
#### Getting the Realm Access Key and Secret Key from a Non Rook Ceph Cluster
The access key and the secret key of the system user can be found in the output of running the following command on a non-rook ceph cluster:
```console
radosgw-admin user info --uid="realm-a-system-user"
```
```json
{
...
"keys": [
{
"user": "realm-a-system-user"
"access_key": "aSw4blZIKV9nKEU5VC0="
"secret_key": "JSlDXFt5TlgjSV9QOE9XUndrLiI5JEo9YDBsJg==",
}
],
...
}
```
Then base64 encode the each of the keys and create a `.yaml` file for the Kubernetes secret from the following template.
Only the `access-key`, `secret-key`, and `namespace` sections need to be replaced.
```yaml
apiVersion: v1
data:
access-key: YVN3NGJsWklLVjluS0VVNVZDMD0=
secret-key: SlNsRFhGdDVUbGdqU1Y5UU9FOVhVbmRyTGlJNUpFbzlZREJzSmc9PQ==
kind: Secret
metadata:
name: realm-a-keys
namespace: $NEW_ROOK_CLUSTER_NAMESPACE
type: kubernetes.io/rook
```
Finally, create a kubernetes secret on the pulling Rook Ceph cluster with the new secrets yaml file.
```console
kubectl create -f realm-a-keys.yaml
```
#### Pulling a Realm on a New Rook Ceph Cluster
Once the admin knows the endpoint and the secret for the keys has been created, the admin should create:
1. A [CephObjectRealm](../../CRDs/Object-Storage/ceph-object-multisite-crd.md#object-realm-settings) matching to the realm on the other Ceph cluster, with an endpoint as described above.
1. A [CephObjectZoneGroup](../../CRDs/Object-Storage/ceph-object-multisite-crd.md#object-zone-group-settings) matching the master zone group name or the master CephObjectZoneGroup from the cluster the the realm was pulled from.
1. A [CephObjectZone](../../CRDs/Object-Storage/ceph-object-multisite-crd.md#object-zone-settings) referring to the CephObjectZoneGroup created above.
1. A CephObjectStore referring to the new CephObjectZone resource.
object-multisite-pull-realm.yaml (with changes) in the [examples](https://github.com/rook/rook/blob/master/deploy/examples/) directory can be used to create the multisite CRDs.
```console
kubectl create -f object-multisite-pull-realm.yaml
```
## Multisite Cleanup
Multisite configuration must be cleaned up by hand. Deleting a realm/zone group/zone CR will not delete the underlying Ceph realm, zone group, zone, or the pools associated with a zone.
### Realm Deletion
Changes made to the resource's configuration or deletion of the resource are not reflected on the Ceph cluster.
When the ceph-object-realm resource is deleted or modified, the realm is not deleted from the Ceph cluster. Realm deletion must be done via the toolbox.
#### Deleting a Realm
The Rook toolbox can modify the Ceph Multisite state via the radosgw-admin command.
The following command, run via the toolbox, deletes the realm.
```console
radosgw-admin realm delete --rgw-realm=realm-a
```
### Zone Group Deletion
Changes made to the resource's configuration or deletion of the resource are not reflected on the Ceph cluster.
When the ceph-object-zone group resource is deleted or modified, the zone group is not deleted from the Ceph cluster. Zone Group deletion must be done through the toolbox.
#### Deleting a Zone Group
The Rook toolbox can modify the Ceph Multisite state via the radosgw-admin command.
The following command, run via the toolbox, deletes the zone group.
```console
radosgw-admin zonegroup delete --rgw-realm=realm-a --rgw-zonegroup=zone-group-a
radosgw-admin period update --commit --rgw-realm=realm-a --rgw-zonegroup=zone-group-a
```
### Deleting and Reconfiguring the Ceph Object Zone
Changes made to the resource's configuration or deletion of the resource are not reflected on the Ceph cluster.
When the ceph-object-zone resource is deleted or modified, the zone is not deleted from the Ceph cluster. Zone deletion must be done through the toolbox.
#### Changing the Master Zone
The Rook toolbox can change the master zone in a zone group.
```console
radosgw-admin zone modify --rgw-realm=realm-a --rgw-zonegroup=zone-group-a --rgw-zone=zone-a --master
radosgw-admin zonegroup modify --rgw-realm=realm-a --rgw-zonegroup=zone-group-a --master
radosgw-admin period update --commit --rgw-realm=realm-a --rgw-zonegroup=zone-group-a --rgw-zone=zone-a
```
#### Deleting Zone
The Rook toolbox can modify the Ceph Multisite state via the radosgw-admin command.
There are two scenarios possible when deleting a zone.
The following commands, run via the toolbox, deletes the zone if there is only one zone in the zone group.
```console
radosgw-admin zone delete --rgw-realm=realm-a --rgw-zonegroup=zone-group-a --rgw-zone=zone-a
radosgw-admin period update --commit --rgw-realm=realm-a --rgw-zonegroup=zone-group-a --rgw-zone=zone-a
```
In the other scenario, there are more than one zones in a zone group.
Care must be taken when changing which zone is the master zone.
Please read the following [documentation](https://docs.ceph.com/docs/master/radosgw/multisite/#changing-the-metadata-master-zone) before running the below commands:
The following commands, run via toolboxes, remove the zone from the zone group first, then delete the zone.
```console
radosgw-admin zonegroup rm --rgw-realm=realm-a --rgw-zonegroup=zone-group-a --rgw-zone=zone-a
radosgw-admin period update --commit --rgw-realm=realm-a --rgw-zonegroup=zone-group-a --rgw-zone=zone-a
radosgw-admin zone delete --rgw-realm=realm-a --rgw-zonegroup=zone-group-a --rgw-zone=zone-a
radosgw-admin period update --commit --rgw-realm=realm-a --rgw-zonegroup=zone-group-a --rgw-zone=zone-a
```
When a zone is deleted, the pools for that zone are not deleted.
#### Deleting Pools for a Zone
The Rook toolbox can delete pools. Deleting pools should be done with caution.
The following [documentation](https://docs.ceph.com/docs/master/rados/operations/pools/) on pools should be read before deleting any pools.
When a zone is created the following pools are created for each zone:
```console
$ZONE_NAME.rgw.control
$ZONE_NAME.rgw.meta
$ZONE_NAME.rgw.log
$ZONE_NAME.rgw.buckets.index
$ZONE_NAME.rgw.buckets.non-ec
$ZONE_NAME.rgw.buckets.data
```
Here is an example command to delete the .rgw.buckets.data pool for zone-a.
```console
ceph osd pool rm zone-a.rgw.buckets.data zone-a.rgw.buckets.data --yes-i-really-really-mean-it
```
In this command the pool name **must** be mentioned twice for the pool to be removed.
#### Removing an Object Store from a Zone
When an object-store (created in a zone) is deleted, the endpoint for that object store is removed from that zone, via
```console
kubectl delete -f object-store.yaml
```
Removing object store(s) from the master zone of the master zone group should be done with caution. When all of these object-stores are deleted the period cannot be updated and that realm cannot be pulled.
Documentation/Storage-Configuration/Object-Storage-RGW/object-storage.md
+369
View File
@@ -0,0 +1,369 @@
---
title: Object Storage Overview
---
Object storage exposes an S3 API to the storage cluster for applications to put and get data.
## Prerequisites
This guide assumes a Rook cluster as explained in the [Quickstart](../../Getting-Started/quickstart.md).
## Configure an Object Store
Rook has the ability to either deploy an object store in Kubernetes or to connect to an external RGW service.
Most commonly, the object store will be configured locally by Rook.
Alternatively, if you have an existing Ceph cluster with Rados Gateways, see the
[external section](#connect-to-an-external-object-store) to consume it from Rook.
### Create a Local Object Store
The below sample will create a `CephObjectStore` that starts the RGW service in the cluster with an S3 API.
!!! note
This sample requires *at least 3 bluestore OSDs*, with each OSD located on a *different node*.
The OSDs must be located on different nodes, because the [`failureDomain`](../../CRDs/Block-Storage/ceph-block-pool-crd.md#spec) is set to `host` and the `erasureCoded` chunk settings require at least 3 different OSDs (2 `dataChunks` + 1 `codingChunks`).
See the [Object Store CRD](../../CRDs/Object-Storage/ceph-object-store-crd.md#object-store-settings), for more detail on the settings available for a `CephObjectStore`.
```yaml
apiVersion: ceph.rook.io/v1
kind: CephObjectStore
metadata:
name: my-store
namespace: rook-ceph
spec:
metadataPool:
failureDomain: host
replicated:
size: 3
dataPool:
failureDomain: host
erasureCoded:
dataChunks: 2
codingChunks: 1
preservePoolsOnDelete: true
gateway:
sslCertificateRef:
port: 80
# securePort: 443
instances: 1
healthCheck:
bucket:
disabled: false
interval: 60s
```
After the `CephObjectStore` is created, the Rook operator will then create all the pools and other resources necessary to start the service. This may take a minute to complete.
Create an object store:
```console
kubectl create -f object.yaml
```
To confirm the object store is configured, wait for the RGW pod(s) to start:
```console
kubectl -n rook-ceph get pod -l app=rook-ceph-rgw
```
### Connect to an External Object Store
Rook can connect to existing RGW gateways to work in conjunction with the external mode of the `CephCluster` CRD.
If you have an external `CephCluster` CR, you can instruct Rook to consume external gateways with the following:
```yaml
apiVersion: ceph.rook.io/v1
kind: CephObjectStore
metadata:
name: external-store
namespace: rook-ceph
spec:
gateway:
port: 8080
externalRgwEndpoints:
- ip: 192.168.39.182
healthCheck:
bucket:
enabled: true
interval: 60s
```
You can use the existing `object-external.yaml` file.
When ready the ceph-object-controller will output a message in the Operator log similar to this one:
```console
ceph-object-controller: ceph object store gateway service >running at 10.100.28.138:8080
```
You can now get and access the store via:
```console
$ kubectl -n rook-ceph get svc -l app=rook-ceph-rgw
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
rook-ceph-rgw-my-store ClusterIP 10.100.28.138 <none> 8080/TCP 6h59m
```
Any pod from your cluster can now access this endpoint:
```console
$ curl 10.100.28.138:8080
<?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>
```
It is also possible to use the internally registered DNS name:
```console
$ curl rook-ceph-rgw-my-store.rook-ceph:8080
<?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>
```
The DNS name is created with the following schema `rook-ceph-rgw-$STORE_NAME.$NAMESPACE`.
## Create a Bucket
Now that the object store is configured, next we need to create a bucket where a client can read and write objects. A bucket can be created by defining a storage class, similar to the pattern used by block and file storage.
First, define the storage class that will allow object clients to create a bucket.
The storage class defines the object storage system, the bucket retention policy, and other properties required by the administrator. Save the following as `storageclass-bucket-delete.yaml` (the example is named as such due to the `Delete` reclaim policy).
```yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: rook-ceph-bucket
# Change "rook-ceph" provisioner prefix to match the operator namespace if needed
provisioner: rook-ceph.ceph.rook.io/bucket
reclaimPolicy: Delete
parameters:
objectStoreName: my-store
objectStoreNamespace: rook-ceph
```
If you’ve deployed the Rook operator in a namespace other than `rook-ceph`, change the prefix in the provisioner to match the namespace you used. For example, if the Rook operator is running in the namespace `my-namespace` the provisioner value should be `my-namespace.ceph.rook.io/bucket`.
```console
kubectl create -f storageclass-bucket-delete.yaml
```
Based on this storage class, an object client can now request a bucket by creating an Object Bucket Claim (OBC).
When the OBC is created, the Rook-Ceph bucket provisioner will create a new bucket. Notice that the OBC
references the storage class that was created above.
Save the following as `object-bucket-claim-delete.yaml` (the example is named as such due to the `Delete` reclaim policy):
```yaml
apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
name: ceph-bucket
spec:
generateBucketName: ceph-bkt
storageClassName: rook-ceph-bucket
```
```console
kubectl create -f object-bucket-claim-delete.yaml
```
Now that the claim is created, the operator will create the bucket as well as generate other artifacts to enable access to the bucket. A secret and ConfigMap are created with the same name as the OBC and in the same namespace.
The secret contains credentials used by the application pod to access the bucket.
The ConfigMap contains bucket endpoint information and is also consumed by the pod.
See the [Object Bucket Claim Documentation](ceph-object-bucket-claim.md) for more details on the `CephObjectBucketClaims`.
### Client Connections
The following commands extract key pieces of information from the secret and configmap:"
```console
#config-map, secret, OBC will part of default if no specific name space mentioned
export AWS_HOST=$(kubectl -n default get cm ceph-bucket -o jsonpath='{.data.BUCKET_HOST}')
export PORT=$(kubectl -n default get cm ceph-bucket -o jsonpath='{.data.BUCKET_PORT}')
export BUCKET_NAME=$(kubectl -n default get cm ceph-bucket -o jsonpath='{.data.BUCKET_NAME}')
export AWS_ACCESS_KEY_ID=$(kubectl -n default get secret ceph-bucket -o jsonpath='{.data.AWS_ACCESS_KEY_ID}' | base64 --decode)
export AWS_SECRET_ACCESS_KEY=$(kubectl -n default get secret ceph-bucket -o jsonpath='{.data.AWS_SECRET_ACCESS_KEY}' | base64 --decode)
```
## Consume the Object Storage
Now that you have the object store configured and a bucket created, you can consume the
object storage from an S3 client.
This section will guide you through testing the connection to the `CephObjectStore` and uploading and downloading from it.
Run the following commands after you have connected to the [Rook toolbox](../../Troubleshooting/ceph-toolbox.md).
### Connection Environment Variables
To simplify the s3 client commands, you will want to set the four environment variables for use by your client (ie. inside the toolbox).
See above for retrieving the variables for a bucket created by an `ObjectBucketClaim`.
```console
export AWS_HOST=<host>
export PORT=<port>
export AWS_ACCESS_KEY_ID=<accessKey>
export AWS_SECRET_ACCESS_KEY=<secretKey>
```
* `Host`: The DNS host name where the rgw service is found in the cluster. Assuming you are using the default `rook-ceph` cluster, it will be `rook-ceph-rgw-my-store.rook-ceph.svc`.
* `Port`: The endpoint where the rgw service is listening. Run `kubectl -n rook-ceph get svc rook-ceph-rgw-my-store`, to get the port.
* `Access key`: The user's `access_key` as printed above
* `Secret key`: The user's `secret_key` as printed above
The variables for the user generated in this example might be:
```console
export AWS_HOST=rook-ceph-rgw-my-store.rook-ceph.svc
export PORT=80
export AWS_ACCESS_KEY_ID=XEZDB3UJ6X7HVBE7X7MA
export AWS_SECRET_ACCESS_KEY=7yGIZON7EhFORz0I40BFniML36D2rl8CQQ5kXU6l
```
The access key and secret key can be retrieved as described in the section above on [client connections](#client-connections) or
below in the section [creating a user](#create-a-user) if you are not creating the buckets with an `ObjectBucketClaim`.
### Configure s5cmd
To test the `CephObjectStore`, set the object store credentials in the toolbox pod for the `s5cmd` tool.
```console
mkdir ~/.aws
cat > ~/.aws/credentials << EOF
[default]
aws_access_key_id = ${AWS_ACCESS_KEY_ID}
aws_secret_access_key = ${AWS_SECRET_ACCESS_KEY}
EOF
```
### PUT or GET an object
Upload a file to the newly created bucket
```console
echo "Hello Rook" > /tmp/rookObj
s5cmd --endpoint-url http://$AWS_HOST:$PORT cp /tmp/rookObj s3://$BUCKET_NAME
```
Download and verify the file from the bucket
```console
s5cmd --endpoint-url http://$AWS_HOST:$PORT cp s3://$BUCKET_NAME/rookObj /tmp/rookObj-download
cat /tmp/rookObj-download
```
## Access External to the Cluster
Rook sets up the object storage so pods will have access internal to the cluster. If your applications are running outside the cluster,
you will need to setup an external service through a `NodePort`.
First, note the service that exposes RGW internal to the cluster. We will leave this service intact and create a new service for external access.
```console
$ kubectl -n rook-ceph get service rook-ceph-rgw-my-store
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
rook-ceph-rgw-my-store 10.3.0.177 <none> 80/TCP 2m
```
Save the external service as `rgw-external.yaml`:
```yaml
apiVersion: v1
kind: Service
metadata:
name: rook-ceph-rgw-my-store-external
namespace: rook-ceph
labels:
app: rook-ceph-rgw
rook_cluster: rook-ceph
rook_object_store: my-store
spec:
ports:
- name: rgw
port: 80
protocol: TCP
targetPort: 80
selector:
app: rook-ceph-rgw
rook_cluster: rook-ceph
rook_object_store: my-store
sessionAffinity: None
type: NodePort
```
Now create the external service.
```console
kubectl create -f rgw-external.yaml
```
See both rgw services running and notice what port the external service is running on:
```console
$ kubectl -n rook-ceph get service rook-ceph-rgw-my-store rook-ceph-rgw-my-store-external
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
rook-ceph-rgw-my-store ClusterIP 10.104.82.228 <none> 80/TCP 4m
rook-ceph-rgw-my-store-external NodePort 10.111.113.237 <none> 80:31536/TCP 39s
```
Internally the rgw service is running on port `80`. The external port in this case is `31536`. Now you can access the `CephObjectStore` from anywhere! All you need is the hostname for any machine in the cluster, the external port, and the user credentials.
## Create a User
If you need to create an independent set of user credentials to access the S3 endpoint,
create a `CephObjectStoreUser`. The user will be used to connect to the RGW service in the cluster using the S3 API.
The user will be independent of any object bucket claims that you might have created in the earlier
instructions in this document.
See the [Object Store User CRD](../../CRDs/Object-Storage/ceph-object-store-user-crd.md) for more detail on the settings available for a `CephObjectStoreUser`.
```yaml
apiVersion: ceph.rook.io/v1
kind: CephObjectStoreUser
metadata:
name: my-user
namespace: rook-ceph
spec:
store: my-store
displayName: "my display name"
```
When the `CephObjectStoreUser` is created, the Rook operator will then create the RGW user on the specified `CephObjectStore` and store the Access Key and Secret Key in a kubernetes secret in the same namespace as the `CephObjectStoreUser`.
```console
# Create the object store user
kubectl create -f object-user.yaml
```
```console
# To confirm the object store user is configured, describe the secret
$ kubectl -n rook-ceph describe secret rook-ceph-object-user-my-store-my-user
Name: rook-ceph-object-user-my-store-my-user
Namespace: rook-ceph
Labels: app=rook-ceph-rgw
rook_cluster=rook-ceph
rook_object_store=my-store
Annotations: <none>
Type: kubernetes.io/rook
Data
====
AccessKey: 20 bytes
SecretKey: 40 bytes
```
The AccessKey and SecretKey data fields can be mounted in a pod as an environment variable. More information on consuming
kubernetes secrets can be found in the [K8s secret documentation](https://kubernetes.io/docs/concepts/configuration/secret/)
To directly retrieve the secrets:
```console
kubectl -n rook-ceph get secret rook-ceph-object-user-my-store-my-user -o jsonpath='{.data.AccessKey}' | base64 --decode
kubectl -n rook-ceph get secret rook-ceph-object-user-my-store-my-user -o jsonpath='{.data.SecretKey}' | base64 --decode
```
## Object Multisite
Multisite is a feature of Ceph that allows object stores to replicate its data over multiple Ceph clusters.
Multisite also allows object stores to be independent and isloated from other object stores in a cluster.
For more information on multisite please read the [ceph multisite overview](ceph-object-multisite.md) for how to run it.