feedc0de/my-rook-config
1
0
Fork 0
forked from rook/rook
Code Pull Requests Activity

object: use advertise endpoint for admin ops

Browse Source 
RGW can only serve a single certificate. This limitation means that the
prior behavior of using the default service for admin ops when TLS is
enabled may mean it requires additional complex certificate management
to make sure the object store uses a certificate valid for Rook internal
admin ops and user connections.

This is needlessly complex for users. Instead, change Rook's behavior
and documentation to clarify that it will use the same endpoint intended
for S3 client applications. This means that users have a more
straightforward path to enabling both Rook and consuming applications.

More info: https://github.com/rook/rook/issues/14530

Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
This commit is contained in:
Blaine Gardner
2024-08-05 14:32:59 -06:00
parent 9c98467f3c
commit b4a2285aa6
4 changed files with 81 additions and 91 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Documentation/Storage-Configuration/Object-Storage-RGW/object-storage.md
+42 -26
View File
@@ -200,7 +200,7 @@ Then create a secret with the user credentials:
kubectl -n rook-ceph create secret generic --type="kubernetes.io/rook" rgw-admin-ops-user --from-literal=accessKey=<access key of the user> --from-literal=secretKey=<secret key of the user>
```
If you have an external `CephCluster` CR, you can instruct Rook to consume external gateways with the following:
For an external CephCluster, configure Rook to consume external RGW servers with the following:
```yaml
apiVersion: ceph.rook.io/v1
@@ -216,42 +216,35 @@ spec:
# hostname: example.com
```
Use the existing `object-external.yaml` file. Even though multiple endpoints can be specified, it is recommend to use only one endpoint. This endpoint is randomly added to `configmap` of OBC and secret of the `cephobjectstoreuser`. Rook never guarantees the randomly picked endpoint is a working one or not.
If there are multiple endpoints, please add load balancer in front of them and use the load balancer endpoint in the `externalRgwEndpoints` list.
See `object-external.yaml` for a more detailed example.
When ready, the message in the `cephobjectstore` status similar to this one:
```console
kubectl -n rook-ceph get cephobjectstore external-store
NAME PHASE
external-store Ready
```
Any pod from your cluster can now access this endpoint:
```console
$ curl 192.168.39.182:8080
<?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>
```
Even though multiple `externalRgwEndpoints` can be specified, it is best to use a single endpoint.
Only the first endpoint in the list will be advertised to any consuming resources like
CephObjectStoreUsers, ObjectBucketClaims, or COSI resources. If there are multiple external RGW
endpoints, add load balancer in front of them, then use the single load balancer endpoint in the
`externalRgwEndpoints` list.
## Object store endpoint
The CephObjectStore resource `status.info` contains `endpoint` (and `secureEndpoint`) fields, which
report the endpoint that can be used to access the object store as a client.
report the endpoint that can be used to access the object store as a client. This endpoint is also
advertised as the default endpoint for CephObjectStoreUsers, ObjectBucketClaims, and
Container Object Store Interface (COSI) resources.
Each object store also creates a Kubernetes service that can be used as a client endpoint from
within the Kubernetes cluster. The DNS name of the service is
`rook-ceph-rgw-<objectStoreName>.<objectStoreNamespace>.svc`. This service DNS name is the default
`endpoint` (and `secureEndpoint`).
For [external clusters](#connect-to-an-external-object-store), the default endpoints contain the
first `spec.gateway.externalRgwEndpoint` instead of the service DNS name.
For [external clusters](#connect-to-an-external-object-store), the default endpoint is the first
`spec.gateway.externalRgwEndpoint` instead of the service DNS name.
Rook always uses the default endpoint to perform management operations against the object store.
When TLS is enabled, the TLS certificate must always specify the default endpoint DNS name to allow
secure management operations. TLS configuration specification can be found in object
[gateway `securePort` documentation](../../CRDs/Object-Storage/ceph-object-store-crd.md#gateway-settings).
The advertised endpoint can be overridden using `advertiseEndpoint` in the
[`spec.hosting` config](../../CRDs/Object-Storage/ceph-object-store-crd.md#hosting-settings).
Rook always uses the advertised endpoint to perform management operations against the object store.
When [TLS is enabled](#enable-tls), the TLS certificate must always specify the endpoint DNS name to
allow secure management operations.
## Create a Bucket
@@ -508,6 +501,29 @@ kubectl -n rook-ceph get secret rook-ceph-object-user-my-store-my-user -o jsonpa
kubectl -n rook-ceph get secret rook-ceph-object-user-my-store-my-user -o jsonpath='{.data.SecretKey}' | base64 --decode
```
## Enable TLS
TLS is critical for securing object storage data access, and it is assumed as a default by many S3
clients. TLS is enabled for CephObjectStores by configuring
[`gateway` options](../../CRDs/Object-Storage/ceph-object-store-crd.md#gateway-settings).
Set `securePort`, and give Rook access to a TLS certificate using `sslCertificateRef`.
`caBundleRef` may be necessary as well to give the deployed gateway (RGW) access to the TLS
certificate's CA signing bundle.
Ceph RGW only supports a **single** TLS certificate. If the given TLS certificate is a concatenation
of multiple certificates, only the first certificate will be used by the RGW as the server
certificate. Therefore, the TLS certificate given must include all endpoints that clients will use
for access as subject alternate names (SANs).
The [CephObjectStore service endpoint](#object-store-endpoint) must be added as a SAN on the TLS
certificate. If it is not possible to add the service DNS name as a SAN on the TLS certificate,
set `hosting.advertiseEndpoint` to a TLS-approved endpoint to help ensure Rook and clients use
secure data access.
!!! note
OpenShift users can use add `service.beta.openshift.io/serving-cert-secret-name` as a service
annotation instead of using `sslCertificateRef`.
## Virtual host-style Bucket Access
The Ceph Object Gateway supports accessing buckets using
@@ -530,7 +546,7 @@ Wildcard addressing can be configured in myriad ways. Some options:
The minimum recommended `hosting` configuration is exemplified below. It is important to ensure that
Rook advertises the wildcard-addressable endpoint as a priority over the default. TLS is also
recommended for security.
recommended for security, and the configured TLS certificate should specify the advertise endpoint.
```yaml
spec: