ceph: add rgw endpoint healthcheck

We have introduced a new goroutine to check the state of the rgw
endpoint. It will run every minute and perform operations on a bucket.
The success or failure will be reported as part of the status field of
the CephObjectStore CR.

A good status will look like:

status:
  endpointStatus:
    lastChanged: "2020-06-25T13:47:45Z"
    lastChecked: "2020-06-25T13:48:46Z"
  phase: Connected

A failed status:

status:
  endpointStatus:
    details: |-
      error creating bucket "rook-ceph-internal-s3-bucket-checker": RequestError: send request failed
      caused by: Put http://rook-ceph-rgw-my-store.rook-ceph:8080/rook-ceph-internal-s3-bucket-checker: dial tcp 10.108.189.148:8080: connect: connection refused
    health: ERROR

This check works for both converged and external modes. Note that the
CephObjectStore CRD has a new field called "externalRgwEndpoints" which
allows you to define a list of IP addresses pointing to rgws.

Closes: https://github.com/rook/rook/issues/5692
Signed-off-by: Sébastien Han <seb@redhat.com>
This commit is contained in:
Sébastien Han committed 2020-07-02 16:34:10 +02:00
1 parent 7229f5826e
commit e4eaa91ede
34 files changed
+1318 -418

No files matched your search

+40
View File
@@ -94,11 +94,24 @@ The gateway settings correspond to the RGW daemon settings.
* `port`: The port on which the Object service will be reachable. If host networking is enabled, the RGW daemons will also listen on that port. If running on SDN, the RGW daemon listening port will be 8080 internally.
* `securePort`: The secure port on which RGW pods will be listening. An SSL certificate must be specified.
* `instances`: The number of pods that will be started to load balance this object store.
* `externalRgwEndpoints`: A list of IP addresses to connect to external existing Rados Gateways (works with external mode). This setting will be ignored if the `CephCluster` does not have `external` spec enabled. Refer to the [external cluster section](ceph-cluster-crd.md#external-cluster) for more details.
* `annotations`: Key value pair list of annotations to add.
* `placement`: The Kubernetes placement settings to determine where the RGW pods should be started in the cluster.
* `resources`: Set resource requests/limits for the Gateway Pod(s), see [Resource Requirements/Limits](ceph-cluster-crd.md#resource-requirementslimits).
* `priorityClassName`: Set priority class name for the Gateway Pod(s)
Example of external rgw endpoints to connect to:
```yaml
gateway:
port: 80
externalRgwEndpoints:
- ip: 192.168.39.182
```
This will create a service with the endpoint `192.168.39.182` on port `80`, pointing to the Ceph object external gateway.
All the other settings from the gateway section will be ignored, except for `securePort`.
## Zone Settings
The [zone](ceph-object-multisite.md) settings allow the object store to join custom created [ceph-object-zone](ceph-object-multisite-crd.md).
@@ -119,3 +132,30 @@ execution attack.
Rook will not overwrite an existing `mime.types` ConfigMap so that user modifications will not be
destroyed. If the object store is destroyed and recreated, the ConfigMap will also be destroyed and
created anew.
## Health settings
Rook-Ceph will be default monitor the state of the object store endpoints.
The following CRD settings are available:
* `healthCheck`: main object store health monitoring section
Here is a complete example:
```yaml
healthCheck:
bucket:
enabled: true
interval: 60s
```
The endpoint health check procedure is the following:
1. Create an S3 user
2. Create a bucket with that user
3. PUT the file in the object store
4. GET the file from the object store
5. Verify object consistency
6. Update CR health status check
Rook-Ceph always keeps the bucket and the user for the health check, it just does a PUT and GET of an s3 object since creating a bucket is an expensive operation.
+76 -6
View File
@@ -12,7 +12,14 @@ Object storage exposes an S3 API to the storage cluster for applications to put
This guide assumes a Rook cluster as explained in the [Quickstart](ceph-quickstart.md).
## Create an Object Store
## Configure an Object Store
Rook has the ability to either deploy an object store in Kubernetes or to connect to an external RGW service.
Most commonly, the object store will be configured locally by Rook.
Alternatively, if you have an existing Ceph cluster with Rados Gateways, see the
[external section](#connect-to-an-external-object-store) to consume it from Rook.
### Create a Local Object Store
The below sample will create a `CephObjectStore` that starts the RGW service in the cluster with an S3 API.
@@ -45,6 +52,10 @@ spec:
port: 80
securePort:
instances: 1
healthCheck:
bucket:
enabled: true
interval: 60s
```
After the `CephObjectStore` is created, the Rook operator will then create all the pools and other resources necessary to start the service. This may take a minute to complete.
@@ -57,7 +68,61 @@ kubectl create -f object.yaml
kubectl -n rook-ceph get pod -l app=rook-ceph-rgw
```
### Connect to an External Object Store
Rook can connect to existing RGW gateways to work in conjunction with the external mode of the `CephCluster` CRD.
If you have an external `CephCluster` CR, you can instruct Rook to consume external gateways with the following:
```yaml
apiVersion: ceph.rook.io/v1
kind: CephObjectStore
metadata:
name: external-store
namespace: rook-ceph
spec:
gateway:
port: 8080
externalRgwEndpoints:
- ip: 192.168.39.182
healthCheck:
bucket:
enabled: true
interval: 60s
```
You can use the existing `object-external.yaml` file.
When ready the ceph-object-controller will output a message in the Operator log similar to this one:
```log
ceph-object-controller: ceph object store gateway service running at 10.100.28.138:8080
```
You can now get and access the store via:
```console
kubectl -n rook-ceph get svc -l app=rook-ceph-rgw
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
rook-ceph-rgw-my-store ClusterIP 10.100.28.138 <none> 8080/TCP 6h59m
```
Any pod from your cluster can now access this endpoint:
```console
curl 10.100.28.138:8080
<?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>
```
It is also possible to use the internally registered DNS name:
```console
curl rook-ceph-rgw-my-store.rook-ceph:8080
<?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>
```
The DNS name is created with the following schema `rook-ceph-rgw-$STORE_NAME.$NAMESPACE`.
## Create a Bucket
Now that the object store is configured, next we need to create a bucket where a client can read and write objects. A bucket can be created by defining a storage class, similar to the pattern used by block and file storage.
First, define the storage class that will allow object clients to create a bucket.
The storage class defines the object storage system, the bucket retention policy, and other properties required by the administrator. Save the following as `storageclass-bucket-delete.yaml` (the example is named as such due to the `Delete` reclaim policy).
@@ -74,7 +139,8 @@ parameters:
objectStoreNamespace: rook-ceph
region: us-east-1
```
```bash
```console
kubectl create -f storageclass-bucket-delete.yaml
```
@@ -92,9 +158,11 @@ spec:
generateBucketName: ceph-bkt
storageClassName: rook-ceph-bucket
```
```bash
```console
kubectl create -f object-bucket-claim-delete.yaml
```
Now that the claim is created, the operator will create the bucket as well as generate other artifacts to enable access to the bucket. A secret and ConfigMap are created with the same name as the OBC and in the same namespace.
The secret contains credentials used by the application pod to access the bucket.
The ConfigMap contains bucket endpoint information and is also consumed by the pod.
@@ -151,7 +219,8 @@ below in the section [creating a user](#create-a-user) if you are not creating t
### Install s3cmd
To test the `CephObjectStore` we will install the `s3cmd` tool into the toolbox pod.
```bash
```console
yum --assumeyes install s3cmd
```
@@ -249,7 +318,7 @@ spec:
When the `CephObjectStoreUser` is created, the Rook operator will then create the RGW user on the specified `CephObjectStore` and store the Access Key and Secret Key in a kubernetes secret in the same namespace as the `CephObjectStoreUser`.
```bash
```console
# Create the object store user
kubectl create -f object-user.yaml
@@ -275,7 +344,8 @@ The AccessKey and SecretKey data fields can be mounted in a pod as an environmen
kubernetes secrets can be found in the [K8s secret documentation](https://kubernetes.io/docs/concepts/configuration/secret/)
To directly retrieve the secrets:
```bash
```console
kubectl -n rook-ceph get secret rook-ceph-object-user-my-store-my-user -o yaml | grep AccessKey | awk '{print $2}' | base64 --decode
kubectl -n rook-ceph get secret rook-ceph-object-user-my-store-my-user -o yaml | grep SecretKey | awk '{print $2}' | base64 --decode
```
+6 -2
View File
@@ -13,10 +13,14 @@
- OSD changes:
- OSD on PVC now supports multipath device.
- Added [admission controller](Documentation/admission-controller-usage.md) support for CRD validations.
- Support for Ceph CRDs is provided. Some validations for CephClusters are included and additional validations can be added for other CRDs
- Can be extended to add support for other providers
- Support for Ceph CRDs is provided. Some validations for CephClusters are included and additional validations can be added for other CRDs
- Can be extended to add support for other providers
- OBC changes:
- Updated lib bucket provisioner version to support multithread and required change can be found in [operator.yaml](cluster/examples/kubernetes/ceph/operator.yaml#L449)
- Can be extended to add support for other providers
- The Rook operator reflects the health of the CephObjectStore in its status field
- The CephObjectStore CR supports connecting to external Ceph Rados Gateways, refer to the [external object section](Documentation/ceph-object.html#connect-to-external-object-store)
- The CephObjectStore CR runs health checks on the object store endpoint, refer to the [health check section](Documentation/ceph-object-store-crd.html#health-settings)
### EdgeFS
@@ -407,6 +407,15 @@ spec:
- force
preservePoolsOnDelete:
type: boolean
healthCheck:
properties:
bucket:
properties:
enabled:
type: boolean
interval:
type: string
subresources:
subresources:
status: {}
---
@@ -431,6 +431,12 @@ spec:
securePort: {}
instances:
type: integer
externalRgwEndpoints:
type: array
items:
properties:
ip:
type: string
annotations: {}
placement: {}
resources: {}
@@ -484,6 +490,14 @@ spec:
- force
preservePoolsOnDelete:
type: boolean
healthCheck:
properties:
bucket:
properties:
enabled:
type: boolean
interval:
type: string
subresources:
status: {}
# OLM: END CEPH OBJECT STORE CRD
@@ -289,6 +289,7 @@ class RadosJSON:
"osd", ("allow rwx pool={0}.rgw.meta, " +
"allow r pool=.rgw.root, " +
"allow rw pool={0}.rgw.control, " +
"allow rx pool={0}.rgw.log, " +
"allow x pool={0}.rgw.buckets.index").format(self._arg_parser.rgw_pool_prefix)],
"format": "json"}
ret_val, json_out, err_msg = self._common_cmd_json_gen(cmd_json)
@@ -32,7 +32,7 @@ fi
}
function createCheckerKey() {
checkerKey=$(ceph auth get-or-create "$CLIENT_CHECKER_NAME" mon 'allow r, allow command quorum_status' osd 'allow rwx pool='"$RGW_POOL_PREFIX"'.rgw.meta, allow r pool=.rgw.root, allow rw pool='"$RGW_POOL_PREFIX"'.rgw.control, allow x pool='"$RGW_POOL_PREFIX"'.rgw.buckets.index'|awk '/key =/ { print $3}')
checkerKey=$(ceph auth get-or-create "$CLIENT_CHECKER_NAME" mon 'allow r, allow command quorum_status' osd 'allow rwx pool='"$RGW_POOL_PREFIX"'.rgw.meta, allow r pool=.rgw.root, allow rw pool='"$RGW_POOL_PREFIX"'.rgw.control, allow x pool='"$RGW_POOL_PREFIX"'.rgw.buckets.index, allow x pool='"$RGW_POOL_PREFIX"'.rgw.log'|awk '/key =/ { print $3}')
echo "export ROOK_EXTERNAL_USER_SECRET=$checkerKey"
echo "export ROOK_EXTERNAL_USERNAME=$CLIENT_CHECKER_NAME"
}
@@ -0,0 +1,22 @@
#################################################################################################################
# Create an object store with settings for replication in a production environment. A minimum of 3 hosts with
# OSDs are required in this example.
# kubectl create -f object.yaml
#################################################################################################################
apiVersion: ceph.rook.io/v1
kind: CephObjectStore
metadata:
name: external-store
namespace: rook-ceph
spec:
gateway:
# The port on which **ALL** the gateway(s) are listening on.
# Passing a single IP from a load-balancer is also valid.
port: 80
externalRgwEndpoints:
- ip: 192.168.39.182
healthCheck:
bucket:
enabled: true
interval: 60s
@@ -84,3 +84,8 @@ spec:
# priorityClassName: my-priority-class
#zone:
#name: zone-a
# service endpoint healthcheck
healthCheck:
endpoint:
enabled: true
interval: 60s
@@ -56,7 +56,7 @@ data:
# (Optional) set user created priorityclassName for csi provisioner pods.
# CSI_PROVISIONER_PRIORITY_CLASSNAME: "system-cluster-critical"
# CSI CephFS plugin daemonset update strategy, supported values are OnDelete and RollingUpdate.
# Default value is RollingUpdate.
# CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY: "OnDelete"
@@ -207,4 +207,104 @@ spec:
- name: Health
type: string
description: Ceph Health
JSONPath: .status.ceph.health
JSONPath: .status.ceph.health
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: cephobjectstores.ceph.rook.io
spec:
group: ceph.rook.io
names:
kind: CephObjectStore
listKind: CephObjectStoreList
plural: cephobjectstores
singular: cephobjectstore
scope: Namespaced
version: v1
validation:
openAPIV3Schema:
properties:
spec:
properties:
gateway:
properties:
type:
type: string
sslCertificateRef: {}
port:
type: integer
minimum: 1
maximum: 65535
securePort: {}
instances:
type: integer
externalRgwEndpoints:
type: array
items:
properties:
ip:
type: string
annotations: {}
placement: {}
resources: {}
metadataPool:
properties:
failureDomain:
type: string
replicated:
properties:
size:
type: integer
requireSafeReplicaSize:
type: boolean
erasureCoded:
properties:
dataChunks:
type: integer
codingChunks:
type: integer
compressionMode:
type: string
enum:
- ""
- none
- passive
- aggressive
- force
dataPool:
properties:
failureDomain:
type: string
replicated:
properties:
size:
type: integer
requireSafeReplicaSize:
type: boolean
erasureCoded:
properties:
dataChunks:
type: integer
codingChunks:
type: integer
compressionMode:
type: string
enum:
- ""
- none
- passive
- aggressive
- force
preservePoolsOnDelete:
type: boolean
healthCheck:
properties:
bucket:
properties:
enabled:
type: boolean
interval:
type: string
subresources:
status: {}
+31 -2
View File
@@ -190,6 +190,7 @@ const (
ConditionFailure ConditionType = "Failure"
ConditionUpgrading ConditionType = "Upgrading"
ConditionDeleting ConditionType = "Deleting"
ConditionHealthy ConditionType = "Healthy"
// DefaultFailureDomain for PoolSpec
DefaultFailureDomain = "host"
)
@@ -364,8 +365,8 @@ type MetadataServerSpec struct {
type CephObjectStore struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata"`
Spec ObjectStoreSpec `json:"spec"`
Status *Status `json:"status"`
Spec ObjectStoreSpec `json:"spec"`
Status *ObjectStoreStatus `json:"status"`
}
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
@@ -392,6 +393,18 @@ type ObjectStoreSpec struct {
// The multisite info
Zone ZoneSpec `json:"zone"`
// The rgw endpoint healthcheck
HealthCheck HealthCheckSpec `json:"healthCheck"`
}
type HealthCheckSpec struct {
Bucket BucketHealthCheckSpec `json:"bucket,omitempty"`
}
type BucketHealthCheckSpec struct {
Enabled bool `json:"enabled,omitempty"`
Interval string `json:"interval,omitempty"`
}
type GatewaySpec struct {
@@ -421,6 +434,9 @@ type GatewaySpec struct {
// PriorityClassName sets priority classes on the rgw pods
PriorityClassName string `json:"priorityClassName,omitempty"`
// ExternalRgwEndpoints points to external rgw endpoint(s)
ExternalRgwEndpoints []v1.EndpointAddress `json:"externalRgwEndpoints,omitempty"`
}
type ZoneSpec struct {
@@ -428,6 +444,19 @@ type ZoneSpec struct {
Name string `json:"name"`
}
type ObjectStoreStatus struct {
Phase ConditionType `json:"phase,omitempty"`
Message string `json:"message,omitempty"`
BucketStatus *BucketStatus `json:"bucketStatus,omitempty"`
}
type BucketStatus struct {
Health ConditionType `json:"health,omitempty"`
Details string `json:"details,omitempty"`
LastChecked string `json:"lastChecked,omitempty"`
LastChanged string `json:"lastChanged,omitempty"`
}
// +genclient
// +genclient:noStatus
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
@@ -26,6 +26,38 @@ import (
runtime "k8s.io/apimachinery/pkg/runtime"
)
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *BucketHealthCheckSpec) DeepCopyInto(out *BucketHealthCheckSpec) {
*out = *in
return
}
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new BucketHealthCheckSpec.
func (in *BucketHealthCheckSpec) DeepCopy() *BucketHealthCheckSpec {
if in == nil {
return nil
}
out := new(BucketHealthCheckSpec)
in.DeepCopyInto(out)
return out
}
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *BucketStatus) DeepCopyInto(out *BucketStatus) {
*out = *in
return
}
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new BucketStatus.
func (in *BucketStatus) DeepCopy() *BucketStatus {
if in == nil {
return nil
}
out := new(BucketStatus)
in.DeepCopyInto(out)
return out
}
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *CephBlockPool) DeepCopyInto(out *CephBlockPool) {
*out = *in
@@ -431,8 +463,8 @@ func (in *CephObjectStore) DeepCopyInto(out *CephObjectStore) {
in.Spec.DeepCopyInto(&out.Spec)
if in.Status != nil {
in, out := &in.Status, &out.Status
*out = new(Status)
**out = **in
*out = new(ObjectStoreStatus)
(*in).DeepCopyInto(*out)
}
return
}
@@ -1116,6 +1148,13 @@ func (in *GatewaySpec) DeepCopyInto(out *GatewaySpec) {
}
}
in.Resources.DeepCopyInto(&out.Resources)
if in.ExternalRgwEndpoints != nil {
in, out := &in.ExternalRgwEndpoints, &out.ExternalRgwEndpoints
*out = make([]corev1.EndpointAddress, len(*in))
for i := range *in {
(*in)[i].DeepCopyInto(&(*out)[i])
}
}
return
}
@@ -1129,6 +1168,23 @@ func (in *GatewaySpec) DeepCopy() *GatewaySpec {
return out
}
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *HealthCheckSpec) DeepCopyInto(out *HealthCheckSpec) {
*out = *in
out.Bucket = in.Bucket
return
}
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new HealthCheckSpec.
func (in *HealthCheckSpec) DeepCopy() *HealthCheckSpec {
if in == nil {
return nil
}
out := new(HealthCheckSpec)
in.DeepCopyInto(out)
return out
}
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *MetadataServerSpec) DeepCopyInto(out *MetadataServerSpec) {
*out = *in
@@ -1286,6 +1342,7 @@ func (in *ObjectStoreSpec) DeepCopyInto(out *ObjectStoreSpec) {
in.DataPool.DeepCopyInto(&out.DataPool)
in.Gateway.DeepCopyInto(&out.Gateway)
out.Zone = in.Zone
out.HealthCheck = in.HealthCheck
return
}
@@ -1299,6 +1356,27 @@ func (in *ObjectStoreSpec) DeepCopy() *ObjectStoreSpec {
return out
}
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *ObjectStoreStatus) DeepCopyInto(out *ObjectStoreStatus) {
*out = *in
if in.BucketStatus != nil {
in, out := &in.BucketStatus, &out.BucketStatus
*out = new(BucketStatus)
**out = **in
}
return
}
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ObjectStoreStatus.
func (in *ObjectStoreStatus) DeepCopy() *ObjectStoreStatus {
if in == nil {
return nil
}
out := new(ObjectStoreStatus)
in.DeepCopyInto(out)
return out
}
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *ObjectStoreUserSpec) DeepCopyInto(out *ObjectStoreUserSpec) {
*out = *in
+6 -1
View File
@@ -29,6 +29,7 @@ type Context struct {
Name string
ClusterName string
RunAsUser string
UID string
}
// NewContext creates a new object store context.
@@ -52,7 +53,11 @@ func runAdminCommand(c *Context, args ...string) (string, error) {
// If the objectStoreName is not passed in the storage class
// This means we are pointing to an external cluster so these commands are not needed
// simply because the external cluster mode does not support that yet
if c.Name != "" {
//
// The following conditions tries to determine if the cluster is external
// When connecting to an external cluster, the Ceph user is different than client.admin
// This is not perfect though since "client.admin" is somehow supported...
if c.Name != "" && c.RunAsUser == client.AdminUsername {
options := []string{
fmt.Sprintf("--rgw-realm=%s", c.Name),
fmt.Sprintf("--rgw-zonegroup=%s", c.Name),
+1 -1
View File
@@ -211,7 +211,7 @@ func GetBucket(c *Context, bucket string) (*ObjectBucket, int, error) {
return &ObjectBucket{Name: bucket, ObjectBucketMetadata: ObjectBucketMetadata{Owner: metadata.Owner, CreatedAt: metadata.CreatedAt}, ObjectBucketStats: *stat}, RGWErrorNone, nil
}
func DeleteBucket(c *Context, bucketName string, purge bool) (int, error) {
func DeleteObjectBucket(c *Context, bucketName string, purge bool) (int, error) {
options := []string{"bucket", "rm", "--bucket", bucketName}
if purge {
options = append(options, "--purge-objects")
+11 -11
View File
@@ -84,7 +84,7 @@ func (p Provisioner) Provision(options *apibkt.BucketOptions) (*bktv1alpha1.Obje
return nil, errors.Wrap(err, "Provision: can't create ceph user")
}
s3svc, err := NewS3Agent(p.accessKeyID, p.secretAccessKey, p.getObjectStoreEndpoint())
s3svc, err := cephObject.NewS3Agent(p.accessKeyID, p.secretAccessKey, p.getObjectStoreEndpoint())
if err != nil {
return nil, err
}
@@ -149,7 +149,7 @@ func (p Provisioner) Grant(options *apibkt.BucketOptions) (*bktv1alpha1.ObjectBu
return nil, errors.Wrapf(err, "could not get user (user: %s)", stats.Owner)
}
s3svc, err := NewS3Agent(*objectUser.AccessKey, *objectUser.SecretKey, p.getObjectStoreEndpoint())
s3svc, err := cephObject.NewS3Agent(*objectUser.AccessKey, *objectUser.SecretKey, p.getObjectStoreEndpoint())
if err != nil {
p.deleteOBCResource("")
return nil, err
@@ -166,15 +166,15 @@ func (p Provisioner) Grant(options *apibkt.BucketOptions) (*bktv1alpha1.ObjectBu
}
}
statement := NewPolicyStatement().
statement := cephObject.NewPolicyStatement().
WithSID(p.cephUserName).
ForPrincipals(p.cephUserName).
ForResources(p.bucketName).
ForSubResources(p.bucketName).
Allows().
Actions(AllowedActions...)
Actions(cephObject.AllowedActions...)
if policy == nil {
policy = NewBucketPolicy(*statement)
policy = cephObject.NewBucketPolicy(*statement)
} else {
policy = policy.ModifyBucketPolicy(*statement)
}
@@ -240,7 +240,7 @@ func (p Provisioner) Revoke(ob *bktv1alpha1.ObjectBucket) error {
return nil
}
s3svc, err := NewS3Agent(*user.AccessKey, *user.SecretKey, p.getObjectStoreEndpoint())
s3svc, err := cephObject.NewS3Agent(*user.AccessKey, *user.SecretKey, p.getObjectStoreEndpoint())
if err != nil {
return err
}
@@ -260,15 +260,15 @@ func (p Provisioner) Revoke(ob *bktv1alpha1.ObjectBucket) error {
}
if bucket.Owner == p.cephUserName {
statement := NewPolicyStatement().
statement := cephObject.NewPolicyStatement().
WithSID(p.cephUserName).
ForPrincipals(p.cephUserName).
ForResources(p.bucketName).
ForSubResources(p.bucketName).
Denies().
Actions(AllowedActions...)
Actions(cephObject.AllowedActions...)
if policy == nil {
policy = NewBucketPolicy(*statement)
policy = cephObject.NewBucketPolicy(*statement)
} else {
policy = policy.ModifyBucketPolicy(*statement)
}
@@ -420,13 +420,13 @@ func (p *Provisioner) setObjectStoreDomainName(sc *storagev1.StorageClass) error
if err != nil {
return err
}
p.storeDomainName = fmt.Sprintf("%s-%s.%s", prefixObjectStoreSvc, name, namespace)
p.storeDomainName = cephObject.BuildDomainName(name, namespace)
return nil
}
func (p *Provisioner) setObjectStorePort(sc *storagev1.StorageClass) error {
name := getObjectStoreName(sc)
name = fmt.Sprintf("%s-%s", prefixObjectStoreSvc, name)
name = fmt.Sprintf("%s-%s", cephObject.AppName, name)
namespace := getObjectStoreNameSpace(sc)
// also ensure the service exists and get the appropriate clusterIP port
svc, err := getService(p.context.Clientset, namespace, name)
@@ -24,6 +24,7 @@ import (
rookclient "github.com/rook/rook/pkg/client/clientset/versioned/fake"
"github.com/rook/rook/pkg/clusterd"
"github.com/rook/rook/pkg/daemon/ceph/client"
"github.com/rook/rook/pkg/operator/ceph/object"
"github.com/rook/rook/pkg/operator/test"
"github.com/stretchr/testify/assert"
v1 "k8s.io/api/core/v1"
@@ -85,7 +86,7 @@ func TestPopulateDomainAndPort(t *testing.T) {
}
svc := &v1.Service{
ObjectMeta: metav1.ObjectMeta{
Name: fmt.Sprintf("%s-%s", prefixObjectStoreSvc, store),
Name: fmt.Sprintf("%s-%s", object.AppName, store),
Namespace: namespace,
},
Spec: v1.ServiceSpec{
@@ -100,7 +100,7 @@ func (p *Provisioner) deleteOBCResource(bucketName string) error {
}
if len(bucketName) > 0 {
// delete bucket with purge option to remove all objects
errCode, err := cephObject.DeleteBucket(p.objectContext, bucketName, true)
errCode, err := cephObject.DeleteObjectBucket(p.objectContext, bucketName, true)
if errCode == cephObject.RGWErrorNone {
logger.Infof("bucket %q successfully deleted", p.bucketName)
-1
View File
@@ -41,7 +41,6 @@ var logger = capnslog.NewPackageLogger("github.com/rook/rook", "op-bucket-prov")
const (
genUserLen = 8
cephUser = "cephUser"
prefixObjectStoreSvc = "rook-ceph-rgw"
accessKeyIdKey = "accessKeyID"
secretSecretKeyKey = "secretAccessKey"
objectStoreName = "objectStoreName"
+152 -98
View File
@@ -20,6 +20,7 @@ import (
"context"
"fmt"
"reflect"
"strconv"
"strings"
"syscall"
"time"
@@ -34,7 +35,6 @@ import (
"github.com/rook/rook/pkg/operator/ceph/cluster/mon"
opconfig "github.com/rook/rook/pkg/operator/ceph/config"
opcontroller "github.com/rook/rook/pkg/operator/ceph/controller"
"github.com/rook/rook/pkg/operator/k8sutil"
"github.com/rook/rook/pkg/util/exec"
appsv1 "k8s.io/api/apps/v1"
corev1 "k8s.io/api/core/v1"
@@ -76,12 +76,18 @@ var controllerTypeMeta = metav1.TypeMeta{
// ReconcileCephObjectStore reconciles a cephObjectStore object
type ReconcileCephObjectStore struct {
client client.Client
bktclient bktclient.Interface
scheme *runtime.Scheme
context *clusterd.Context
cephClusterSpec *cephv1.ClusterSpec
clusterInfo *cephconfig.ClusterInfo
client client.Client
bktclient bktclient.Interface
scheme *runtime.Scheme
context *clusterd.Context
cephClusterSpec *cephv1.ClusterSpec
clusterInfo *cephconfig.ClusterInfo
objectStoreChannels map[string]*objectStoreHealth
}
type objectStoreHealth struct {
stopChan chan struct{}
monitoringRunning bool
}
// Add creates a new cephObjectStore Controller and adds it to the Manager. The Manager will set fields on the Controller
@@ -97,10 +103,11 @@ func newReconciler(mgr manager.Manager, context *clusterd.Context) reconcile.Rec
cephv1.AddToScheme(mgr.GetScheme())
return &ReconcileCephObjectStore{
client: mgr.GetClient(),
scheme: mgrScheme,
context: context,
bktclient: bktclient.NewForConfigOrDie(context.KubeConfig),
client: mgr.GetClient(),
scheme: mgrScheme,
context: context,
bktclient: bktclient.NewForConfigOrDie(context.KubeConfig),
objectStoreChannels: make(map[string]*objectStoreHealth),
}
}
@@ -161,7 +168,7 @@ func (r *ReconcileCephObjectStore) reconcile(request reconcile.Request) (reconci
// The CR was just created, initializing status fields
if cephObjectStore.Status == nil {
updateStatus(r.client, request.NamespacedName, k8sutil.Created)
updateStatusPhase(r.client, request.NamespacedName, cephv1.ConditionProgressing)
}
// Make sure a CephCluster is present otherwise do nothing
@@ -188,20 +195,39 @@ func (r *ReconcileCephObjectStore) reconcile(request reconcile.Request) (reconci
}
r.cephClusterSpec = &cephCluster.Spec
// Initialize the channel for this object store
// This allows us to track multiple ObjectStores in the same namespace
_, ok := r.objectStoreChannels[cephObjectStore.Name]
if !ok {
r.objectStoreChannels[cephObjectStore.Name] = &objectStoreHealth{
stopChan: make(chan struct{}),
monitoringRunning: false,
}
}
// Populate clusterInfo
// Always populate it during each reconcile
clusterInfo, _, _, err := mon.LoadClusterInfo(r.context, request.NamespacedName.Namespace)
if err != nil {
return reconcile.Result{}, errors.Wrap(err, "failed to populate cluster info")
var clusterInfo *cephconfig.ClusterInfo
if r.cephClusterSpec.External.Enable {
clusterInfo = mon.PopulateExternalClusterInfo(r.context, request.NamespacedName.Namespace)
} else {
clusterInfo, _, _, err = mon.LoadClusterInfo(r.context, request.NamespacedName.Namespace)
if err != nil {
return reconcile.Result{}, errors.Wrap(err, "failed to populate cluster info")
}
}
r.clusterInfo = clusterInfo
// Populate CephVersion
currentCephVersion, err := cephclient.LeastUptodateDaemonVersion(r.context, r.clusterInfo.Name, opconfig.MonType)
if err != nil {
return reconcile.Result{}, errors.Wrapf(err, "failed to retrieve current ceph %q version", opconfig.MonType)
// This only needed when bootstrapping rgw pods
// Might improve if we set the version by running a 'ceph version' command on a goroutine and update it every now and then
if !r.cephClusterSpec.External.Enable {
currentCephVersion, err := cephclient.LeastUptodateDaemonVersion(r.context, r.clusterInfo.Name, opconfig.MonType)
if err != nil {
return reconcile.Result{}, errors.Wrapf(err, "failed to retrieve current ceph %q version", opconfig.MonType)
}
r.clusterInfo.CephVersion = currentCephVersion
}
r.clusterInfo.CephVersion = currentCephVersion
// Set a finalizer so we can do cleanup before the object goes away
err = opcontroller.AddFinalizerIfNotPresent(r.client, cephObjectStore)
@@ -220,7 +246,17 @@ func (r *ReconcileCephObjectStore) reconcile(request reconcile.Request) (reconci
return response, nil
}
cfg := clusterConfig{context: r.context, store: cephObjectStore}
// Close the channel to stop the healthcheck of the endpoint
close(r.objectStoreChannels[cephObjectStore.Name].stopChan)
// Remove object store from the map
delete(r.objectStoreChannels, cephObjectStore.Name)
cfg := clusterConfig{
context: r.context,
store: cephObjectStore,
clusterSpec: r.cephClusterSpec,
}
err = cfg.deleteStore()
if err != nil {
return reconcile.Result{}, errors.Wrapf(err, "failed to delete store %q", cephObjectStore.Name)
@@ -237,35 +273,25 @@ func (r *ReconcileCephObjectStore) reconcile(request reconcile.Request) (reconci
}
// validate the store settings
if err := validateStore(r.context, cephObjectStore); err != nil {
if err := r.validateStore(cephObjectStore); err != nil {
return reconcile.Result{}, errors.Wrapf(err, "invalid object store %q arguments", cephObjectStore.Name)
}
// CREATE/UPDATE
logger.Info("reconciling object store deployments")
reconcileResponse, err = r.reconcileCreateObjectStore(cephObjectStore, request.NamespacedName)
if err != nil {
return r.setFailedStatus(request.NamespacedName, "failed to create object store deployments", err)
}
// Set Ready status, we are done reconciling
updateStatus(r.client, request.NamespacedName, k8sutil.ReadyStatus)
updateStatusPhase(r.client, request.NamespacedName, cephv1.ConditionReady)
// Return and do not requeue
logger.Debug("done reconciling")
return reconcile.Result{}, nil
}
func (r *ReconcileCephObjectStore) reconcileCreateObjectStore(cephObjectStore *cephv1.CephObjectStore, name types.NamespacedName) (reconcile.Result, error) {
if r.cephClusterSpec.External.Enable {
_, err := opcontroller.ValidateCephVersionsBetweenLocalAndExternalClusters(r.context, cephObjectStore.Namespace, r.clusterInfo.CephVersion)
if err != nil {
// This handles the case where the operator is running, the external cluster has been upgraded and a CR creation is called
// If that's a major version upgrade we fail, if it's a minor version, we continue, it's not ideal but not critical
return reconcile.Result{}, errors.Wrap(err, "refusing to run new crd")
}
}
func (r *ReconcileCephObjectStore) reconcileCreateObjectStore(cephObjectStore *cephv1.CephObjectStore, namespacedName types.NamespacedName) (reconcile.Result, error) {
cfg := clusterConfig{
context: r.context,
clusterInfo: r.clusterInfo,
@@ -277,47 +303,80 @@ func (r *ReconcileCephObjectStore) reconcileCreateObjectStore(cephObjectStore *c
scheme: r.scheme,
skipUpgradeChecks: r.cephClusterSpec.SkipUpgradeChecks,
}
objContext := NewContext(r.context, cephObjectStore.Name, cephObjectStore.Namespace)
objContext.UID = string(cephObjectStore.UID)
// Reconcile realm/zonegroup/zone CRs & update their names
realmName, zoneGroupName, zoneName, reconcileResponse, err := r.reconcileMultisiteCRs(cephObjectStore)
if err != nil {
return reconcileResponse, err
}
var serviceIP string
var err error
// Reconcile Ceph Zone if Multisite
if cephObjectStore.Spec.IsMultisite() {
reconcileResponse, err := r.reconcileCephZone(cephObjectStore, zoneGroupName, realmName)
if r.cephClusterSpec.External.Enable {
// Assign the cephx user to run Ceph commands with
objContext.RunAsUser = r.clusterInfo.ExternalCred.Username
logger.Info("reconciling external object store")
// RECONCILE SERVICE
logger.Info("reconciling object store service")
serviceIP, err = cfg.reconcileService(cephObjectStore)
if err != nil {
return r.setFailedStatus(namespacedName, "failed to reconcile service", err)
}
// RECONCILE ENDPOINTS
// Always add the endpoint AFTER the service otherwise it will get overridden
logger.Info("reconciling external object store endpoint")
err = cfg.reconcileExternalEndpoint(cephObjectStore)
if err != nil {
return r.setFailedStatus(namespacedName, "failed to reconcile external endpoint", err)
}
} else {
logger.Info("reconciling object store deployments")
// Reconcile realm/zonegroup/zone CRs & update their names
realmName, zoneGroupName, zoneName, reconcileResponse, err := r.reconcileMultisiteCRs(cephObjectStore)
if err != nil {
return reconcileResponse, err
}
// Reconcile Ceph Zone if Multisite
if cephObjectStore.Spec.IsMultisite() {
reconcileResponse, err := r.reconcileCephZone(cephObjectStore, zoneGroupName, realmName)
if err != nil {
return reconcileResponse, err
}
}
// RECONCILE SERVICE
logger.Debug("reconciling object store service")
serviceIP, err := cfg.reconcileService(cephObjectStore)
if err != nil {
return r.setFailedStatus(namespacedName, "failed to reconcile service", err)
}
// RECONCILE POOLS
logger.Info("reconciling object store pools")
err = createPools(objContext, cephObjectStore.Spec)
if err != nil {
return r.setFailedStatus(namespacedName, "failed to create object pools", err)
}
// RECONCILE REALM
logger.Infof("setting multisite settings for object store %q", cephObjectStore.Name)
err = setMultisite(objContext, serviceIP, cephObjectStore.Spec, realmName, zoneGroupName, zoneName)
if err != nil {
return r.setFailedStatus(namespacedName, "failed to configure multisite for object store", err)
}
err = cfg.createOrUpdateStore(realmName, zoneGroupName, zoneName)
if err != nil {
return reconcile.Result{}, errors.Wrapf(err, "failed to create object store %q", cephObjectStore.Name)
}
}
// RECONCILE SERVICE
logger.Debug("reconciling object store service")
serviceIP, err := cfg.reconcileService(cephObjectStore)
if err != nil {
return r.setFailedStatus(name, "failed to reconcile service", err)
}
objContext := NewContext(r.context, cephObjectStore.Name, cephObjectStore.Namespace)
// RECONCILE POOLS
logger.Info("reconciling object store pools")
err = createPools(objContext, cephObjectStore.Spec)
if err != nil {
return r.setFailedStatus(name, "failed to create object pools", err)
}
// RECONCILE REALM
logger.Infof("setting multisite settings for object store %q", cephObjectStore.Name)
err = setMultisite(objContext, serviceIP, cephObjectStore.Spec, realmName, zoneGroupName, zoneName)
if err != nil {
return r.setFailedStatus(name, "failed to configure multisite for object store", err)
}
err = cfg.createOrUpdateStore(realmName, zoneGroupName, zoneName)
if err != nil {
return reconcile.Result{}, errors.Wrapf(err, "failed to create object store %q", cephObjectStore.Name)
// Start monitoring
if cephObjectStore.Spec.HealthCheck.Bucket.Enabled {
r.startMonitoring(cephObjectStore, objContext, serviceIP, namespacedName)
}
return reconcile.Result{}, nil
@@ -378,38 +437,10 @@ func (r *ReconcileCephObjectStore) reconcileMultisiteCRs(cephObjectStore *cephv1
return cephObjectStore.Name, cephObjectStore.Name, cephObjectStore.Name, reconcile.Result{}, nil
}
func (r *ReconcileCephObjectStore) setFailedStatus(name types.NamespacedName, errMessage string, err error) (reconcile.Result, error) {
updateStatus(r.client, name, k8sutil.ReconcileFailedStatus)
return reconcile.Result{}, errors.Wrapf(err, "%s", errMessage)
}
// updateStatus updates an object with a given status
func updateStatus(client client.Client, name types.NamespacedName, status string) {
objectStore := &cephv1.CephObjectStore{}
if err := client.Get(context.TODO(), name, objectStore); err != nil {
if kerrors.IsNotFound(err) {
logger.Debug("CephObjectStore resource not found. Ignoring since object must be deleted.")
return
}
logger.Warningf("failed to retrieve object store %q to update status to %q. %v", name, status, err)
return
}
if objectStore.Status == nil {
objectStore.Status = &cephv1.Status{}
}
objectStore.Status.Phase = status
if err := opcontroller.UpdateStatus(client, objectStore); err != nil {
logger.Errorf("failed to set object store %q status to %q. %v", name, status, err)
return
}
logger.Debugf("object store %q status updated to %q", name, status)
}
func (r *ReconcileCephObjectStore) verifyObjectBucketCleanup(objectstore *cephv1.CephObjectStore) (reconcile.Result, bool) {
bktProvsioner := GetObjectBucketProvisioner(r.context, objectstore.Namespace)
bktProvsioner = strings.Replace(bktProvsioner, "/", "-", -1)
selector := fmt.Sprintf("bucket-provisioner=%s", bktProvsioner)
bktProvisioner := GetObjectBucketProvisioner(r.context, objectstore.Namespace)
bktProvisioner = strings.Replace(bktProvisioner, "/", "-", -1)
selector := fmt.Sprintf("bucket-provisioner=%s", bktProvisioner)
objectBuckets, err := r.bktclient.ObjectbucketV1alpha1().ObjectBuckets().List(metav1.ListOptions{LabelSelector: selector})
if err != nil {
logger.Errorf("failed to delete object store. failed to list buckets for objectstore %q in namespace %q", objectstore.Name, objectstore.Namespace)
@@ -429,3 +460,26 @@ func (r *ReconcileCephObjectStore) verifyObjectBucketCleanup(objectstore *cephv1
logger.Errorf("failed to delete object store. buckets for objectstore %q in namespace %q are not cleaned up. remaining buckets: %+v", objectstore.Name, objectstore.Namespace, bucketNames)
return opcontroller.WaitForRequeueIfFinalizerBlocked, false
}
func (r *ReconcileCephObjectStore) startMonitoring(objectstore *cephv1.CephObjectStore, objContext *Context, serviceIP string, namespacedName types.NamespacedName) {
// Start monitoring object store
if r.objectStoreChannels[objectstore.Name].monitoringRunning {
logger.Debug("external rgw endpoint monitoring go routine already running!")
return
}
// Set the monitoring flag so we don't start more than one go routine
r.objectStoreChannels[objectstore.Name].monitoringRunning = true
var port string
if objectstore.Spec.Gateway.Port != 0 {
port = strconv.Itoa(int(objectstore.Spec.Gateway.Port))
} else if objectstore.Spec.Gateway.SecurePort != 0 {
port = strconv.Itoa(int(objectstore.Spec.Gateway.SecurePort))
}
rgwChecker := newBucketChecker(r.context, objContext, serviceIP, port, r.client, namespacedName, &objectstore.Spec.HealthCheck)
logger.Info("starting rgw healthcheck")
go rgwChecker.checkObjectStore(r.objectStoreChannels[objectstore.Name].stopChan)
}
+19 -4
View File
@@ -197,7 +197,12 @@ func TestCephObjectStoreController(t *testing.T) {
// Create a fake client to mock API calls.
cl := fake.NewFakeClientWithScheme(s, object...)
// Create a ReconcileCephObjectStore object with the scheme and fake client.
r := &ReconcileCephObjectStore{client: cl, scheme: s, context: c}
r := &ReconcileCephObjectStore{
client: cl,
scheme: s,
context: c,
objectStoreChannels: make(map[string]*objectStoreHealth),
}
// Mock request to simulate Reconcile() being called on an event for a
// watched resource .
@@ -234,7 +239,12 @@ func TestCephObjectStoreController(t *testing.T) {
// Create a fake client to mock API calls.
cl = fake.NewFakeClientWithScheme(s, object...)
// Create a ReconcileCephObjectStore object with the scheme and fake client.
r = &ReconcileCephObjectStore{client: cl, scheme: s, context: c}
r = &ReconcileCephObjectStore{
client: cl,
scheme: s,
context: c,
objectStoreChannels: make(map[string]*objectStoreHealth),
}
logger.Info("STARTING PHASE 2")
res, err = r.Reconcile(req)
assert.NoError(t, err)
@@ -305,7 +315,12 @@ func TestCephObjectStoreController(t *testing.T) {
c.Executor = executor
// Create a ReconcileCephObjectStore object with the scheme and fake client.
r = &ReconcileCephObjectStore{client: cl, scheme: s, context: c}
r = &ReconcileCephObjectStore{
client: cl,
scheme: s,
context: c,
objectStoreChannels: make(map[string]*objectStoreHealth),
}
logger.Info("STARTING PHASE 3")
res, err = r.Reconcile(req)
@@ -313,6 +328,6 @@ func TestCephObjectStoreController(t *testing.T) {
assert.False(t, res.Requeue)
err = r.client.Get(context.TODO(), req.NamespacedName, objectStore)
assert.NoError(t, err)
assert.Equal(t, "Ready", objectStore.Status.Phase, objectStore)
assert.Equal(t, cephv1.ConditionReady, objectStore.Status.Phase, objectStore)
logger.Info("PHASE 3 DONE")
}
+260
View File
@@ -0,0 +1,260 @@
/*
Copyright 2020 The Rook Authors. All rights reserved.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package object
import (
"fmt"
"time"
"github.com/pkg/errors"
cephv1 "github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
"github.com/rook/rook/pkg/clusterd"
"github.com/rook/rook/pkg/operator/k8sutil"
"k8s.io/apimachinery/pkg/types"
"sigs.k8s.io/controller-runtime/pkg/client"
)
const (
s3UserHealthCheckName = "rook-ceph-internal-s3-user-checker"
s3HealthCheckBucketName = "rook-ceph-bucket-checker"
defaultStatusCheckInterval = 1 * time.Minute
s3HealthCheckObjectBody = "Test Rook Object Data"
s3HealthCheckObjectBodyMD5 = "5f286306a2227a156ba770800e71b796"
s3HealthCheckObjectKey = "rookHealthCheckTestObject"
contentType = "plain/text"
)
// bucketChecker aggregates the mon/cluster info needed to check the health of the monitors
type bucketChecker struct {
context *clusterd.Context
objContext *Context
interval time.Duration
serviceIP string
port string
client client.Client
namespacedName types.NamespacedName
healthCheckSpec *cephv1.HealthCheckSpec
}
// newbucketChecker creates a new HealthChecker object
func newBucketChecker(context *clusterd.Context, objContext *Context, serviceIP, port string, client client.Client, namespacedName types.NamespacedName, healthCheckSpec *cephv1.HealthCheckSpec) *bucketChecker {
c := &bucketChecker{
context: context,
objContext: objContext,
interval: defaultStatusCheckInterval,
serviceIP: serviceIP,
port: port,
namespacedName: namespacedName,
client: client,
healthCheckSpec: healthCheckSpec,
}
// allow overriding the check interval with an env var on the operator
checkInterval := healthCheckSpec.Bucket.Interval
if checkInterval != "" {
if duration, err := time.ParseDuration(checkInterval); err == nil {
logger.Infof("ceph rgw status check interval for object store %q is %q", namespacedName.Name, checkInterval)
c.interval = duration
}
}
return c
}
// checkObjectStore periodically checks the health of the cluster
func (c *bucketChecker) checkObjectStore(stopCh chan struct{}) {
for {
select {
case <-stopCh:
// purge bucket and s3 user
// Needed for external mode where in converged everything goes away with the CR deletion
c.cleanupHealthCheck()
logger.Infof("stopping monitoring of rgw endpoints for object store %q", c.namespacedName.Name)
return
case <-time.After(c.interval):
logger.Debug("checking rgw health of object store %q", c.namespacedName.Name)
err := c.checkObjectStoreHealth()
if err != nil {
updateStatusBucket(c.client, c.namespacedName, cephv1.ConditionFailure, err.Error())
logger.Warningf("failed to check rgw health for object store %q. %v", c.namespacedName.Name, err)
}
}
}
}
func (c *bucketChecker) checkObjectStoreHealth() error {
/*
0. purge the s3 object by default
1. create an S3 user
2. always use the same user
3. if already exists just re-hydrate the s3 credentials
4. create a bucket with that user or use the existing one (always use the same bucket)
5. create a check file
6. get the hash of the file
7. PUT the file
8. GET the file
9. compare hashes
10. delete object on bucket
11. update CR health status check
Always keep the bucket and the user for the health check, just do PUT and GET because bucket creation is expensive
*/
var s3AccessKey string
var s3SecretKey string
s3endpoint := fmt.Sprintf("%s:%s", BuildDomainName(c.objContext.Name, c.namespacedName.Namespace), c.port)
// Generate unique user and bucket name
bucketName := genUniqueBucketName(c.objContext.UID)
userConfig := c.genUserConfig()
// Create S3 user
logger.Debugf("creating s3 user object %q for object store %q", userConfig.UserID, c.namespacedName.Name)
user, rgwerr, err := CreateUser(c.objContext, userConfig)
if err != nil {
if rgwerr == ErrorCodeFileExists {
user, _, err = GetUser(c.objContext, userConfig.UserID)
if err != nil {
return errors.Wrapf(err, "failed to get details from ceph object user %q for object store %q", user.UserID, c.namespacedName.Name)
}
} else {
return errors.Wrapf(err, "failed to create object user %q. error code %d for object store %q", userConfig.UserID, rgwerr, c.namespacedName.Name)
}
}
// Set access and secret key
s3AccessKey = *user.AccessKey
s3SecretKey = *user.SecretKey
// Initiate s3 agent
logger.Debugf("initializing s3 connection for object store %q", c.namespacedName.Name)
s3client, err := NewS3Agent(s3AccessKey, s3SecretKey, s3endpoint)
if err != nil {
return errors.Wrap(err, "failed to initialize s3 connection")
}
// Force purge the s3 object before starting anything
err = cleanupObjectHealthCheck(s3client, c.objContext.UID)
if err != nil {
return errors.Wrapf(err, "failed to perform object cleanup for object store %q", c.namespacedName.Name)
}
// Bucket health test
err = c.testBucketHealth(s3client, bucketName)
if err != nil {
return errors.Wrapf(err, "failed to run bucket health checks for object store %q", c.namespacedName.Name)
}
logger.Debug("successfully checked object store endpoint for object store %q", c.namespacedName.Name)
// Update the EndpointStatus in the CR to reflect the healthyness
updateStatusBucket(c.client, c.namespacedName, cephv1.ConditionHealthy, "")
return nil
}
func cleanupObjectHealthCheck(s3client *S3Agent, objectStoreUID string) error {
bucketToDelete := genUniqueBucketName(objectStoreUID)
logger.Infof("deleting object %q from bucket %q", s3HealthCheckObjectKey, bucketToDelete)
_, err := s3client.DeleteObjectInBucket(bucketToDelete, s3HealthCheckObjectKey)
return err
}
func (c *bucketChecker) cleanupHealthCheck() {
bucketToDelete := genUniqueBucketName(c.objContext.UID)
logger.Infof("deleting object %q from bucket %q in object store %q", s3HealthCheckObjectKey, bucketToDelete, c.namespacedName.Name)
_, err := DeleteObjectBucket(c.objContext, bucketToDelete, true)
if err != nil {
logger.Errorf("failed to delete bucket %q for object store %q. %v", bucketToDelete, c.namespacedName.Name, err)
}
userToDelete := c.genUserConfig()
output, err := DeleteUser(c.objContext, userToDelete.UserID)
if err != nil {
logger.Errorf("failed to delete object user %q for object store %q. %s. %v", userToDelete.UserID, c.namespacedName.Name, output, err)
} else {
logger.Debugf("successfully deleted object user %q for object store %q", userToDelete.UserID, c.namespacedName.Name)
}
}
func toCustomResourceStatus(currentStatus *cephv1.BucketStatus, details string, health cephv1.ConditionType) *cephv1.BucketStatus {
s := &cephv1.BucketStatus{
Health: health,
LastChecked: time.Now().UTC().Format(time.RFC3339),
Details: details,
}
if currentStatus != nil {
s.LastChanged = currentStatus.LastChanged
if currentStatus.Details != s.Details {
s.LastChanged = s.LastChecked
}
}
return s
}
func genUniqueBucketName(uuid string) string {
return fmt.Sprintf("%s-%s", s3HealthCheckBucketName, uuid)
}
func (c *bucketChecker) genUserConfig() ObjectUser {
userName := fmt.Sprintf("%s-%s", s3UserHealthCheckName, c.objContext.UID)
return ObjectUser{
UserID: userName,
DisplayName: &userName,
}
}
func (c *bucketChecker) testBucketHealth(s3client *S3Agent, bucket string) error {
// Purge on exit
defer cleanupObjectHealthCheck(s3client, c.objContext.UID)
// Create S3 bucket
logger.Debugf("creating bucket %q", bucket)
err := s3client.CreateBucket(bucket)
if err != nil {
return errors.Wrapf(err, "failed to create bucket %q for object store %q", bucket, c.namespacedName.Name)
}
// Put an object into the bucket
logger.Debugf("putting object %q in bucket %q for object store %q", s3HealthCheckObjectKey, bucket, c.namespacedName.Name)
_, err = s3client.PutObjectInBucket(bucket, string(s3HealthCheckObjectBody), s3HealthCheckObjectKey, contentType)
if err != nil {
return errors.Wrapf(err, "failed to put object %q in bucket %q for object store %q", s3HealthCheckObjectKey, bucket, c.namespacedName.Name)
}
// Get the object from the bucket
logger.Debugf("getting object %q in bucket %q for object store %q", s3HealthCheckObjectKey, bucket, c.namespacedName.Name)
read, err := s3client.GetObjectInBucket(bucket, s3HealthCheckObjectKey)
if err != nil {
return errors.Wrapf(err, "failed to get object %q in bucket %q for object store %q", s3HealthCheckObjectKey, bucket, c.namespacedName.Name)
}
// Compare the old and the existing object
logger.Debugf("comparing objects hash for object store %q", c.namespacedName.Name)
oldHash := k8sutil.Hash(s3HealthCheckObjectBody)
currentHash := k8sutil.Hash(read)
if currentHash != oldHash {
return errors.Wrapf(err, "wrong file content, old file hash is %q and new one is %q for object store %q", oldHash, currentHash, c.namespacedName.Name)
}
return nil
}
@@ -1,5 +1,5 @@
/*
Copyright 2018 The Kubernetes Authors.
Copyright 2020 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
@@ -14,113 +14,15 @@ See the License for the specific language governing permissions and
limitations under the License.
*/
package bucket
package object
import (
"fmt"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/awserr"
"github.com/aws/aws-sdk-go/aws/credentials"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/s3"
oerrors "github.com/kube-object-storage/lib-bucket-provisioner/pkg/provisioner/api/errors"
"github.com/pkg/errors"
"k8s.io/apimachinery/pkg/util/json"
)
// S3Agent wraps the s3.S3 structure to allow for wrapper methods
type S3Agent struct {
client *s3.S3
}
func NewS3Agent(accessKey, secretKey, endpoint string) (*S3Agent, error) {
const cephRegion = "us-east-1"
sess, err := session.NewSession(
aws.NewConfig().
WithRegion(cephRegion).
WithCredentials(credentials.NewStaticCredentials(accessKey, secretKey, "")).
WithEndpoint(endpoint).
WithS3ForcePathStyle(true).
WithMaxRetries(20).
WithDisableSSL(true))
if err != nil {
return nil, err
}
svc := s3.New(sess)
return &S3Agent{
client: svc,
}, nil
}
// CreateBucket creates a bucket with the given name
func (s S3Agent) CreateBucket(name string) error {
logger.Infof("creating bucket %q", name)
bucketInput := &s3.CreateBucketInput{
Bucket: &name,
}
_, err := s.client.CreateBucket(bucketInput)
if err != nil {
if aerr, ok := err.(awserr.Error); ok {
logger.Infof("DEBUG: after s3 call, ok=%v, aerr=%v", ok, aerr)
switch aerr.Code() {
case s3.ErrCodeBucketAlreadyExists:
msg := fmt.Sprintf("Bucket %q already exists", name)
logger.Errorf(msg)
return oerrors.NewBucketExistsError(msg)
case s3.ErrCodeBucketAlreadyOwnedByYou:
msg := fmt.Sprintf("Bucket %q already owned by you", name)
logger.Errorf(msg)
return oerrors.NewBucketExistsError(msg)
}
}
return errors.Wrapf(err, "bucket %q could not be created", name)
}
logger.Infof("successfully created bucket %q", name)
return nil
}
// PutBucketPolicy applies the policy to the bucket
func (s S3Agent) PutBucketPolicy(bucket string, policy BucketPolicy) (*s3.PutBucketPolicyOutput, error) {
confirmRemoveSelfBucketAccess := false
serializedPolicy, _ := json.Marshal(policy)
consumablePolicy := string(serializedPolicy)
p := &s3.PutBucketPolicyInput{
Bucket: &bucket,
ConfirmRemoveSelfBucketAccess: &confirmRemoveSelfBucketAccess,
Policy: &consumablePolicy,
}
out, err := s.client.PutBucketPolicy(p)
if err != nil {
return out, err
}
return out, nil
}
func (s S3Agent) GetBucketPolicy(bucket string) (*BucketPolicy, error) {
out, err := s.client.GetBucketPolicy(&s3.GetBucketPolicyInput{
Bucket: &bucket,
})
if err != nil {
return nil, err
}
policy := &BucketPolicy{}
err = json.Unmarshal([]byte(*out.Policy), policy)
if err != nil {
return nil, err
}
return policy, nil
}
// //////////////
// Policy
// //////////////
type action string
const (
@@ -258,6 +160,41 @@ func NewBucketPolicy(ps ...PolicyStatement) *BucketPolicy {
return bp
}
// PutBucketPolicy applies the policy to the bucket
func (s *S3Agent) PutBucketPolicy(bucket string, policy BucketPolicy) (*s3.PutBucketPolicyOutput, error) {
confirmRemoveSelfBucketAccess := false
serializedPolicy, _ := json.Marshal(policy)
consumablePolicy := string(serializedPolicy)
p := &s3.PutBucketPolicyInput{
Bucket: &bucket,
ConfirmRemoveSelfBucketAccess: &confirmRemoveSelfBucketAccess,
Policy: &consumablePolicy,
}
out, err := s.Client.PutBucketPolicy(p)
if err != nil {
return out, err
}
return out, nil
}
func (s *S3Agent) GetBucketPolicy(bucket string) (*BucketPolicy, error) {
out, err := s.Client.GetBucketPolicy(&s3.GetBucketPolicyInput{
Bucket: &bucket,
})
if err != nil {
return nil, err
}
policy := &BucketPolicy{}
err = json.Unmarshal([]byte(*out.Policy), policy)
if err != nil {
return nil, err
}
return policy, nil
}
// ModifyBucketPolicy new and old statement SIDs and overwrites on a match.
// This allows users to Get, modify, and Replace existing statements as well as
// add new ones.
+31 -17
View File
@@ -267,25 +267,27 @@ func (c *clusterConfig) deleteLegacyDaemons() {
func (c *clusterConfig) deleteStore() error {
logger.Infof("deleting object store %q from namespace %q", c.store.Name, c.store.Namespace)
// Delete rgw CephX keys and configuration in centralized mon database
for i := 0; i < int(c.store.Spec.Gateway.Instances); i++ {
daemonLetterID := k8sutil.IndexToName(i)
depNameToRemove := fmt.Sprintf("%s-%s-%s", AppName, c.store.Name, daemonLetterID)
if !c.clusterSpec.External.Enable {
// Delete rgw CephX keys and configuration in centralized mon database
for i := 0; i < int(c.store.Spec.Gateway.Instances); i++ {
daemonLetterID := k8sutil.IndexToName(i)
depNameToRemove := fmt.Sprintf("%s-%s-%s", AppName, c.store.Name, daemonLetterID)
err := c.deleteRgwCephObjects(depNameToRemove)
err := c.deleteRgwCephObjects(depNameToRemove)
if err != nil {
return err
}
}
// Delete the realm and pools
objContext := NewContext(c.context, c.store.Name, c.store.Namespace)
err := deleteRealmAndPools(objContext, c.store.Spec)
if err != nil {
return err
return errors.Wrap(err, "failed to delete the realm and pools")
}
}
// Delete the realm and pools
objContext := NewContext(c.context, c.store.Name, c.store.Namespace)
err := deleteRealmAndPools(objContext, c.store.Spec)
if err != nil {
return errors.Wrap(err, "failed to delete the realm and pools")
}
logger.Infof("completed deleting object store %q from namespace %q", c.store.Name, c.store.Namespace)
logger.Infof("successfully deleted object store %q from namespace %q", c.store.Name, c.store.Namespace)
return nil
}
@@ -316,7 +318,7 @@ func (c *clusterConfig) storeLabelSelector() string {
}
// Validate the object store arguments
func validateStore(context *clusterd.Context, s *cephv1.CephObjectStore) error {
func (r *ReconcileCephObjectStore) validateStore(s *cephv1.CephObjectStore) error {
if s.Name == "" {
return errors.New("missing name")
}
@@ -331,16 +333,23 @@ func validateStore(context *clusterd.Context, s *cephv1.CephObjectStore) error {
// Validate the pool settings, but allow for empty pools specs in case they have already been created
// such as by the ceph mgr
if !emptyPool(s.Spec.MetadataPool) {
if err := pool.ValidatePoolSpec(context, s.Namespace, &s.Spec.MetadataPool); err != nil {
if err := pool.ValidatePoolSpec(r.context, s.Namespace, &s.Spec.MetadataPool); err != nil {
return errors.Wrap(err, "invalid metadata pool spec")
}
}
if !emptyPool(s.Spec.DataPool) {
if err := pool.ValidatePoolSpec(context, s.Namespace, &s.Spec.DataPool); err != nil {
if err := pool.ValidatePoolSpec(r.context, s.Namespace, &s.Spec.DataPool); err != nil {
return errors.Wrap(err, "invalid data pool spec")
}
}
// Fail if we detected an external CephCluster CR and the list of endpoints is empty
if r.cephClusterSpec.External.Enable {
if len(s.Spec.Gateway.ExternalRgwEndpoints) == 0 {
return errors.New("ceph cluster is external but externalRgwEndpoints list is empty")
}
}
return nil
}
@@ -351,3 +360,8 @@ func (c *clusterConfig) generateSecretName(id string) string {
func emptyPool(pool cephv1.PoolSpec) bool {
return reflect.DeepEqual(pool, cephv1.PoolSpec{})
}
// BuildDomainName build the dns name to reach out the service endpoint
func BuildDomainName(name, namespace string) string {
return fmt.Sprintf("%s-%s.%s", AppName, name, namespace)
}
+7
View File
@@ -154,3 +154,10 @@ func TestEmptyPoolSpec(t *testing.T) {
p = cephv1.PoolSpec{ErasureCoded: cephv1.ErasureCodedSpec{CodingChunks: 1}}
assert.False(t, emptyPool(p))
}
func TestBuildDomainName(t *testing.T) {
name := "my-store"
ns := "rook-ceph"
dns := BuildDomainName(name, ns)
assert.Equal(t, "rook-ceph-rgw-my-store.rook-ceph", dns)
}
+156
View File
@@ -0,0 +1,156 @@
/*
Copyright 2018 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package object
import (
"bytes"
"net/http"
"strings"
"time"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/awserr"
"github.com/aws/aws-sdk-go/aws/credentials"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/s3"
"github.com/pkg/errors"
)
// S3Agent wraps the s3.S3 structure to allow for wrapper methods
type S3Agent struct {
Client *s3.S3
}
func NewS3Agent(accessKey, secretKey, endpoint string) (*S3Agent, error) {
const cephRegion = "us-east-1"
sess, err := session.NewSession(
aws.NewConfig().
WithRegion(cephRegion).
WithCredentials(credentials.NewStaticCredentials(accessKey, secretKey, "")).
WithEndpoint(endpoint).
WithS3ForcePathStyle(true).
WithMaxRetries(20).
WithDisableSSL(true).
WithHTTPClient(&http.Client{
Timeout: time.Second * 15,
}),
)
if err != nil {
return nil, err
}
svc := s3.New(sess)
return &S3Agent{
Client: svc,
}, nil
}
// CreateBucket creates a bucket with the given name
func (s *S3Agent) CreateBucket(name string) error {
logger.Infof("creating bucket %q", name)
bucketInput := &s3.CreateBucketInput{
Bucket: &name,
}
_, err := s.Client.CreateBucket(bucketInput)
if err != nil {
if aerr, ok := err.(awserr.Error); ok {
logger.Debugf("DEBUG: after s3 call, ok=%v, aerr=%v", ok, aerr)
switch aerr.Code() {
case s3.ErrCodeBucketAlreadyExists:
logger.Debugf("bucket %q already exists", name)
return nil
case s3.ErrCodeBucketAlreadyOwnedByYou:
logger.Debugf("bucket %q already owned by you", name)
return nil
}
}
return errors.Wrapf(err, "failed to create bucket %q", name)
}
logger.Infof("successfully created bucket %q", name)
return nil
}
// DeleteBucket function deletes given bucket using s3 client
func (s *S3Agent) DeleteBucket(name string) (bool, error) {
_, err := s.Client.DeleteBucket(&s3.DeleteBucketInput{
Bucket: aws.String(name),
})
if err != nil {
logger.Errorf("failed to delete bucket. %v", err)
return false, err
}
return true, nil
}
// PutObjectInBucket function puts an object in a bucket using s3 client
func (s *S3Agent) PutObjectInBucket(bucketname string, body string, key string,
contentType string) (bool, error) {
_, err := s.Client.PutObject(&s3.PutObjectInput{
Body: strings.NewReader(body),
Bucket: &bucketname,
Key: &key,
ContentType: &contentType,
})
if err != nil {
logger.Errorf("failed to put object in bucket. %v", err)
return false, err
}
return true, nil
}
// GetObjectInBucket function retrieves an object from a bucket using s3 client
func (s *S3Agent) GetObjectInBucket(bucketname string, key string) (string, error) {
result, err := s.Client.GetObject(&s3.GetObjectInput{
Bucket: aws.String(bucketname),
Key: aws.String(key),
})
if err != nil {
logger.Errorf("failed to retrieve object from bucket. %v", err)
return "ERROR_ OBJECT NOT FOUND", err
}
buf := new(bytes.Buffer)
buf.ReadFrom(result.Body)
return buf.String(), nil
}
// DeleteObjectInBucket function deletes given bucket using s3 client
func (s *S3Agent) DeleteObjectInBucket(bucketname string, key string) (bool, error) {
_, err := s.Client.DeleteObject(&s3.DeleteObjectInput{
Bucket: aws.String(bucketname),
Key: aws.String(key),
})
if err != nil {
if aerr, ok := err.(awserr.Error); ok {
switch aerr.Code() {
case s3.ErrCodeNoSuchBucket:
return true, nil
case s3.ErrCodeNoSuchKey:
return true, nil
}
}
logger.Errorf("failed to delete object from bucket. %v", err)
return false, err
}
return true, nil
}
+58 -1
View File
@@ -233,18 +233,63 @@ func (c *clusterConfig) generateService(cephObjectStore *cephv1.CephObjectStore)
}
destPort := c.generateLiveProbePort()
// When the cluster is external we must use the same one as the gateways are listening on
if c.clusterSpec.External.Enable {
destPort.IntVal = cephObjectStore.Spec.Gateway.Port
}
addPort(svc, "http", cephObjectStore.Spec.Gateway.Port, destPort.IntVal)
addPort(svc, "https", cephObjectStore.Spec.Gateway.SecurePort, cephObjectStore.Spec.Gateway.SecurePort)
return svc
}
func (c *clusterConfig) generateEndpoint(cephObjectStore *cephv1.CephObjectStore) *v1.Endpoints {
labels := getLabels(cephObjectStore.Name, cephObjectStore.Namespace)
endpoints := &v1.Endpoints{
ObjectMeta: metav1.ObjectMeta{
Name: instanceName(cephObjectStore.Name),
Namespace: cephObjectStore.Namespace,
Labels: labels,
},
Subsets: []v1.EndpointSubset{
{
Addresses: cephObjectStore.Spec.Gateway.ExternalRgwEndpoints,
},
},
}
addPortToEndpoint(endpoints, "http", cephObjectStore.Spec.Gateway.Port)
addPortToEndpoint(endpoints, "https", cephObjectStore.Spec.Gateway.SecurePort)
return endpoints
}
func (c *clusterConfig) reconcileExternalEndpoint(cephObjectStore *cephv1.CephObjectStore) error {
logger.Info("reconciling external object store service")
endpoint := c.generateEndpoint(cephObjectStore)
// Set owner ref to the parent object
err := controllerutil.SetControllerReference(cephObjectStore, endpoint, c.scheme)
if err != nil {
return errors.Wrap(err, "failed to set owner reference to ceph object store endpoint")
}
_, err = k8sutil.CreateOrUpdateEndpoint(c.context.Clientset, cephObjectStore.Namespace, endpoint)
if err != nil {
return errors.Wrapf(err, "failed to create or update object store %q service", cephObjectStore.Name)
}
return nil
}
func (c *clusterConfig) reconcileService(cephObjectStore *cephv1.CephObjectStore) (string, error) {
service := c.generateService(cephObjectStore)
// Set owner ref to the parent object
err := controllerutil.SetControllerReference(cephObjectStore, service, c.scheme)
if err != nil {
return "", errors.Wrap(err, "failed to set owner reference to ceph object store")
return "", errors.Wrap(err, "failed to set owner reference to ceph object store service")
}
svc, err := k8sutil.CreateOrUpdateService(c.context.Clientset, cephObjectStore.Namespace, service)
@@ -268,6 +313,18 @@ func addPort(service *v1.Service, name string, port, destPort int32) {
})
}
func addPortToEndpoint(endpoints *v1.Endpoints, name string, port int32) {
if port == 0 {
return
}
endpoints.Subsets[0].Ports = append(endpoints.Subsets[0].Ports, v1.EndpointPort{
Name: name,
Port: port,
Protocol: v1.ProtocolTCP,
},
)
}
func getLabels(name, namespace string) map[string]string {
labels := controller.PodLabels(AppName, namespace, "rgw", name)
labels["rook_object_store"] = name
+29 -7
View File
@@ -133,34 +133,56 @@ func TestSSLPodSpec(t *testing.T) {
func TestValidateSpec(t *testing.T) {
context := &clusterd.Context{Executor: &exectest.MockExecutor{}}
r := &ReconcileCephObjectStore{
context: context,
cephClusterSpec: &cephv1.ClusterSpec{
External: cephv1.ExternalSpec{
Enable: false,
},
},
}
// valid store
s := simpleStore()
err := validateStore(context, s)
err := r.validateStore(s)
assert.Nil(t, err)
// no name
s.Name = ""
err = validateStore(context, s)
err = r.validateStore(s)
assert.NotNil(t, err)
s.Name = "default"
err = validateStore(context, s)
err = r.validateStore(s)
assert.Nil(t, err)
// no namespace
s.Namespace = ""
err = validateStore(context, s)
err = r.validateStore(s)
assert.NotNil(t, err)
s.Namespace = "mycluster"
err = validateStore(context, s)
err = r.validateStore(s)
assert.Nil(t, err)
// no replication or EC is valid
s.Spec.MetadataPool.Replicated.Size = 0
err = validateStore(context, s)
err = r.validateStore(s)
assert.Nil(t, err)
s.Spec.MetadataPool.Replicated.Size = 1
err = validateStore(context, s)
err = r.validateStore(s)
assert.Nil(t, err)
// external with no endpoints, failure
r.cephClusterSpec.External.Enable = true
err = r.validateStore(s)
assert.NotNil(t, err)
// external with endpoints, success
s.Spec.Gateway.ExternalRgwEndpoints = []v1.EndpointAddress{
{
IP: "192.168.0.1",
},
}
err = r.validateStore(s)
assert.Nil(t, err)
}
+78
View File
@@ -0,0 +1,78 @@
/*
Copyright 2020 The Rook Authors. All rights reserved.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package object
import (
"context"
"github.com/pkg/errors"
cephv1 "github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
opcontroller "github.com/rook/rook/pkg/operator/ceph/controller"
kerrors "k8s.io/apimachinery/pkg/api/errors"
"k8s.io/apimachinery/pkg/types"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/reconcile"
)
func (r *ReconcileCephObjectStore) setFailedStatus(name types.NamespacedName, errMessage string, err error) (reconcile.Result, error) {
updateStatusPhase(r.client, name, cephv1.ConditionFailure)
return reconcile.Result{}, errors.Wrapf(err, "%s", errMessage)
}
// updateStatusPhase updates an object with a given status
func updateStatusPhase(client client.Client, namespacedName types.NamespacedName, status cephv1.ConditionType) {
objectStore := &cephv1.CephObjectStore{}
if err := client.Get(context.TODO(), namespacedName, objectStore); err != nil {
if kerrors.IsNotFound(err) {
logger.Debug("CephObjectStore resource not found. Ignoring since object must be deleted.")
return
}
logger.Warningf("failed to retrieve object store %q to update status to %q. %v", namespacedName, status, err)
return
}
if objectStore.Status == nil {
objectStore.Status = &cephv1.ObjectStoreStatus{}
}
objectStore.Status.Phase = status
if err := opcontroller.UpdateStatus(client, objectStore); err != nil {
logger.Errorf("failed to set object store %q status to %q. %v", namespacedName, status, err)
return
}
logger.Debugf("object store %q status updated to %q", namespacedName, status)
}
// updateStatusBucket updates an object with a given status
func updateStatusBucket(client client.Client, name types.NamespacedName, phase cephv1.ConditionType, details string) {
objectStore := &cephv1.CephObjectStore{}
if err := client.Get(context.TODO(), name, objectStore); err != nil {
if kerrors.IsNotFound(err) {
logger.Debug("CephObjectStore resource not found. Ignoring since object must be deleted.")
return
}
logger.Warningf("failed to retrieve object store %q to update status to %v. %v", name, phase, err)
return
}
objectStore.Status.BucketStatus = toCustomResourceStatus(objectStore.Status.BucketStatus, details, phase)
if err := opcontroller.UpdateStatus(client, objectStore); err != nil {
logger.Errorf("failed to set object store %q status to %v. %v", name, phase, err)
return
}
logger.Debugf("object store %q status updated to %v", name, phase)
}
+15 -14
View File
@@ -73,7 +73,7 @@ func decodeUser(data string) (*ObjectUser, int, error) {
var user rgwUserInfo
err := json.Unmarshal([]byte(data), &user)
if err != nil {
return nil, RGWErrorParse, errors.Wrap(err, "Failed to unmarshal json")
return nil, RGWErrorParse, errors.Wrapf(err, "failed to unmarshal json. %s", data)
}
rookUser := ObjectUser{UserID: user.UserID, DisplayName: &user.DisplayName, Email: &user.Email}
@@ -88,7 +88,7 @@ func decodeUser(data string) (*ObjectUser, int, error) {
// GetUser returns the user with the given ID.
func GetUser(c *Context, id string) (*ObjectUser, int, error) {
logger.Infof("Getting user: %s", id)
logger.Infof("getting s3 user %q", id)
// note: err is set for non-existent user but result output is also empty
result, err := runAdminCommand(c, "user", "info", "--uid", id)
@@ -103,7 +103,7 @@ func GetUser(c *Context, id string) (*ObjectUser, int, error) {
// CreateUser creates a new user with the information given.
func CreateUser(c *Context, user ObjectUser) (*ObjectUser, int, error) {
logger.Debugf("Creating user: %s", user.UserID)
logger.Debugf("creating s3 user %q", user.UserID)
if strings.TrimSpace(user.UserID) == "" {
return nil, RGWErrorBadData, errors.New("userId cannot be empty")
@@ -126,15 +126,16 @@ func CreateUser(c *Context, user ObjectUser) (*ObjectUser, int, error) {
result, err := runAdminCommand(c, args...)
if err != nil {
return nil, RGWErrorUnknown, errors.Wrap(err, "failed to create user")
}
if strings.Contains(result, "could not create user: unable to create user, user: ") {
return nil, ErrorCodeFileExists, errors.New("s3 user already exists")
}
if strings.HasPrefix(result, "could not create user: unable to create user, user: ") && strings.HasSuffix(result, " exists") {
return nil, ErrorCodeFileExists, errors.New("user already exists")
}
if strings.Contains(result, "could not create user: unable to create user, email: ") && strings.Contains(result, " is the email address an existing user") {
return nil, RGWErrorBadData, errors.New("email already in use")
}
if strings.HasPrefix(result, "could not create user: unable to create user, email: ") && strings.HasSuffix(result, " is the email address an existing user") {
return nil, RGWErrorBadData, errors.New("email already in use")
// We don't know what happened
return nil, RGWErrorUnknown, errors.Wrap(err, "failed to create s3 user")
}
return decodeUser(result)
@@ -142,7 +143,7 @@ func CreateUser(c *Context, user ObjectUser) (*ObjectUser, int, error) {
// UpdateUser updates the user whose ID matches the user.
func UpdateUser(c *Context, user ObjectUser) (*ObjectUser, int, error) {
logger.Infof("Updating user: %s", user.UserID)
logger.Infof("updating s3 user %q", user.UserID)
args := []string{"user", "modify", "--uid", user.UserID}
@@ -155,11 +156,11 @@ func UpdateUser(c *Context, user ObjectUser) (*ObjectUser, int, error) {
body, err := runAdminCommand(c, args...)
if err != nil {
return nil, RGWErrorUnknown, errors.Wrap(err, "failed to update user")
return nil, RGWErrorUnknown, errors.Wrap(err, "failed to update s3 user")
}
if body == "could not modify user: unable to modify user, user not found" {
return nil, RGWErrorNotFound, errors.New("user not found")
return nil, RGWErrorNotFound, errors.New("s3 user not found")
}
return decodeUser(body)
@@ -179,7 +180,7 @@ func DeleteUser(c *Context, id string, opts ...string) (string, error) {
}
}
return result, errors.Wrap(err, "failed to delete user")
return result, errors.Wrap(err, "failed to delete s3 user")
}
func SetQuotaUserBucketMax(c *Context, id string, max int) (string, int, error) {
+43
View File
@@ -0,0 +1,43 @@
/*
Copyright 2020 The Rook Authors. All rights reserved.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package k8sutil
import (
"fmt"
v1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/api/errors"
"k8s.io/client-go/kubernetes"
)
// CreateOrUpdateEndpoint creates a service or updates the service declaratively if it already exists.
func CreateOrUpdateEndpoint(clientset kubernetes.Interface, namespace string, endpointDefinition *v1.Endpoints) (*v1.Endpoints, error) {
name := endpointDefinition.Name
logger.Debugf("creating endpoint %q. %v", name, endpointDefinition.Subsets)
ep, err := clientset.CoreV1().Endpoints(namespace).Create(endpointDefinition)
if err != nil {
if !errors.IsAlreadyExists(err) {
return nil, fmt.Errorf("failed to create endpoint %q. %v", name, err)
}
ep, err = clientset.CoreV1().Endpoints(namespace).Update(endpointDefinition)
if err != nil {
return nil, fmt.Errorf("failed to update endpoint %q. %v", name, err)
}
}
return ep, err
}
@@ -488,6 +488,15 @@ spec:
- force
preservePoolsOnDelete:
type: boolean
healthCheck:
properties:
bucket:
properties:
enabled:
type: boolean
interval:
type: string
subresources:
subresources:
status: {}
---
@@ -2209,6 +2218,10 @@ spec:
securePort:
instances: ` + strconv.Itoa(replicaCount) + `
allNodes: false
healthCheck:
bucket:
enabled: true
interval: 10s
`
}
-143
View File
@@ -1,143 +0,0 @@
/*
Copyright 2016 The Rook Authors. All rights reserved.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package utils
import (
"bytes"
"strings"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/credentials"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/s3"
)
// S3Helper contains pointer to s3 client and wrappers for basic object store operations
type S3Helper struct {
s3client *s3.S3
}
// CreateNewS3Helper creates a s3 client for specified endpoint and creds
func CreateNewS3Helper(endpoint string, keyID string, keySecret string) *S3Helper {
creds := credentials.NewStaticCredentials(keyID, keySecret, "")
// create aws s3 config, must use 'us-east-1' default aws region for ceph object store
awsConfig := aws.NewConfig().
WithRegion("us-east-1").
WithCredentials(creds).
WithEndpoint(endpoint).
WithS3ForcePathStyle(true).
WithDisableSSL(true).
WithMaxRetries(20)
// create new session
ses := session.New()
// create new s3 client connection
c := s3.New(ses, awsConfig)
return &S3Helper{c}
}
// CreateBucket function creates bucket using s3 client
func (h *S3Helper) CreateBucket(name string) (bool, error) {
_, err := h.s3client.CreateBucket(&s3.CreateBucketInput{
Bucket: aws.String(name),
})
if err != nil {
logger.Errorf("error encountered while creating bucket : %v", err)
return false, err
}
return true, nil
}
// DeleteBucket function deletes given bucket using s3 client
func (h *S3Helper) DeleteBucket(name string) (bool, error) {
_, err := h.s3client.DeleteBucket(&s3.DeleteBucketInput{
Bucket: aws.String(name),
})
if err != nil {
logger.Errorf("error encountered while deleting bucket : %v", err)
return false, err
}
return true, nil
}
// PutObjectInBucket function puts an object in a bucket using s3 client
func (h *S3Helper) PutObjectInBucket(bucketname string, body string, key string,
contentType string) (bool, error) {
_, err := h.s3client.PutObject(&s3.PutObjectInput{
Body: strings.NewReader(body),
Bucket: &bucketname,
Key: &key,
ContentType: &contentType,
})
if err != nil {
logger.Errorf("error encountered while putting object in bucket : %v", err)
return false, err
}
return true, nil
}
// GetObjectInBucket function retrieves an object from a bucket using s3 client
func (h *S3Helper) GetObjectInBucket(bucketname string, key string) (string, error) {
result, err := h.s3client.GetObject(&s3.GetObjectInput{
Bucket: aws.String(bucketname),
Key: aws.String(key),
})
if err != nil {
logger.Errorf("error encountered while retrieving object from bucket : %v", err)
return "ERROR_ OBJECT NOT FOUND", err
}
buf := new(bytes.Buffer)
buf.ReadFrom(result.Body)
return buf.String(), nil
}
// DeleteObjectInBucket function deletes given bucket using s3 client
func (h *S3Helper) DeleteObjectInBucket(bucketname string, key string) (bool, error) {
_, err := h.s3client.DeleteObject(&s3.DeleteObjectInput{
Bucket: aws.String(bucketname),
Key: aws.String(key),
})
if err != nil {
logger.Errorf("error encountered while deleting object from bucket : %v", err)
return false, err
}
return true, nil
}
// IsBucketPresent function returns true if a bucket is present and false if it's not present
func (h *S3Helper) IsBucketPresent(bucketname string) (bool, error) {
_, err := h.s3client.HeadBucket(&s3.HeadBucketInput{
Bucket: aws.String(bucketname),
})
if err != nil && strings.Contains(err.Error(), "NotFound") {
return false, nil
} else if err == nil {
return true, nil
}
return false, err
}
+12 -3
View File
@@ -21,12 +21,14 @@ import (
"time"
cephv1 "github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
rgw "github.com/rook/rook/pkg/operator/ceph/object"
"github.com/rook/rook/tests/framework/clients"
"github.com/rook/rook/tests/framework/utils"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/stretchr/testify/suite"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
var (
@@ -70,14 +72,20 @@ func runObjectE2ETest(helper *clients.TestClient, k8sh *utils.K8sHelper, s suite
logger.Infof("Object store created successfully")
// check that ObjectUser is created
logger.Infof("Waiting 10 seconds to ensure user was created")
time.Sleep(10 * time.Second)
logger.Infof("Waiting 15 seconds to ensure user was created")
time.Sleep(15 * time.Second)
logger.Infof("Checking to see if the user secret has been created")
i = 0
for i = 0; i < 4 && helper.ObjectUserClient.UserSecretExists(namespace, storeName, userid) == false; i++ {
logger.Infof("(%d) secret check sleeping for 5 seconds ...", i)
time.Sleep(5 * time.Second)
}
// Check object store status
objectStore, err := k8sh.RookClientset.CephV1().CephObjectStores(namespace).Get(storeName, metav1.GetOptions{})
assert.Nil(s.T(), err)
assert.Equal(s.T(), cephv1.ConditionHealthy, objectStore.Status.BucketStatus.Health)
assert.True(s.T(), helper.ObjectUserClient.UserSecretExists(namespace, storeName, userid))
userInfo, err := helper.ObjectUserClient.GetUser(namespace, storeName, userid)
require.NoError(s.T(), err)
@@ -136,7 +144,8 @@ func runObjectE2ETest(helper *clients.TestClient, k8sh *utils.K8sHelper, s suite
s3endpoint, _ := helper.ObjectClient.GetEndPointUrl(namespace, storeName)
s3AccessKey, _ := helper.BucketClient.GetAccessKey(obcName)
s3SecretKey, _ := helper.BucketClient.GetSecretKey(obcName)
s3client := utils.CreateNewS3Helper(s3endpoint, s3AccessKey, s3SecretKey)
s3client, err := rgw.NewS3Agent(s3AccessKey, s3SecretKey, s3endpoint)
require.Nil(s.T(), err)
logger.Infof("endpoint (%s) Accesskey (%s) secret (%s)", s3endpoint, s3AccessKey, s3SecretKey)
logger.Infof("Step 5 : Put Object on bucket")