ceph: add rgw endpoint healthcheck
We have introduced a new goroutine to check the state of the rgw
endpoint. It will run every minute and perform operations on a bucket.
The success or failure will be reported as part of the status field of
the CephObjectStore CR.
A good status will look like:
status:
endpointStatus:
lastChanged: "2020-06-25T13:47:45Z"
lastChecked: "2020-06-25T13:48:46Z"
phase: Connected
A failed status:
status:
endpointStatus:
details: |-
error creating bucket "rook-ceph-internal-s3-bucket-checker": RequestError: send request failed
caused by: Put http://rook-ceph-rgw-my-store.rook-ceph:8080/rook-ceph-internal-s3-bucket-checker: dial tcp 10.108.189.148:8080: connect: connection refused
health: ERROR
This check works for both converged and external modes. Note that the
CephObjectStore CRD has a new field called "externalRgwEndpoints" which
allows you to define a list of IP addresses pointing to rgws.
Closes: https://github.com/rook/rook/issues/5692
Signed-off-by: Sébastien Han <seb@redhat.com>
This commit is contained in:
1 parent
7229f5826e
commit
e4eaa91ede
34 files changed
+1318
-418
No files matched your search
@@ -94,11 +94,24 @@ The gateway settings correspond to the RGW daemon settings.
|
||||
* `port`: The port on which the Object service will be reachable. If host networking is enabled, the RGW daemons will also listen on that port. If running on SDN, the RGW daemon listening port will be 8080 internally.
|
||||
* `securePort`: The secure port on which RGW pods will be listening. An SSL certificate must be specified.
|
||||
* `instances`: The number of pods that will be started to load balance this object store.
|
||||
* `externalRgwEndpoints`: A list of IP addresses to connect to external existing Rados Gateways (works with external mode). This setting will be ignored if the `CephCluster` does not have `external` spec enabled. Refer to the [external cluster section](ceph-cluster-crd.md#external-cluster) for more details.
|
||||
* `annotations`: Key value pair list of annotations to add.
|
||||
* `placement`: The Kubernetes placement settings to determine where the RGW pods should be started in the cluster.
|
||||
* `resources`: Set resource requests/limits for the Gateway Pod(s), see [Resource Requirements/Limits](ceph-cluster-crd.md#resource-requirementslimits).
|
||||
* `priorityClassName`: Set priority class name for the Gateway Pod(s)
|
||||
|
||||
Example of external rgw endpoints to connect to:
|
||||
|
||||
```yaml
|
||||
gateway:
|
||||
port: 80
|
||||
externalRgwEndpoints:
|
||||
- ip: 192.168.39.182
|
||||
```
|
||||
|
||||
This will create a service with the endpoint `192.168.39.182` on port `80`, pointing to the Ceph object external gateway.
|
||||
All the other settings from the gateway section will be ignored, except for `securePort`.
|
||||
|
||||
## Zone Settings
|
||||
|
||||
The [zone](ceph-object-multisite.md) settings allow the object store to join custom created [ceph-object-zone](ceph-object-multisite-crd.md).
|
||||
@@ -119,3 +132,30 @@ execution attack.
|
||||
Rook will not overwrite an existing `mime.types` ConfigMap so that user modifications will not be
|
||||
destroyed. If the object store is destroyed and recreated, the ConfigMap will also be destroyed and
|
||||
created anew.
|
||||
|
||||
## Health settings
|
||||
|
||||
Rook-Ceph will be default monitor the state of the object store endpoints.
|
||||
The following CRD settings are available:
|
||||
|
||||
* `healthCheck`: main object store health monitoring section
|
||||
|
||||
Here is a complete example:
|
||||
|
||||
```yaml
|
||||
healthCheck:
|
||||
bucket:
|
||||
enabled: true
|
||||
interval: 60s
|
||||
```
|
||||
|
||||
The endpoint health check procedure is the following:
|
||||
|
||||
1. Create an S3 user
|
||||
2. Create a bucket with that user
|
||||
3. PUT the file in the object store
|
||||
4. GET the file from the object store
|
||||
5. Verify object consistency
|
||||
6. Update CR health status check
|
||||
|
||||
Rook-Ceph always keeps the bucket and the user for the health check, it just does a PUT and GET of an s3 object since creating a bucket is an expensive operation.
|
||||
@@ -12,7 +12,14 @@ Object storage exposes an S3 API to the storage cluster for applications to put
|
||||
|
||||
This guide assumes a Rook cluster as explained in the [Quickstart](ceph-quickstart.md).
|
||||
|
||||
## Create an Object Store
|
||||
## Configure an Object Store
|
||||
|
||||
Rook has the ability to either deploy an object store in Kubernetes or to connect to an external RGW service.
|
||||
Most commonly, the object store will be configured locally by Rook.
|
||||
Alternatively, if you have an existing Ceph cluster with Rados Gateways, see the
|
||||
[external section](#connect-to-an-external-object-store) to consume it from Rook.
|
||||
|
||||
### Create a Local Object Store
|
||||
|
||||
The below sample will create a `CephObjectStore` that starts the RGW service in the cluster with an S3 API.
|
||||
|
||||
@@ -45,6 +52,10 @@ spec:
|
||||
port: 80
|
||||
securePort:
|
||||
instances: 1
|
||||
healthCheck:
|
||||
bucket:
|
||||
enabled: true
|
||||
interval: 60s
|
||||
```
|
||||
|
||||
After the `CephObjectStore` is created, the Rook operator will then create all the pools and other resources necessary to start the service. This may take a minute to complete.
|
||||
@@ -57,7 +68,61 @@ kubectl create -f object.yaml
|
||||
kubectl -n rook-ceph get pod -l app=rook-ceph-rgw
|
||||
```
|
||||
|
||||
### Connect to an External Object Store
|
||||
|
||||
Rook can connect to existing RGW gateways to work in conjunction with the external mode of the `CephCluster` CRD.
|
||||
If you have an external `CephCluster` CR, you can instruct Rook to consume external gateways with the following:
|
||||
|
||||
```yaml
|
||||
apiVersion: ceph.rook.io/v1
|
||||
kind: CephObjectStore
|
||||
metadata:
|
||||
name: external-store
|
||||
namespace: rook-ceph
|
||||
spec:
|
||||
gateway:
|
||||
port: 8080
|
||||
externalRgwEndpoints:
|
||||
- ip: 192.168.39.182
|
||||
healthCheck:
|
||||
bucket:
|
||||
enabled: true
|
||||
interval: 60s
|
||||
```
|
||||
|
||||
You can use the existing `object-external.yaml` file.
|
||||
When ready the ceph-object-controller will output a message in the Operator log similar to this one:
|
||||
|
||||
```log
|
||||
ceph-object-controller: ceph object store gateway service running at 10.100.28.138:8080
|
||||
```
|
||||
|
||||
You can now get and access the store via:
|
||||
|
||||
```console
|
||||
kubectl -n rook-ceph get svc -l app=rook-ceph-rgw
|
||||
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
|
||||
rook-ceph-rgw-my-store ClusterIP 10.100.28.138 <none> 8080/TCP 6h59m
|
||||
```
|
||||
|
||||
Any pod from your cluster can now access this endpoint:
|
||||
|
||||
```console
|
||||
curl 10.100.28.138:8080
|
||||
<?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>
|
||||
```
|
||||
|
||||
It is also possible to use the internally registered DNS name:
|
||||
|
||||
```console
|
||||
curl rook-ceph-rgw-my-store.rook-ceph:8080
|
||||
<?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>
|
||||
```
|
||||
|
||||
The DNS name is created with the following schema `rook-ceph-rgw-$STORE_NAME.$NAMESPACE`.
|
||||
|
||||
## Create a Bucket
|
||||
|
||||
Now that the object store is configured, next we need to create a bucket where a client can read and write objects. A bucket can be created by defining a storage class, similar to the pattern used by block and file storage.
|
||||
First, define the storage class that will allow object clients to create a bucket.
|
||||
The storage class defines the object storage system, the bucket retention policy, and other properties required by the administrator. Save the following as `storageclass-bucket-delete.yaml` (the example is named as such due to the `Delete` reclaim policy).
|
||||
@@ -74,7 +139,8 @@ parameters:
|
||||
objectStoreNamespace: rook-ceph
|
||||
region: us-east-1
|
||||
```
|
||||
```bash
|
||||
|
||||
```console
|
||||
kubectl create -f storageclass-bucket-delete.yaml
|
||||
```
|
||||
|
||||
@@ -92,9 +158,11 @@ spec:
|
||||
generateBucketName: ceph-bkt
|
||||
storageClassName: rook-ceph-bucket
|
||||
```
|
||||
```bash
|
||||
|
||||
```console
|
||||
kubectl create -f object-bucket-claim-delete.yaml
|
||||
```
|
||||
|
||||
Now that the claim is created, the operator will create the bucket as well as generate other artifacts to enable access to the bucket. A secret and ConfigMap are created with the same name as the OBC and in the same namespace.
|
||||
The secret contains credentials used by the application pod to access the bucket.
|
||||
The ConfigMap contains bucket endpoint information and is also consumed by the pod.
|
||||
@@ -151,7 +219,8 @@ below in the section [creating a user](#create-a-user) if you are not creating t
|
||||
### Install s3cmd
|
||||
|
||||
To test the `CephObjectStore` we will install the `s3cmd` tool into the toolbox pod.
|
||||
```bash
|
||||
|
||||
```console
|
||||
yum --assumeyes install s3cmd
|
||||
```
|
||||
|
||||
@@ -249,7 +318,7 @@ spec:
|
||||
|
||||
When the `CephObjectStoreUser` is created, the Rook operator will then create the RGW user on the specified `CephObjectStore` and store the Access Key and Secret Key in a kubernetes secret in the same namespace as the `CephObjectStoreUser`.
|
||||
|
||||
```bash
|
||||
```console
|
||||
# Create the object store user
|
||||
kubectl create -f object-user.yaml
|
||||
|
||||
@@ -275,7 +344,8 @@ The AccessKey and SecretKey data fields can be mounted in a pod as an environmen
|
||||
kubernetes secrets can be found in the [K8s secret documentation](https://kubernetes.io/docs/concepts/configuration/secret/)
|
||||
|
||||
To directly retrieve the secrets:
|
||||
```bash
|
||||
|
||||
```console
|
||||
kubectl -n rook-ceph get secret rook-ceph-object-user-my-store-my-user -o yaml | grep AccessKey | awk '{print $2}' | base64 --decode
|
||||
kubectl -n rook-ceph get secret rook-ceph-object-user-my-store-my-user -o yaml | grep SecretKey | awk '{print $2}' | base64 --decode
|
||||
```
|
||||
|
||||
@@ -13,10 +13,14 @@
|
||||
- OSD changes:
|
||||
- OSD on PVC now supports multipath device.
|
||||
- Added [admission controller](Documentation/admission-controller-usage.md) support for CRD validations.
|
||||
- Support for Ceph CRDs is provided. Some validations for CephClusters are included and additional validations can be added for other CRDs
|
||||
- Can be extended to add support for other providers
|
||||
- Support for Ceph CRDs is provided. Some validations for CephClusters are included and additional validations can be added for other CRDs
|
||||
- Can be extended to add support for other providers
|
||||
- OBC changes:
|
||||
- Updated lib bucket provisioner version to support multithread and required change can be found in [operator.yaml](cluster/examples/kubernetes/ceph/operator.yaml#L449)
|
||||
- Can be extended to add support for other providers
|
||||
- The Rook operator reflects the health of the CephObjectStore in its status field
|
||||
- The CephObjectStore CR supports connecting to external Ceph Rados Gateways, refer to the [external object section](Documentation/ceph-object.html#connect-to-external-object-store)
|
||||
- The CephObjectStore CR runs health checks on the object store endpoint, refer to the [health check section](Documentation/ceph-object-store-crd.html#health-settings)
|
||||
|
||||
### EdgeFS
|
||||
|
||||
|
||||
@@ -407,6 +407,15 @@ spec:
|
||||
- force
|
||||
preservePoolsOnDelete:
|
||||
type: boolean
|
||||
healthCheck:
|
||||
properties:
|
||||
bucket:
|
||||
properties:
|
||||
enabled:
|
||||
type: boolean
|
||||
interval:
|
||||
type: string
|
||||
subresources:
|
||||
subresources:
|
||||
status: {}
|
||||
---
|
||||
|
||||
@@ -431,6 +431,12 @@ spec:
|
||||
securePort: {}
|
||||
instances:
|
||||
type: integer
|
||||
externalRgwEndpoints:
|
||||
type: array
|
||||
items:
|
||||
properties:
|
||||
ip:
|
||||
type: string
|
||||
annotations: {}
|
||||
placement: {}
|
||||
resources: {}
|
||||
@@ -484,6 +490,14 @@ spec:
|
||||
- force
|
||||
preservePoolsOnDelete:
|
||||
type: boolean
|
||||
healthCheck:
|
||||
properties:
|
||||
bucket:
|
||||
properties:
|
||||
enabled:
|
||||
type: boolean
|
||||
interval:
|
||||
type: string
|
||||
subresources:
|
||||
status: {}
|
||||
# OLM: END CEPH OBJECT STORE CRD
|
||||
|
||||
@@ -289,6 +289,7 @@ class RadosJSON:
|
||||
"osd", ("allow rwx pool={0}.rgw.meta, " +
|
||||
"allow r pool=.rgw.root, " +
|
||||
"allow rw pool={0}.rgw.control, " +
|
||||
"allow rx pool={0}.rgw.log, " +
|
||||
"allow x pool={0}.rgw.buckets.index").format(self._arg_parser.rgw_pool_prefix)],
|
||||
"format": "json"}
|
||||
ret_val, json_out, err_msg = self._common_cmd_json_gen(cmd_json)
|
||||
|
||||
@@ -32,7 +32,7 @@ fi
|
||||
}
|
||||
|
||||
function createCheckerKey() {
|
||||
checkerKey=$(ceph auth get-or-create "$CLIENT_CHECKER_NAME" mon 'allow r, allow command quorum_status' osd 'allow rwx pool='"$RGW_POOL_PREFIX"'.rgw.meta, allow r pool=.rgw.root, allow rw pool='"$RGW_POOL_PREFIX"'.rgw.control, allow x pool='"$RGW_POOL_PREFIX"'.rgw.buckets.index'|awk '/key =/ { print $3}')
|
||||
checkerKey=$(ceph auth get-or-create "$CLIENT_CHECKER_NAME" mon 'allow r, allow command quorum_status' osd 'allow rwx pool='"$RGW_POOL_PREFIX"'.rgw.meta, allow r pool=.rgw.root, allow rw pool='"$RGW_POOL_PREFIX"'.rgw.control, allow x pool='"$RGW_POOL_PREFIX"'.rgw.buckets.index, allow x pool='"$RGW_POOL_PREFIX"'.rgw.log'|awk '/key =/ { print $3}')
|
||||
echo "export ROOK_EXTERNAL_USER_SECRET=$checkerKey"
|
||||
echo "export ROOK_EXTERNAL_USERNAME=$CLIENT_CHECKER_NAME"
|
||||
}
|
||||
|
||||
@@ -0,0 +1,22 @@
|
||||
#################################################################################################################
|
||||
# Create an object store with settings for replication in a production environment. A minimum of 3 hosts with
|
||||
# OSDs are required in this example.
|
||||
# kubectl create -f object.yaml
|
||||
#################################################################################################################
|
||||
|
||||
apiVersion: ceph.rook.io/v1
|
||||
kind: CephObjectStore
|
||||
metadata:
|
||||
name: external-store
|
||||
namespace: rook-ceph
|
||||
spec:
|
||||
gateway:
|
||||
# The port on which **ALL** the gateway(s) are listening on.
|
||||
# Passing a single IP from a load-balancer is also valid.
|
||||
port: 80
|
||||
externalRgwEndpoints:
|
||||
- ip: 192.168.39.182
|
||||
healthCheck:
|
||||
bucket:
|
||||
enabled: true
|
||||
interval: 60s
|
||||
@@ -84,3 +84,8 @@ spec:
|
||||
# priorityClassName: my-priority-class
|
||||
#zone:
|
||||
#name: zone-a
|
||||
# service endpoint healthcheck
|
||||
healthCheck:
|
||||
endpoint:
|
||||
enabled: true
|
||||
interval: 60s
|
||||
@@ -56,7 +56,7 @@ data:
|
||||
|
||||
# (Optional) set user created priorityclassName for csi provisioner pods.
|
||||
# CSI_PROVISIONER_PRIORITY_CLASSNAME: "system-cluster-critical"
|
||||
|
||||
|
||||
# CSI CephFS plugin daemonset update strategy, supported values are OnDelete and RollingUpdate.
|
||||
# Default value is RollingUpdate.
|
||||
# CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY: "OnDelete"
|
||||
|
||||
@@ -207,4 +207,104 @@ spec:
|
||||
- name: Health
|
||||
type: string
|
||||
description: Ceph Health
|
||||
JSONPath: .status.ceph.health
|
||||
JSONPath: .status.ceph.health
|
||||
---
|
||||
apiVersion: apiextensions.k8s.io/v1beta1
|
||||
kind: CustomResourceDefinition
|
||||
metadata:
|
||||
name: cephobjectstores.ceph.rook.io
|
||||
spec:
|
||||
group: ceph.rook.io
|
||||
names:
|
||||
kind: CephObjectStore
|
||||
listKind: CephObjectStoreList
|
||||
plural: cephobjectstores
|
||||
singular: cephobjectstore
|
||||
scope: Namespaced
|
||||
version: v1
|
||||
validation:
|
||||
openAPIV3Schema:
|
||||
properties:
|
||||
spec:
|
||||
properties:
|
||||
gateway:
|
||||
properties:
|
||||
type:
|
||||
type: string
|
||||
sslCertificateRef: {}
|
||||
port:
|
||||
type: integer
|
||||
minimum: 1
|
||||
maximum: 65535
|
||||
securePort: {}
|
||||
instances:
|
||||
type: integer
|
||||
externalRgwEndpoints:
|
||||
type: array
|
||||
items:
|
||||
properties:
|
||||
ip:
|
||||
type: string
|
||||
annotations: {}
|
||||
placement: {}
|
||||
resources: {}
|
||||
metadataPool:
|
||||
properties:
|
||||
failureDomain:
|
||||
type: string
|
||||
replicated:
|
||||
properties:
|
||||
size:
|
||||
type: integer
|
||||
requireSafeReplicaSize:
|
||||
type: boolean
|
||||
erasureCoded:
|
||||
properties:
|
||||
dataChunks:
|
||||
type: integer
|
||||
codingChunks:
|
||||
type: integer
|
||||
compressionMode:
|
||||
type: string
|
||||
enum:
|
||||
- ""
|
||||
- none
|
||||
- passive
|
||||
- aggressive
|
||||
- force
|
||||
dataPool:
|
||||
properties:
|
||||
failureDomain:
|
||||
type: string
|
||||
replicated:
|
||||
properties:
|
||||
size:
|
||||
type: integer
|
||||
requireSafeReplicaSize:
|
||||
type: boolean
|
||||
erasureCoded:
|
||||
properties:
|
||||
dataChunks:
|
||||
type: integer
|
||||
codingChunks:
|
||||
type: integer
|
||||
compressionMode:
|
||||
type: string
|
||||
enum:
|
||||
- ""
|
||||
- none
|
||||
- passive
|
||||
- aggressive
|
||||
- force
|
||||
preservePoolsOnDelete:
|
||||
type: boolean
|
||||
healthCheck:
|
||||
properties:
|
||||
bucket:
|
||||
properties:
|
||||
enabled:
|
||||
type: boolean
|
||||
interval:
|
||||
type: string
|
||||
subresources:
|
||||
status: {}
|
||||
@@ -190,6 +190,7 @@ const (
|
||||
ConditionFailure ConditionType = "Failure"
|
||||
ConditionUpgrading ConditionType = "Upgrading"
|
||||
ConditionDeleting ConditionType = "Deleting"
|
||||
ConditionHealthy ConditionType = "Healthy"
|
||||
// DefaultFailureDomain for PoolSpec
|
||||
DefaultFailureDomain = "host"
|
||||
)
|
||||
@@ -364,8 +365,8 @@ type MetadataServerSpec struct {
|
||||
type CephObjectStore struct {
|
||||
metav1.TypeMeta `json:",inline"`
|
||||
metav1.ObjectMeta `json:"metadata"`
|
||||
Spec ObjectStoreSpec `json:"spec"`
|
||||
Status *Status `json:"status"`
|
||||
Spec ObjectStoreSpec `json:"spec"`
|
||||
Status *ObjectStoreStatus `json:"status"`
|
||||
}
|
||||
|
||||
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
|
||||
@@ -392,6 +393,18 @@ type ObjectStoreSpec struct {
|
||||
|
||||
// The multisite info
|
||||
Zone ZoneSpec `json:"zone"`
|
||||
|
||||
// The rgw endpoint healthcheck
|
||||
HealthCheck HealthCheckSpec `json:"healthCheck"`
|
||||
}
|
||||
|
||||
type HealthCheckSpec struct {
|
||||
Bucket BucketHealthCheckSpec `json:"bucket,omitempty"`
|
||||
}
|
||||
|
||||
type BucketHealthCheckSpec struct {
|
||||
Enabled bool `json:"enabled,omitempty"`
|
||||
Interval string `json:"interval,omitempty"`
|
||||
}
|
||||
|
||||
type GatewaySpec struct {
|
||||
@@ -421,6 +434,9 @@ type GatewaySpec struct {
|
||||
|
||||
// PriorityClassName sets priority classes on the rgw pods
|
||||
PriorityClassName string `json:"priorityClassName,omitempty"`
|
||||
|
||||
// ExternalRgwEndpoints points to external rgw endpoint(s)
|
||||
ExternalRgwEndpoints []v1.EndpointAddress `json:"externalRgwEndpoints,omitempty"`
|
||||
}
|
||||
|
||||
type ZoneSpec struct {
|
||||
@@ -428,6 +444,19 @@ type ZoneSpec struct {
|
||||
Name string `json:"name"`
|
||||
}
|
||||
|
||||
type ObjectStoreStatus struct {
|
||||
Phase ConditionType `json:"phase,omitempty"`
|
||||
Message string `json:"message,omitempty"`
|
||||
BucketStatus *BucketStatus `json:"bucketStatus,omitempty"`
|
||||
}
|
||||
|
||||
type BucketStatus struct {
|
||||
Health ConditionType `json:"health,omitempty"`
|
||||
Details string `json:"details,omitempty"`
|
||||
LastChecked string `json:"lastChecked,omitempty"`
|
||||
LastChanged string `json:"lastChanged,omitempty"`
|
||||
}
|
||||
|
||||
// +genclient
|
||||
// +genclient:noStatus
|
||||
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
|
||||
|
||||
@@ -26,6 +26,38 @@ import (
|
||||
runtime "k8s.io/apimachinery/pkg/runtime"
|
||||
)
|
||||
|
||||
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
|
||||
func (in *BucketHealthCheckSpec) DeepCopyInto(out *BucketHealthCheckSpec) {
|
||||
*out = *in
|
||||
return
|
||||
}
|
||||
|
||||
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new BucketHealthCheckSpec.
|
||||
func (in *BucketHealthCheckSpec) DeepCopy() *BucketHealthCheckSpec {
|
||||
if in == nil {
|
||||
return nil
|
||||
}
|
||||
out := new(BucketHealthCheckSpec)
|
||||
in.DeepCopyInto(out)
|
||||
return out
|
||||
}
|
||||
|
||||
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
|
||||
func (in *BucketStatus) DeepCopyInto(out *BucketStatus) {
|
||||
*out = *in
|
||||
return
|
||||
}
|
||||
|
||||
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new BucketStatus.
|
||||
func (in *BucketStatus) DeepCopy() *BucketStatus {
|
||||
if in == nil {
|
||||
return nil
|
||||
}
|
||||
out := new(BucketStatus)
|
||||
in.DeepCopyInto(out)
|
||||
return out
|
||||
}
|
||||
|
||||
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
|
||||
func (in *CephBlockPool) DeepCopyInto(out *CephBlockPool) {
|
||||
*out = *in
|
||||
@@ -431,8 +463,8 @@ func (in *CephObjectStore) DeepCopyInto(out *CephObjectStore) {
|
||||
in.Spec.DeepCopyInto(&out.Spec)
|
||||
if in.Status != nil {
|
||||
in, out := &in.Status, &out.Status
|
||||
*out = new(Status)
|
||||
**out = **in
|
||||
*out = new(ObjectStoreStatus)
|
||||
(*in).DeepCopyInto(*out)
|
||||
}
|
||||
return
|
||||
}
|
||||
@@ -1116,6 +1148,13 @@ func (in *GatewaySpec) DeepCopyInto(out *GatewaySpec) {
|
||||
}
|
||||
}
|
||||
in.Resources.DeepCopyInto(&out.Resources)
|
||||
if in.ExternalRgwEndpoints != nil {
|
||||
in, out := &in.ExternalRgwEndpoints, &out.ExternalRgwEndpoints
|
||||
*out = make([]corev1.EndpointAddress, len(*in))
|
||||
for i := range *in {
|
||||
(*in)[i].DeepCopyInto(&(*out)[i])
|
||||
}
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
@@ -1129,6 +1168,23 @@ func (in *GatewaySpec) DeepCopy() *GatewaySpec {
|
||||
return out
|
||||
}
|
||||
|
||||
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
|
||||
func (in *HealthCheckSpec) DeepCopyInto(out *HealthCheckSpec) {
|
||||
*out = *in
|
||||
out.Bucket = in.Bucket
|
||||
return
|
||||
}
|
||||
|
||||
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new HealthCheckSpec.
|
||||
func (in *HealthCheckSpec) DeepCopy() *HealthCheckSpec {
|
||||
if in == nil {
|
||||
return nil
|
||||
}
|
||||
out := new(HealthCheckSpec)
|
||||
in.DeepCopyInto(out)
|
||||
return out
|
||||
}
|
||||
|
||||
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
|
||||
func (in *MetadataServerSpec) DeepCopyInto(out *MetadataServerSpec) {
|
||||
*out = *in
|
||||
@@ -1286,6 +1342,7 @@ func (in *ObjectStoreSpec) DeepCopyInto(out *ObjectStoreSpec) {
|
||||
in.DataPool.DeepCopyInto(&out.DataPool)
|
||||
in.Gateway.DeepCopyInto(&out.Gateway)
|
||||
out.Zone = in.Zone
|
||||
out.HealthCheck = in.HealthCheck
|
||||
return
|
||||
}
|
||||
|
||||
@@ -1299,6 +1356,27 @@ func (in *ObjectStoreSpec) DeepCopy() *ObjectStoreSpec {
|
||||
return out
|
||||
}
|
||||
|
||||
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
|
||||
func (in *ObjectStoreStatus) DeepCopyInto(out *ObjectStoreStatus) {
|
||||
*out = *in
|
||||
if in.BucketStatus != nil {
|
||||
in, out := &in.BucketStatus, &out.BucketStatus
|
||||
*out = new(BucketStatus)
|
||||
**out = **in
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ObjectStoreStatus.
|
||||
func (in *ObjectStoreStatus) DeepCopy() *ObjectStoreStatus {
|
||||
if in == nil {
|
||||
return nil
|
||||
}
|
||||
out := new(ObjectStoreStatus)
|
||||
in.DeepCopyInto(out)
|
||||
return out
|
||||
}
|
||||
|
||||
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
|
||||
func (in *ObjectStoreUserSpec) DeepCopyInto(out *ObjectStoreUserSpec) {
|
||||
*out = *in
|
||||
|
||||
@@ -29,6 +29,7 @@ type Context struct {
|
||||
Name string
|
||||
ClusterName string
|
||||
RunAsUser string
|
||||
UID string
|
||||
}
|
||||
|
||||
// NewContext creates a new object store context.
|
||||
@@ -52,7 +53,11 @@ func runAdminCommand(c *Context, args ...string) (string, error) {
|
||||
// If the objectStoreName is not passed in the storage class
|
||||
// This means we are pointing to an external cluster so these commands are not needed
|
||||
// simply because the external cluster mode does not support that yet
|
||||
if c.Name != "" {
|
||||
//
|
||||
// The following conditions tries to determine if the cluster is external
|
||||
// When connecting to an external cluster, the Ceph user is different than client.admin
|
||||
// This is not perfect though since "client.admin" is somehow supported...
|
||||
if c.Name != "" && c.RunAsUser == client.AdminUsername {
|
||||
options := []string{
|
||||
fmt.Sprintf("--rgw-realm=%s", c.Name),
|
||||
fmt.Sprintf("--rgw-zonegroup=%s", c.Name),
|
||||
|
||||
@@ -211,7 +211,7 @@ func GetBucket(c *Context, bucket string) (*ObjectBucket, int, error) {
|
||||
return &ObjectBucket{Name: bucket, ObjectBucketMetadata: ObjectBucketMetadata{Owner: metadata.Owner, CreatedAt: metadata.CreatedAt}, ObjectBucketStats: *stat}, RGWErrorNone, nil
|
||||
}
|
||||
|
||||
func DeleteBucket(c *Context, bucketName string, purge bool) (int, error) {
|
||||
func DeleteObjectBucket(c *Context, bucketName string, purge bool) (int, error) {
|
||||
options := []string{"bucket", "rm", "--bucket", bucketName}
|
||||
if purge {
|
||||
options = append(options, "--purge-objects")
|
||||
|
||||
@@ -84,7 +84,7 @@ func (p Provisioner) Provision(options *apibkt.BucketOptions) (*bktv1alpha1.Obje
|
||||
return nil, errors.Wrap(err, "Provision: can't create ceph user")
|
||||
}
|
||||
|
||||
s3svc, err := NewS3Agent(p.accessKeyID, p.secretAccessKey, p.getObjectStoreEndpoint())
|
||||
s3svc, err := cephObject.NewS3Agent(p.accessKeyID, p.secretAccessKey, p.getObjectStoreEndpoint())
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@@ -149,7 +149,7 @@ func (p Provisioner) Grant(options *apibkt.BucketOptions) (*bktv1alpha1.ObjectBu
|
||||
return nil, errors.Wrapf(err, "could not get user (user: %s)", stats.Owner)
|
||||
}
|
||||
|
||||
s3svc, err := NewS3Agent(*objectUser.AccessKey, *objectUser.SecretKey, p.getObjectStoreEndpoint())
|
||||
s3svc, err := cephObject.NewS3Agent(*objectUser.AccessKey, *objectUser.SecretKey, p.getObjectStoreEndpoint())
|
||||
if err != nil {
|
||||
p.deleteOBCResource("")
|
||||
return nil, err
|
||||
@@ -166,15 +166,15 @@ func (p Provisioner) Grant(options *apibkt.BucketOptions) (*bktv1alpha1.ObjectBu
|
||||
}
|
||||
}
|
||||
|
||||
statement := NewPolicyStatement().
|
||||
statement := cephObject.NewPolicyStatement().
|
||||
WithSID(p.cephUserName).
|
||||
ForPrincipals(p.cephUserName).
|
||||
ForResources(p.bucketName).
|
||||
ForSubResources(p.bucketName).
|
||||
Allows().
|
||||
Actions(AllowedActions...)
|
||||
Actions(cephObject.AllowedActions...)
|
||||
if policy == nil {
|
||||
policy = NewBucketPolicy(*statement)
|
||||
policy = cephObject.NewBucketPolicy(*statement)
|
||||
} else {
|
||||
policy = policy.ModifyBucketPolicy(*statement)
|
||||
}
|
||||
@@ -240,7 +240,7 @@ func (p Provisioner) Revoke(ob *bktv1alpha1.ObjectBucket) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
s3svc, err := NewS3Agent(*user.AccessKey, *user.SecretKey, p.getObjectStoreEndpoint())
|
||||
s3svc, err := cephObject.NewS3Agent(*user.AccessKey, *user.SecretKey, p.getObjectStoreEndpoint())
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -260,15 +260,15 @@ func (p Provisioner) Revoke(ob *bktv1alpha1.ObjectBucket) error {
|
||||
}
|
||||
|
||||
if bucket.Owner == p.cephUserName {
|
||||
statement := NewPolicyStatement().
|
||||
statement := cephObject.NewPolicyStatement().
|
||||
WithSID(p.cephUserName).
|
||||
ForPrincipals(p.cephUserName).
|
||||
ForResources(p.bucketName).
|
||||
ForSubResources(p.bucketName).
|
||||
Denies().
|
||||
Actions(AllowedActions...)
|
||||
Actions(cephObject.AllowedActions...)
|
||||
if policy == nil {
|
||||
policy = NewBucketPolicy(*statement)
|
||||
policy = cephObject.NewBucketPolicy(*statement)
|
||||
} else {
|
||||
policy = policy.ModifyBucketPolicy(*statement)
|
||||
}
|
||||
@@ -420,13 +420,13 @@ func (p *Provisioner) setObjectStoreDomainName(sc *storagev1.StorageClass) error
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
p.storeDomainName = fmt.Sprintf("%s-%s.%s", prefixObjectStoreSvc, name, namespace)
|
||||
p.storeDomainName = cephObject.BuildDomainName(name, namespace)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (p *Provisioner) setObjectStorePort(sc *storagev1.StorageClass) error {
|
||||
name := getObjectStoreName(sc)
|
||||
name = fmt.Sprintf("%s-%s", prefixObjectStoreSvc, name)
|
||||
name = fmt.Sprintf("%s-%s", cephObject.AppName, name)
|
||||
namespace := getObjectStoreNameSpace(sc)
|
||||
// also ensure the service exists and get the appropriate clusterIP port
|
||||
svc, err := getService(p.context.Clientset, namespace, name)
|
||||
|
||||
@@ -24,6 +24,7 @@ import (
|
||||
rookclient "github.com/rook/rook/pkg/client/clientset/versioned/fake"
|
||||
"github.com/rook/rook/pkg/clusterd"
|
||||
"github.com/rook/rook/pkg/daemon/ceph/client"
|
||||
"github.com/rook/rook/pkg/operator/ceph/object"
|
||||
"github.com/rook/rook/pkg/operator/test"
|
||||
"github.com/stretchr/testify/assert"
|
||||
v1 "k8s.io/api/core/v1"
|
||||
@@ -85,7 +86,7 @@ func TestPopulateDomainAndPort(t *testing.T) {
|
||||
}
|
||||
svc := &v1.Service{
|
||||
ObjectMeta: metav1.ObjectMeta{
|
||||
Name: fmt.Sprintf("%s-%s", prefixObjectStoreSvc, store),
|
||||
Name: fmt.Sprintf("%s-%s", object.AppName, store),
|
||||
Namespace: namespace,
|
||||
},
|
||||
Spec: v1.ServiceSpec{
|
||||
|
||||
@@ -100,7 +100,7 @@ func (p *Provisioner) deleteOBCResource(bucketName string) error {
|
||||
}
|
||||
if len(bucketName) > 0 {
|
||||
// delete bucket with purge option to remove all objects
|
||||
errCode, err := cephObject.DeleteBucket(p.objectContext, bucketName, true)
|
||||
errCode, err := cephObject.DeleteObjectBucket(p.objectContext, bucketName, true)
|
||||
|
||||
if errCode == cephObject.RGWErrorNone {
|
||||
logger.Infof("bucket %q successfully deleted", p.bucketName)
|
||||
|
||||
@@ -41,7 +41,6 @@ var logger = capnslog.NewPackageLogger("github.com/rook/rook", "op-bucket-prov")
|
||||
const (
|
||||
genUserLen = 8
|
||||
cephUser = "cephUser"
|
||||
prefixObjectStoreSvc = "rook-ceph-rgw"
|
||||
accessKeyIdKey = "accessKeyID"
|
||||
secretSecretKeyKey = "secretAccessKey"
|
||||
objectStoreName = "objectStoreName"
|
||||
|
||||
@@ -20,6 +20,7 @@ import (
|
||||
"context"
|
||||
"fmt"
|
||||
"reflect"
|
||||
"strconv"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
@@ -34,7 +35,6 @@ import (
|
||||
"github.com/rook/rook/pkg/operator/ceph/cluster/mon"
|
||||
opconfig "github.com/rook/rook/pkg/operator/ceph/config"
|
||||
opcontroller "github.com/rook/rook/pkg/operator/ceph/controller"
|
||||
"github.com/rook/rook/pkg/operator/k8sutil"
|
||||
"github.com/rook/rook/pkg/util/exec"
|
||||
appsv1 "k8s.io/api/apps/v1"
|
||||
corev1 "k8s.io/api/core/v1"
|
||||
@@ -76,12 +76,18 @@ var controllerTypeMeta = metav1.TypeMeta{
|
||||
|
||||
// ReconcileCephObjectStore reconciles a cephObjectStore object
|
||||
type ReconcileCephObjectStore struct {
|
||||
client client.Client
|
||||
bktclient bktclient.Interface
|
||||
scheme *runtime.Scheme
|
||||
context *clusterd.Context
|
||||
cephClusterSpec *cephv1.ClusterSpec
|
||||
clusterInfo *cephconfig.ClusterInfo
|
||||
client client.Client
|
||||
bktclient bktclient.Interface
|
||||
scheme *runtime.Scheme
|
||||
context *clusterd.Context
|
||||
cephClusterSpec *cephv1.ClusterSpec
|
||||
clusterInfo *cephconfig.ClusterInfo
|
||||
objectStoreChannels map[string]*objectStoreHealth
|
||||
}
|
||||
|
||||
type objectStoreHealth struct {
|
||||
stopChan chan struct{}
|
||||
monitoringRunning bool
|
||||
}
|
||||
|
||||
// Add creates a new cephObjectStore Controller and adds it to the Manager. The Manager will set fields on the Controller
|
||||
@@ -97,10 +103,11 @@ func newReconciler(mgr manager.Manager, context *clusterd.Context) reconcile.Rec
|
||||
cephv1.AddToScheme(mgr.GetScheme())
|
||||
|
||||
return &ReconcileCephObjectStore{
|
||||
client: mgr.GetClient(),
|
||||
scheme: mgrScheme,
|
||||
context: context,
|
||||
bktclient: bktclient.NewForConfigOrDie(context.KubeConfig),
|
||||
client: mgr.GetClient(),
|
||||
scheme: mgrScheme,
|
||||
context: context,
|
||||
bktclient: bktclient.NewForConfigOrDie(context.KubeConfig),
|
||||
objectStoreChannels: make(map[string]*objectStoreHealth),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -161,7 +168,7 @@ func (r *ReconcileCephObjectStore) reconcile(request reconcile.Request) (reconci
|
||||
|
||||
// The CR was just created, initializing status fields
|
||||
if cephObjectStore.Status == nil {
|
||||
updateStatus(r.client, request.NamespacedName, k8sutil.Created)
|
||||
updateStatusPhase(r.client, request.NamespacedName, cephv1.ConditionProgressing)
|
||||
}
|
||||
|
||||
// Make sure a CephCluster is present otherwise do nothing
|
||||
@@ -188,20 +195,39 @@ func (r *ReconcileCephObjectStore) reconcile(request reconcile.Request) (reconci
|
||||
}
|
||||
r.cephClusterSpec = &cephCluster.Spec
|
||||
|
||||
// Initialize the channel for this object store
|
||||
// This allows us to track multiple ObjectStores in the same namespace
|
||||
_, ok := r.objectStoreChannels[cephObjectStore.Name]
|
||||
if !ok {
|
||||
r.objectStoreChannels[cephObjectStore.Name] = &objectStoreHealth{
|
||||
stopChan: make(chan struct{}),
|
||||
monitoringRunning: false,
|
||||
}
|
||||
}
|
||||
|
||||
// Populate clusterInfo
|
||||
// Always populate it during each reconcile
|
||||
clusterInfo, _, _, err := mon.LoadClusterInfo(r.context, request.NamespacedName.Namespace)
|
||||
if err != nil {
|
||||
return reconcile.Result{}, errors.Wrap(err, "failed to populate cluster info")
|
||||
var clusterInfo *cephconfig.ClusterInfo
|
||||
if r.cephClusterSpec.External.Enable {
|
||||
clusterInfo = mon.PopulateExternalClusterInfo(r.context, request.NamespacedName.Namespace)
|
||||
} else {
|
||||
clusterInfo, _, _, err = mon.LoadClusterInfo(r.context, request.NamespacedName.Namespace)
|
||||
if err != nil {
|
||||
return reconcile.Result{}, errors.Wrap(err, "failed to populate cluster info")
|
||||
}
|
||||
}
|
||||
r.clusterInfo = clusterInfo
|
||||
|
||||
// Populate CephVersion
|
||||
currentCephVersion, err := cephclient.LeastUptodateDaemonVersion(r.context, r.clusterInfo.Name, opconfig.MonType)
|
||||
if err != nil {
|
||||
return reconcile.Result{}, errors.Wrapf(err, "failed to retrieve current ceph %q version", opconfig.MonType)
|
||||
// This only needed when bootstrapping rgw pods
|
||||
// Might improve if we set the version by running a 'ceph version' command on a goroutine and update it every now and then
|
||||
if !r.cephClusterSpec.External.Enable {
|
||||
currentCephVersion, err := cephclient.LeastUptodateDaemonVersion(r.context, r.clusterInfo.Name, opconfig.MonType)
|
||||
if err != nil {
|
||||
return reconcile.Result{}, errors.Wrapf(err, "failed to retrieve current ceph %q version", opconfig.MonType)
|
||||
}
|
||||
r.clusterInfo.CephVersion = currentCephVersion
|
||||
}
|
||||
r.clusterInfo.CephVersion = currentCephVersion
|
||||
|
||||
// Set a finalizer so we can do cleanup before the object goes away
|
||||
err = opcontroller.AddFinalizerIfNotPresent(r.client, cephObjectStore)
|
||||
@@ -220,7 +246,17 @@ func (r *ReconcileCephObjectStore) reconcile(request reconcile.Request) (reconci
|
||||
return response, nil
|
||||
}
|
||||
|
||||
cfg := clusterConfig{context: r.context, store: cephObjectStore}
|
||||
// Close the channel to stop the healthcheck of the endpoint
|
||||
close(r.objectStoreChannels[cephObjectStore.Name].stopChan)
|
||||
|
||||
// Remove object store from the map
|
||||
delete(r.objectStoreChannels, cephObjectStore.Name)
|
||||
|
||||
cfg := clusterConfig{
|
||||
context: r.context,
|
||||
store: cephObjectStore,
|
||||
clusterSpec: r.cephClusterSpec,
|
||||
}
|
||||
err = cfg.deleteStore()
|
||||
if err != nil {
|
||||
return reconcile.Result{}, errors.Wrapf(err, "failed to delete store %q", cephObjectStore.Name)
|
||||
@@ -237,35 +273,25 @@ func (r *ReconcileCephObjectStore) reconcile(request reconcile.Request) (reconci
|
||||
}
|
||||
|
||||
// validate the store settings
|
||||
if err := validateStore(r.context, cephObjectStore); err != nil {
|
||||
if err := r.validateStore(cephObjectStore); err != nil {
|
||||
return reconcile.Result{}, errors.Wrapf(err, "invalid object store %q arguments", cephObjectStore.Name)
|
||||
}
|
||||
|
||||
// CREATE/UPDATE
|
||||
logger.Info("reconciling object store deployments")
|
||||
reconcileResponse, err = r.reconcileCreateObjectStore(cephObjectStore, request.NamespacedName)
|
||||
if err != nil {
|
||||
return r.setFailedStatus(request.NamespacedName, "failed to create object store deployments", err)
|
||||
}
|
||||
|
||||
// Set Ready status, we are done reconciling
|
||||
updateStatus(r.client, request.NamespacedName, k8sutil.ReadyStatus)
|
||||
updateStatusPhase(r.client, request.NamespacedName, cephv1.ConditionReady)
|
||||
|
||||
// Return and do not requeue
|
||||
logger.Debug("done reconciling")
|
||||
return reconcile.Result{}, nil
|
||||
}
|
||||
|
||||
func (r *ReconcileCephObjectStore) reconcileCreateObjectStore(cephObjectStore *cephv1.CephObjectStore, name types.NamespacedName) (reconcile.Result, error) {
|
||||
if r.cephClusterSpec.External.Enable {
|
||||
_, err := opcontroller.ValidateCephVersionsBetweenLocalAndExternalClusters(r.context, cephObjectStore.Namespace, r.clusterInfo.CephVersion)
|
||||
if err != nil {
|
||||
// This handles the case where the operator is running, the external cluster has been upgraded and a CR creation is called
|
||||
// If that's a major version upgrade we fail, if it's a minor version, we continue, it's not ideal but not critical
|
||||
return reconcile.Result{}, errors.Wrap(err, "refusing to run new crd")
|
||||
}
|
||||
}
|
||||
|
||||
func (r *ReconcileCephObjectStore) reconcileCreateObjectStore(cephObjectStore *cephv1.CephObjectStore, namespacedName types.NamespacedName) (reconcile.Result, error) {
|
||||
cfg := clusterConfig{
|
||||
context: r.context,
|
||||
clusterInfo: r.clusterInfo,
|
||||
@@ -277,47 +303,80 @@ func (r *ReconcileCephObjectStore) reconcileCreateObjectStore(cephObjectStore *c
|
||||
scheme: r.scheme,
|
||||
skipUpgradeChecks: r.cephClusterSpec.SkipUpgradeChecks,
|
||||
}
|
||||
objContext := NewContext(r.context, cephObjectStore.Name, cephObjectStore.Namespace)
|
||||
objContext.UID = string(cephObjectStore.UID)
|
||||
|
||||
// Reconcile realm/zonegroup/zone CRs & update their names
|
||||
realmName, zoneGroupName, zoneName, reconcileResponse, err := r.reconcileMultisiteCRs(cephObjectStore)
|
||||
if err != nil {
|
||||
return reconcileResponse, err
|
||||
}
|
||||
var serviceIP string
|
||||
var err error
|
||||
|
||||
// Reconcile Ceph Zone if Multisite
|
||||
if cephObjectStore.Spec.IsMultisite() {
|
||||
reconcileResponse, err := r.reconcileCephZone(cephObjectStore, zoneGroupName, realmName)
|
||||
if r.cephClusterSpec.External.Enable {
|
||||
// Assign the cephx user to run Ceph commands with
|
||||
objContext.RunAsUser = r.clusterInfo.ExternalCred.Username
|
||||
|
||||
logger.Info("reconciling external object store")
|
||||
|
||||
// RECONCILE SERVICE
|
||||
logger.Info("reconciling object store service")
|
||||
serviceIP, err = cfg.reconcileService(cephObjectStore)
|
||||
if err != nil {
|
||||
return r.setFailedStatus(namespacedName, "failed to reconcile service", err)
|
||||
}
|
||||
|
||||
// RECONCILE ENDPOINTS
|
||||
// Always add the endpoint AFTER the service otherwise it will get overridden
|
||||
logger.Info("reconciling external object store endpoint")
|
||||
err = cfg.reconcileExternalEndpoint(cephObjectStore)
|
||||
if err != nil {
|
||||
return r.setFailedStatus(namespacedName, "failed to reconcile external endpoint", err)
|
||||
}
|
||||
|
||||
} else {
|
||||
logger.Info("reconciling object store deployments")
|
||||
|
||||
// Reconcile realm/zonegroup/zone CRs & update their names
|
||||
realmName, zoneGroupName, zoneName, reconcileResponse, err := r.reconcileMultisiteCRs(cephObjectStore)
|
||||
if err != nil {
|
||||
return reconcileResponse, err
|
||||
}
|
||||
|
||||
// Reconcile Ceph Zone if Multisite
|
||||
if cephObjectStore.Spec.IsMultisite() {
|
||||
reconcileResponse, err := r.reconcileCephZone(cephObjectStore, zoneGroupName, realmName)
|
||||
if err != nil {
|
||||
return reconcileResponse, err
|
||||
}
|
||||
}
|
||||
|
||||
// RECONCILE SERVICE
|
||||
logger.Debug("reconciling object store service")
|
||||
serviceIP, err := cfg.reconcileService(cephObjectStore)
|
||||
if err != nil {
|
||||
return r.setFailedStatus(namespacedName, "failed to reconcile service", err)
|
||||
}
|
||||
|
||||
// RECONCILE POOLS
|
||||
logger.Info("reconciling object store pools")
|
||||
err = createPools(objContext, cephObjectStore.Spec)
|
||||
if err != nil {
|
||||
return r.setFailedStatus(namespacedName, "failed to create object pools", err)
|
||||
}
|
||||
|
||||
// RECONCILE REALM
|
||||
logger.Infof("setting multisite settings for object store %q", cephObjectStore.Name)
|
||||
err = setMultisite(objContext, serviceIP, cephObjectStore.Spec, realmName, zoneGroupName, zoneName)
|
||||
if err != nil {
|
||||
return r.setFailedStatus(namespacedName, "failed to configure multisite for object store", err)
|
||||
}
|
||||
|
||||
err = cfg.createOrUpdateStore(realmName, zoneGroupName, zoneName)
|
||||
if err != nil {
|
||||
return reconcile.Result{}, errors.Wrapf(err, "failed to create object store %q", cephObjectStore.Name)
|
||||
}
|
||||
}
|
||||
|
||||
// RECONCILE SERVICE
|
||||
logger.Debug("reconciling object store service")
|
||||
serviceIP, err := cfg.reconcileService(cephObjectStore)
|
||||
if err != nil {
|
||||
return r.setFailedStatus(name, "failed to reconcile service", err)
|
||||
}
|
||||
|
||||
objContext := NewContext(r.context, cephObjectStore.Name, cephObjectStore.Namespace)
|
||||
|
||||
// RECONCILE POOLS
|
||||
logger.Info("reconciling object store pools")
|
||||
err = createPools(objContext, cephObjectStore.Spec)
|
||||
if err != nil {
|
||||
return r.setFailedStatus(name, "failed to create object pools", err)
|
||||
}
|
||||
|
||||
// RECONCILE REALM
|
||||
logger.Infof("setting multisite settings for object store %q", cephObjectStore.Name)
|
||||
err = setMultisite(objContext, serviceIP, cephObjectStore.Spec, realmName, zoneGroupName, zoneName)
|
||||
if err != nil {
|
||||
return r.setFailedStatus(name, "failed to configure multisite for object store", err)
|
||||
}
|
||||
|
||||
err = cfg.createOrUpdateStore(realmName, zoneGroupName, zoneName)
|
||||
if err != nil {
|
||||
return reconcile.Result{}, errors.Wrapf(err, "failed to create object store %q", cephObjectStore.Name)
|
||||
// Start monitoring
|
||||
if cephObjectStore.Spec.HealthCheck.Bucket.Enabled {
|
||||
r.startMonitoring(cephObjectStore, objContext, serviceIP, namespacedName)
|
||||
}
|
||||
|
||||
return reconcile.Result{}, nil
|
||||
@@ -378,38 +437,10 @@ func (r *ReconcileCephObjectStore) reconcileMultisiteCRs(cephObjectStore *cephv1
|
||||
return cephObjectStore.Name, cephObjectStore.Name, cephObjectStore.Name, reconcile.Result{}, nil
|
||||
}
|
||||
|
||||
func (r *ReconcileCephObjectStore) setFailedStatus(name types.NamespacedName, errMessage string, err error) (reconcile.Result, error) {
|
||||
updateStatus(r.client, name, k8sutil.ReconcileFailedStatus)
|
||||
return reconcile.Result{}, errors.Wrapf(err, "%s", errMessage)
|
||||
}
|
||||
|
||||
// updateStatus updates an object with a given status
|
||||
func updateStatus(client client.Client, name types.NamespacedName, status string) {
|
||||
objectStore := &cephv1.CephObjectStore{}
|
||||
if err := client.Get(context.TODO(), name, objectStore); err != nil {
|
||||
if kerrors.IsNotFound(err) {
|
||||
logger.Debug("CephObjectStore resource not found. Ignoring since object must be deleted.")
|
||||
return
|
||||
}
|
||||
logger.Warningf("failed to retrieve object store %q to update status to %q. %v", name, status, err)
|
||||
return
|
||||
}
|
||||
if objectStore.Status == nil {
|
||||
objectStore.Status = &cephv1.Status{}
|
||||
}
|
||||
|
||||
objectStore.Status.Phase = status
|
||||
if err := opcontroller.UpdateStatus(client, objectStore); err != nil {
|
||||
logger.Errorf("failed to set object store %q status to %q. %v", name, status, err)
|
||||
return
|
||||
}
|
||||
logger.Debugf("object store %q status updated to %q", name, status)
|
||||
}
|
||||
|
||||
func (r *ReconcileCephObjectStore) verifyObjectBucketCleanup(objectstore *cephv1.CephObjectStore) (reconcile.Result, bool) {
|
||||
bktProvsioner := GetObjectBucketProvisioner(r.context, objectstore.Namespace)
|
||||
bktProvsioner = strings.Replace(bktProvsioner, "/", "-", -1)
|
||||
selector := fmt.Sprintf("bucket-provisioner=%s", bktProvsioner)
|
||||
bktProvisioner := GetObjectBucketProvisioner(r.context, objectstore.Namespace)
|
||||
bktProvisioner = strings.Replace(bktProvisioner, "/", "-", -1)
|
||||
selector := fmt.Sprintf("bucket-provisioner=%s", bktProvisioner)
|
||||
objectBuckets, err := r.bktclient.ObjectbucketV1alpha1().ObjectBuckets().List(metav1.ListOptions{LabelSelector: selector})
|
||||
if err != nil {
|
||||
logger.Errorf("failed to delete object store. failed to list buckets for objectstore %q in namespace %q", objectstore.Name, objectstore.Namespace)
|
||||
@@ -429,3 +460,26 @@ func (r *ReconcileCephObjectStore) verifyObjectBucketCleanup(objectstore *cephv1
|
||||
logger.Errorf("failed to delete object store. buckets for objectstore %q in namespace %q are not cleaned up. remaining buckets: %+v", objectstore.Name, objectstore.Namespace, bucketNames)
|
||||
return opcontroller.WaitForRequeueIfFinalizerBlocked, false
|
||||
}
|
||||
|
||||
func (r *ReconcileCephObjectStore) startMonitoring(objectstore *cephv1.CephObjectStore, objContext *Context, serviceIP string, namespacedName types.NamespacedName) {
|
||||
// Start monitoring object store
|
||||
if r.objectStoreChannels[objectstore.Name].monitoringRunning {
|
||||
logger.Debug("external rgw endpoint monitoring go routine already running!")
|
||||
return
|
||||
}
|
||||
|
||||
// Set the monitoring flag so we don't start more than one go routine
|
||||
r.objectStoreChannels[objectstore.Name].monitoringRunning = true
|
||||
|
||||
var port string
|
||||
|
||||
if objectstore.Spec.Gateway.Port != 0 {
|
||||
port = strconv.Itoa(int(objectstore.Spec.Gateway.Port))
|
||||
} else if objectstore.Spec.Gateway.SecurePort != 0 {
|
||||
port = strconv.Itoa(int(objectstore.Spec.Gateway.SecurePort))
|
||||
}
|
||||
|
||||
rgwChecker := newBucketChecker(r.context, objContext, serviceIP, port, r.client, namespacedName, &objectstore.Spec.HealthCheck)
|
||||
logger.Info("starting rgw healthcheck")
|
||||
go rgwChecker.checkObjectStore(r.objectStoreChannels[objectstore.Name].stopChan)
|
||||
}
|
||||
@@ -197,7 +197,12 @@ func TestCephObjectStoreController(t *testing.T) {
|
||||
// Create a fake client to mock API calls.
|
||||
cl := fake.NewFakeClientWithScheme(s, object...)
|
||||
// Create a ReconcileCephObjectStore object with the scheme and fake client.
|
||||
r := &ReconcileCephObjectStore{client: cl, scheme: s, context: c}
|
||||
r := &ReconcileCephObjectStore{
|
||||
client: cl,
|
||||
scheme: s,
|
||||
context: c,
|
||||
objectStoreChannels: make(map[string]*objectStoreHealth),
|
||||
}
|
||||
|
||||
// Mock request to simulate Reconcile() being called on an event for a
|
||||
// watched resource .
|
||||
@@ -234,7 +239,12 @@ func TestCephObjectStoreController(t *testing.T) {
|
||||
// Create a fake client to mock API calls.
|
||||
cl = fake.NewFakeClientWithScheme(s, object...)
|
||||
// Create a ReconcileCephObjectStore object with the scheme and fake client.
|
||||
r = &ReconcileCephObjectStore{client: cl, scheme: s, context: c}
|
||||
r = &ReconcileCephObjectStore{
|
||||
client: cl,
|
||||
scheme: s,
|
||||
context: c,
|
||||
objectStoreChannels: make(map[string]*objectStoreHealth),
|
||||
}
|
||||
logger.Info("STARTING PHASE 2")
|
||||
res, err = r.Reconcile(req)
|
||||
assert.NoError(t, err)
|
||||
@@ -305,7 +315,12 @@ func TestCephObjectStoreController(t *testing.T) {
|
||||
c.Executor = executor
|
||||
|
||||
// Create a ReconcileCephObjectStore object with the scheme and fake client.
|
||||
r = &ReconcileCephObjectStore{client: cl, scheme: s, context: c}
|
||||
r = &ReconcileCephObjectStore{
|
||||
client: cl,
|
||||
scheme: s,
|
||||
context: c,
|
||||
objectStoreChannels: make(map[string]*objectStoreHealth),
|
||||
}
|
||||
|
||||
logger.Info("STARTING PHASE 3")
|
||||
res, err = r.Reconcile(req)
|
||||
@@ -313,6 +328,6 @@ func TestCephObjectStoreController(t *testing.T) {
|
||||
assert.False(t, res.Requeue)
|
||||
err = r.client.Get(context.TODO(), req.NamespacedName, objectStore)
|
||||
assert.NoError(t, err)
|
||||
assert.Equal(t, "Ready", objectStore.Status.Phase, objectStore)
|
||||
assert.Equal(t, cephv1.ConditionReady, objectStore.Status.Phase, objectStore)
|
||||
logger.Info("PHASE 3 DONE")
|
||||
}
|
||||
@@ -0,0 +1,260 @@
|
||||
/*
|
||||
Copyright 2020 The Rook Authors. All rights reserved.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
*/
|
||||
|
||||
package object
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/pkg/errors"
|
||||
cephv1 "github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
|
||||
"github.com/rook/rook/pkg/clusterd"
|
||||
"github.com/rook/rook/pkg/operator/k8sutil"
|
||||
"k8s.io/apimachinery/pkg/types"
|
||||
"sigs.k8s.io/controller-runtime/pkg/client"
|
||||
)
|
||||
|
||||
const (
|
||||
s3UserHealthCheckName = "rook-ceph-internal-s3-user-checker"
|
||||
s3HealthCheckBucketName = "rook-ceph-bucket-checker"
|
||||
defaultStatusCheckInterval = 1 * time.Minute
|
||||
s3HealthCheckObjectBody = "Test Rook Object Data"
|
||||
s3HealthCheckObjectBodyMD5 = "5f286306a2227a156ba770800e71b796"
|
||||
s3HealthCheckObjectKey = "rookHealthCheckTestObject"
|
||||
contentType = "plain/text"
|
||||
)
|
||||
|
||||
// bucketChecker aggregates the mon/cluster info needed to check the health of the monitors
|
||||
type bucketChecker struct {
|
||||
context *clusterd.Context
|
||||
objContext *Context
|
||||
interval time.Duration
|
||||
serviceIP string
|
||||
port string
|
||||
client client.Client
|
||||
namespacedName types.NamespacedName
|
||||
healthCheckSpec *cephv1.HealthCheckSpec
|
||||
}
|
||||
|
||||
// newbucketChecker creates a new HealthChecker object
|
||||
func newBucketChecker(context *clusterd.Context, objContext *Context, serviceIP, port string, client client.Client, namespacedName types.NamespacedName, healthCheckSpec *cephv1.HealthCheckSpec) *bucketChecker {
|
||||
c := &bucketChecker{
|
||||
context: context,
|
||||
objContext: objContext,
|
||||
interval: defaultStatusCheckInterval,
|
||||
serviceIP: serviceIP,
|
||||
port: port,
|
||||
namespacedName: namespacedName,
|
||||
client: client,
|
||||
healthCheckSpec: healthCheckSpec,
|
||||
}
|
||||
|
||||
// allow overriding the check interval with an env var on the operator
|
||||
checkInterval := healthCheckSpec.Bucket.Interval
|
||||
if checkInterval != "" {
|
||||
if duration, err := time.ParseDuration(checkInterval); err == nil {
|
||||
logger.Infof("ceph rgw status check interval for object store %q is %q", namespacedName.Name, checkInterval)
|
||||
c.interval = duration
|
||||
}
|
||||
}
|
||||
|
||||
return c
|
||||
}
|
||||
|
||||
// checkObjectStore periodically checks the health of the cluster
|
||||
func (c *bucketChecker) checkObjectStore(stopCh chan struct{}) {
|
||||
|
||||
for {
|
||||
select {
|
||||
case <-stopCh:
|
||||
// purge bucket and s3 user
|
||||
// Needed for external mode where in converged everything goes away with the CR deletion
|
||||
c.cleanupHealthCheck()
|
||||
logger.Infof("stopping monitoring of rgw endpoints for object store %q", c.namespacedName.Name)
|
||||
return
|
||||
|
||||
case <-time.After(c.interval):
|
||||
logger.Debug("checking rgw health of object store %q", c.namespacedName.Name)
|
||||
err := c.checkObjectStoreHealth()
|
||||
if err != nil {
|
||||
updateStatusBucket(c.client, c.namespacedName, cephv1.ConditionFailure, err.Error())
|
||||
logger.Warningf("failed to check rgw health for object store %q. %v", c.namespacedName.Name, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (c *bucketChecker) checkObjectStoreHealth() error {
|
||||
/*
|
||||
0. purge the s3 object by default
|
||||
1. create an S3 user
|
||||
2. always use the same user
|
||||
3. if already exists just re-hydrate the s3 credentials
|
||||
4. create a bucket with that user or use the existing one (always use the same bucket)
|
||||
5. create a check file
|
||||
6. get the hash of the file
|
||||
7. PUT the file
|
||||
8. GET the file
|
||||
9. compare hashes
|
||||
10. delete object on bucket
|
||||
11. update CR health status check
|
||||
|
||||
Always keep the bucket and the user for the health check, just do PUT and GET because bucket creation is expensive
|
||||
*/
|
||||
|
||||
var s3AccessKey string
|
||||
var s3SecretKey string
|
||||
s3endpoint := fmt.Sprintf("%s:%s", BuildDomainName(c.objContext.Name, c.namespacedName.Namespace), c.port)
|
||||
|
||||
// Generate unique user and bucket name
|
||||
bucketName := genUniqueBucketName(c.objContext.UID)
|
||||
userConfig := c.genUserConfig()
|
||||
|
||||
// Create S3 user
|
||||
logger.Debugf("creating s3 user object %q for object store %q", userConfig.UserID, c.namespacedName.Name)
|
||||
user, rgwerr, err := CreateUser(c.objContext, userConfig)
|
||||
if err != nil {
|
||||
if rgwerr == ErrorCodeFileExists {
|
||||
user, _, err = GetUser(c.objContext, userConfig.UserID)
|
||||
if err != nil {
|
||||
return errors.Wrapf(err, "failed to get details from ceph object user %q for object store %q", user.UserID, c.namespacedName.Name)
|
||||
}
|
||||
} else {
|
||||
return errors.Wrapf(err, "failed to create object user %q. error code %d for object store %q", userConfig.UserID, rgwerr, c.namespacedName.Name)
|
||||
}
|
||||
}
|
||||
// Set access and secret key
|
||||
s3AccessKey = *user.AccessKey
|
||||
s3SecretKey = *user.SecretKey
|
||||
|
||||
// Initiate s3 agent
|
||||
logger.Debugf("initializing s3 connection for object store %q", c.namespacedName.Name)
|
||||
s3client, err := NewS3Agent(s3AccessKey, s3SecretKey, s3endpoint)
|
||||
if err != nil {
|
||||
return errors.Wrap(err, "failed to initialize s3 connection")
|
||||
}
|
||||
|
||||
// Force purge the s3 object before starting anything
|
||||
err = cleanupObjectHealthCheck(s3client, c.objContext.UID)
|
||||
if err != nil {
|
||||
return errors.Wrapf(err, "failed to perform object cleanup for object store %q", c.namespacedName.Name)
|
||||
}
|
||||
|
||||
// Bucket health test
|
||||
err = c.testBucketHealth(s3client, bucketName)
|
||||
if err != nil {
|
||||
return errors.Wrapf(err, "failed to run bucket health checks for object store %q", c.namespacedName.Name)
|
||||
}
|
||||
|
||||
logger.Debug("successfully checked object store endpoint for object store %q", c.namespacedName.Name)
|
||||
|
||||
// Update the EndpointStatus in the CR to reflect the healthyness
|
||||
updateStatusBucket(c.client, c.namespacedName, cephv1.ConditionHealthy, "")
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func cleanupObjectHealthCheck(s3client *S3Agent, objectStoreUID string) error {
|
||||
bucketToDelete := genUniqueBucketName(objectStoreUID)
|
||||
logger.Infof("deleting object %q from bucket %q", s3HealthCheckObjectKey, bucketToDelete)
|
||||
_, err := s3client.DeleteObjectInBucket(bucketToDelete, s3HealthCheckObjectKey)
|
||||
|
||||
return err
|
||||
}
|
||||
|
||||
func (c *bucketChecker) cleanupHealthCheck() {
|
||||
bucketToDelete := genUniqueBucketName(c.objContext.UID)
|
||||
logger.Infof("deleting object %q from bucket %q in object store %q", s3HealthCheckObjectKey, bucketToDelete, c.namespacedName.Name)
|
||||
|
||||
_, err := DeleteObjectBucket(c.objContext, bucketToDelete, true)
|
||||
if err != nil {
|
||||
logger.Errorf("failed to delete bucket %q for object store %q. %v", bucketToDelete, c.namespacedName.Name, err)
|
||||
}
|
||||
|
||||
userToDelete := c.genUserConfig()
|
||||
output, err := DeleteUser(c.objContext, userToDelete.UserID)
|
||||
if err != nil {
|
||||
logger.Errorf("failed to delete object user %q for object store %q. %s. %v", userToDelete.UserID, c.namespacedName.Name, output, err)
|
||||
} else {
|
||||
logger.Debugf("successfully deleted object user %q for object store %q", userToDelete.UserID, c.namespacedName.Name)
|
||||
}
|
||||
}
|
||||
|
||||
func toCustomResourceStatus(currentStatus *cephv1.BucketStatus, details string, health cephv1.ConditionType) *cephv1.BucketStatus {
|
||||
s := &cephv1.BucketStatus{
|
||||
Health: health,
|
||||
LastChecked: time.Now().UTC().Format(time.RFC3339),
|
||||
Details: details,
|
||||
}
|
||||
|
||||
if currentStatus != nil {
|
||||
s.LastChanged = currentStatus.LastChanged
|
||||
if currentStatus.Details != s.Details {
|
||||
s.LastChanged = s.LastChecked
|
||||
}
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
func genUniqueBucketName(uuid string) string {
|
||||
return fmt.Sprintf("%s-%s", s3HealthCheckBucketName, uuid)
|
||||
}
|
||||
|
||||
func (c *bucketChecker) genUserConfig() ObjectUser {
|
||||
userName := fmt.Sprintf("%s-%s", s3UserHealthCheckName, c.objContext.UID)
|
||||
|
||||
return ObjectUser{
|
||||
UserID: userName,
|
||||
DisplayName: &userName,
|
||||
}
|
||||
}
|
||||
|
||||
func (c *bucketChecker) testBucketHealth(s3client *S3Agent, bucket string) error {
|
||||
// Purge on exit
|
||||
defer cleanupObjectHealthCheck(s3client, c.objContext.UID)
|
||||
|
||||
// Create S3 bucket
|
||||
logger.Debugf("creating bucket %q", bucket)
|
||||
err := s3client.CreateBucket(bucket)
|
||||
if err != nil {
|
||||
return errors.Wrapf(err, "failed to create bucket %q for object store %q", bucket, c.namespacedName.Name)
|
||||
}
|
||||
|
||||
// Put an object into the bucket
|
||||
logger.Debugf("putting object %q in bucket %q for object store %q", s3HealthCheckObjectKey, bucket, c.namespacedName.Name)
|
||||
_, err = s3client.PutObjectInBucket(bucket, string(s3HealthCheckObjectBody), s3HealthCheckObjectKey, contentType)
|
||||
if err != nil {
|
||||
return errors.Wrapf(err, "failed to put object %q in bucket %q for object store %q", s3HealthCheckObjectKey, bucket, c.namespacedName.Name)
|
||||
}
|
||||
|
||||
// Get the object from the bucket
|
||||
logger.Debugf("getting object %q in bucket %q for object store %q", s3HealthCheckObjectKey, bucket, c.namespacedName.Name)
|
||||
read, err := s3client.GetObjectInBucket(bucket, s3HealthCheckObjectKey)
|
||||
if err != nil {
|
||||
return errors.Wrapf(err, "failed to get object %q in bucket %q for object store %q", s3HealthCheckObjectKey, bucket, c.namespacedName.Name)
|
||||
}
|
||||
|
||||
// Compare the old and the existing object
|
||||
logger.Debugf("comparing objects hash for object store %q", c.namespacedName.Name)
|
||||
oldHash := k8sutil.Hash(s3HealthCheckObjectBody)
|
||||
currentHash := k8sutil.Hash(read)
|
||||
if currentHash != oldHash {
|
||||
return errors.Wrapf(err, "wrong file content, old file hash is %q and new one is %q for object store %q", oldHash, currentHash, c.namespacedName.Name)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
+37
-100
@@ -1,5 +1,5 @@
|
||||
/*
|
||||
Copyright 2018 The Kubernetes Authors.
|
||||
Copyright 2020 The Kubernetes Authors.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
@@ -14,113 +14,15 @@ See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
*/
|
||||
|
||||
package bucket
|
||||
package object
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
"github.com/aws/aws-sdk-go/aws/awserr"
|
||||
"github.com/aws/aws-sdk-go/aws/credentials"
|
||||
"github.com/aws/aws-sdk-go/aws/session"
|
||||
"github.com/aws/aws-sdk-go/service/s3"
|
||||
oerrors "github.com/kube-object-storage/lib-bucket-provisioner/pkg/provisioner/api/errors"
|
||||
"github.com/pkg/errors"
|
||||
"k8s.io/apimachinery/pkg/util/json"
|
||||
)
|
||||
|
||||
// S3Agent wraps the s3.S3 structure to allow for wrapper methods
|
||||
type S3Agent struct {
|
||||
client *s3.S3
|
||||
}
|
||||
|
||||
func NewS3Agent(accessKey, secretKey, endpoint string) (*S3Agent, error) {
|
||||
const cephRegion = "us-east-1"
|
||||
|
||||
sess, err := session.NewSession(
|
||||
aws.NewConfig().
|
||||
WithRegion(cephRegion).
|
||||
WithCredentials(credentials.NewStaticCredentials(accessKey, secretKey, "")).
|
||||
WithEndpoint(endpoint).
|
||||
WithS3ForcePathStyle(true).
|
||||
WithMaxRetries(20).
|
||||
WithDisableSSL(true))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
svc := s3.New(sess)
|
||||
return &S3Agent{
|
||||
client: svc,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// CreateBucket creates a bucket with the given name
|
||||
func (s S3Agent) CreateBucket(name string) error {
|
||||
logger.Infof("creating bucket %q", name)
|
||||
bucketInput := &s3.CreateBucketInput{
|
||||
Bucket: &name,
|
||||
}
|
||||
_, err := s.client.CreateBucket(bucketInput)
|
||||
if err != nil {
|
||||
if aerr, ok := err.(awserr.Error); ok {
|
||||
logger.Infof("DEBUG: after s3 call, ok=%v, aerr=%v", ok, aerr)
|
||||
switch aerr.Code() {
|
||||
case s3.ErrCodeBucketAlreadyExists:
|
||||
msg := fmt.Sprintf("Bucket %q already exists", name)
|
||||
logger.Errorf(msg)
|
||||
return oerrors.NewBucketExistsError(msg)
|
||||
case s3.ErrCodeBucketAlreadyOwnedByYou:
|
||||
msg := fmt.Sprintf("Bucket %q already owned by you", name)
|
||||
logger.Errorf(msg)
|
||||
return oerrors.NewBucketExistsError(msg)
|
||||
}
|
||||
}
|
||||
return errors.Wrapf(err, "bucket %q could not be created", name)
|
||||
}
|
||||
|
||||
logger.Infof("successfully created bucket %q", name)
|
||||
return nil
|
||||
}
|
||||
|
||||
// PutBucketPolicy applies the policy to the bucket
|
||||
func (s S3Agent) PutBucketPolicy(bucket string, policy BucketPolicy) (*s3.PutBucketPolicyOutput, error) {
|
||||
|
||||
confirmRemoveSelfBucketAccess := false
|
||||
serializedPolicy, _ := json.Marshal(policy)
|
||||
consumablePolicy := string(serializedPolicy)
|
||||
|
||||
p := &s3.PutBucketPolicyInput{
|
||||
Bucket: &bucket,
|
||||
ConfirmRemoveSelfBucketAccess: &confirmRemoveSelfBucketAccess,
|
||||
Policy: &consumablePolicy,
|
||||
}
|
||||
out, err := s.client.PutBucketPolicy(p)
|
||||
if err != nil {
|
||||
return out, err
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func (s S3Agent) GetBucketPolicy(bucket string) (*BucketPolicy, error) {
|
||||
out, err := s.client.GetBucketPolicy(&s3.GetBucketPolicyInput{
|
||||
Bucket: &bucket,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
policy := &BucketPolicy{}
|
||||
err = json.Unmarshal([]byte(*out.Policy), policy)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return policy, nil
|
||||
}
|
||||
|
||||
// //////////////
|
||||
// Policy
|
||||
// //////////////
|
||||
|
||||
type action string
|
||||
|
||||
const (
|
||||
@@ -258,6 +160,41 @@ func NewBucketPolicy(ps ...PolicyStatement) *BucketPolicy {
|
||||
return bp
|
||||
}
|
||||
|
||||
// PutBucketPolicy applies the policy to the bucket
|
||||
func (s *S3Agent) PutBucketPolicy(bucket string, policy BucketPolicy) (*s3.PutBucketPolicyOutput, error) {
|
||||
|
||||
confirmRemoveSelfBucketAccess := false
|
||||
serializedPolicy, _ := json.Marshal(policy)
|
||||
consumablePolicy := string(serializedPolicy)
|
||||
|
||||
p := &s3.PutBucketPolicyInput{
|
||||
Bucket: &bucket,
|
||||
ConfirmRemoveSelfBucketAccess: &confirmRemoveSelfBucketAccess,
|
||||
Policy: &consumablePolicy,
|
||||
}
|
||||
out, err := s.Client.PutBucketPolicy(p)
|
||||
if err != nil {
|
||||
return out, err
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func (s *S3Agent) GetBucketPolicy(bucket string) (*BucketPolicy, error) {
|
||||
out, err := s.Client.GetBucketPolicy(&s3.GetBucketPolicyInput{
|
||||
Bucket: &bucket,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
policy := &BucketPolicy{}
|
||||
err = json.Unmarshal([]byte(*out.Policy), policy)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return policy, nil
|
||||
}
|
||||
|
||||
// ModifyBucketPolicy new and old statement SIDs and overwrites on a match.
|
||||
// This allows users to Get, modify, and Replace existing statements as well as
|
||||
// add new ones.
|
||||
@@ -267,25 +267,27 @@ func (c *clusterConfig) deleteLegacyDaemons() {
|
||||
func (c *clusterConfig) deleteStore() error {
|
||||
logger.Infof("deleting object store %q from namespace %q", c.store.Name, c.store.Namespace)
|
||||
|
||||
// Delete rgw CephX keys and configuration in centralized mon database
|
||||
for i := 0; i < int(c.store.Spec.Gateway.Instances); i++ {
|
||||
daemonLetterID := k8sutil.IndexToName(i)
|
||||
depNameToRemove := fmt.Sprintf("%s-%s-%s", AppName, c.store.Name, daemonLetterID)
|
||||
if !c.clusterSpec.External.Enable {
|
||||
// Delete rgw CephX keys and configuration in centralized mon database
|
||||
for i := 0; i < int(c.store.Spec.Gateway.Instances); i++ {
|
||||
daemonLetterID := k8sutil.IndexToName(i)
|
||||
depNameToRemove := fmt.Sprintf("%s-%s-%s", AppName, c.store.Name, daemonLetterID)
|
||||
|
||||
err := c.deleteRgwCephObjects(depNameToRemove)
|
||||
err := c.deleteRgwCephObjects(depNameToRemove)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
// Delete the realm and pools
|
||||
objContext := NewContext(c.context, c.store.Name, c.store.Namespace)
|
||||
err := deleteRealmAndPools(objContext, c.store.Spec)
|
||||
if err != nil {
|
||||
return err
|
||||
return errors.Wrap(err, "failed to delete the realm and pools")
|
||||
}
|
||||
}
|
||||
|
||||
// Delete the realm and pools
|
||||
objContext := NewContext(c.context, c.store.Name, c.store.Namespace)
|
||||
err := deleteRealmAndPools(objContext, c.store.Spec)
|
||||
if err != nil {
|
||||
return errors.Wrap(err, "failed to delete the realm and pools")
|
||||
}
|
||||
|
||||
logger.Infof("completed deleting object store %q from namespace %q", c.store.Name, c.store.Namespace)
|
||||
logger.Infof("successfully deleted object store %q from namespace %q", c.store.Name, c.store.Namespace)
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -316,7 +318,7 @@ func (c *clusterConfig) storeLabelSelector() string {
|
||||
}
|
||||
|
||||
// Validate the object store arguments
|
||||
func validateStore(context *clusterd.Context, s *cephv1.CephObjectStore) error {
|
||||
func (r *ReconcileCephObjectStore) validateStore(s *cephv1.CephObjectStore) error {
|
||||
if s.Name == "" {
|
||||
return errors.New("missing name")
|
||||
}
|
||||
@@ -331,16 +333,23 @@ func validateStore(context *clusterd.Context, s *cephv1.CephObjectStore) error {
|
||||
// Validate the pool settings, but allow for empty pools specs in case they have already been created
|
||||
// such as by the ceph mgr
|
||||
if !emptyPool(s.Spec.MetadataPool) {
|
||||
if err := pool.ValidatePoolSpec(context, s.Namespace, &s.Spec.MetadataPool); err != nil {
|
||||
if err := pool.ValidatePoolSpec(r.context, s.Namespace, &s.Spec.MetadataPool); err != nil {
|
||||
return errors.Wrap(err, "invalid metadata pool spec")
|
||||
}
|
||||
}
|
||||
if !emptyPool(s.Spec.DataPool) {
|
||||
if err := pool.ValidatePoolSpec(context, s.Namespace, &s.Spec.DataPool); err != nil {
|
||||
if err := pool.ValidatePoolSpec(r.context, s.Namespace, &s.Spec.DataPool); err != nil {
|
||||
return errors.Wrap(err, "invalid data pool spec")
|
||||
}
|
||||
}
|
||||
|
||||
// Fail if we detected an external CephCluster CR and the list of endpoints is empty
|
||||
if r.cephClusterSpec.External.Enable {
|
||||
if len(s.Spec.Gateway.ExternalRgwEndpoints) == 0 {
|
||||
return errors.New("ceph cluster is external but externalRgwEndpoints list is empty")
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -351,3 +360,8 @@ func (c *clusterConfig) generateSecretName(id string) string {
|
||||
func emptyPool(pool cephv1.PoolSpec) bool {
|
||||
return reflect.DeepEqual(pool, cephv1.PoolSpec{})
|
||||
}
|
||||
|
||||
// BuildDomainName build the dns name to reach out the service endpoint
|
||||
func BuildDomainName(name, namespace string) string {
|
||||
return fmt.Sprintf("%s-%s.%s", AppName, name, namespace)
|
||||
}
|
||||
@@ -154,3 +154,10 @@ func TestEmptyPoolSpec(t *testing.T) {
|
||||
p = cephv1.PoolSpec{ErasureCoded: cephv1.ErasureCodedSpec{CodingChunks: 1}}
|
||||
assert.False(t, emptyPool(p))
|
||||
}
|
||||
|
||||
func TestBuildDomainName(t *testing.T) {
|
||||
name := "my-store"
|
||||
ns := "rook-ceph"
|
||||
dns := BuildDomainName(name, ns)
|
||||
assert.Equal(t, "rook-ceph-rgw-my-store.rook-ceph", dns)
|
||||
}
|
||||
@@ -0,0 +1,156 @@
|
||||
/*
|
||||
Copyright 2018 The Kubernetes Authors.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
*/
|
||||
|
||||
package object
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"net/http"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
"github.com/aws/aws-sdk-go/aws/awserr"
|
||||
"github.com/aws/aws-sdk-go/aws/credentials"
|
||||
"github.com/aws/aws-sdk-go/aws/session"
|
||||
"github.com/aws/aws-sdk-go/service/s3"
|
||||
"github.com/pkg/errors"
|
||||
)
|
||||
|
||||
// S3Agent wraps the s3.S3 structure to allow for wrapper methods
|
||||
type S3Agent struct {
|
||||
Client *s3.S3
|
||||
}
|
||||
|
||||
func NewS3Agent(accessKey, secretKey, endpoint string) (*S3Agent, error) {
|
||||
const cephRegion = "us-east-1"
|
||||
|
||||
sess, err := session.NewSession(
|
||||
aws.NewConfig().
|
||||
WithRegion(cephRegion).
|
||||
WithCredentials(credentials.NewStaticCredentials(accessKey, secretKey, "")).
|
||||
WithEndpoint(endpoint).
|
||||
WithS3ForcePathStyle(true).
|
||||
WithMaxRetries(20).
|
||||
WithDisableSSL(true).
|
||||
WithHTTPClient(&http.Client{
|
||||
Timeout: time.Second * 15,
|
||||
}),
|
||||
)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
svc := s3.New(sess)
|
||||
return &S3Agent{
|
||||
Client: svc,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// CreateBucket creates a bucket with the given name
|
||||
func (s *S3Agent) CreateBucket(name string) error {
|
||||
logger.Infof("creating bucket %q", name)
|
||||
bucketInput := &s3.CreateBucketInput{
|
||||
Bucket: &name,
|
||||
}
|
||||
_, err := s.Client.CreateBucket(bucketInput)
|
||||
if err != nil {
|
||||
if aerr, ok := err.(awserr.Error); ok {
|
||||
logger.Debugf("DEBUG: after s3 call, ok=%v, aerr=%v", ok, aerr)
|
||||
switch aerr.Code() {
|
||||
case s3.ErrCodeBucketAlreadyExists:
|
||||
logger.Debugf("bucket %q already exists", name)
|
||||
return nil
|
||||
case s3.ErrCodeBucketAlreadyOwnedByYou:
|
||||
logger.Debugf("bucket %q already owned by you", name)
|
||||
return nil
|
||||
}
|
||||
}
|
||||
return errors.Wrapf(err, "failed to create bucket %q", name)
|
||||
}
|
||||
|
||||
logger.Infof("successfully created bucket %q", name)
|
||||
return nil
|
||||
}
|
||||
|
||||
// DeleteBucket function deletes given bucket using s3 client
|
||||
func (s *S3Agent) DeleteBucket(name string) (bool, error) {
|
||||
_, err := s.Client.DeleteBucket(&s3.DeleteBucketInput{
|
||||
Bucket: aws.String(name),
|
||||
})
|
||||
if err != nil {
|
||||
logger.Errorf("failed to delete bucket. %v", err)
|
||||
return false, err
|
||||
|
||||
}
|
||||
return true, nil
|
||||
}
|
||||
|
||||
// PutObjectInBucket function puts an object in a bucket using s3 client
|
||||
func (s *S3Agent) PutObjectInBucket(bucketname string, body string, key string,
|
||||
contentType string) (bool, error) {
|
||||
_, err := s.Client.PutObject(&s3.PutObjectInput{
|
||||
Body: strings.NewReader(body),
|
||||
Bucket: &bucketname,
|
||||
Key: &key,
|
||||
ContentType: &contentType,
|
||||
})
|
||||
if err != nil {
|
||||
logger.Errorf("failed to put object in bucket. %v", err)
|
||||
return false, err
|
||||
|
||||
}
|
||||
return true, nil
|
||||
}
|
||||
|
||||
// GetObjectInBucket function retrieves an object from a bucket using s3 client
|
||||
func (s *S3Agent) GetObjectInBucket(bucketname string, key string) (string, error) {
|
||||
result, err := s.Client.GetObject(&s3.GetObjectInput{
|
||||
Bucket: aws.String(bucketname),
|
||||
Key: aws.String(key),
|
||||
})
|
||||
|
||||
if err != nil {
|
||||
logger.Errorf("failed to retrieve object from bucket. %v", err)
|
||||
return "ERROR_ OBJECT NOT FOUND", err
|
||||
|
||||
}
|
||||
buf := new(bytes.Buffer)
|
||||
buf.ReadFrom(result.Body)
|
||||
|
||||
return buf.String(), nil
|
||||
}
|
||||
|
||||
// DeleteObjectInBucket function deletes given bucket using s3 client
|
||||
func (s *S3Agent) DeleteObjectInBucket(bucketname string, key string) (bool, error) {
|
||||
_, err := s.Client.DeleteObject(&s3.DeleteObjectInput{
|
||||
Bucket: aws.String(bucketname),
|
||||
Key: aws.String(key),
|
||||
})
|
||||
if err != nil {
|
||||
if aerr, ok := err.(awserr.Error); ok {
|
||||
switch aerr.Code() {
|
||||
case s3.ErrCodeNoSuchBucket:
|
||||
return true, nil
|
||||
case s3.ErrCodeNoSuchKey:
|
||||
return true, nil
|
||||
}
|
||||
}
|
||||
logger.Errorf("failed to delete object from bucket. %v", err)
|
||||
return false, err
|
||||
|
||||
}
|
||||
return true, nil
|
||||
}
|
||||
@@ -233,18 +233,63 @@ func (c *clusterConfig) generateService(cephObjectStore *cephv1.CephObjectStore)
|
||||
}
|
||||
|
||||
destPort := c.generateLiveProbePort()
|
||||
|
||||
// When the cluster is external we must use the same one as the gateways are listening on
|
||||
if c.clusterSpec.External.Enable {
|
||||
destPort.IntVal = cephObjectStore.Spec.Gateway.Port
|
||||
}
|
||||
addPort(svc, "http", cephObjectStore.Spec.Gateway.Port, destPort.IntVal)
|
||||
addPort(svc, "https", cephObjectStore.Spec.Gateway.SecurePort, cephObjectStore.Spec.Gateway.SecurePort)
|
||||
|
||||
return svc
|
||||
}
|
||||
|
||||
func (c *clusterConfig) generateEndpoint(cephObjectStore *cephv1.CephObjectStore) *v1.Endpoints {
|
||||
labels := getLabels(cephObjectStore.Name, cephObjectStore.Namespace)
|
||||
|
||||
endpoints := &v1.Endpoints{
|
||||
ObjectMeta: metav1.ObjectMeta{
|
||||
Name: instanceName(cephObjectStore.Name),
|
||||
Namespace: cephObjectStore.Namespace,
|
||||
Labels: labels,
|
||||
},
|
||||
Subsets: []v1.EndpointSubset{
|
||||
{
|
||||
Addresses: cephObjectStore.Spec.Gateway.ExternalRgwEndpoints,
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
addPortToEndpoint(endpoints, "http", cephObjectStore.Spec.Gateway.Port)
|
||||
addPortToEndpoint(endpoints, "https", cephObjectStore.Spec.Gateway.SecurePort)
|
||||
|
||||
return endpoints
|
||||
}
|
||||
|
||||
func (c *clusterConfig) reconcileExternalEndpoint(cephObjectStore *cephv1.CephObjectStore) error {
|
||||
logger.Info("reconciling external object store service")
|
||||
|
||||
endpoint := c.generateEndpoint(cephObjectStore)
|
||||
// Set owner ref to the parent object
|
||||
err := controllerutil.SetControllerReference(cephObjectStore, endpoint, c.scheme)
|
||||
if err != nil {
|
||||
return errors.Wrap(err, "failed to set owner reference to ceph object store endpoint")
|
||||
}
|
||||
|
||||
_, err = k8sutil.CreateOrUpdateEndpoint(c.context.Clientset, cephObjectStore.Namespace, endpoint)
|
||||
if err != nil {
|
||||
return errors.Wrapf(err, "failed to create or update object store %q service", cephObjectStore.Name)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func (c *clusterConfig) reconcileService(cephObjectStore *cephv1.CephObjectStore) (string, error) {
|
||||
service := c.generateService(cephObjectStore)
|
||||
// Set owner ref to the parent object
|
||||
err := controllerutil.SetControllerReference(cephObjectStore, service, c.scheme)
|
||||
if err != nil {
|
||||
return "", errors.Wrap(err, "failed to set owner reference to ceph object store")
|
||||
return "", errors.Wrap(err, "failed to set owner reference to ceph object store service")
|
||||
}
|
||||
|
||||
svc, err := k8sutil.CreateOrUpdateService(c.context.Clientset, cephObjectStore.Namespace, service)
|
||||
@@ -268,6 +313,18 @@ func addPort(service *v1.Service, name string, port, destPort int32) {
|
||||
})
|
||||
}
|
||||
|
||||
func addPortToEndpoint(endpoints *v1.Endpoints, name string, port int32) {
|
||||
if port == 0 {
|
||||
return
|
||||
}
|
||||
endpoints.Subsets[0].Ports = append(endpoints.Subsets[0].Ports, v1.EndpointPort{
|
||||
Name: name,
|
||||
Port: port,
|
||||
Protocol: v1.ProtocolTCP,
|
||||
},
|
||||
)
|
||||
}
|
||||
|
||||
func getLabels(name, namespace string) map[string]string {
|
||||
labels := controller.PodLabels(AppName, namespace, "rgw", name)
|
||||
labels["rook_object_store"] = name
|
||||
|
||||
@@ -133,34 +133,56 @@ func TestSSLPodSpec(t *testing.T) {
|
||||
|
||||
func TestValidateSpec(t *testing.T) {
|
||||
context := &clusterd.Context{Executor: &exectest.MockExecutor{}}
|
||||
r := &ReconcileCephObjectStore{
|
||||
context: context,
|
||||
cephClusterSpec: &cephv1.ClusterSpec{
|
||||
External: cephv1.ExternalSpec{
|
||||
Enable: false,
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
// valid store
|
||||
s := simpleStore()
|
||||
err := validateStore(context, s)
|
||||
err := r.validateStore(s)
|
||||
assert.Nil(t, err)
|
||||
|
||||
// no name
|
||||
s.Name = ""
|
||||
err = validateStore(context, s)
|
||||
err = r.validateStore(s)
|
||||
assert.NotNil(t, err)
|
||||
s.Name = "default"
|
||||
err = validateStore(context, s)
|
||||
err = r.validateStore(s)
|
||||
assert.Nil(t, err)
|
||||
|
||||
// no namespace
|
||||
s.Namespace = ""
|
||||
err = validateStore(context, s)
|
||||
err = r.validateStore(s)
|
||||
assert.NotNil(t, err)
|
||||
s.Namespace = "mycluster"
|
||||
err = validateStore(context, s)
|
||||
err = r.validateStore(s)
|
||||
assert.Nil(t, err)
|
||||
|
||||
// no replication or EC is valid
|
||||
s.Spec.MetadataPool.Replicated.Size = 0
|
||||
err = validateStore(context, s)
|
||||
err = r.validateStore(s)
|
||||
assert.Nil(t, err)
|
||||
s.Spec.MetadataPool.Replicated.Size = 1
|
||||
err = validateStore(context, s)
|
||||
err = r.validateStore(s)
|
||||
assert.Nil(t, err)
|
||||
|
||||
// external with no endpoints, failure
|
||||
r.cephClusterSpec.External.Enable = true
|
||||
err = r.validateStore(s)
|
||||
assert.NotNil(t, err)
|
||||
|
||||
// external with endpoints, success
|
||||
s.Spec.Gateway.ExternalRgwEndpoints = []v1.EndpointAddress{
|
||||
{
|
||||
IP: "192.168.0.1",
|
||||
},
|
||||
}
|
||||
err = r.validateStore(s)
|
||||
assert.Nil(t, err)
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,78 @@
|
||||
/*
|
||||
Copyright 2020 The Rook Authors. All rights reserved.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
*/
|
||||
|
||||
package object
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"github.com/pkg/errors"
|
||||
cephv1 "github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
|
||||
opcontroller "github.com/rook/rook/pkg/operator/ceph/controller"
|
||||
kerrors "k8s.io/apimachinery/pkg/api/errors"
|
||||
"k8s.io/apimachinery/pkg/types"
|
||||
"sigs.k8s.io/controller-runtime/pkg/client"
|
||||
"sigs.k8s.io/controller-runtime/pkg/reconcile"
|
||||
)
|
||||
|
||||
func (r *ReconcileCephObjectStore) setFailedStatus(name types.NamespacedName, errMessage string, err error) (reconcile.Result, error) {
|
||||
updateStatusPhase(r.client, name, cephv1.ConditionFailure)
|
||||
return reconcile.Result{}, errors.Wrapf(err, "%s", errMessage)
|
||||
}
|
||||
|
||||
// updateStatusPhase updates an object with a given status
|
||||
func updateStatusPhase(client client.Client, namespacedName types.NamespacedName, status cephv1.ConditionType) {
|
||||
objectStore := &cephv1.CephObjectStore{}
|
||||
if err := client.Get(context.TODO(), namespacedName, objectStore); err != nil {
|
||||
if kerrors.IsNotFound(err) {
|
||||
logger.Debug("CephObjectStore resource not found. Ignoring since object must be deleted.")
|
||||
return
|
||||
}
|
||||
logger.Warningf("failed to retrieve object store %q to update status to %q. %v", namespacedName, status, err)
|
||||
return
|
||||
}
|
||||
if objectStore.Status == nil {
|
||||
objectStore.Status = &cephv1.ObjectStoreStatus{}
|
||||
}
|
||||
|
||||
objectStore.Status.Phase = status
|
||||
if err := opcontroller.UpdateStatus(client, objectStore); err != nil {
|
||||
logger.Errorf("failed to set object store %q status to %q. %v", namespacedName, status, err)
|
||||
return
|
||||
}
|
||||
logger.Debugf("object store %q status updated to %q", namespacedName, status)
|
||||
}
|
||||
|
||||
// updateStatusBucket updates an object with a given status
|
||||
func updateStatusBucket(client client.Client, name types.NamespacedName, phase cephv1.ConditionType, details string) {
|
||||
objectStore := &cephv1.CephObjectStore{}
|
||||
if err := client.Get(context.TODO(), name, objectStore); err != nil {
|
||||
if kerrors.IsNotFound(err) {
|
||||
logger.Debug("CephObjectStore resource not found. Ignoring since object must be deleted.")
|
||||
return
|
||||
}
|
||||
logger.Warningf("failed to retrieve object store %q to update status to %v. %v", name, phase, err)
|
||||
return
|
||||
}
|
||||
|
||||
objectStore.Status.BucketStatus = toCustomResourceStatus(objectStore.Status.BucketStatus, details, phase)
|
||||
if err := opcontroller.UpdateStatus(client, objectStore); err != nil {
|
||||
logger.Errorf("failed to set object store %q status to %v. %v", name, phase, err)
|
||||
return
|
||||
}
|
||||
|
||||
logger.Debugf("object store %q status updated to %v", name, phase)
|
||||
}
|
||||
@@ -73,7 +73,7 @@ func decodeUser(data string) (*ObjectUser, int, error) {
|
||||
var user rgwUserInfo
|
||||
err := json.Unmarshal([]byte(data), &user)
|
||||
if err != nil {
|
||||
return nil, RGWErrorParse, errors.Wrap(err, "Failed to unmarshal json")
|
||||
return nil, RGWErrorParse, errors.Wrapf(err, "failed to unmarshal json. %s", data)
|
||||
}
|
||||
|
||||
rookUser := ObjectUser{UserID: user.UserID, DisplayName: &user.DisplayName, Email: &user.Email}
|
||||
@@ -88,7 +88,7 @@ func decodeUser(data string) (*ObjectUser, int, error) {
|
||||
|
||||
// GetUser returns the user with the given ID.
|
||||
func GetUser(c *Context, id string) (*ObjectUser, int, error) {
|
||||
logger.Infof("Getting user: %s", id)
|
||||
logger.Infof("getting s3 user %q", id)
|
||||
|
||||
// note: err is set for non-existent user but result output is also empty
|
||||
result, err := runAdminCommand(c, "user", "info", "--uid", id)
|
||||
@@ -103,7 +103,7 @@ func GetUser(c *Context, id string) (*ObjectUser, int, error) {
|
||||
|
||||
// CreateUser creates a new user with the information given.
|
||||
func CreateUser(c *Context, user ObjectUser) (*ObjectUser, int, error) {
|
||||
logger.Debugf("Creating user: %s", user.UserID)
|
||||
logger.Debugf("creating s3 user %q", user.UserID)
|
||||
|
||||
if strings.TrimSpace(user.UserID) == "" {
|
||||
return nil, RGWErrorBadData, errors.New("userId cannot be empty")
|
||||
@@ -126,15 +126,16 @@ func CreateUser(c *Context, user ObjectUser) (*ObjectUser, int, error) {
|
||||
|
||||
result, err := runAdminCommand(c, args...)
|
||||
if err != nil {
|
||||
return nil, RGWErrorUnknown, errors.Wrap(err, "failed to create user")
|
||||
}
|
||||
if strings.Contains(result, "could not create user: unable to create user, user: ") {
|
||||
return nil, ErrorCodeFileExists, errors.New("s3 user already exists")
|
||||
}
|
||||
|
||||
if strings.HasPrefix(result, "could not create user: unable to create user, user: ") && strings.HasSuffix(result, " exists") {
|
||||
return nil, ErrorCodeFileExists, errors.New("user already exists")
|
||||
}
|
||||
if strings.Contains(result, "could not create user: unable to create user, email: ") && strings.Contains(result, " is the email address an existing user") {
|
||||
return nil, RGWErrorBadData, errors.New("email already in use")
|
||||
}
|
||||
|
||||
if strings.HasPrefix(result, "could not create user: unable to create user, email: ") && strings.HasSuffix(result, " is the email address an existing user") {
|
||||
return nil, RGWErrorBadData, errors.New("email already in use")
|
||||
// We don't know what happened
|
||||
return nil, RGWErrorUnknown, errors.Wrap(err, "failed to create s3 user")
|
||||
}
|
||||
|
||||
return decodeUser(result)
|
||||
@@ -142,7 +143,7 @@ func CreateUser(c *Context, user ObjectUser) (*ObjectUser, int, error) {
|
||||
|
||||
// UpdateUser updates the user whose ID matches the user.
|
||||
func UpdateUser(c *Context, user ObjectUser) (*ObjectUser, int, error) {
|
||||
logger.Infof("Updating user: %s", user.UserID)
|
||||
logger.Infof("updating s3 user %q", user.UserID)
|
||||
|
||||
args := []string{"user", "modify", "--uid", user.UserID}
|
||||
|
||||
@@ -155,11 +156,11 @@ func UpdateUser(c *Context, user ObjectUser) (*ObjectUser, int, error) {
|
||||
|
||||
body, err := runAdminCommand(c, args...)
|
||||
if err != nil {
|
||||
return nil, RGWErrorUnknown, errors.Wrap(err, "failed to update user")
|
||||
return nil, RGWErrorUnknown, errors.Wrap(err, "failed to update s3 user")
|
||||
}
|
||||
|
||||
if body == "could not modify user: unable to modify user, user not found" {
|
||||
return nil, RGWErrorNotFound, errors.New("user not found")
|
||||
return nil, RGWErrorNotFound, errors.New("s3 user not found")
|
||||
}
|
||||
|
||||
return decodeUser(body)
|
||||
@@ -179,7 +180,7 @@ func DeleteUser(c *Context, id string, opts ...string) (string, error) {
|
||||
}
|
||||
}
|
||||
|
||||
return result, errors.Wrap(err, "failed to delete user")
|
||||
return result, errors.Wrap(err, "failed to delete s3 user")
|
||||
}
|
||||
|
||||
func SetQuotaUserBucketMax(c *Context, id string, max int) (string, int, error) {
|
||||
|
||||
@@ -0,0 +1,43 @@
|
||||
/*
|
||||
Copyright 2020 The Rook Authors. All rights reserved.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
*/
|
||||
|
||||
package k8sutil
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
|
||||
v1 "k8s.io/api/core/v1"
|
||||
"k8s.io/apimachinery/pkg/api/errors"
|
||||
"k8s.io/client-go/kubernetes"
|
||||
)
|
||||
|
||||
// CreateOrUpdateEndpoint creates a service or updates the service declaratively if it already exists.
|
||||
func CreateOrUpdateEndpoint(clientset kubernetes.Interface, namespace string, endpointDefinition *v1.Endpoints) (*v1.Endpoints, error) {
|
||||
name := endpointDefinition.Name
|
||||
logger.Debugf("creating endpoint %q. %v", name, endpointDefinition.Subsets)
|
||||
ep, err := clientset.CoreV1().Endpoints(namespace).Create(endpointDefinition)
|
||||
if err != nil {
|
||||
if !errors.IsAlreadyExists(err) {
|
||||
return nil, fmt.Errorf("failed to create endpoint %q. %v", name, err)
|
||||
}
|
||||
ep, err = clientset.CoreV1().Endpoints(namespace).Update(endpointDefinition)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to update endpoint %q. %v", name, err)
|
||||
}
|
||||
}
|
||||
|
||||
return ep, err
|
||||
}
|
||||
@@ -488,6 +488,15 @@ spec:
|
||||
- force
|
||||
preservePoolsOnDelete:
|
||||
type: boolean
|
||||
healthCheck:
|
||||
properties:
|
||||
bucket:
|
||||
properties:
|
||||
enabled:
|
||||
type: boolean
|
||||
interval:
|
||||
type: string
|
||||
subresources:
|
||||
subresources:
|
||||
status: {}
|
||||
---
|
||||
@@ -2209,6 +2218,10 @@ spec:
|
||||
securePort:
|
||||
instances: ` + strconv.Itoa(replicaCount) + `
|
||||
allNodes: false
|
||||
healthCheck:
|
||||
bucket:
|
||||
enabled: true
|
||||
interval: 10s
|
||||
`
|
||||
}
|
||||
|
||||
|
||||
@@ -1,143 +0,0 @@
|
||||
/*
|
||||
Copyright 2016 The Rook Authors. All rights reserved.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
*/
|
||||
|
||||
package utils
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"strings"
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
"github.com/aws/aws-sdk-go/aws/credentials"
|
||||
"github.com/aws/aws-sdk-go/aws/session"
|
||||
"github.com/aws/aws-sdk-go/service/s3"
|
||||
)
|
||||
|
||||
// S3Helper contains pointer to s3 client and wrappers for basic object store operations
|
||||
type S3Helper struct {
|
||||
s3client *s3.S3
|
||||
}
|
||||
|
||||
// CreateNewS3Helper creates a s3 client for specified endpoint and creds
|
||||
func CreateNewS3Helper(endpoint string, keyID string, keySecret string) *S3Helper {
|
||||
|
||||
creds := credentials.NewStaticCredentials(keyID, keySecret, "")
|
||||
|
||||
// create aws s3 config, must use 'us-east-1' default aws region for ceph object store
|
||||
awsConfig := aws.NewConfig().
|
||||
WithRegion("us-east-1").
|
||||
WithCredentials(creds).
|
||||
WithEndpoint(endpoint).
|
||||
WithS3ForcePathStyle(true).
|
||||
WithDisableSSL(true).
|
||||
WithMaxRetries(20)
|
||||
|
||||
// create new session
|
||||
ses := session.New()
|
||||
|
||||
// create new s3 client connection
|
||||
c := s3.New(ses, awsConfig)
|
||||
|
||||
return &S3Helper{c}
|
||||
}
|
||||
|
||||
// CreateBucket function creates bucket using s3 client
|
||||
func (h *S3Helper) CreateBucket(name string) (bool, error) {
|
||||
_, err := h.s3client.CreateBucket(&s3.CreateBucketInput{
|
||||
Bucket: aws.String(name),
|
||||
})
|
||||
if err != nil {
|
||||
logger.Errorf("error encountered while creating bucket : %v", err)
|
||||
return false, err
|
||||
|
||||
}
|
||||
return true, nil
|
||||
}
|
||||
|
||||
// DeleteBucket function deletes given bucket using s3 client
|
||||
func (h *S3Helper) DeleteBucket(name string) (bool, error) {
|
||||
_, err := h.s3client.DeleteBucket(&s3.DeleteBucketInput{
|
||||
Bucket: aws.String(name),
|
||||
})
|
||||
if err != nil {
|
||||
logger.Errorf("error encountered while deleting bucket : %v", err)
|
||||
return false, err
|
||||
|
||||
}
|
||||
return true, nil
|
||||
}
|
||||
|
||||
// PutObjectInBucket function puts an object in a bucket using s3 client
|
||||
func (h *S3Helper) PutObjectInBucket(bucketname string, body string, key string,
|
||||
contentType string) (bool, error) {
|
||||
_, err := h.s3client.PutObject(&s3.PutObjectInput{
|
||||
Body: strings.NewReader(body),
|
||||
Bucket: &bucketname,
|
||||
Key: &key,
|
||||
ContentType: &contentType,
|
||||
})
|
||||
if err != nil {
|
||||
logger.Errorf("error encountered while putting object in bucket : %v", err)
|
||||
return false, err
|
||||
|
||||
}
|
||||
return true, nil
|
||||
}
|
||||
|
||||
// GetObjectInBucket function retrieves an object from a bucket using s3 client
|
||||
func (h *S3Helper) GetObjectInBucket(bucketname string, key string) (string, error) {
|
||||
result, err := h.s3client.GetObject(&s3.GetObjectInput{
|
||||
Bucket: aws.String(bucketname),
|
||||
Key: aws.String(key),
|
||||
})
|
||||
|
||||
if err != nil {
|
||||
logger.Errorf("error encountered while retrieving object from bucket : %v", err)
|
||||
return "ERROR_ OBJECT NOT FOUND", err
|
||||
|
||||
}
|
||||
buf := new(bytes.Buffer)
|
||||
buf.ReadFrom(result.Body)
|
||||
|
||||
return buf.String(), nil
|
||||
}
|
||||
|
||||
// DeleteObjectInBucket function deletes given bucket using s3 client
|
||||
func (h *S3Helper) DeleteObjectInBucket(bucketname string, key string) (bool, error) {
|
||||
_, err := h.s3client.DeleteObject(&s3.DeleteObjectInput{
|
||||
Bucket: aws.String(bucketname),
|
||||
Key: aws.String(key),
|
||||
})
|
||||
if err != nil {
|
||||
logger.Errorf("error encountered while deleting object from bucket : %v", err)
|
||||
return false, err
|
||||
|
||||
}
|
||||
return true, nil
|
||||
}
|
||||
|
||||
// IsBucketPresent function returns true if a bucket is present and false if it's not present
|
||||
func (h *S3Helper) IsBucketPresent(bucketname string) (bool, error) {
|
||||
_, err := h.s3client.HeadBucket(&s3.HeadBucketInput{
|
||||
Bucket: aws.String(bucketname),
|
||||
})
|
||||
if err != nil && strings.Contains(err.Error(), "NotFound") {
|
||||
return false, nil
|
||||
} else if err == nil {
|
||||
return true, nil
|
||||
}
|
||||
return false, err
|
||||
}
|
||||
@@ -21,12 +21,14 @@ import (
|
||||
|
||||
"time"
|
||||
|
||||
cephv1 "github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
|
||||
rgw "github.com/rook/rook/pkg/operator/ceph/object"
|
||||
"github.com/rook/rook/tests/framework/clients"
|
||||
"github.com/rook/rook/tests/framework/utils"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"github.com/stretchr/testify/suite"
|
||||
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
|
||||
)
|
||||
|
||||
var (
|
||||
@@ -70,14 +72,20 @@ func runObjectE2ETest(helper *clients.TestClient, k8sh *utils.K8sHelper, s suite
|
||||
logger.Infof("Object store created successfully")
|
||||
|
||||
// check that ObjectUser is created
|
||||
logger.Infof("Waiting 10 seconds to ensure user was created")
|
||||
time.Sleep(10 * time.Second)
|
||||
logger.Infof("Waiting 15 seconds to ensure user was created")
|
||||
time.Sleep(15 * time.Second)
|
||||
logger.Infof("Checking to see if the user secret has been created")
|
||||
i = 0
|
||||
for i = 0; i < 4 && helper.ObjectUserClient.UserSecretExists(namespace, storeName, userid) == false; i++ {
|
||||
logger.Infof("(%d) secret check sleeping for 5 seconds ...", i)
|
||||
time.Sleep(5 * time.Second)
|
||||
}
|
||||
|
||||
// Check object store status
|
||||
objectStore, err := k8sh.RookClientset.CephV1().CephObjectStores(namespace).Get(storeName, metav1.GetOptions{})
|
||||
assert.Nil(s.T(), err)
|
||||
assert.Equal(s.T(), cephv1.ConditionHealthy, objectStore.Status.BucketStatus.Health)
|
||||
|
||||
assert.True(s.T(), helper.ObjectUserClient.UserSecretExists(namespace, storeName, userid))
|
||||
userInfo, err := helper.ObjectUserClient.GetUser(namespace, storeName, userid)
|
||||
require.NoError(s.T(), err)
|
||||
@@ -136,7 +144,8 @@ func runObjectE2ETest(helper *clients.TestClient, k8sh *utils.K8sHelper, s suite
|
||||
s3endpoint, _ := helper.ObjectClient.GetEndPointUrl(namespace, storeName)
|
||||
s3AccessKey, _ := helper.BucketClient.GetAccessKey(obcName)
|
||||
s3SecretKey, _ := helper.BucketClient.GetSecretKey(obcName)
|
||||
s3client := utils.CreateNewS3Helper(s3endpoint, s3AccessKey, s3SecretKey)
|
||||
s3client, err := rgw.NewS3Agent(s3AccessKey, s3SecretKey, s3endpoint)
|
||||
require.Nil(s.T(), err)
|
||||
logger.Infof("endpoint (%s) Accesskey (%s) secret (%s)", s3endpoint, s3AccessKey, s3SecretKey)
|
||||
|
||||
logger.Infof("Step 5 : Put Object on bucket")
|
||||
|
||||
Reference in new issue
Block a user