edgefs: support for SMB CRD

Signed-off-by: Dmitry Yusupov <dmitry.yusupov@nexenta.com>
This commit is contained in:
Dmitry Yusupov committed 2020-04-03 13:15:14 -07:00
1 parent 843390a4fc
commit 4920202c10
28 files changed
+1482 -12

No files matched your search

+1 -1
View File
@@ -92,7 +92,7 @@ spec:
* `remoteURL`: I/O flow with destination pointing to the other ISGW endpoint service, locally mounted directory or other (non-EdgeFS) S3 compatible service. Format should be of form PROTOCOL://LOCATION, where PROTOCOL can be of "ccow", "file", "s3" and LOCATION depends on the PROTOCOL. In case of "ccow" LOCATION needs to specify a HOST:PORT of the other EdgeFS site, e.g. ccow://192.168.1.43:14000. In case of "file" LOCATION needs to be a path to a local POSIX directory, e.g. file://volumeDir. In cae of "s3" LOCATION can either point to an existing bucket, e.g. s3://mybucket.aws.com or to the other S3 compatible server IP:HOST, e.g. s3://214.4.34.22:4567. It is important to note that file:// and s3:// protocols only provided for easy integration with existing infrastructure.
* `replicationType`: Defines how tenant's buckets needs to be replicated. Can be specified as `initial`, `continuous` or `initial+continuous` (default). In case of `initial` replication type, ISGW service will only sync served buckets once with ability to manually trigger re-sync again. In case of `continuous` replication type, ISGW service will enable asynchronous syncing from the moment a bucket added to a served list of buckets, so that only recent changes will be synchronized, continuously. In case of `initial+continuous` replication type, ISGW service will first initiate syncing of all existing served buckets contents and once all is copied, it will automatically enable `continuous` mode of operation, guaranteeing that none of the source objects are skipped.
* `metadataOnly`: If set, then only immutable metadata changes will be transferred. This also enables on-demand data fetch and E-LRU cache capability. In case of metadata only transfers, dynamically fetched data will be cached at the accessing endpoints ISGW links locations. If an application would want to read an offset of a data which isn't found, global search request will be formed and distributed across all the connected ISGW bi-directional links serving source bucket. Missing data chunks will be found and fetched, delivered transparently to application. Can be set to either `all` or `versions`. Setting it to `versions` enables extremely fast replication (next second!) of data modifications.
* `dynamicFetchAddr`: Defines local cluster pod service host:port. Used by EdgeFS gateways (S3, S3X, NFS or iSCSI) to distribute missing chunk fetch requests. It needs to be set to other than 0.0.0.0 on access endpoints so to enable dynamic fetch feature. Default is 0.0.0.0:49678.
* `dynamicFetchAddr`: Defines local cluster pod service host:port. Used by EdgeFS gateways (S3, S3X, NFS, SMB or iSCSI) to distribute missing chunk fetch requests. It needs to be set to other than 0.0.0.0 on access endpoints so to enable dynamic fetch feature. Default is 0.0.0.0:49678.
* `localAddr`: Defines local cluster pod service address and port. Used by ISGW link endpoint. Kubernetes ExternalIP will be automatically constructed if address is provided. Default is 0.0.0.0:14000.
* `useEncryptedTunnel`: If true, enables usage of encrypted SSH tunnel. By default, transferred metadata and data chunks are just compressed but unencrypted. Enable encryption if ISGW links crossing boundaries of single data center or multiple clouds.
* `chunkCacheSize`: Limit amount of memory allocated for dynamic chunk cache. By default ISGW pod uses up to 75% of available memory as chunk caching area. This option can influence this allocation strategy.
+3 -2
View File
@@ -10,7 +10,7 @@ indent: true
EdgeFS Data Fabric virtualzing common storage protocols and enables multi-cluster, multi-region data flow topologies.
This guide will walk you through the basic setup of a EdgeFS cluster namespaces and enable you to consume S3 object, NFS file access, and iSCSI block storage
This guide will walk you through the basic setup of a EdgeFS cluster namespaces and enable you to consume S3 object, NFS/SMB file access, and iSCSI block storage
from other pods running in your cluster, in decentralized ways.
## Minimum Version
@@ -54,7 +54,7 @@ kubectl create -f operator.yaml
kubectl create -f cluster.yaml
```
After the cluster is running, you can create [NFS, S3, or iSCSI](#storage) storage to be consumed by other applications in your cluster.
After the cluster is running, you can create [NFS, SMB, S3, or iSCSI](#storage) storage to be consumed by other applications in your cluster.
## Deploy the Rook Operator
@@ -107,6 +107,7 @@ Notice that EdgeFS Targets are running as StatefulSet.
For a walkthrough of the types of Storage CRDs exposed by EdgeFS Rook, see the guides for:
* **[NFS Server](edgefs-nfs-crd.md)**: Create Scale-Out NFS storage to be consumed by multiple pods, simultaneously
* **[SMB Server](edgefs-smb-crd.md)**: Create Scale-Out Microsoft CIFS/SMB storage to be consumed by multiple pods, simultaneously
* **[S3X](edgefs-s3x-crd.md)**: Create an Extended S3 HTTP/2 compatible object and key-value store that is accessible inside or outside the Kubernetes cluster
* **[AWS S3](edgefs-s3-crd.md)**: Create an AWS S3 compatible object store that is accessible inside or outside the Kubernetes cluster
* **[iSCSI Target](edgefs-iscsi-crd.md)**: Create low-latency and high-throughput iSCSI block to be consumed by a pod
+156
View File
@@ -0,0 +1,156 @@
---
title: Scale-Out SMB CRD
weight: 4350
indent: true
---
# EdgeFS Scale-Out SMB CRD
Rook allows creation and customization of EdgeFS SMB/CIFS filesystems through the custom resource definitions (CRDs).
The following settings are available for customization of EdgeFS SMB services.
## Sample
```yaml
apiVersion: edgefs.rook.io/v1
kind: SMB
metadata:
name: smb01
namespace: rook-edgefs
spec:
instances: 3
#relaxedDirUpdates: true
#chunkCacheSize: 1Gi
# A key/value list of annotations
annotations:
# key: value
placement:
# nodeAffinity:
# requiredDuringSchedulingIgnoredDuringExecution:
# nodeSelectorTerms:
# - matchExpressions:
# - key: role
# operator: In
# values:
# - smb-node
# tolerations:
# - key: smb-node
# operator: Exists
# podAffinity:
# podAntiAffinity:
#resourceProfile: embedded
resources:
# limits:
# cpu: "500m"
# memory: "1024Mi"
# requests:
# cpu: "500m"
# memory: "1024Mi"
```
## Metadata
* `name`: The name of the SMB system to create, which must match existing EdgeFS service.
* `namespace`: The namespace of the Rook cluster where the SMB service is created.
* `instances`: The number of active SMB service instances. EdgeFS SMB service is Multi-Head capable, such so that multiple PODs can mount same tenant's buckets via different endpoints.
* `relaxedDirUpdates`: If set to `true` then it will significantly improve performance of directory operations by defering updates, guaranteeing eventual directory consistency. This option is recommended when a bucket exported via single SMB instance and it is not a destination for ISGW Link synchronization.
* `chunkCacheSize`: Limit amount of memory allocated for dynamic chunk cache. By default SMB pod uses up to 75% of available memory as chunk caching area. This option can influence this allocation strategy.
* `annotations`: Key value pair list of annotations to add.
* `placement`: The SMB PODs can be given standard Kubernetes placement restrictions with `nodeAffinity`, `tolerations`, `podAffinity`, and `podAntiAffinity` similar to placement defined for daemons configured by the [cluster CRD](/cluster/examples/kubernetes/edgefs/cluster.yaml).
* `resourceProfile`: SMB pod resource utilization profile (Memory and CPU). Can be `embedded` or `performance` (default). In case of `performance` an SMB pod trying to increase amount of internal I/O resources that results in higher performance at the cost of additional memory allocation and more CPU load. In `embedded` profile case, SMB pod gives preference to preserving memory over I/O and limiting chunk cache (see `chunkCacheSize` option). The `performance` profile is the default unless cluster wide `embedded` option is defined.
* `resources`: Set resource requests/limits for the SMB Pod(s), see [Resource Requirements/Limits](edgefs-cluster-crd.md#resource-requirementslimits).
## Setting up EdgeFS namespace and tenant
For more detailed instructions please refer to [EdgeFS Wiki](https://github.com/Nexenta/edgefs/wiki).
Below is an exampmle procedure to get things initialized and configured.
Before new local namespace (or local site) can be used, it has to be initialized with FlexHash and special purpose root object.
FlexHash consists of dynamically discovered configuration and checkpoint of accepted distribution table. FlexHash is responsible for I/O direction and plays important role in dynamic load balancing logic. It defines so-called Negotiating Groups (typically across zoned 8-24 disks) and final table distribution across all the participating components, e.g. data nodes, service gateways and tools.
Root object holds system information and table of namespaces registered to a local site. Root object is always local and never shared between the sites.
To initialize system and prepare logical definitions, login to the toolbox as shown in this example:
```console
kubectl get po --all-namespaces | grep edgefs-mgr
kubectl exec -it -n rook-edgefs rook-edgefs-mgr-6cb9598469-czr7p -- env COLUMNS=$COLUMNS LINES=$LINES TERM=linux toolbox
```
Assumption at this point is that nodes are all configured and can be seen via the following command:
```console
efscli system status
```
1. Initialize EdgeFS cluster:
Verify that HW (or better say emulated in this case) configuration look normal and accept it:
```console
efscli system init
```
At this point new dynamically discovered configuration checkpoint will be created at $NEDGE_HOME/var/run/flexhash-checkpoint.json
This will also create system "root" object, holding Site's Namespace. Namespace may consist of more then single region.
2. Create new local namespace (or we also call it "Region" or "Segment"):
```console
efscli cluster create Hawaii
```
3. Create logical tenants of cluster namespace "Hawaii", also buckets if needed:
```console
efscli tenant create Hawaii/Cola
efscli bucket create Hawaii/Cola/bk1
efscli tenant create Hawaii/Pepsi
efscli bucket create Hawaii/Pepsi/bk1
```
Now cluster is setup, services can be now created and attached to CSI provisioner.
4. Create SMB service objects for tenants:
```console
efscli service create smb smb-cola
efscli service serve smb-cola Hawaii/Cola/bk1
efscli service config smb-cola X-SMB-OPTS-Cola-bk1 "security = share;guest account = nobody"
efscli service create smb smb-pepsi
efscli service serve smb-pepsi Hawaii/Pepsi/bk1
efscli service config smb-pepsi X-SMB-OPTS-Pepsi-bk1 "security = share;guest account = nobody"
```
Also, notice that we setting password-less configuration for bk1 share.
5. Create your EdgeFS SMB objects:
```yaml
apiVersion: edgefs.rook.io/v1
kind: SMB
metadata:
name: smb-cola
namespace: rook-edgefs
spec:
instances: 1
```
```yaml
apiVersion: edgefs.rook.io/v1
kind: SMB
metadata:
name: smb-pepsi
namespace: rook-edgefs
spec:
instances: 1
```
At this point two SMB services should be available. Install cifs-utils package and verify that you can mount it (substitute CLUSTERIP with corresponding entry from `kubectl get svc` command):
```console
kubectl get -o rook-edgefs svc | grep smb-pepsi
mount -t cifs -o vers=3.0,sec=none //CLUSTERIP/bk1 /tmp/bk1
```
+4 -4
View File
@@ -5,15 +5,15 @@ weight: 4000
# EdgeFS Data Fabric
[EdgeFS](http://edgefs.io) is high-performance and fault-tolerant decentralized data fabric with virtualized access to S3 object, NFS file, NoSQL and iSCSI block.
[EdgeFS](http://edgefs.io) is high-performance and fault-tolerant decentralized data fabric with virtualized access to S3 object, NFS/SMB file, NoSQL and iSCSI block.
EdgeFS is capable of spanning unlimited number of geographically distributed sites (Geo-site), connected with each other as one global name space data fabric running on top of Kubernetes platform, providing persistent, fault-tolerant and high-performance volumes for stateful Kubernetes Applications.
At each Geo-site, EdgeFS nodes deployed as containers (StatefulSet) on physical or virtual Kubernetes nodes, pooling available storage capacity and presenting it via compatible S3/NFS/iSCSI/etc storage emulated protocols for cloud-native applications running on the same or dedicated servers.
At each Geo-site, EdgeFS nodes deployed as containers (StatefulSet) on physical or virtual Kubernetes nodes, pooling available storage capacity and presenting it via compatible S3/NFS/SMB/iSCSI/etc storage emulated protocols for cloud-native applications running on the same or dedicated servers.
## How it works, in a Nutshell?
If you familiar with "git", where all modifications are fully versioned and globally immutable, it is highly likely you already know how it works at its core. Think of it as a world-scale copy-on-write technique. Now, if we can make a parallel for you to understand it better - what EdgeFS does, it expands "git" paradigm to object storage and making Kubernetes Persistent Volumes accessible via emulated storage standard protocols e.g. S3, NFS and even block devices such as iSCSI, in a high-performance and low-latency ways. With fully versioned modifications, fully immutable metadata and data, users data can be transparently replicated, distributed and dynamically pre-fetched across many Geo-sites.
If you familiar with "git", where all modifications are fully versioned and globally immutable, it is highly likely you already know how it works at its core. Think of it as a world-scale copy-on-write technique. Now, if we can make a parallel for you to understand it better - what EdgeFS does, it expands "git" paradigm to object storage and making Kubernetes Persistent Volumes accessible via emulated storage standard protocols e.g. S3, NFS, SMB and even block devices such as iSCSI, in a high-performance and low-latency ways. With fully versioned modifications, fully immutable metadata and data, users data can be transparently replicated, distributed and dynamically pre-fetched across many Geo-sites.
## Design
@@ -25,7 +25,7 @@ mount block devices and filesystems managed by Rook, or can use the S3/S3X API f
automates configuration of storage components and monitors the cluster to ensure the storage remains available
and healthy.
The Rook operator is a simple container that has all that is needed to bootstrap and monitor the storage cluster. The operator will start and monitor StatefulSet storage Targets, gRPC manager and Prometheus Multi-Tenant Dashboard. All the attached devices (or directories) will provide pooled storage site. Storage sites then can be easily connected with each other as one global name space data fabric. The operator manages CRDs for Targets, Scale-out NFS, Object stores (S3/S3X), and iSCSI volumes by initializing the pods and other artifacts necessary to
The Rook operator is a simple container that has all that is needed to bootstrap and monitor the storage cluster. The operator will start and monitor StatefulSet storage Targets, gRPC manager and Prometheus Multi-Tenant Dashboard. All the attached devices (or directories) will provide pooled storage site. Storage sites then can be easily connected with each other as one global name space data fabric. The operator manages CRDs for Targets, Scale-out NFS/SMB, Object stores (S3/S3X), and iSCSI volumes by initializing the pods and other artifacts necessary to
run the services.
The operator will monitor the storage Targets to ensure the cluster is healthy. EdgeFS will dynamically handle services failover, and other adjustments that maybe made as the cluster grows or shrinks.
+10
View File
@@ -142,3 +142,13 @@ Currently we adding new node `node3072ub16` with two drives `sdb` and `sdc` on i
* Save CRD and operator will update all target nodes and related pods of the EdgeFS cluster.
* Login to EdgeFS mgr toolbox and adjust FlexHash table to a new configuration using `efscli system fhtable` command.
## EdgeFS Operator Upgrade
In terms of to bring new Rook Operator features on a running system, it needs to be upgraded first.
And once upgraded, you will need to create new CustomResourceDefinition(s).
1. Edit deployment with `kubectl edit deploy rook-edgefs-operator -n rook-edgefs-system` and set to a new version
2. Apply new CRDs or changes. Look at examples in cluster/examples/kubernetes/edgefs/operator.yaml.
3. Monitor operator logs with `kubectl logs rook-edgefs-operator-ID -n rook-edgefs-system`
+3
View File
@@ -34,6 +34,9 @@ ConfigMap can be used in mix with the already existing Env Vars defined in opera
### EdgeFS
- Enhanced support for Time zones, now geo-distributed name space can be properly identified in case of Kubernetes stretched clusters.
- Added support for Microsoft Windows CIFS/SMB CRD. Now, it is possible to orchestrate Windows CIFS/SMB Scale-Out service managed by [Rook SMB CRD](Documentation/edgefs-smb-crd.md).
### YugabyteDB
- Master and TServer pods for YugabyteDB will have resources requests and limits specified as per YugabyteDB recommendations. This will help avoid the soft/hard memory limit issue.
@@ -1,7 +1,7 @@
apiVersion: edgefs.rook.io/v1
kind: ISCSI
metadata:
name: iscsi01
name: iscsi01 # this name has to match efscli service
namespace: rook-edgefs
spec:
#targetName:
+1 -1
View File
@@ -1,7 +1,7 @@
apiVersion: edgefs.rook.io/v1
kind: NFS
metadata:
name: nfs01
name: nfs01 # this name has to match efscli service
namespace: rook-edgefs
spec:
instances: 1
@@ -71,6 +71,35 @@ spec:
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: smbs.edgefs.rook.io
spec:
group: edgefs.rook.io
names:
kind: SMB
listKind: SMBList
plural: smbs
singular: smb
scope: Namespaced
version: v1
validation:
openAPIV3Schema:
properties:
spec:
properties:
instances:
type: integer
minimum: 1
required:
- instances
additionalPrinterColumns:
- name: Instances
type: string
description: Edgefs's service instances count
JSONPath: .spec.instances
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: nfss.edgefs.rook.io
spec:
+1 -1
View File
@@ -1,7 +1,7 @@
apiVersion: edgefs.rook.io/v1
kind: S3
metadata:
name: s301
name: s301 # this name has to match efscli service
namespace: rook-edgefs
spec:
instances: 1
+1 -1
View File
@@ -1,7 +1,7 @@
apiVersion: edgefs.rook.io/v1
kind: S3X
metadata:
name: s3x01
name: s3x01 # this name has to match efscli service
namespace: rook-edgefs
spec:
instances: 1
@@ -0,0 +1,35 @@
apiVersion: edgefs.rook.io/v1
kind: SMB
metadata:
name: smb01 # this name has to match efscli service
namespace: rook-edgefs
spec:
instances: 1
#placement:
# nodeAffinity:
# requiredDuringSchedulingIgnoredDuringExecution:
# nodeSelectorTerms:
# - matchExpressions:
# - key: rook-edgefs-nodetype
# operator: In
# values:
# - gateway
# tolerations:
# - key: node230
# operator: Exists
# podAffinity:
# podAntiAffinity:
#resources:
# limits:
# cpu: "2"
# memory: "4Gi"
# requests:
# cpu: "2"
# memory: "4Gi"
# A key value list of annotations
annotations:
# all:
# key: value
# mgr:
# prepare:
# target:
+2
View File
@@ -57,6 +57,8 @@ func addKnownTypes(scheme *runtime.Scheme) error {
&ClusterList{},
&NFS{},
&NFSList{},
&SMB{},
&SMBList{},
&S3{},
&S3List{},
&SWIFT{},
+33
View File
@@ -133,6 +133,39 @@ type NFSSpec struct {
// +genclient:noStatus
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
type SMB struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata"`
Spec SMBSpec `json:"spec"`
}
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
type SMBList struct {
metav1.TypeMeta `json:",inline"`
metav1.ListMeta `json:"metadata"`
Items []SMB `json:"items"`
}
// SMBSpec represent the spec of a pool
type SMBSpec struct {
// The annotations-related configuration to add/set on each Pod related object.
Annotations rookv1.Annotations `json:"annotations,omitempty"`
// The affinity to place the SMB pods (default is to place on any available nodes in EdgeFS running namespace)
Placement rookv1.Placement `json:"placement"`
// Resources set resource requests and limits
Resources v1.ResourceRequirements `json:"resources,omitempty"`
// The number of pods in the SMB replicaset
Instances int32 `json:"instances"`
RelaxedDirUpdates bool `json:"relaxedDirUpdates,omitempty"`
ResourceProfile string `json:"resourceProfile,omitempty"`
ChunkCacheSize resource.Quantity `json:"chunkCacheSize,omitempty"`
}
// +genclient
// +genclient:noStatus
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
type S3 struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata"`
@@ -937,6 +937,92 @@ func (in *S3XSpec) DeepCopy() *S3XSpec {
return out
}
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *SMB) DeepCopyInto(out *SMB) {
*out = *in
out.TypeMeta = in.TypeMeta
in.ObjectMeta.DeepCopyInto(&out.ObjectMeta)
in.Spec.DeepCopyInto(&out.Spec)
return
}
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new SMB.
func (in *SMB) DeepCopy() *SMB {
if in == nil {
return nil
}
out := new(SMB)
in.DeepCopyInto(out)
return out
}
// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
func (in *SMB) DeepCopyObject() runtime.Object {
if c := in.DeepCopy(); c != nil {
return c
}
return nil
}
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *SMBList) DeepCopyInto(out *SMBList) {
*out = *in
out.TypeMeta = in.TypeMeta
in.ListMeta.DeepCopyInto(&out.ListMeta)
if in.Items != nil {
in, out := &in.Items, &out.Items
*out = make([]SMB, len(*in))
for i := range *in {
(*in)[i].DeepCopyInto(&(*out)[i])
}
}
return
}
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new SMBList.
func (in *SMBList) DeepCopy() *SMBList {
if in == nil {
return nil
}
out := new(SMBList)
in.DeepCopyInto(out)
return out
}
// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
func (in *SMBList) DeepCopyObject() runtime.Object {
if c := in.DeepCopy(); c != nil {
return c
}
return nil
}
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *SMBSpec) DeepCopyInto(out *SMBSpec) {
*out = *in
if in.Annotations != nil {
in, out := &in.Annotations, &out.Annotations
*out = make(rookiov1.Annotations, len(*in))
for key, val := range *in {
(*out)[key] = val
}
}
in.Placement.DeepCopyInto(&out.Placement)
in.Resources.DeepCopyInto(&out.Resources)
out.ChunkCacheSize = in.ChunkCacheSize.DeepCopy()
return
}
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new SMBSpec.
func (in *SMBSpec) DeepCopy() *SMBSpec {
if in == nil {
return nil
}
out := new(SMBSpec)
in.DeepCopyInto(out)
return out
}
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *SWIFT) DeepCopyInto(out *SWIFT) {
*out = *in
@@ -32,6 +32,7 @@ type EdgefsV1Interface interface {
NFSsGetter
S3sGetter
S3XsGetter
SMBsGetter
SWIFTsGetter
}
@@ -64,6 +65,10 @@ func (c *EdgefsV1Client) S3Xs(namespace string) S3XInterface {
return newS3Xs(c, namespace)
}
func (c *EdgefsV1Client) SMBs(namespace string) SMBInterface {
return newSMBs(c, namespace)
}
func (c *EdgefsV1Client) SWIFTs(namespace string) SWIFTInterface {
return newSWIFTs(c, namespace)
}
@@ -52,6 +52,10 @@ func (c *FakeEdgefsV1) S3Xs(namespace string) v1.S3XInterface {
return &FakeS3Xs{c, namespace}
}
func (c *FakeEdgefsV1) SMBs(namespace string) v1.SMBInterface {
return &FakeSMBs{c, namespace}
}
func (c *FakeEdgefsV1) SWIFTs(namespace string) v1.SWIFTInterface {
return &FakeSWIFTs{c, namespace}
}
@@ -0,0 +1,128 @@
/*
Copyright The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
// Code generated by client-gen. DO NOT EDIT.
package fake
import (
edgefsrookiov1 "github.com/rook/rook/pkg/apis/edgefs.rook.io/v1"
v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
labels "k8s.io/apimachinery/pkg/labels"
schema "k8s.io/apimachinery/pkg/runtime/schema"
types "k8s.io/apimachinery/pkg/types"
watch "k8s.io/apimachinery/pkg/watch"
testing "k8s.io/client-go/testing"
)
// FakeSMBs implements SMBInterface
type FakeSMBs struct {
Fake *FakeEdgefsV1
ns string
}
var smbsResource = schema.GroupVersionResource{Group: "edgefs.rook.io", Version: "v1", Resource: "smbs"}
var smbsKind = schema.GroupVersionKind{Group: "edgefs.rook.io", Version: "v1", Kind: "SMB"}
// Get takes name of the sMB, and returns the corresponding sMB object, and an error if there is any.
func (c *FakeSMBs) Get(name string, options v1.GetOptions) (result *edgefsrookiov1.SMB, err error) {
obj, err := c.Fake.
Invokes(testing.NewGetAction(smbsResource, c.ns, name), &edgefsrookiov1.SMB{})
if obj == nil {
return nil, err
}
return obj.(*edgefsrookiov1.SMB), err
}
// List takes label and field selectors, and returns the list of SMBs that match those selectors.
func (c *FakeSMBs) List(opts v1.ListOptions) (result *edgefsrookiov1.SMBList, err error) {
obj, err := c.Fake.
Invokes(testing.NewListAction(smbsResource, smbsKind, c.ns, opts), &edgefsrookiov1.SMBList{})
if obj == nil {
return nil, err
}
label, _, _ := testing.ExtractFromListOptions(opts)
if label == nil {
label = labels.Everything()
}
list := &edgefsrookiov1.SMBList{ListMeta: obj.(*edgefsrookiov1.SMBList).ListMeta}
for _, item := range obj.(*edgefsrookiov1.SMBList).Items {
if label.Matches(labels.Set(item.Labels)) {
list.Items = append(list.Items, item)
}
}
return list, err
}
// Watch returns a watch.Interface that watches the requested sMBs.
func (c *FakeSMBs) Watch(opts v1.ListOptions) (watch.Interface, error) {
return c.Fake.
InvokesWatch(testing.NewWatchAction(smbsResource, c.ns, opts))
}
// Create takes the representation of a sMB and creates it. Returns the server's representation of the sMB, and an error, if there is any.
func (c *FakeSMBs) Create(sMB *edgefsrookiov1.SMB) (result *edgefsrookiov1.SMB, err error) {
obj, err := c.Fake.
Invokes(testing.NewCreateAction(smbsResource, c.ns, sMB), &edgefsrookiov1.SMB{})
if obj == nil {
return nil, err
}
return obj.(*edgefsrookiov1.SMB), err
}
// Update takes the representation of a sMB and updates it. Returns the server's representation of the sMB, and an error, if there is any.
func (c *FakeSMBs) Update(sMB *edgefsrookiov1.SMB) (result *edgefsrookiov1.SMB, err error) {
obj, err := c.Fake.
Invokes(testing.NewUpdateAction(smbsResource, c.ns, sMB), &edgefsrookiov1.SMB{})
if obj == nil {
return nil, err
}
return obj.(*edgefsrookiov1.SMB), err
}
// Delete takes name of the sMB and deletes it. Returns an error if one occurs.
func (c *FakeSMBs) Delete(name string, options *v1.DeleteOptions) error {
_, err := c.Fake.
Invokes(testing.NewDeleteAction(smbsResource, c.ns, name), &edgefsrookiov1.SMB{})
return err
}
// DeleteCollection deletes a collection of objects.
func (c *FakeSMBs) DeleteCollection(options *v1.DeleteOptions, listOptions v1.ListOptions) error {
action := testing.NewDeleteCollectionAction(smbsResource, c.ns, listOptions)
_, err := c.Fake.Invokes(action, &edgefsrookiov1.SMBList{})
return err
}
// Patch applies the patch and returns the patched sMB.
func (c *FakeSMBs) Patch(name string, pt types.PatchType, data []byte, subresources ...string) (result *edgefsrookiov1.SMB, err error) {
obj, err := c.Fake.
Invokes(testing.NewPatchSubresourceAction(smbsResource, c.ns, name, pt, data, subresources...), &edgefsrookiov1.SMB{})
if obj == nil {
return nil, err
}
return obj.(*edgefsrookiov1.SMB), err
}
@@ -30,4 +30,6 @@ type S3Expansion interface{}
type S3XExpansion interface{}
type SMBExpansion interface{}
type SWIFTExpansion interface{}
@@ -0,0 +1,174 @@
/*
Copyright The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
// Code generated by client-gen. DO NOT EDIT.
package v1
import (
"time"
v1 "github.com/rook/rook/pkg/apis/edgefs.rook.io/v1"
scheme "github.com/rook/rook/pkg/client/clientset/versioned/scheme"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
types "k8s.io/apimachinery/pkg/types"
watch "k8s.io/apimachinery/pkg/watch"
rest "k8s.io/client-go/rest"
)
// SMBsGetter has a method to return a SMBInterface.
// A group's client should implement this interface.
type SMBsGetter interface {
SMBs(namespace string) SMBInterface
}
// SMBInterface has methods to work with SMB resources.
type SMBInterface interface {
Create(*v1.SMB) (*v1.SMB, error)
Update(*v1.SMB) (*v1.SMB, error)
Delete(name string, options *metav1.DeleteOptions) error
DeleteCollection(options *metav1.DeleteOptions, listOptions metav1.ListOptions) error
Get(name string, options metav1.GetOptions) (*v1.SMB, error)
List(opts metav1.ListOptions) (*v1.SMBList, error)
Watch(opts metav1.ListOptions) (watch.Interface, error)
Patch(name string, pt types.PatchType, data []byte, subresources ...string) (result *v1.SMB, err error)
SMBExpansion
}
// sMBs implements SMBInterface
type sMBs struct {
client rest.Interface
ns string
}
// newSMBs returns a SMBs
func newSMBs(c *EdgefsV1Client, namespace string) *sMBs {
return &sMBs{
client: c.RESTClient(),
ns: namespace,
}
}
// Get takes name of the sMB, and returns the corresponding sMB object, and an error if there is any.
func (c *sMBs) Get(name string, options metav1.GetOptions) (result *v1.SMB, err error) {
result = &v1.SMB{}
err = c.client.Get().
Namespace(c.ns).
Resource("smbs").
Name(name).
VersionedParams(&options, scheme.ParameterCodec).
Do().
Into(result)
return
}
// List takes label and field selectors, and returns the list of SMBs that match those selectors.
func (c *sMBs) List(opts metav1.ListOptions) (result *v1.SMBList, err error) {
var timeout time.Duration
if opts.TimeoutSeconds != nil {
timeout = time.Duration(*opts.TimeoutSeconds) * time.Second
}
result = &v1.SMBList{}
err = c.client.Get().
Namespace(c.ns).
Resource("smbs").
VersionedParams(&opts, scheme.ParameterCodec).
Timeout(timeout).
Do().
Into(result)
return
}
// Watch returns a watch.Interface that watches the requested sMBs.
func (c *sMBs) Watch(opts metav1.ListOptions) (watch.Interface, error) {
var timeout time.Duration
if opts.TimeoutSeconds != nil {
timeout = time.Duration(*opts.TimeoutSeconds) * time.Second
}
opts.Watch = true
return c.client.Get().
Namespace(c.ns).
Resource("smbs").
VersionedParams(&opts, scheme.ParameterCodec).
Timeout(timeout).
Watch()
}
// Create takes the representation of a sMB and creates it. Returns the server's representation of the sMB, and an error, if there is any.
func (c *sMBs) Create(sMB *v1.SMB) (result *v1.SMB, err error) {
result = &v1.SMB{}
err = c.client.Post().
Namespace(c.ns).
Resource("smbs").
Body(sMB).
Do().
Into(result)
return
}
// Update takes the representation of a sMB and updates it. Returns the server's representation of the sMB, and an error, if there is any.
func (c *sMBs) Update(sMB *v1.SMB) (result *v1.SMB, err error) {
result = &v1.SMB{}
err = c.client.Put().
Namespace(c.ns).
Resource("smbs").
Name(sMB.Name).
Body(sMB).
Do().
Into(result)
return
}
// Delete takes name of the sMB and deletes it. Returns an error if one occurs.
func (c *sMBs) Delete(name string, options *metav1.DeleteOptions) error {
return c.client.Delete().
Namespace(c.ns).
Resource("smbs").
Name(name).
Body(options).
Do().
Error()
}
// DeleteCollection deletes a collection of objects.
func (c *sMBs) DeleteCollection(options *metav1.DeleteOptions, listOptions metav1.ListOptions) error {
var timeout time.Duration
if listOptions.TimeoutSeconds != nil {
timeout = time.Duration(*listOptions.TimeoutSeconds) * time.Second
}
return c.client.Delete().
Namespace(c.ns).
Resource("smbs").
VersionedParams(&listOptions, scheme.ParameterCodec).
Timeout(timeout).
Body(options).
Do().
Error()
}
// Patch applies the patch and returns the patched sMB.
func (c *sMBs) Patch(name string, pt types.PatchType, data []byte, subresources ...string) (result *v1.SMB, err error) {
result = &v1.SMB{}
err = c.client.Patch(pt).
Namespace(c.ns).
Resource("smbs").
SubResource(subresources...).
Name(name).
Body(data).
Do().
Into(result)
return
}
@@ -36,6 +36,8 @@ type Interface interface {
S3s() S3Informer
// S3Xs returns a S3XInformer.
S3Xs() S3XInformer
// SMBs returns a SMBInformer.
SMBs() SMBInformer
// SWIFTs returns a SWIFTInformer.
SWIFTs() SWIFTInformer
}
@@ -81,6 +83,11 @@ func (v *version) S3Xs() S3XInformer {
return &s3XInformer{factory: v.factory, namespace: v.namespace, tweakListOptions: v.tweakListOptions}
}
// SMBs returns a SMBInformer.
func (v *version) SMBs() SMBInformer {
return &sMBInformer{factory: v.factory, namespace: v.namespace, tweakListOptions: v.tweakListOptions}
}
// SWIFTs returns a SWIFTInformer.
func (v *version) SWIFTs() SWIFTInformer {
return &sWIFTInformer{factory: v.factory, namespace: v.namespace, tweakListOptions: v.tweakListOptions}
@@ -0,0 +1,89 @@
/*
Copyright The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
// Code generated by informer-gen. DO NOT EDIT.
package v1
import (
time "time"
edgefsrookiov1 "github.com/rook/rook/pkg/apis/edgefs.rook.io/v1"
versioned "github.com/rook/rook/pkg/client/clientset/versioned"
internalinterfaces "github.com/rook/rook/pkg/client/informers/externalversions/internalinterfaces"
v1 "github.com/rook/rook/pkg/client/listers/edgefs.rook.io/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
runtime "k8s.io/apimachinery/pkg/runtime"
watch "k8s.io/apimachinery/pkg/watch"
cache "k8s.io/client-go/tools/cache"
)
// SMBInformer provides access to a shared informer and lister for
// SMBs.
type SMBInformer interface {
Informer() cache.SharedIndexInformer
Lister() v1.SMBLister
}
type sMBInformer struct {
factory internalinterfaces.SharedInformerFactory
tweakListOptions internalinterfaces.TweakListOptionsFunc
namespace string
}
// NewSMBInformer constructs a new informer for SMB type.
// Always prefer using an informer factory to get a shared informer instead of getting an independent
// one. This reduces memory footprint and number of connections to the server.
func NewSMBInformer(client versioned.Interface, namespace string, resyncPeriod time.Duration, indexers cache.Indexers) cache.SharedIndexInformer {
return NewFilteredSMBInformer(client, namespace, resyncPeriod, indexers, nil)
}
// NewFilteredSMBInformer constructs a new informer for SMB type.
// Always prefer using an informer factory to get a shared informer instead of getting an independent
// one. This reduces memory footprint and number of connections to the server.
func NewFilteredSMBInformer(client versioned.Interface, namespace string, resyncPeriod time.Duration, indexers cache.Indexers, tweakListOptions internalinterfaces.TweakListOptionsFunc) cache.SharedIndexInformer {
return cache.NewSharedIndexInformer(
&cache.ListWatch{
ListFunc: func(options metav1.ListOptions) (runtime.Object, error) {
if tweakListOptions != nil {
tweakListOptions(&options)
}
return client.EdgefsV1().SMBs(namespace).List(options)
},
WatchFunc: func(options metav1.ListOptions) (watch.Interface, error) {
if tweakListOptions != nil {
tweakListOptions(&options)
}
return client.EdgefsV1().SMBs(namespace).Watch(options)
},
},
&edgefsrookiov1.SMB{},
resyncPeriod,
indexers,
)
}
func (f *sMBInformer) defaultInformer(client versioned.Interface, resyncPeriod time.Duration) cache.SharedIndexInformer {
return NewFilteredSMBInformer(client, f.namespace, resyncPeriod, cache.Indexers{cache.NamespaceIndex: cache.MetaNamespaceIndexFunc}, f.tweakListOptions)
}
func (f *sMBInformer) Informer() cache.SharedIndexInformer {
return f.factory.InformerFor(&edgefsrookiov1.SMB{}, f.defaultInformer)
}
func (f *sMBInformer) Lister() v1.SMBLister {
return v1.NewSMBLister(f.Informer().GetIndexer())
}
@@ -66,6 +66,14 @@ type S3XListerExpansion interface{}
// S3XNamespaceLister.
type S3XNamespaceListerExpansion interface{}
// SMBListerExpansion allows custom methods to be added to
// SMBLister.
type SMBListerExpansion interface{}
// SMBNamespaceListerExpansion allows custom methods to be added to
// SMBNamespaceLister.
type SMBNamespaceListerExpansion interface{}
// SWIFTListerExpansion allows custom methods to be added to
// SWIFTLister.
type SWIFTListerExpansion interface{}
@@ -0,0 +1,94 @@
/*
Copyright The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
// Code generated by lister-gen. DO NOT EDIT.
package v1
import (
v1 "github.com/rook/rook/pkg/apis/edgefs.rook.io/v1"
"k8s.io/apimachinery/pkg/api/errors"
"k8s.io/apimachinery/pkg/labels"
"k8s.io/client-go/tools/cache"
)
// SMBLister helps list SMBs.
type SMBLister interface {
// List lists all SMBs in the indexer.
List(selector labels.Selector) (ret []*v1.SMB, err error)
// SMBs returns an object that can list and get SMBs.
SMBs(namespace string) SMBNamespaceLister
SMBListerExpansion
}
// sMBLister implements the SMBLister interface.
type sMBLister struct {
indexer cache.Indexer
}
// NewSMBLister returns a new SMBLister.
func NewSMBLister(indexer cache.Indexer) SMBLister {
return &sMBLister{indexer: indexer}
}
// List lists all SMBs in the indexer.
func (s *sMBLister) List(selector labels.Selector) (ret []*v1.SMB, err error) {
err = cache.ListAll(s.indexer, selector, func(m interface{}) {
ret = append(ret, m.(*v1.SMB))
})
return ret, err
}
// SMBs returns an object that can list and get SMBs.
func (s *sMBLister) SMBs(namespace string) SMBNamespaceLister {
return sMBNamespaceLister{indexer: s.indexer, namespace: namespace}
}
// SMBNamespaceLister helps list and get SMBs.
type SMBNamespaceLister interface {
// List lists all SMBs in the indexer for a given namespace.
List(selector labels.Selector) (ret []*v1.SMB, err error)
// Get retrieves the SMB from the indexer for a given namespace and name.
Get(name string) (*v1.SMB, error)
SMBNamespaceListerExpansion
}
// sMBNamespaceLister implements the SMBNamespaceLister
// interface.
type sMBNamespaceLister struct {
indexer cache.Indexer
namespace string
}
// List lists all SMBs in the indexer for a given namespace.
func (s sMBNamespaceLister) List(selector labels.Selector) (ret []*v1.SMB, err error) {
err = cache.ListAllByNamespace(s.indexer, s.namespace, selector, func(m interface{}) {
ret = append(ret, m.(*v1.SMB))
})
return ret, err
}
// Get retrieves the SMB from the indexer for a given namespace and name.
func (s sMBNamespaceLister) Get(name string) (*v1.SMB, error) {
obj, exists, err := s.indexer.GetByKey(s.namespace + "/" + name)
if err != nil {
return nil, err
}
if !exists {
return nil, errors.NewNotFound(v1.Resource("smb"), name)
}
return obj.(*v1.SMB), nil
}
+14
View File
@@ -32,6 +32,7 @@ import (
"github.com/rook/rook/pkg/operator/edgefs/nfs"
"github.com/rook/rook/pkg/operator/edgefs/s3"
"github.com/rook/rook/pkg/operator/edgefs/s3x"
"github.com/rook/rook/pkg/operator/edgefs/smb"
"github.com/rook/rook/pkg/operator/edgefs/swift"
"github.com/rook/rook/pkg/operator/k8sutil"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
@@ -170,6 +171,19 @@ func (c *ClusterController) onAdd(obj interface{}) {
cluster.Spec.UseHostLocalTime)
NFSController.StartWatch(cluster.stopCh)
// Start SMB service CRD watcher
SMBController := smb.NewSMBController(c.context,
cluster.Namespace,
c.containerImage,
cluster.Spec.Network,
cluster.Spec.DataDirHostPath, cluster.Spec.DataVolumeSize,
edgefsv1.GetTargetPlacement(cluster.Spec.Placement),
cluster.Spec.Resources,
cluster.Spec.ResourceProfile,
cluster.ownerRef,
cluster.Spec.UseHostLocalTime)
SMBController.StartWatch(cluster.stopCh)
// Start S3 service CRD watcher
S3Controller := s3.NewS3Controller(c.context,
cluster.Namespace,
+1 -1
View File
@@ -173,7 +173,7 @@ func (c *ISCSIController) ParentClusterChanged(cluster edgefsv1.ClusterSpec) {
iscsis, err := c.context.RookClientset.EdgefsV1().ISCSIs(c.namespace).List(metav1.ListOptions{})
if err != nil {
logger.Errorf("failed to retrieve NFSes to update the Edgefs version. %+v", err)
logger.Errorf("failed to retrieve ISCSIes to update the Edgefs version. %+v", err)
return
}
for _, iscsi := range iscsis.Items {
+225
View File
@@ -0,0 +1,225 @@
/*
Copyright 2019 The Rook Authors. All rights reserved.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
// scale-out, multi-cloud SMB
package smb
import (
"fmt"
"reflect"
"github.com/coreos/pkg/capnslog"
"github.com/google/go-cmp/cmp"
edgefsv1 "github.com/rook/rook/pkg/apis/edgefs.rook.io/v1"
rookv1 "github.com/rook/rook/pkg/apis/rook.io/v1"
"github.com/rook/rook/pkg/clusterd"
"github.com/rook/rook/pkg/operator/k8sutil"
v1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/api/resource"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/tools/cache"
)
const (
customResourceName = "smb"
customResourceNamePlural = "smbs"
)
var logger = capnslog.NewPackageLogger("github.com/rook/rook", "edgefs-op-smb")
// SMBResource represents the smb custom resource
var SMBResource = k8sutil.CustomResource{
Name: customResourceName,
Plural: customResourceNamePlural,
Group: edgefsv1.CustomResourceGroup,
Version: edgefsv1.Version,
Kind: reflect.TypeOf(edgefsv1.SMB{}).Name(),
}
// SMBController represents a controller object for smb custom resources
type SMBController struct {
context *clusterd.Context
namespace string
rookImage string
NetworkSpec rookv1.NetworkSpec
dataDirHostPath string
dataVolumeSize resource.Quantity
annotations rookv1.Annotations
placement rookv1.Placement
resources v1.ResourceRequirements
resourceProfile string
ownerRef metav1.OwnerReference
useHostLocalTime bool
}
// NewSMBController create controller for watching smb custom resources created
func NewSMBController(
context *clusterd.Context,
namespace string,
rookImage string,
NetworkSpec rookv1.NetworkSpec,
dataDirHostPath string,
dataVolumeSize resource.Quantity,
placement rookv1.Placement,
resources v1.ResourceRequirements,
resourceProfile string,
ownerRef metav1.OwnerReference,
useHostLocalTime bool,
) *SMBController {
return &SMBController{
context: context,
namespace: namespace,
rookImage: rookImage,
NetworkSpec: NetworkSpec,
dataDirHostPath: dataDirHostPath,
dataVolumeSize: dataVolumeSize,
placement: placement,
resources: resources,
resourceProfile: resourceProfile,
ownerRef: ownerRef,
useHostLocalTime: useHostLocalTime,
}
}
// StartWatch watches for instances of SMB custom resources and acts on them
func (c *SMBController) StartWatch(stopCh chan struct{}) {
resourceHandlerFuncs := cache.ResourceEventHandlerFuncs{
AddFunc: c.onAdd,
UpdateFunc: c.onUpdate,
DeleteFunc: c.onDelete,
}
logger.Infof("start watching smb resources in namespace %s", c.namespace)
go k8sutil.WatchCR(SMBResource, c.namespace, resourceHandlerFuncs, c.context.RookClientset.EdgefsV1().RESTClient(), &edgefsv1.SMB{}, stopCh)
}
func (c *SMBController) onAdd(obj interface{}) {
smb, err := getSMBObject(obj)
if err != nil {
logger.Errorf("failed to get smb object: %+v", err)
return
}
if err = c.CreateService(*smb, c.serviceOwners(smb)); err != nil {
logger.Errorf("failed to create smb %s. %+v", smb.Name, err)
}
}
func (c *SMBController) onUpdate(oldObj, newObj interface{}) {
oldService, err := getSMBObject(oldObj)
if err != nil {
logger.Errorf("failed to get old smb object: %+v", err)
return
}
newService, err := getSMBObject(newObj)
if err != nil {
logger.Errorf("failed to get new smb object: %+v", err)
return
}
if !serviceChanged(oldService.Spec, newService.Spec) {
logger.Debugf("smb %s did not change", newService.Name)
return
}
logger.Infof("applying smb %s changes", newService.Name)
if err = c.UpdateService(*newService, c.serviceOwners(newService)); err != nil {
logger.Errorf("failed to create (modify) smb %s. %+v", newService.Name, err)
}
}
func (c *SMBController) onDelete(obj interface{}) {
smb, err := getSMBObject(obj)
if err != nil {
logger.Errorf("failed to get smb object: %+v", err)
return
}
if err = c.DeleteService(*smb); err != nil {
logger.Errorf("failed to delete smb %s. %+v", smb.Name, err)
}
}
func (c *SMBController) ParentClusterChanged(cluster edgefsv1.ClusterSpec) {
if c.rookImage == cluster.EdgefsImageName {
logger.Infof("No need to update the smb service, the same images present")
return
}
// update controller options by updated cluster spec
c.rookImage = cluster.EdgefsImageName
smbes, err := c.context.RookClientset.EdgefsV1().SMBs(c.namespace).List(metav1.ListOptions{})
if err != nil {
logger.Errorf("failed to retrieve SMBes to update the Edgefs version. %+v", err)
return
}
for _, smb := range smbes.Items {
logger.Infof("updating the Edgefs version for smb %s to %s", smb.Name, cluster.EdgefsImageName)
err := c.UpdateService(smb, nil)
if err != nil {
logger.Errorf("failed to update smb %s. %+v", smb.Name, err)
} else {
logger.Infof("updated smb %s to Edgefs version %s", smb.Name, cluster.EdgefsImageName)
}
}
}
func (c *SMBController) serviceOwners(service *edgefsv1.SMB) []metav1.OwnerReference {
// Only set the cluster crd as the owner of the SMB resources.
// If the SMB crd is deleted, the operator will explicitly remove the SMB resources.
// If the SMB crd still exists when the cluster crd is deleted, this will make sure the SMB
// resources are cleaned up.
return []metav1.OwnerReference{c.ownerRef}
}
func serviceChanged(oldService, newService edgefsv1.SMBSpec) bool {
var diff string
// any change in the crd will trigger an orchestration
if !reflect.DeepEqual(oldService, newService) {
func() {
defer func() {
if err := recover(); err != nil {
logger.Warningf("Encountered an issue getting smb service change differences: %v", err)
}
}()
// resource.Quantity has non-exportable fields, so we use its comparator method
resourceQtyComparer := cmp.Comparer(func(x, y resource.Quantity) bool { return x.Cmp(y) == 0 })
diff = cmp.Diff(oldService, newService, resourceQtyComparer)
logger.Infof("The SMB Service has changed. diff=%s", diff)
}()
}
if len(diff) > 0 {
return true
}
return false
}
func getSMBObject(obj interface{}) (smb *edgefsv1.SMB, err error) {
var ok bool
smb, ok = obj.(*edgefsv1.SMB)
if ok {
// the smb object is of the latest type, simply return it
return smb.DeepCopy(), nil
}
return nil, fmt.Errorf("not a known smb object: %+v", obj)
}
+365
View File
@@ -0,0 +1,365 @@
/*
Copyright 2019 The Rook Authors. All rights reserved.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
// Package smb for the Edgefs manager.
package smb
import (
"fmt"
edgefsv1 "github.com/rook/rook/pkg/apis/edgefs.rook.io/v1"
"github.com/rook/rook/pkg/clusterd"
"github.com/rook/rook/pkg/operator/k8sutil"
apps "k8s.io/api/apps/v1"
v1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
const (
appName = "rook-edgefs-smb"
/* Volumes definitions */
serviceAccountName = "rook-edgefs-cluster"
dataVolumeName = "edgefs-datadir"
stateVolumeFolder = ".state"
etcVolumeFolder = ".etc"
)
// Start the rgw manager
func (c *SMBController) CreateService(s edgefsv1.SMB, ownerRefs []metav1.OwnerReference) error {
return c.CreateOrUpdate(s, false, ownerRefs)
}
func (c *SMBController) UpdateService(s edgefsv1.SMB, ownerRefs []metav1.OwnerReference) error {
return c.CreateOrUpdate(s, true, ownerRefs)
}
// Start the smb instance
func (c *SMBController) CreateOrUpdate(s edgefsv1.SMB, update bool, ownerRefs []metav1.OwnerReference) error {
logger.Infof("starting update=%v service=%s", update, s.Name)
logger.Infof("SMB Base image is %s", c.rookImage)
// validate SMB service settings
if err := validateService(c.context, s); err != nil {
return fmt.Errorf("invalid SMB service %s arguments. %+v", s.Name, err)
}
// check if SMB service already exists
exists, err := serviceExists(c.context, s)
if err == nil && exists {
if !update {
logger.Infof("SMB service %s exists in namespace %s", s.Name, s.Namespace)
return nil
}
logger.Infof("SMB service %s exists in namespace %s. checking for updates", s.Name, s.Namespace)
}
// start the deployment
deployment := c.makeDeployment(s.Name, s.Namespace, c.rookImage, s.Spec)
if _, err := c.context.Clientset.AppsV1().Deployments(s.Namespace).Create(deployment); err != nil {
if !errors.IsAlreadyExists(err) {
return fmt.Errorf("failed to create %s deployment. %+v", appName, err)
}
logger.Infof("%s deployment already exists", appName)
if _, err := c.context.Clientset.AppsV1().Deployments(s.Namespace).Update(deployment); err != nil {
return fmt.Errorf("failed to update %s deployment. %+v", appName, err)
}
logger.Infof("%s deployment updated", appName)
} else {
logger.Infof("%s deployment started", appName)
}
// create the smb service
service := c.makeSMBService(instanceName(s.Name), s.Name, s.Namespace)
if _, err := c.context.Clientset.CoreV1().Services(s.Namespace).Create(service); err != nil {
if !errors.IsAlreadyExists(err) {
return fmt.Errorf("failed to create smb service. %+v", err)
}
logger.Infof("smb service %s already exists", service)
} else {
logger.Infof("smb service %s started", service)
}
return nil
}
func (c *SMBController) makeSMBService(name, svcname, namespace string) *v1.Service {
labels := getLabels(name, svcname, namespace)
svc := &v1.Service{
ObjectMeta: metav1.ObjectMeta{
Name: name,
Namespace: namespace,
Labels: labels,
},
Spec: v1.ServiceSpec{
Selector: labels,
Type: v1.ServiceTypeClusterIP,
Ports: []v1.ServicePort{
{Name: "grpc", Port: 49000, Protocol: v1.ProtocolTCP},
{Name: "netbios-ns", Port: 137, Protocol: v1.ProtocolTCP},
{Name: "netbios-ns-udp", Port: 137, Protocol: v1.ProtocolUDP},
{Name: "netbios-dgm", Port: 138, Protocol: v1.ProtocolTCP},
{Name: "netbios-dgm-udp", Port: 138, Protocol: v1.ProtocolUDP},
{Name: "netbios-ssn", Port: 139, Protocol: v1.ProtocolTCP},
{Name: "microsoft-ds", Port: 445, Protocol: v1.ProtocolTCP},
},
},
}
k8sutil.SetOwnerRef(&svc.ObjectMeta, &c.ownerRef)
return svc
}
func (c *SMBController) makeDeployment(svcname, namespace, rookImage string, smbSpec edgefsv1.SMBSpec) *apps.Deployment {
name := instanceName(svcname)
volumes := []v1.Volume{}
if c.useHostLocalTime {
volumes = append(volumes, edgefsv1.GetHostLocalTimeVolume())
volumes = append(volumes, edgefsv1.GetHostTimeZoneVolume())
}
if c.dataVolumeSize.Value() > 0 {
// dataVolume case
volumes = append(volumes, v1.Volume{
Name: dataVolumeName,
VolumeSource: v1.VolumeSource{
PersistentVolumeClaim: &v1.PersistentVolumeClaimVolumeSource{
ClaimName: dataVolumeName,
},
},
})
} else {
// dataDir case
volumes = append(volumes, v1.Volume{
Name: dataVolumeName,
VolumeSource: v1.VolumeSource{
HostPath: &v1.HostPathVolumeSource{
Path: c.dataDirHostPath,
},
},
})
}
podSpec := v1.PodTemplateSpec{
ObjectMeta: metav1.ObjectMeta{
Name: name,
Labels: getLabels(name, svcname, namespace),
},
Spec: v1.PodSpec{
Containers: []v1.Container{c.smbContainer(svcname, name, rookImage, smbSpec)},
RestartPolicy: v1.RestartPolicyAlways,
Volumes: volumes,
HostIPC: true,
HostNetwork: c.NetworkSpec.IsHost(),
NodeSelector: map[string]string{namespace: "cluster"},
ServiceAccountName: serviceAccountName,
},
}
if c.NetworkSpec.IsHost() {
podSpec.Spec.DNSPolicy = v1.DNSClusterFirstWithHostNet
} else if c.NetworkSpec.IsMultus() {
if err := k8sutil.ApplyMultus(c.NetworkSpec, &podSpec.ObjectMeta); err != nil {
logger.Errorf("failed to apply multus spec to podspec metadata for smb. %v", err)
}
}
smbSpec.Annotations.ApplyToObjectMeta(&podSpec.ObjectMeta)
// apply current SMB CRD options to pod's specification
smbSpec.Placement.ApplyToPodSpec(&podSpec.Spec)
d := &apps.Deployment{
ObjectMeta: metav1.ObjectMeta{
Name: name,
Namespace: namespace,
},
Spec: apps.DeploymentSpec{
Selector: &metav1.LabelSelector{
MatchLabels: podSpec.Labels,
},
Template: podSpec,
Replicas: &smbSpec.Instances,
},
}
k8sutil.SetOwnerRef(&d.ObjectMeta, &c.ownerRef)
smbSpec.Annotations.ApplyToObjectMeta(&d.ObjectMeta)
return d
}
func (c *SMBController) smbContainer(svcname, name, containerImage string, smbSpec edgefsv1.SMBSpec) v1.Container {
runAsUser := int64(0)
readOnlyRootFilesystem := false
securityContext := &v1.SecurityContext{
RunAsUser: &runAsUser,
ReadOnlyRootFilesystem: &readOnlyRootFilesystem,
Capabilities: &v1.Capabilities{
Add: []v1.Capability{"SYS_NICE", "SYS_RESOURCE", "IPC_LOCK"},
},
}
volumeMounts := []v1.VolumeMount{
{
Name: dataVolumeName,
MountPath: "/opt/nedge/etc.target",
SubPath: etcVolumeFolder,
},
{
Name: dataVolumeName,
MountPath: "/opt/nedge/var/run",
SubPath: stateVolumeFolder,
},
}
if c.useHostLocalTime {
volumeMounts = append(volumeMounts, edgefsv1.GetHostLocalTimeVolumeMount())
volumeMounts = append(volumeMounts, edgefsv1.GetHostTimeZoneVolumeMount())
}
cont := v1.Container{
Name: name,
Image: containerImage,
ImagePullPolicy: v1.PullAlways,
Args: []string{"smb"},
Env: []v1.EnvVar{
{
Name: "CCOW_LOG_LEVEL",
Value: "5",
},
{
Name: "CCOW_SVCNAME",
Value: svcname,
},
{
Name: "HOST_HOSTNAME",
ValueFrom: &v1.EnvVarSource{
FieldRef: &v1.ObjectFieldSelector{
FieldPath: "spec.nodeName",
},
},
},
{
Name: "K8S_NAMESPACE",
ValueFrom: &v1.EnvVarSource{
FieldRef: &v1.ObjectFieldSelector{
FieldPath: "metadata.namespace",
},
},
},
},
SecurityContext: securityContext,
Resources: smbSpec.Resources,
Ports: []v1.ContainerPort{
{Name: "grpc", ContainerPort: 49000, Protocol: v1.ProtocolTCP},
{Name: "netbios-ns", ContainerPort: 137, Protocol: v1.ProtocolTCP},
{Name: "netbios-ns-udp", ContainerPort: 137, Protocol: v1.ProtocolUDP},
{Name: "netbios-dgm", ContainerPort: 138, Protocol: v1.ProtocolTCP},
{Name: "netbios-dgm-udp", ContainerPort: 138, Protocol: v1.ProtocolUDP},
{Name: "netbios-ssn", ContainerPort: 139, Protocol: v1.ProtocolTCP},
{Name: "microsoft-ds", ContainerPort: 445, Protocol: v1.ProtocolTCP},
},
VolumeMounts: volumeMounts,
}
if smbSpec.RelaxedDirUpdates {
cont.Env = append(cont.Env, v1.EnvVar{
Name: "EFSSMB_RELAXED_DIR_UPDATES",
Value: "1",
})
}
cont.Env = append(cont.Env, edgefsv1.GetInitiatorEnvArr("smb",
c.resourceProfile == "embedded" || smbSpec.ResourceProfile == "embedded",
smbSpec.ChunkCacheSize, smbSpec.Resources)...)
return cont
}
// Delete SMB service and possibly some artifacts.
func (c *SMBController) DeleteService(s edgefsv1.SMB) error {
// check if service exists
exists, err := serviceExists(c.context, s)
if err != nil {
return fmt.Errorf("failed to detect if there is a SMB service to delete. %+v", err)
}
if !exists {
logger.Infof("SMB service %s does not exist in namespace %s", s.Name, s.Namespace)
return nil
}
logger.Infof("Deleting SMB service %s from namespace %s", s.Name, s.Namespace)
var gracePeriod int64
propagation := metav1.DeletePropagationForeground
options := &metav1.DeleteOptions{GracePeriodSeconds: &gracePeriod, PropagationPolicy: &propagation}
// Delete the smb service
err = c.context.Clientset.CoreV1().Services(s.Namespace).Delete(instanceName(s.Name), options)
if err != nil && !errors.IsNotFound(err) {
logger.Warningf("failed to delete SMB service. %+v", err)
}
// Make a best effort to delete the SMB pods
err = k8sutil.DeleteDeployment(c.context.Clientset, s.Namespace, instanceName(s.Name))
if err != nil {
logger.Warningf(err.Error())
}
logger.Infof("Completed deleting SMB service %s", s.Name)
return nil
}
func getLabels(name, svcname, namespace string) map[string]string {
return map[string]string{
k8sutil.AppAttr: appName,
k8sutil.ClusterAttr: namespace,
"edgefs_svcname": svcname,
"edgefs_svctype": "smb",
}
}
// Validate the SMB arguments
func validateService(context *clusterd.Context, s edgefsv1.SMB) error {
if s.Name == "" {
return fmt.Errorf("missing name")
}
if s.Namespace == "" {
return fmt.Errorf("missing namespace")
}
return nil
}
func instanceName(svcname string) string {
return fmt.Sprintf("%s-%s", appName, svcname)
}
// Check if the SMB service exists
func serviceExists(context *clusterd.Context, s edgefsv1.SMB) (bool, error) {
_, err := context.Clientset.AppsV1().Deployments(s.Namespace).Get(instanceName(s.Name), metav1.GetOptions{})
if err == nil {
// the deployment was found
return true, nil
}
if err != nil && !errors.IsNotFound(err) {
return false, err
}
// not found
return false, nil
}