Merge pull request #2758 from ashishranjan738/nfs

enhance(NFS): adds dynamic provisoning for nfs
This commit is contained in:
Jared Watts authored and GitHub committed 2019-04-26 10:08:51 -07:00
commit 21ddc83b84
23 files changed
+923 -433

No files matched your search

+30 -86
View File
@@ -32,8 +32,9 @@ You can check if the operator is up and running with:
```console
kubectl -n rook-nfs-system get pod
NAME READY STATUS RESTARTS AGE
rook-nfs-operator-b8d6d955d-cdq2v 1/1 Running 0 1m
NAME READY STATUS RESTARTS AGE
rook-nfs-operator-879f5bf8b-gnwht 1/1 Running 0 29m
rook-nfs-provisioner-65f4874c8f-kkz6b 1/1 Running 0 29m
```
## Create and Initialize NFS Server
@@ -61,7 +62,7 @@ kind: Namespace
metadata:
name: rook-nfs
---
# A default StorageClass must be present
# A default storageclass must be present
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
@@ -80,15 +81,19 @@ metadata:
name: rook-nfs
namespace: rook-nfs
spec:
serviceAccountName: rook-nfs
replicas: 1
exports:
- name: nfs-share
- name: share1
server:
accessMode: ReadWrite
squash: "none"
# A Persistent Volume Claim must be created before creating NFS CRD instance.
persistentVolumeClaim:
claimName: nfs-default-claim
# A key/value list of annotations
annotations:
# key: value
```
With the `nfs.yaml` file saved, now create the NFS server as shown:
@@ -118,54 +123,33 @@ rook-nfs-0 1/1 Running 0 2m
If the NFS server pod is in the `Running` state, then we have successfully created an exported NFS share that clients can start to access over the network.
### Accessing the Export
With PR https://github.com/rook/rook/pull/2758 rook starts supporting dynamic provisioning with NFS. This example will be showing how dynamic provisioning feature can be used for nfs.
To access the export from another pod, you must first manually create a `PersistentVolume` with the connection information.
This experience is a bit cumbersome, but will be improved in the future with dynamic provisioning support.
First, find the current IP address of your NFS server pod using the following command:
```console
kubectl -n rook-nfs get service -l app=rook-nfs -o jsonpath='{.items[0].spec.clusterIP}'
```
Copy this IP address and insert it into the following content in place of the `<Cluster IP>` placeholder.
Then also replace the `/<Claim name>` value with the `claimName` that was configured in the `nfs.yaml`.
In this case, it will be `nfs-default-claim` and it will look like `path: "/nfs-default-claim"`.
Finally, save the updated content to a file called `pv.yaml`.
Once the NFS Operator and an instance of NFSServer is deployed. A storageclass similar to below example has to be created to dynamically provisioning volumes.
```yaml
apiVersion: v1
kind: PersistentVolume
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: rook-nfs-pv
namespace: rook-nfs
spec:
capacity:
storage: 1Mi
accessModes:
- ReadWriteMany
nfs:
# FIXME: replace <Cluster IP> with the current IP of the NFS server pod.
# Run the below command to get the current IP address:
# kubectl -n rook-nfs get service -l app=rook-nfs -o jsonpath='{.items[0].spec.clusterIP}'
server: <Cluster IP>
path: "/<Claim name>"
labels:
app: rook-nfs
name: rook-nfs-share1
parameters:
exportName: share1
nfsServerName: rook-nfs
nfsServerNamespace: rook-nfs
provisioner: rook.io/nfs-provisioner
reclaimPolicy: Delete
volumeBindingMode: Immediate
```
With the `pv.yaml` file saved, we can create the PV object:
#### Note: The storageclass need to have the following 3 parameters passed.
1. `exportName`: It tells the provisioner which export to use for provisioning the volumes.
2. `nfsServerName`: It is the name of the NFSServer instance.
3. `nfsServerNamespace`: It namespace where the NFSServer instance is running.
```console
kubectl create -f pv.yaml
```
### Consuming the Export
Now we can consume the PV that we just created by creating an example web server app that uses a `PersistentVolumeClaim` to claim the exported volume.
There are 2 pods that comprise this example:
1. A web server pod that will read and display the contents of the NFS share
1. A writer pod that will write random data to the NFS share so the website will continually update
First, save the PVC definition in a file called `pvc.yaml`:
Once the above storageclass has been created create a PV claim referencing the storageclass as shown in the example given below.
```yaml
apiVersion: v1
@@ -173,50 +157,14 @@ kind: PersistentVolumeClaim
metadata:
name: rook-nfs-pv-claim
spec:
storageClassName: "rook-nfs-share1"
accessModes:
- ReadWriteMany
storageClassName: ""
resources:
requests:
storage: 1Mi
```
Then create the PVC:
```console
kubectl create -f pvc.yaml
```
Start both the busybox pod (writer) and the web server from the `cluster/examples/kubernetes/nfs` folder:
```console
kubectl create -f busybox-rc.yaml
kubectl create -f web-rc.yaml
```
Let's confirm that the expected busybox writer pods and web server pods are **all** up and in the `Running` state:
```console
kubectl get pod -l app=nfs-demo
```
In order to be able to reach the web server over the network, let's create a service for it:
```console
kubectl create -f web-service.yaml
```
We can then use the busybox writer pod we launched before to check that nginx is serving the data appropriately.
In the below 1-liner command, we use `kubectl exec` to run a command in the busybox writer pod that uses `wget` to retrieve the web page that the web server pod is hosting. As the busybox writer pod continues to write a new timestamp, we should see the returned output also update every ~10 seconds or so.
```console
> echo; kubectl exec $(kubectl get pod -l name=nfs-busybox -o jsonpath='{.items[0].metadata.name}') -- wget -qO- http://$(kubectl get services nfs-web -o jsonpath='{.spec.clusterIP}'); echo
Thu Oct 22 19:28:55 UTC 2015
nfs-busybox-w3s4t
```
## Rook Ceph volume example
In this alternative example, we will use a different underlying volume as an export for the NFS server.
@@ -273,11 +221,7 @@ Create the NFS server instance that you saved in `nfs-ceph.yaml`:
kubectl create -f nfs-ceph.yaml
```
After the NFS server pod is running, follow the same instructions from the previous example to access and consume the NFS share, with the following exception:
* Replace the `/<Claim name>` value in the PV definition in the [Accessing the Export](nfs.md#accessing-the-export) section with the `claimName` that was configured in the `nfs-ceph.yaml`. In this case it will be `nfs-ceph-claim` and will look like `path: "/nfs-ceph-claim"`.
After that, follow the rest of the instructions in the [Accessing the Export](nfs.md#accessing-the-export) section and then the [Consuming the Export](nfs.md#consuming-the-export) section to consume the NFS volume.
After the NFS server pod is running, follow the same instructions from the previous example to access and consume the NFS share.
## Teardown
Generated
+10 -1
View File
@@ -438,6 +438,14 @@
revision = "adf5a7427709b9deb95d29d3fa8a2bf9cfd388f1"
version = "v1.2"
[[projects]]
digest = "1:cf31692c14422fa27c83a05292eb5cbe0fb2775972e8f1f8446a71549bd8980b"
name = "github.com/pkg/errors"
packages = ["."]
pruneopts = "UT"
revision = "ba968bfe8b2f7e042a574c888954fccecfa385b4"
version = "v0.8.1"
[[projects]]
digest = "1:0028cb19b2e4c3112225cd871870f2d9cf49b9b4276531f03438a88e94be86fe"
name = "github.com/pmezard/go-difflib"
@@ -1318,7 +1326,6 @@
revision = "21c4ce38f2a793ec01e925ddc31216500183b773"
[[projects]]
branch = "release-1.14"
digest = "1:a2147c72c6ac436c7cbd3ffaefb3a8ca26264dd60fed5aec22cf4ed6d6c42690"
name = "sigs.k8s.io/sig-storage-lib-external-provisioner"
packages = [
@@ -1328,6 +1335,7 @@
]
pruneopts = "UT"
revision = "3eea7193b4c5c034f885913c03105178e9e23a4f"
version = "v3.1.0"
[[projects]]
digest = "1:fef5992ddf85e9ccad899b11338339858936a10fbce96b9d5c43767669d95398"
@@ -1367,6 +1375,7 @@
"github.com/google/uuid",
"github.com/icrowley/fake",
"github.com/jbw976/go-ps",
"github.com/pkg/errors",
"github.com/rook/operator-kit",
"github.com/spf13/cobra",
"github.com/spf13/pflag",
+13 -1
View File
@@ -92,7 +92,7 @@ ignored = [
[[constraint]]
name = "sigs.k8s.io/sig-storage-lib-external-provisioner"
branch = "release-1.14"
version = "v3.1.0"
# Kubernetes 1.14 dependencies on random package versions:
[[override]]
@@ -113,3 +113,15 @@ ignored = [
[[override]]
name = "bitbucket.org/ww/goautoneg"
source = "github.com/rancher/goautoneg"
[[override]]
name = "k8s.io/apiextensions-apiserver"
version = "kubernetes-1.14.0"
[[override]]
name = "k8s.io/apimachinery"
version = "kubernetes-1.14.0"
[[override]]
name = "k8s.io/client-go"
version = "v11.0.0"
+41 -41
View File
@@ -1,41 +1,41 @@
apiVersion: v1
kind: Namespace
metadata:
name: rook-nfs
---
# A rook ceph cluster must be running
# Create a rook ceph cluster using examples in rook/cluster/examples/kubernetes/ceph
# Refer to https://rook.io/docs/rook/master/ceph-quickstart.html for a quick rook cluster setup
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: nfs-ceph-claim
namespace: rook-nfs
spec:
storageClassName: rook-ceph-block
accessModes:
- ReadWriteMany
resources:
requests:
storage: 2Gi
---
apiVersion: nfs.rook.io/v1alpha1
kind: NFSServer
metadata:
name: rook-nfs
namespace: rook-nfs
spec:
replicas: 1
exports:
- name: nfs-share
server:
accessMode: ReadWrite
squash: "none"
# A Persistent Volume Claim must be created before creating NFS CRD instance.
# Create a Ceph cluster for using this example
# Create a ceph PVC after creating the rook ceph cluster using ceph-pvc.yaml
persistentVolumeClaim:
claimName: nfs-ceph-claim
# A key value list of annotations
annotations:
# key: value
apiVersion: v1
kind: Namespace
metadata:
name: rook-nfs
---
# A rook ceph cluster must be running
# Create a rook ceph cluster using examples in rook/cluster/examples/kubernetes/ceph
# Refer to https://rook.io/docs/rook/master/ceph-quickstart.html for a quick rook cluster setup
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: nfs-ceph-claim
namespace: rook-nfs
spec:
storageClassName: rook-ceph-block
accessModes:
- ReadWriteMany
resources:
requests:
storage: 2Gi
---
apiVersion: nfs.rook.io/v1alpha1
kind: NFSServer
metadata:
name: rook-nfs
namespace: rook-nfs
spec:
replicas: 1
exports:
- name: share1
server:
accessMode: ReadWrite
squash: "none"
# A Persistent Volume Claim must be created before creating NFS CRD instance.
# Create a Ceph cluster for using this example
# Create a ceph PVC after creating the rook ceph cluster using ceph-pvc.yaml
persistentVolumeClaim:
claimName: nfs-ceph-claim
# A key value list of annotations
annotations:
# key: value
+2 -1
View File
@@ -22,9 +22,10 @@ metadata:
name: rook-nfs
namespace: rook-nfs
spec:
serviceAccountName: rook-nfs
replicas: 1
exports:
- name: nfs-share
- name: share1
server:
accessMode: ReadWrite
squash: "none"
+206 -106
View File
@@ -1,106 +1,206 @@
apiVersion: v1
kind: Namespace
metadata:
name: rook-nfs-system
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: nfsservers.nfs.rook.io
spec:
group: nfs.rook.io
names:
kind: NFSServer
listKind: NFSServerList
plural: nfsservers
singular: nfsserver
scope: Namespaced
version: v1alpha1
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: rook-nfs-operator
rules:
- apiGroups:
- ""
resources:
- namespaces
- configmaps
- pods
- services
verbs:
- get
- watch
- create
- apiGroups:
- apps
resources:
- statefulsets
verbs:
- get
- create
- apiGroups:
- nfs.rook.io
resources:
- "*"
verbs:
- "*"
- apiGroups:
- rook.io
resources:
- "*"
verbs:
- "*"
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: rook-nfs-operator
namespace: rook-nfs-system
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: rook-nfs-operator
namespace: rook-nfs-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: rook-nfs-operator
subjects:
- kind: ServiceAccount
name: rook-nfs-operator
namespace: rook-nfs-system
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: rook-nfs-operator
namespace: rook-nfs-system
labels:
app: rook-nfs-operator
spec:
replicas: 1
selector:
matchLabels:
app: rook-nfs-operator
template:
metadata:
labels:
app: rook-nfs-operator
spec:
serviceAccountName: rook-nfs-operator
containers:
- name: rook-nfs-operator
image: rook/nfs:master
args: ["nfs", "operator"]
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
apiVersion: v1
kind: Namespace
metadata:
name: rook-nfs-system
---
# NFS operator deployment specifications
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: nfsservers.nfs.rook.io
spec:
group: nfs.rook.io
names:
kind: NFSServer
listKind: NFSServerList
plural: nfsservers
singular: nfsserver
scope: Namespaced
version: v1alpha1
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: rook-nfs-operator
rules:
- apiGroups:
- ""
resources:
- namespaces
- configmaps
- pods
- services
- storageclasses
verbs:
- get
- watch
- create
- apiGroups:
- apps
resources:
- statefulsets
verbs:
- get
- create
- apiGroups:
- nfs.rook.io
resources:
- "*"
verbs:
- "*"
- apiGroups:
- rook.io
resources:
- "*"
verbs:
- "*"
- apiGroups:
- storage.k8s.io
resources:
- storageclasses
verbs:
- get
- create
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: rook-nfs-operator
namespace: rook-nfs-system
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: rook-nfs-operator
namespace: rook-nfs-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: rook-nfs-operator
subjects:
- kind: ServiceAccount
name: rook-nfs-operator
namespace: rook-nfs-system
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: rook-nfs-operator
namespace: rook-nfs-system
labels:
app: rook-nfs-operator
spec:
replicas: 1
selector:
matchLabels:
app: rook-nfs-operator
template:
metadata:
labels:
app: rook-nfs-operator
spec:
serviceAccountName: rook-nfs-operator
containers:
- name: rook-nfs-operator
image: rook/nfs:master
imagePullPolicy: IfNotPresent
args: ["nfs", "operator"]
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
---
# NFS provisioner deployment specification
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: rook-nfs-provisioner-runner
rules:
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "create", "delete"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["create", "update", "patch"]
- apiGroups: [""]
resources: ["services", "endpoints"]
verbs: ["get"]
- apiGroups: ["extensions"]
resources: ["podsecuritypolicies"]
resourceNames: ["nfs-provisioner"]
verbs: ["use"]
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
- apiGroups:
- nfs.rook.io
resources:
- "*"
verbs:
- "*"
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: run-nfs-provisioner
subjects:
- kind: ServiceAccount
name: rook-nfs-provisioner
# replace with namespace where provisioner is deployed
namespace: rook-nfs-system
roleRef:
kind: ClusterRole
name: rook-nfs-provisioner-runner
apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: rook-nfs-provisioner
namespace: rook-nfs-system
---
kind: Deployment
apiVersion: apps/v1
metadata:
name: rook-nfs-provisioner
namespace: rook-nfs-system
spec:
selector:
matchLabels:
app: rook-nfs-provisioner
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
app: rook-nfs-provisioner
spec:
serviceAccount: rook-nfs-provisioner
containers:
- name: rook-nfs-provisioner
image: rook/nfs:master
imagePullPolicy: IfNotPresent
args: ["nfs", "provisioner","--provisioner=rook.io/nfs-provisioner"]
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
-19
View File
@@ -1,19 +0,0 @@
apiVersion: v1
kind: PersistentVolume
metadata:
name: rook-nfs-pv
namespace: rook-nfs
annotations:
volume.beta.kubernetes.io/mount-options: "vers=4.1"
spec:
storageClassName: rook-nfs-sc
capacity:
storage: 1Mi
accessModes:
- ReadWriteMany
nfs:
# FIXME: replace <Cluster IP> with the current IP of the NFS server pod.
# Run the below command to get the current IP address:
# kubectl -n rook-nfs get service -l app=rook-nfs -o jsonpath='{.items[0].spec.clusterIP}'
server: <Cluster IP>
path: "/<Claim name>"
+2 -1
View File
@@ -1,9 +1,10 @@
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: rook-nfs-pv-claim
spec:
storageClassName: rook-nfs-sc
storageClassName: "rook-nfs-share1"
accessModes:
- ReadWriteMany
resources:
+13
View File
@@ -0,0 +1,13 @@
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
labels:
app: rook-nfs
name: rook-nfs-share1
parameters:
exportName: share1
nfsServerName: rook-nfs
nfsServerNamespace: rook-nfs
provisioner: rook.io/nfs-provisioner
reclaimPolicy: Delete
volumeBindingMode: Immediate
+2
View File
@@ -36,6 +36,8 @@ var (
func init() {
Cmd.AddCommand(operatorCmd)
Cmd.AddCommand(provisonerCmd)
Cmd.AddCommand(serverCmd)
}
func createContext() *clusterd.Context {
+70
View File
@@ -0,0 +1,70 @@
/*
Copyright 2019 The Rook Authors. All rights reserved.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package nfs
import (
"errors"
"fmt"
"github.com/rook/rook/cmd/rook/rook"
"github.com/rook/rook/pkg/operator/nfs"
"github.com/rook/rook/pkg/util/flags"
"github.com/spf13/cobra"
"k8s.io/apimachinery/pkg/util/wait"
"sigs.k8s.io/sig-storage-lib-external-provisioner/controller"
)
var provisonerCmd = &cobra.Command{
Use: "provisioner",
Short: "Runs the NFS provisioner for provisioning volumes",
Long: "Runs the NFS provisioner for provisioning volumes from the rook provisioned nfs servers",
}
var (
provisioner *string
)
func init() {
flags.SetFlagsFromEnv(provisonerCmd.Flags(), rook.RookEnvVarPrefix)
flags.SetLoggingFlags(provisonerCmd.Flags())
provisioner = provisonerCmd.Flags().String("provisioner", "", "Name of the provisioner. The provisioner will only provision volumes for claims that request a StorageClass with a provisioner field set equal to this name.")
provisonerCmd.RunE = startProvisioner
}
func startProvisioner(cmd *cobra.Command, args []string) error {
rook.SetLogLevel()
rook.LogStartupInfo(serverCmd.Flags())
if len(*provisioner) == 0 {
return errors.New("--provisioner is a required parameter")
}
clientset, _, rookClientset, err := rook.GetClientset()
if err != nil {
rook.TerminateFatal(fmt.Errorf("failed to get k8s clients. %+v", err))
}
serverVersion, err := clientset.Discovery().ServerVersion()
if err != nil {
logger.Fatalf("Error getting server version: %v", err)
}
clientNFSProvisioner := nfs.NewNFSProvisioner(clientset, rookClientset)
pc := controller.NewProvisionController(clientset, *provisioner, clientNFSProvisioner, serverVersion.GitVersion)
pc.Run(wait.NeverStop)
return nil
}
+70
View File
@@ -0,0 +1,70 @@
/*
Copyright 2018 The Rook Authors. All rights reserved.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package nfs
import (
"errors"
"github.com/rook/rook/cmd/rook/rook"
"github.com/rook/rook/pkg/operator/nfs"
"github.com/rook/rook/pkg/util/flags"
"github.com/spf13/cobra"
)
var serverCmd = &cobra.Command{
Use: "server",
Short: "Runs the NFS server to deploy and manage NFS server in kubernetes clusters",
Long: `Runs the NFS operator to deploy and manage NFS server in kubernetes clusters.
https://github.com/rook/rook`,
}
var (
ganeshaConfigPath *string
)
func init() {
flags.SetFlagsFromEnv(serverCmd.Flags(), rook.RookEnvVarPrefix)
flags.SetLoggingFlags(serverCmd.Flags())
ganeshaConfigPath = serverCmd.Flags().String("ganeshaConfigPath", "", "ConfigPath of nfs ganesha")
serverCmd.RunE = startServer
}
func startServer(cmd *cobra.Command, args []string) error {
rook.SetLogLevel()
rook.LogStartupInfo(serverCmd.Flags())
if len(*ganeshaConfigPath) == 0 {
return errors.New("--ganeshaConfigPath is a required parameter")
}
logger.Infof("Setting up NFS server!")
err := nfs.Setup(*ganeshaConfigPath)
if err != nil {
logger.Fatalf("Error setting up NFS server: %v", err)
}
logger.Infof("starting NFS server")
// This blocks until server exits (presumably due to an error)
err = nfs.Run(*ganeshaConfigPath)
if err != nil {
logger.Errorf("NFS server Exited Unexpectedly with err: %v", err)
}
return nil
}
+15 -25
View File
@@ -12,39 +12,29 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#Portions of this file came from https://github.com/kubernetes-incubator/external-storage/blob/master/nfs/deploy/docker/Dockerfile, which uses the same license.
#Portions of this file came from https://github.com/mitcdh/docker-nfs-ganesha/blob/master/Dockerfile, which uses the same license.
FROM NFS_BASEIMAGE
# Build ganesha from source, installing deps and removing them in one line.
# Why?
# 1. Root_Id_Squash, only present in >= 2.4.0.3 which is not yet packaged
# 2. Set NFS_V4_RECOV_ROOT to /export
# 3. Use device major/minor as fsid major/minor to work on OverlayFS
RUN dnf install -y tar gcc cmake autoconf libtool bison flex make gcc-c++ krb5-devel dbus-devel jemalloc-devel libnfsidmap-devel patch libnsl2-devel && dnf clean all \
&& curl -L https://github.com/nfs-ganesha/nfs-ganesha/archive/V2.4.1.tar.gz | tar zx \
&& curl -L https://github.com/nfs-ganesha/nfs-ganesha/commit/c583534b166be198755d905c82a7687d83b458d8.patch -o /nfs-ganesha.patch \
&& curl -L https://github.com/nfs-ganesha/ntirpc/archive/v1.4.4.tar.gz | tar zx \
&& rm -r nfs-ganesha-2.4.1/src/libntirpc \
&& mv ntirpc-1.4.4 nfs-ganesha-2.4.1/src/libntirpc \
&& cd nfs-ganesha-2.4.1 \
&& cmake -DCMAKE_BUILD_TYPE=Release -DBUILD_CONFIG=vfs_only src/ \
&& patch -p1 < /nfs-ganesha.patch \
&& make \
&& make install \
&& cp src/scripts/ganeshactl/org.ganesha.nfsd.conf /etc/dbus-1/system.d/ \
&& cd .. \
&& rm -rf nfs-ganesha-2.4.1 /nfs-ganesha.patch \
&& dnf remove -y tar gcc cmake autoconf libtool bison flex make gcc-c++ krb5-devel dbus-devel jemalloc-devel libnfsidmap-devel patch && dnf clean all
RUN dnf install -y dbus-x11 rpcbind hostname nfs-utils xfsprogs jemalloc libnfsidmap && dnf clean all
RUN mkdir -p /var/run/dbus
RUN mkdir -p /export
COPY start.sh /
RUN DEBIAN_FRONTEND=noninteractive \
&& apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 3FE869A9 \
&& echo "deb http://ppa.launchpad.net/gluster/nfs-ganesha-2.5/ubuntu xenial main" > /etc/apt/sources.list.d/nfs-ganesha-2.5.list \
&& echo "deb http://ppa.launchpad.net/gluster/libntirpc-1.5/ubuntu xenial main" > /etc/apt/sources.list.d/libntirpc-1.5.list \
&& echo "deb http://ppa.launchpad.net/gluster/glusterfs-3.13/ubuntu xenial main" > /etc/apt/sources.list.d/glusterfs-3.13.list \
&& apt-get update \
&& apt-get install -y netbase nfs-common dbus nfs-ganesha nfs-ganesha-vfs glusterfs-common \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* \
&& mkdir -p /run/rpcbind /export /var/run/dbus \
&& touch /run/rpcbind/rpcbind.xdr /run/rpcbind/portmap.xdr \
&& chmod 755 /run/rpcbind/* \
&& chown messagebus:messagebus /var/run/dbus
EXPOSE 2049 38465-38467 662 111/udp 111
COPY rook /usr/local/bin/
+1 -2
View File
@@ -19,7 +19,7 @@ include ../image.mk
NFS_IMAGE = $(BUILD_REGISTRY)/nfs-$(GOARCH)
NFS_BASE ?= fedora:28
NFS_BASE ?= ubuntu:xenial
ifeq ($(GOARCH),amd64)
NFS_BASEIMAGE = $(NFS_BASE)
@@ -41,7 +41,6 @@ IMAGE_FILENAME := $(IMAGE_OUTPUT_DIR)/nfs.tar.gz
do.build:
@echo === docker build $(NFS_IMAGE)
@cp Dockerfile $(TEMP)
@cp start.sh $(TEMP)
@cp $(OUTPUT_DIR)/bin/linux_$(GOARCH)/rook $(TEMP)
@cd $(TEMP) && $(SED_CMD) 's|NFS_BASEIMAGE|$(NFS_BASEIMAGE)|g' Dockerfile
@$(DOCKERCMD) build $(BUILD_ARGS) \
-57
View File
@@ -1,57 +0,0 @@
#!/bin/bash
# Copyright 2018 The Rook Authors. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#Portions of this file came from https://github.com/mitcdh/docker-nfs-ganesha/blob/master/start.sh
set -e
# Options for starting Ganesha
: ${GANESHA_LOGFILE:="/dev/stdout"}
: ${GANESHA_CONFIGFILE:="/nfs-ganesha/config/nfs-ganesha-config"}
: ${GANESHA_OPTIONS:="-N NIV_DEBUG"} # NIV_DEBUG
: ${GANESHA_EPOCH:=""}
function init_rpc {
echo "Starting rpcbind"
rpcbind || return 0
rpc.statd -L || return 0
rpc.idmapd || return 0
sleep 1
}
function init_dbus {
echo "Starting dbus"
rm -f /var/run/dbus/system_bus_socket
rm -f /var/run/dbus/pid
dbus-uuidgen --ensure
dbus-daemon --system --fork
sleep 1
}
function startup_script {
if [ -f "${STARTUP_SCRIPT}" ]; then
/bin/sh ${STARTUP_SCRIPT}
fi
}
startup_script
init_rpc
init_dbus
echo "Starting Ganesha NFS"
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib
exec /usr/bin/ganesha.nfsd -F -L ${GANESHA_LOGFILE} -f ${GANESHA_CONFIGFILE} ${GANESHA_OPTIONS}
+2 -2
View File
@@ -17,7 +17,7 @@ package v1alpha1
import (
rookalpha "github.com/rook/rook/pkg/apis/rook.io/v1alpha2"
"k8s.io/api/core/v1"
v1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
@@ -45,7 +45,7 @@ type NFSServerList struct {
Items []NFSServer `json:"items"`
}
// NFSSpec represents the spec of NFS daemon
// NFSServerSpec represents the spec of NFS daemon
type NFSServerSpec struct {
// The annotations-related configuration to add/set on each Pod related object.
Annotations rookalpha.Annotations `json:"annotations,omitempty"`
+49 -50
View File
@@ -28,7 +28,7 @@ import (
"github.com/rook/rook/pkg/clusterd"
"github.com/rook/rook/pkg/operator/k8sutil"
appsv1 "k8s.io/api/apps/v1"
"k8s.io/api/core/v1"
v1 "k8s.io/api/core/v1"
apiextensionsv1beta1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1beta1"
"k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
@@ -40,9 +40,7 @@ import (
const (
customResourceName = "nfsserver"
customResourceNamePlural = "nfsservers"
appName = "rook-nfs"
nfsConfigMapName = "nfs-ganesha-config"
nfsConfigMapPath = "/nfs-ganesha/config"
NFSConfigMapPath = "/nfs-ganesha/config"
nfsPort = 2049
rpcPort = 111
noneMode = "none"
@@ -99,7 +97,7 @@ type nfsServer struct {
func newNfsServer(c *nfsv1alpha1.NFSServer, context *clusterd.Context) *nfsServer {
return &nfsServer{
name: appName,
name: c.Name,
context: context,
namespace: c.Namespace,
spec: c.Spec,
@@ -134,9 +132,9 @@ func getServerConfig(exports []nfsv1alpha1.ExportsSpec) map[string]map[string]st
return claimConfigOpt
}
func createAppLabels() map[string]string {
func createAppLabels(nfsServer *nfsServer) map[string]string {
return map[string]string{
k8sutil.AppAttr: appName,
k8sutil.AppAttr: nfsServer.name,
}
}
@@ -155,32 +153,33 @@ func createServicePorts() []v1.ServicePort {
}
}
func (c *Controller) createNFSService(nfsServer *nfsServer) error {
func (c *Controller) createNFSService(nfsServer *nfsServer) (*v1.Service, error) {
// This service is meant to be used by clients to access NFS.
nfsService := &v1.Service{
ObjectMeta: metav1.ObjectMeta{
Name: nfsServer.name,
Namespace: nfsServer.namespace,
OwnerReferences: []metav1.OwnerReference{nfsServer.ownerRef},
Labels: createAppLabels(),
Labels: createAppLabels(nfsServer),
},
Spec: v1.ServiceSpec{
Selector: createAppLabels(),
Selector: createAppLabels(nfsServer),
Type: v1.ServiceTypeClusterIP,
Ports: createServicePorts(),
},
}
if _, err := c.context.Clientset.CoreV1().Services(nfsServer.namespace).Create(nfsService); err != nil {
svc, err := c.context.Clientset.CoreV1().Services(nfsServer.namespace).Create(nfsService)
if err != nil {
if !errors.IsAlreadyExists(err) {
return err
return nil, err
}
logger.Infof("nfs service %s already exists in namespace %s", nfsService.Name, nfsService.Namespace)
} else {
logger.Infof("nfs service %s started in namespace %s", nfsService.Name, nfsService.Namespace)
}
return nil
return svc, nil
}
func createCephNFSExport(id int, path string, access string, squash string) string {
@@ -241,13 +240,13 @@ func (c *Controller) createNFSConfigMap(nfsServer *nfsServer) error {
configMap := &v1.ConfigMap{
ObjectMeta: metav1.ObjectMeta{
Name: nfsConfigMapName,
Name: nfsServer.name,
Namespace: nfsServer.namespace,
OwnerReferences: []metav1.OwnerReference{nfsServer.ownerRef},
Labels: createAppLabels(),
Labels: createAppLabels(nfsServer),
},
Data: map[string]string{
nfsConfigMapName: nfsGaneshaConfig,
nfsServer.name: nfsGaneshaConfig,
},
}
_, err := c.context.Clientset.CoreV1().ConfigMaps(nfsServer.namespace).Create(configMap)
@@ -258,25 +257,23 @@ func (c *Controller) createNFSConfigMap(nfsServer *nfsServer) error {
return nil
}
func getPVCNameList(spec *nfsv1alpha1.NFSServerSpec) []string {
exports := spec.Exports
pvcNameList := make([]string, 0)
for _, export := range exports {
func getPVCMount(spec *nfsv1alpha1.NFSServerSpec) map[string]string {
pvcMounts := map[string]string{}
for _, export := range spec.Exports {
claimName := export.PersistentVolumeClaim.ClaimName
if claimName != "" {
pvcNameList = append(pvcNameList, claimName)
pvcMounts[export.Name] = export.PersistentVolumeClaim.ClaimName
}
}
return pvcNameList
return pvcMounts
}
func createPVCSpecList(spec *nfsv1alpha1.NFSServerSpec) []v1.Volume {
func createPVCSpecList(nfsServer *nfsServer) []v1.Volume {
pvcSpecList := make([]v1.Volume, 0)
pvcNameList := getPVCNameList(spec)
for _, claimName := range pvcNameList {
pvcNameList := getPVCMount(&nfsServer.spec)
for shareName, claimName := range pvcNameList {
pvcSpecList = append(pvcSpecList, v1.Volume{
Name: claimName,
Name: shareName,
VolumeSource: v1.VolumeSource{
PersistentVolumeClaim: &v1.PersistentVolumeClaimVolumeSource{
ClaimName: claimName,
@@ -288,14 +285,14 @@ func createPVCSpecList(spec *nfsv1alpha1.NFSServerSpec) []v1.Volume {
configMapSrc := &v1.ConfigMapVolumeSource{
Items: []v1.KeyToPath{
{
Key: nfsConfigMapName,
Path: nfsConfigMapName,
Key: nfsServer.name,
Path: nfsServer.name,
},
},
}
configMapSrc.Name = nfsConfigMapName
configMapSrc.Name = nfsServer.name
configMapVol := v1.Volume{
Name: nfsConfigMapName,
Name: nfsServer.name,
VolumeSource: v1.VolumeSource{
ConfigMap: configMapSrc,
},
@@ -305,38 +302,39 @@ func createPVCSpecList(spec *nfsv1alpha1.NFSServerSpec) []v1.Volume {
return pvcSpecList
}
func createVolumeMountList(spec *nfsv1alpha1.NFSServerSpec) []v1.VolumeMount {
func createVolumeMountList(nfsServer *nfsServer) []v1.VolumeMount {
volumeMountList := make([]v1.VolumeMount, 0)
pvcNameList := getPVCNameList(spec)
for _, claimName := range pvcNameList {
pvcMount := getPVCMount(&nfsServer.spec)
for shareName, claimName := range pvcMount {
volumeMountList = append(volumeMountList, v1.VolumeMount{
Name: claimName,
Name: shareName,
MountPath: "/" + claimName,
})
}
configMapVolMount := v1.VolumeMount{
Name: nfsConfigMapName,
MountPath: nfsConfigMapPath,
Name: nfsServer.name,
MountPath: NFSConfigMapPath,
}
volumeMountList = append(volumeMountList, configMapVolMount)
return volumeMountList
}
func (c *Controller) createNfsPodSpec(nfsServer *nfsServer) v1.PodTemplateSpec {
func (c *Controller) createNfsPodSpec(nfsServer *nfsServer, service *v1.Service) v1.PodTemplateSpec {
nfsPodSpec := v1.PodTemplateSpec{
ObjectMeta: metav1.ObjectMeta{
Name: nfsServer.name,
Namespace: nfsServer.namespace,
Labels: createAppLabels(),
Labels: createAppLabels(nfsServer),
},
Spec: v1.PodSpec{
Containers: []v1.Container{
{
Name: nfsServer.name,
Image: c.containerImage,
Command: []string{"/start.sh"},
ImagePullPolicy: "IfNotPresent",
Name: nfsServer.name,
Image: c.containerImage,
Args: []string{"nfs", "server", "--ganeshaConfigPath=" + NFSConfigMapPath + "/" + nfsServer.name},
Ports: []v1.ContainerPort{
{
Name: "nfs-port",
@@ -347,7 +345,7 @@ func (c *Controller) createNfsPodSpec(nfsServer *nfsServer) v1.PodTemplateSpec {
ContainerPort: int32(rpcPort),
},
},
VolumeMounts: createVolumeMountList(&nfsServer.spec),
VolumeMounts: createVolumeMountList(nfsServer),
SecurityContext: &v1.SecurityContext{
Capabilities: &v1.Capabilities{
Add: []v1.Capability{
@@ -358,29 +356,29 @@ func (c *Controller) createNfsPodSpec(nfsServer *nfsServer) v1.PodTemplateSpec {
},
},
},
Volumes: createPVCSpecList(&nfsServer.spec),
Volumes: createPVCSpecList(nfsServer),
},
}
return nfsPodSpec
}
func (c *Controller) createNfsStatefulSet(nfsServer *nfsServer, replicas int32) error {
func (c *Controller) createNfsStatefulSet(nfsServer *nfsServer, replicas int32, service *v1.Service) error {
appsClient := c.context.Clientset.AppsV1()
nfsPodSpec := c.createNfsPodSpec(nfsServer)
nfsPodSpec := c.createNfsPodSpec(nfsServer, service)
statefulSet := appsv1.StatefulSet{
ObjectMeta: metav1.ObjectMeta{
Name: nfsServer.name,
Namespace: nfsServer.namespace,
Labels: createAppLabels(),
Labels: createAppLabels(nfsServer),
OwnerReferences: []metav1.OwnerReference{nfsServer.ownerRef},
},
Spec: appsv1.StatefulSetSpec{
Replicas: &replicas,
Selector: &metav1.LabelSelector{
MatchLabels: createAppLabels(),
MatchLabels: createAppLabels(nfsServer),
},
Template: nfsPodSpec,
ServiceName: nfsServer.name,
@@ -415,7 +413,8 @@ func (c *Controller) onAdd(obj interface{}) {
}
logger.Infof("creating nfs server service in namespace %s", nfsServer.namespace)
if err := c.createNFSService(nfsServer); err != nil {
svc, err := c.createNFSService(nfsServer)
if err != nil {
logger.Errorf("Unable to create NFS service %+v", err)
}
@@ -425,7 +424,7 @@ func (c *Controller) onAdd(obj interface{}) {
}
logger.Infof("creating nfs server stateful set in namespace %s", nfsServer.namespace)
if err := c.createNfsStatefulSet(nfsServer, int32(nfsServer.spec.Replicas)); err != nil {
if err := c.createNfsStatefulSet(nfsServer, int32(nfsServer.spec.Replicas), svc); err != nil {
logger.Errorf("Unable to create NFS stateful set %+v", err)
}
}
+8 -6
View File
@@ -25,11 +25,13 @@ import (
"github.com/rook/rook/pkg/operator/k8sutil"
testop "github.com/rook/rook/pkg/operator/test"
"github.com/stretchr/testify/assert"
"k8s.io/api/core/v1"
v1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/kubernetes/fake"
)
const appName = "nfs-server-X"
func TestValidateNFSServerSpec(t *testing.T) {
// first, test that a good NFSServerSpec is good
@@ -90,7 +92,7 @@ func TestOnAdd(t *testing.T) {
namespace := "rook-nfs-test"
nfsserver := &nfsv1alpha1.NFSServer{
ObjectMeta: metav1.ObjectMeta{
Name: "nfs-server-X",
Name: appName,
Namespace: namespace,
},
Spec: nfsv1alpha1.NFSServerSpec{
@@ -122,13 +124,13 @@ func TestOnAdd(t *testing.T) {
controller.onAdd(nfsserver)
// verify client service
clientService, err := clientset.CoreV1().Services(namespace).Get(appName, metav1.GetOptions{})
clientService, err := clientset.CoreV1().Services(namespace).Get(nfsserver.GetName(), metav1.GetOptions{})
assert.Nil(t, err)
assert.NotNil(t, clientService)
assert.Equal(t, v1.ServiceTypeClusterIP, clientService.Spec.Type)
// verify nfs-ganesha config in the configmap
configMap, err := clientset.CoreV1().ConfigMaps(namespace).Get(nfsConfigMapName, metav1.GetOptions{})
configMap, err := clientset.CoreV1().ConfigMaps(namespace).Get(nfsserver.GetName(), metav1.GetOptions{})
assert.Nil(t, err)
assert.NotNil(t, configMap)
nfsGaneshaConfig := `
@@ -149,7 +151,7 @@ NFS_Core_Param
{
fsid_device = true;
}`
assert.Equal(t, nfsGaneshaConfig, configMap.Data[nfsConfigMapName])
assert.Equal(t, nfsGaneshaConfig, configMap.Data[nfsserver.GetName()])
// verify stateful set
ss, err := clientset.AppsV1().StatefulSets(namespace).Get(appName, metav1.GetOptions{})
@@ -161,7 +163,7 @@ NFS_Core_Param
container := ss.Spec.Template.Spec.Containers[0]
assert.Equal(t, 2, len(container.VolumeMounts))
expectedVolumeMounts := []v1.VolumeMount{{Name: "test-claim", MountPath: "/test-claim"}, {Name: "nfs-ganesha-config", MountPath: "/nfs-ganesha/config"}}
expectedVolumeMounts := []v1.VolumeMount{{Name: "export-test", MountPath: "/test-claim"}, {Name: nfsserver.GetName(), MountPath: "/nfs-ganesha/config"}}
assert.Equal(t, expectedVolumeMounts, container.VolumeMounts)
}
+152
View File
@@ -0,0 +1,152 @@
/*
Copyright 2019 The Rook Authors. All rights reserved.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package nfs
import (
"fmt"
"github.com/pkg/errors"
rookclient "github.com/rook/rook/pkg/client/clientset/versioned"
v1 "k8s.io/api/core/v1"
storage "k8s.io/api/storage/v1"
"k8s.io/client-go/kubernetes"
"k8s.io/kubernetes/pkg/apis/core/v1/helper"
"sigs.k8s.io/sig-storage-lib-external-provisioner/controller"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
type nfsProvisioner struct {
client kubernetes.Interface
rookClient rookclient.Interface
}
var _ controller.Provisioner = &nfsProvisioner{}
// NewNFSProvisioner returns an instance of nfsProvisioner
func NewNFSProvisioner(clientset kubernetes.Interface, rookClientset rookclient.Interface) *nfsProvisioner {
return &nfsProvisioner{clientset, rookClientset}
}
func (p *nfsProvisioner) Provision(options controller.VolumeOptions) (*v1.PersistentVolume, error) {
logger.Infof("nfs provisioner: VolumeOptions %v", options)
// Get the storage class for this volume.
storageClass, err := p.getClassForPVC(options.PVC)
if err != nil {
return nil, err
}
if options.PVC.Spec.Selector != nil {
return nil, fmt.Errorf("claim Selector is not supported")
}
nfsServerName, present := storageClass.Parameters["nfsServerName"]
if !present {
return nil, errors.Errorf("NFS share Path not found in the storageclass: %v", storageClass.GetName())
}
nfsNamespace, present := storageClass.Parameters["nfsServerNamespace"]
if !present {
return nil, errors.Errorf("NFS share Path not found in the storageclass: %v", storageClass.GetName())
}
exportName, present := storageClass.Parameters["exportName"]
if !present {
return nil, errors.Errorf("NFS share Path not found in the storageclass: %v", storageClass.GetName())
}
nfsVolumeSource, err := p.getNFSVolumeSource(nfsServerName, nfsNamespace, exportName)
if err != nil {
return nil, err
}
pv := &v1.PersistentVolume{
ObjectMeta: metav1.ObjectMeta{
Name: options.PVName,
},
Spec: v1.PersistentVolumeSpec{
PersistentVolumeReclaimPolicy: options.PersistentVolumeReclaimPolicy,
AccessModes: options.PVC.Spec.AccessModes,
MountOptions: options.MountOptions,
Capacity: v1.ResourceList{
v1.ResourceName(v1.ResourceStorage): options.PVC.Spec.Resources.Requests[v1.ResourceName(v1.ResourceStorage)],
},
PersistentVolumeSource: v1.PersistentVolumeSource{
NFS: nfsVolumeSource,
},
},
}
return pv, nil
}
func (p *nfsProvisioner) Delete(volume *v1.PersistentVolume) error {
return nil
}
// getClassForPV returns StorageClass
func (p *nfsProvisioner) getClassForPV(pv *v1.PersistentVolume) (*storage.StorageClass, error) {
if p.client == nil {
return nil, fmt.Errorf("Cannot get kube client")
}
className := helper.GetPersistentVolumeClass(pv)
if className == "" {
return nil, fmt.Errorf("Volume has no storage class")
}
class, err := p.client.StorageV1().StorageClasses().Get(className, metav1.GetOptions{})
if err != nil {
return nil, err
}
return class, nil
}
// getClassForPVC returns StorageClass
func (p *nfsProvisioner) getClassForPVC(pvc *v1.PersistentVolumeClaim) (*storage.StorageClass, error) {
if p.client == nil {
return nil, fmt.Errorf("Cannot get kube client")
}
className := helper.GetPersistentVolumeClaimClass(pvc)
if className == "" {
return nil, fmt.Errorf("Volume has no storage class")
}
class, err := p.client.StorageV1().StorageClasses().Get(className, metav1.GetOptions{})
if err != nil {
return nil, err
}
return class, nil
}
// getNFSVolumeSource returns the nfs source configuration for the pv
func (p *nfsProvisioner) getNFSVolumeSource(nfsServerName, namespace, exportName string) (*v1.NFSVolumeSource, error) {
nfsServer, err := p.rookClient.NfsV1alpha1().NFSServers(namespace).Get(nfsServerName, metav1.GetOptions{})
if err != nil {
return nil, err
}
nfsService, err := p.client.CoreV1().Services(namespace).Get(nfsServerName, metav1.GetOptions{})
if err != nil {
return nil, err
}
for _, export := range nfsServer.Spec.Exports {
if export.Name == exportName {
return &v1.NFSVolumeSource{
ReadOnly: false,
Server: nfsService.Spec.ClusterIP,
Path: "/" + export.PersistentVolumeClaim.ClaimName,
}, nil
}
}
return nil, errors.Errorf("exportName not Found")
}
+101
View File
@@ -0,0 +1,101 @@
/*
Copyright 2019 The Rook Authors. All rights reserved.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
// Portion of this file is comming from https://github.com/kubernetes-incubator/external-storage/blob/master/nfs/pkg/server/server.go
package nfs
import (
"fmt"
"os/exec"
"syscall"
)
const (
ganeshaLog = "/dev/stdout"
ganeshaOptions = "NIV_DEBUG"
)
// Setup sets up various prerequisites and settings for the server. If an error
// is encountered at any point it returns it instantly
func Setup(ganeshaConfig string) error {
// Start rpcbind if it is not started yet
cmd := exec.Command("rpcinfo", "127.0.0.1")
if err := cmd.Run(); err != nil {
cmd = exec.Command("rpcbind", "-w")
if out, err := cmd.CombinedOutput(); err != nil {
return fmt.Errorf("Starting rpcbind failed with error: %v, output: %s", err, out)
}
}
cmd = exec.Command("rpc.statd")
if out, err := cmd.CombinedOutput(); err != nil {
return fmt.Errorf("rpc.statd failed with error: %v, output: %s", err, out)
}
// Start dbus, needed for ganesha dynamic exports
cmd = exec.Command("dbus-daemon", "--system")
if out, err := cmd.CombinedOutput(); err != nil {
return fmt.Errorf("dbus-daemon failed with error: %v, output: %s", err, out)
}
err := setRlimitNOFILE()
if err != nil {
logger.Warningf("Error setting RLIMIT_NOFILE, there may be 'Too many open files' errors later: %v", err)
}
return nil
}
// Run : run the NFS server in the foreground until it exits
// Ideally, it should never exit when run in foreground mode
// We force foreground to allow the provisioner process to restart
// the server if it crashes - daemonization prevents us from using Wait()
// for this purpose
func Run(ganeshaConfig string) error {
// Start ganesha.nfsd
logger.Infof("Running NFS server!")
cmd := exec.Command("ganesha.nfsd", "-F", "-L", ganeshaLog, "-f", ganeshaConfig, "-N", ganeshaOptions)
if out, err := cmd.CombinedOutput(); err != nil {
return fmt.Errorf("ganesha.nfsd failed with error: %v, output: %s", err, out)
}
return nil
}
func setRlimitNOFILE() error {
var rlimit syscall.Rlimit
err := syscall.Getrlimit(syscall.RLIMIT_NOFILE, &rlimit)
if err != nil {
return fmt.Errorf("error getting RLIMIT_NOFILE: %v", err)
}
logger.Infof("starting RLIMIT_NOFILE rlimit.Cur %d, rlimit.Max %d", rlimit.Cur, rlimit.Max)
rlimit.Max = 1024 * 1024
rlimit.Cur = 1024 * 1024
err = syscall.Setrlimit(syscall.RLIMIT_NOFILE, &rlimit)
if err != nil {
return err
}
err = syscall.Getrlimit(syscall.RLIMIT_NOFILE, &rlimit)
if err != nil {
return fmt.Errorf("error getting RLIMIT_NOFILE: %v", err)
}
logger.Infof("ending RLIMIT_NOFILE rlimit.Cur %d, rlimit.Max %d", rlimit.Cur, rlimit.Max)
return nil
}
// Stop stops the NFS server.
func Stop() {
// /bin/dbus-send --system --dest=org.ganesha.nfsd --type=method_call /org/ganesha/nfsd/admin org.ganesha.nfsd.admin.shutdown
}
+5 -25
View File
@@ -113,12 +113,12 @@ func (h *NFSInstaller) CreateNFSServer(namespace string, count int, storageClass
return fmt.Errorf("Failed to create nfs server: %+v ", err)
}
if err := h.k8shelper.WaitForPodCount("app=rook-nfs", namespace, 1); err != nil {
if err := h.k8shelper.WaitForPodCount("app="+namespace, namespace, 1); err != nil {
logger.Errorf("nfs server pods in namespace %s not found", namespace)
return err
}
err := h.k8shelper.WaitForLabeledPodsToRun("app=rook-nfs", namespace)
err := h.k8shelper.WaitForLabeledPodsToRun("app="+namespace, namespace)
if err != nil {
logger.Errorf("nfs server pods in namespace %s are not running", namespace)
return err
@@ -132,17 +132,7 @@ func (h *NFSInstaller) CreateNFSServer(namespace string, count int, storageClass
func (h *NFSInstaller) CreateNFSServerVolume(namespace string) error {
logger.Info("creating volume from nfs server in namespace %s", namespace)
clusterIP, err := h.GetNFSServerClusterIP(namespace)
if err != nil {
return err
}
nfsServerPV := h.manifests.GetNFSServerPV(namespace, clusterIP)
nfsServerPVC := h.manifests.GetNFSServerPVC()
logger.Info("creating nfs server pv")
if _, err := h.k8shelper.KubectlWithStdin(nfsServerPV, createFromStdinArgs...); err != nil {
return err
}
nfsServerPVC := h.manifests.GetNFSServerPVC(namespace)
logger.Info("creating nfs server pvc")
if _, err := h.k8shelper.KubectlWithStdin(nfsServerPVC, createFromStdinArgs...); err != nil {
@@ -193,6 +183,8 @@ func (h *NFSInstaller) UninstallNFSServer(systemNamespace, namespace string) {
checkError(h.T(), err, "cannot uninstall hostpath provisioner")
h.k8shelper.Clientset.RbacV1beta1().ClusterRoleBindings().Delete("anon-user-access", nil)
h.k8shelper.Clientset.RbacV1beta1().ClusterRoleBindings().Delete("run-nfs-client-provisioner", nil)
h.k8shelper.Clientset.RbacV1beta1().ClusterRoles().Delete("nfs-client-provisioner-runner", nil)
logger.Infof("done removing the operator from namespace %s", systemNamespace)
}
@@ -202,15 +194,3 @@ func (h *NFSInstaller) GatherAllNFSServerLogs(systemNamespace, namespace, testNa
h.k8shelper.GetLogs("rook-nfs-operator", Env.HostType, systemNamespace, testName)
h.k8shelper.GetLogs("rook-nfs", Env.HostType, namespace, testName)
}
// GetNFSServerClusterIP gets the nfs server cluster ip on which it serves
func (h *NFSInstaller) GetNFSServerClusterIP(namespace string) (string, error) {
clusterIP := ""
service, err := h.k8shelper.GetService("rook-nfs", namespace)
if err != nil {
logger.Errorf("nfs server service in namespace %s is not active", namespace)
return clusterIP, err
}
clusterIP = service.Spec.ClusterIP
return clusterIP, nil
}
+130 -9
View File
@@ -43,10 +43,10 @@ spec:
// GetNFSServerOperator returns the NFSServer operator definition
func (i *NFSManifests) GetNFSServerOperator(namespace string) string {
return `kind: Namespace
apiVersion: v1
return `apiVersion: v1
kind: Namespace
metadata:
name: ` + namespace + `
name: ` + namespace + `
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
@@ -54,12 +54,13 @@ metadata:
name: rook-nfs-operator
rules:
- apiGroups:
- ""
- "*"
resources:
- namespaces
- configmaps
- pods
- services
- storageclasses
verbs:
- get
- watch
@@ -124,6 +125,7 @@ spec:
serviceAccountName: rook-nfs-operator
containers:
- name: rook-nfs-operator
imagePullPolicy: IfNotPresent
image: rook/nfs:master
args: ["nfs", "operator"]
env:
@@ -135,6 +137,93 @@ spec:
valueFrom:
fieldRef:
fieldPath: metadata.namespace
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: rook-nfs-provisioner-runner
rules:
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "create", "delete"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["create", "update", "patch"]
- apiGroups: [""]
resources: ["services", "endpoints"]
verbs: ["get"]
- apiGroups: ["extensions"]
resources: ["podsecuritypolicies"]
resourceNames: ["nfs-provisioner"]
verbs: ["use"]
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
- apiGroups:
- nfs.rook.io
resources:
- "*"
verbs:
- "*"
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: run-nfs-provisioner
subjects:
- kind: ServiceAccount
name: rook-nfs-provisioner
# replace with namespace where provisioner is deployed
namespace: ` + namespace + `
roleRef:
kind: ClusterRole
name: rook-nfs-provisioner-runner
apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: rook-nfs-provisioner
namespace: ` + namespace + `
---
kind: Deployment
apiVersion: apps/v1
metadata:
name: rook-nfs-provisioner
namespace: ` + namespace + `
spec:
selector:
matchLabels:
app: rook-nfs-provisioner
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
app: rook-nfs-provisioner
spec:
serviceAccount: rook-nfs-provisioner
containers:
- name: rook-nfs-provisioner
image: rook/nfs:master
imagePullPolicy: IfNotPresent
args: ["nfs", "provisioner","--provisioner=rook.io/nfs-provisioner"]
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
`
}
@@ -177,13 +266,43 @@ spec:
}
// GetNFSServerPVC returns NFSServer PVC definition
func (i *NFSManifests) GetNFSServerPVC() string {
return `apiVersion: v1
func (i *NFSManifests) GetNFSServerPVC(namespace string) string {
return `
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
labels:
app: rook-nfs
name: nfs-ns-nfs-share
parameters:
exportName: nfs-share
nfsServerName: ` + namespace + `
nfsServerNamespace: ` + namespace + `
provisioner: rook.io/nfs-provisioner
reclaimPolicy: Delete
volumeBindingMode: Immediate
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
labels:
app: rook-nfs
name: nfs-ns-nfs-share1
parameters:
exportName: nfs-share1
nfsServerName: ` + namespace + `
nfsServerNamespace: ` + namespace + `
provisioner: rook.io/nfs-provisioner
reclaimPolicy: Delete
volumeBindingMode: Immediate
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: nfs-pv-claim
spec:
storageClassName: nfs-sc
storageClassName: nfs-ns-nfs-share
accessModes:
- ReadWriteMany
resources:
@@ -195,7 +314,7 @@ kind: PersistentVolumeClaim
metadata:
name: nfs-pv-claim-bigger
spec:
storageClassName: nfs-sc
storageClassName: nfs-ns-nfs-share1
accessModes:
- ReadWriteMany
resources:
@@ -206,7 +325,8 @@ spec:
// GetNFSServer returns NFSServer CRD instance definition
func (i *NFSManifests) GetNFSServer(namespace string, count int, storageClassName string) string {
return `apiVersion: v1
return `
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: test-claim
@@ -238,6 +358,7 @@ metadata:
name: ` + namespace + `
namespace: ` + namespace + `
spec:
serviceAccountName: ` + namespace + `
replicas: ` + strconv.Itoa(count) + `
exports:
- name: nfs-share
+1 -1
View File
@@ -105,7 +105,7 @@ func (suite *NfsSuite) TestNfsServerInstallation() {
"1 rook-nfs-operator must be in Running state")
// verify nfs server instances are running OK
assert.True(suite.T(), suite.k8shelper.CheckPodCountAndState("rook-nfs", suite.namespace, suite.instanceCount, "Running"),
assert.True(suite.T(), suite.k8shelper.CheckPodCountAndState(suite.namespace, suite.namespace, suite.instanceCount, "Running"),
fmt.Sprintf("%d rook-nfs pods must be in Running state", suite.instanceCount))
// verify bigger export is running OK