diff --git a/Documentation/cluster-crd.md b/Documentation/cluster-crd.md index 5728f9c05..b85a70a7b 100644 --- a/Documentation/cluster-crd.md +++ b/Documentation/cluster-crd.md @@ -81,11 +81,11 @@ Below are the settings available, both at the cluster and individual node level, Placement configuration for the cluster services. It includes the following keys: `mgr`, `mon`, `osd` and `all`. Each service will have its placement configuration generated by merging the generic configuration under `all` with the most specific one (which will override any attributes). -A Placement configuration is specified (according to the kubernetes [PodSpec](https://kubernetes.io/docs/api-reference/v1.6/#podspec-v1-core)) as: -- `nodeAffinity`: kubernetes [NodeAffinity](https://kubernetes.io/docs/api-reference/v1.6/#nodeaffinity-v1-core) -- `podAffinity`: kubernetes [PodAffinity](https://kubernetes.io/docs/api-reference/v1.6/#podaffinity-v1-core) -- `podAntiAffinity`: kubernetes [PodAntiAffinity](https://kubernetes.io/docs/api-reference/v1.6/#podantiaffinity-v1-core) -- `tolerations`: list of kubernetes [Toleration](https://kubernetes.io/docs/api-reference/v1.6/#toleration-v1-core) +A Placement configuration is specified (according to the kubernetes PodSpec) as: +- `nodeAffinity`: kubernetes [NodeAffinity](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#node-affinity-beta-feature) +- `podAffinity`: kubernetes [PodAffinity](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#inter-pod-affinity-and-anti-affinity-beta-feature) +- `podAntiAffinity`: kubernetes [PodAntiAffinity](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#inter-pod-affinity-and-anti-affinity-beta-feature) +- `tolerations`: list of kubernetes [Toleration](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/) The `mon` pod does not allow `Pod` affinity or anti-affinity. This is because of the mons having built-in anti-affinity with each other through the operator. The operator chooses which nodes are to run a mon on. Each mon is then tied to a node with a node selector using a hostname. diff --git a/Documentation/common-issues.md b/Documentation/common-issues.md index c0c48f288..18240b706 100644 --- a/Documentation/common-issues.md +++ b/Documentation/common-issues.md @@ -88,7 +88,6 @@ There are many Ceph sub-commands to look at and manipulate Ceph objects, well be * Persistent Volume and/or Claim are failing to be created and bound: [Volume Creation](#volume-creation) * `rook-agent` pod is failing to mount and format the volume: [Rook Agent Mounting](#volume-mounting) * You are using Kubernetes 1.7.x or earlier and the Kubelet has not been restarted after `rook-agent` is in the `Running` status: [Restart Kubelet](#kubelet-restart) -* You are using Kubernetes 1.6.x and the attach-detach controller has not been disabled: [Disable attach-detach controller](./quickstart.md#disable-attacher-detacher-controller) ## Investigation Details If you see some of the symptoms above, it's because the requested Rook storage for your pod is not being created and mounted successfully. @@ -142,8 +141,8 @@ kubectl -n rook-system delete pod -l app=rook-operator ``` ### Kubelet Restart -#### **Kubernetes 1.7.x and earlier only** -If the `rook-agent` pods are all in the `Running` state then another thing to confirm is that **if you are running on Kubernetes 1.7.x or earlier**, the Kubelet must be restarted after the `rook-agent` pods are running. +#### **Kubernetes 1.7.x only** +If the `rook-agent` pods are all in the `Running` state then another thing to confirm is that **if you are running on Kubernetes 1.7.x**, the Kubelet must be restarted after the `rook-agent` pods are running. A symptom of this can be found in the Kubelet's log/journal, with the following error saying `no volume plugin matched`: ```console diff --git a/Documentation/helm-operator.md b/Documentation/helm-operator.md index 2281ab7de..07df1fb57 100644 --- a/Documentation/helm-operator.md +++ b/Documentation/helm-operator.md @@ -14,7 +14,7 @@ This chart bootstraps a [rook-operator](https://github.com/rook/rook) deployment ## Prerequisites -- Kubernetes 1.6+ +- Kubernetes 1.7+ ### RBAC diff --git a/Documentation/k8s-pre-reqs.md b/Documentation/k8s-pre-reqs.md index 2064539d4..2fcc6816f 100644 --- a/Documentation/k8s-pre-reqs.md +++ b/Documentation/k8s-pre-reqs.md @@ -9,10 +9,7 @@ Rook can be installed on any existing Kubernetes clusters as long as it meets th ## Minimum Version -Kubernetes v1.6 or higher is targeted by Rook (while Rook is in alpha it will track the latest release to use the latest features). - -Support is available for Kubernetes v1.5.2, although your mileage may vary. -You will need to use the yaml files from the [1.5 folder](/cluster/examples/kubernetes/1.5). +Kubernetes v1.7 or higher is supported by Rook. ## Privileges and RBAC diff --git a/Documentation/monitoring.md b/Documentation/monitoring.md index a7999ecf2..ad6663b07 100644 --- a/Documentation/monitoring.md +++ b/Documentation/monitoring.md @@ -10,8 +10,6 @@ If you do not have Prometheus running, follow the steps below to enable monitori contains a Prometheus instance, it will automatically discover Rooks scrape endpoint using the standard `prometheus.io/scrape` and `prometheus.io/port` annotations. -Note that these steps work only for Kubernetes 1.7.x or higher. For Kubernetes 1.6.x or older, refer to the Rook 0.5.0 [documentation](https://github.com/rook/rook/blob/release-0.5/Documentation/monitoring.md) and use these [manifest files](https://github.com/rook/rook/tree/release-0.5/cluster/examples/kubernetes/monitoring). - ## Prometheus Operator diff --git a/Documentation/object-store-crd.md b/Documentation/object-store-crd.md index 59db0c107..4b50814da 100644 --- a/Documentation/object-store-crd.md +++ b/Documentation/object-store-crd.md @@ -57,12 +57,6 @@ spec: ## Object Store Settings -### Kind -If you are using a version of Kubernetes **earlier than 1.7**, you will need to slightly modify the `kind` to be compatible with TPRs (deprecated in 1.7). Notice the different casing. -```yaml -kind: Objectstore -``` - ### Metadata - `name`: The name of the object store to create, which will be reflected in the pool and other resource names. diff --git a/Documentation/quickstart.md b/Documentation/quickstart.md index bd47cb35d..58356e91c 100644 --- a/Documentation/quickstart.md +++ b/Documentation/quickstart.md @@ -13,10 +13,7 @@ from other pods running in your cluster. ## Minimum Version -Kubernetes **v1.6** or higher is targeted by Rook (while Rook is in alpha it will track the latest release to use the latest features). - -Support is available for Kubernetes **v1.5.2**, although your mileage may vary. -You will need to use the yaml files from the [1.5 folder](/cluster/examples/kubernetes/1.5). +Kubernetes **v1.7** or higher is supported by Rook. ## Prerequisites @@ -51,15 +48,10 @@ You can also deploy the operator with the [Rook Helm Chart](helm-operator.md). --- ### **Restart Kubelet** -**(K8S 1.7.x and older only)** +**(K8S 1.7.x only)** For versions of Kubernetes prior to 1.8, the Kubelet process on all nodes will require a restart after the Rook operator and Rook agents have been deployed. As part of their initial setup, the Rook agents deploy and configure a Flexvolume plugin in order to integrate with Kubernetes' volume controller framework. In Kubernetes v1.8+, the [dynamic Flexvolume plugin discovery](https://github.com/kubernetes/community/blob/master/contributors/devel/flexvolume.md#dynamic-plugin-discovery) will find and initialize our plugin, but in older versions of Kubernetes a manual restart of the Kubelet will be required. -### **Disable Attacher-detacher controller** -**(K8S 1.6.x only)** - -For Kubernetes 1.6, it is also necessary to pass the `--enable-controller-attach-detach=false` flag to Kubelet when you restart it. This is a workaround for a [Kubernetes issue](https://github.com/kubernetes/kubernetes/issues/47109) that only affects 1.6. - --- ## Create a Rook Cluster diff --git a/Jenkinsfile b/Jenkinsfile index aa29cda6e..d312fad3e 100644 --- a/Jenkinsfile +++ b/Jenkinsfile @@ -67,8 +67,6 @@ pipeline { stash name: 'repo-amd64',includes: 'rook-amd64.tar,build/common.sh,_output/tests/linux_amd64/,_output/charts/,tests/scripts/' script{ def data = [ - "gce_1.6.1": "v1.6.1", - "gce_1.6.x": "v1.6.13", "aws_1.7.x": "v1.7.11", "aws_1.8.x": "v1.8.5", "gce_1.9.x": "v1.9.1" diff --git a/PendingReleaseNotes.md b/PendingReleaseNotes.md index 961d001e5..8a16dfc16 100644 --- a/PendingReleaseNotes.md +++ b/PendingReleaseNotes.md @@ -8,6 +8,7 @@ - Rook-Operator no longer creates the resources CRD's or TPR's at the runtime. Instead, those resources are provisioned during deployment via `helm` or `kubectl`. ## Breaking Changes +- Removed support for Kubernetes 1.6, including the legacy Third Party Resources (TPRs). ### Removal of the API service and rookctl tool The [REST API service](https://github.com/rook/rook/issues/1122) has been removed. All cluster configuration is now accomplished through the diff --git a/cluster/charts/rook/templates/clusterrole.yaml b/cluster/charts/rook/templates/clusterrole.yaml index dce7e6109..88027df19 100644 --- a/cluster/charts/rook/templates/clusterrole.yaml +++ b/cluster/charts/rook/templates/clusterrole.yaml @@ -42,18 +42,6 @@ rules: - create - update - delete -- apiGroups: - - extensions - resources: - - thirdpartyresources - resourceNames: - - cluster.rook.io - - pool.rook.io - - objectstore.rook.io - - filesystem.rook.io - - volumeattachment.rook.io - verbs: - - get - apiGroups: - apiextensions.k8s.io resources: diff --git a/cluster/charts/rook/templates/resources.yaml b/cluster/charts/rook/templates/resources.yaml index f7ca92f4b..041e2c250 100644 --- a/cluster/charts/rook/templates/resources.yaml +++ b/cluster/charts/rook/templates/resources.yaml @@ -1,4 +1,3 @@ -{{- if .Capabilities.APIVersions.Has "apiextensions.k8s.io/v1beta1" }} --- apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition @@ -70,46 +69,3 @@ spec: scope: Namespaced version: v1alpha1 --- -{{- else }} ---- -apiVersion: extensions/v1beta1 -description: ThirdPartyResource for cluster -kind: ThirdPartyResource -metadata: - name: cluster.rook.io -versions: -- name: v1alpha1 ---- -apiVersion: extensions/v1beta1 -description: ThirdPartyResource for filesystem -kind: ThirdPartyResource -metadata: - name: filesystem.rook.io -versions: -- name: v1alpha1 ---- -apiVersion: extensions/v1beta1 -description: ThirdPartyResource for objectstore -kind: ThirdPartyResource -metadata: - name: objectstore.rook.io -versions: -- name: v1alpha1 ---- -apiVersion: extensions/v1beta1 -description: ThirdPartyResource for pool -kind: ThirdPartyResource -metadata: - name: pool.rook.io -versions: -- name: v1alpha1 ---- -apiVersion: extensions/v1beta1 -description: ThirdPartyResource for volumeattachment -kind: ThirdPartyResource -metadata: - name: volumeattachment.rook.io -versions: -- name: v1alpha1 ---- -{{- end }} \ No newline at end of file diff --git a/cluster/charts/rook/templates/wait-on-resources-job.yaml b/cluster/charts/rook/templates/wait-on-resources-job.yaml index d8756bdcc..fd048461d 100644 --- a/cluster/charts/rook/templates/wait-on-resources-job.yaml +++ b/cluster/charts/rook/templates/wait-on-resources-job.yaml @@ -22,21 +22,12 @@ spec: command: - ./kubectl - get - {{- if .Capabilities.APIVersions.Has "apiextensions.k8s.io/v1beta1" }} - customresourcedefinitions - clusters.rook.io - pools.rook.io - objectstores.rook.io - filesystems.rook.io - volumeattachments.rook.io - {{- else }} - - thirdpartyresources - - cluster.rook.io - - pool.rook.io - - objectstore.rook.io - - filesystem.rook.io - - volumeattachment.rook.io - {{- end }} restartPolicy: OnFailure {{- if .Values.rbacEnable }} serviceAccountName: rook-operator diff --git a/cluster/examples/kubernetes/1.5/kube-registry.yaml b/cluster/examples/kubernetes/1.5/kube-registry.yaml deleted file mode 120000 index 920a8c92d..000000000 --- a/cluster/examples/kubernetes/1.5/kube-registry.yaml +++ /dev/null @@ -1 +0,0 @@ -../kube-registry.yaml \ No newline at end of file diff --git a/cluster/examples/kubernetes/1.5/mysql.yaml b/cluster/examples/kubernetes/1.5/mysql.yaml deleted file mode 100644 index d8a570243..000000000 --- a/cluster/examples/kubernetes/1.5/mysql.yaml +++ /dev/null @@ -1,60 +0,0 @@ -apiVersion: v1 -kind: Service -metadata: - name: wordpress-mysql - labels: - app: wordpress -spec: - ports: - - port: 3306 - selector: - app: wordpress - tier: mysql - clusterIP: None ---- -apiVersion: v1 -kind: PersistentVolumeClaim -metadata: - name: mysql-pv-claim - annotations: - volume.beta.kubernetes.io/storage-class: rook-block - labels: - app: wordpress -spec: - accessModes: - - ReadWriteOnce - resources: - requests: - storage: 20Gi ---- -apiVersion: extensions/v1beta1 -kind: Deployment -metadata: - name: wordpress-mysql - labels: - app: wordpress -spec: - strategy: - type: Recreate - template: - metadata: - labels: - app: wordpress - tier: mysql - spec: - containers: - - image: mysql:5.6 - name: mysql - env: - - name: MYSQL_ROOT_PASSWORD - value: changeme - ports: - - containerPort: 3306 - name: mysql - volumeMounts: - - name: mysql-persistent-storage - mountPath: /var/lib/mysql - volumes: - - name: mysql-persistent-storage - persistentVolumeClaim: - claimName: mysql-pv-claim diff --git a/cluster/examples/kubernetes/1.5/rgw-external.yaml b/cluster/examples/kubernetes/1.5/rgw-external.yaml deleted file mode 120000 index 515b44861..000000000 --- a/cluster/examples/kubernetes/1.5/rgw-external.yaml +++ /dev/null @@ -1 +0,0 @@ -../rgw-external.yaml \ No newline at end of file diff --git a/cluster/examples/kubernetes/1.5/rook-client.yaml b/cluster/examples/kubernetes/1.5/rook-client.yaml deleted file mode 120000 index 263b888aa..000000000 --- a/cluster/examples/kubernetes/1.5/rook-client.yaml +++ /dev/null @@ -1 +0,0 @@ -../rook-client.yaml \ No newline at end of file diff --git a/cluster/examples/kubernetes/1.5/rook-cluster.yaml b/cluster/examples/kubernetes/1.5/rook-cluster.yaml deleted file mode 120000 index bdc1f3c25..000000000 --- a/cluster/examples/kubernetes/1.5/rook-cluster.yaml +++ /dev/null @@ -1 +0,0 @@ -../rook-cluster.yaml \ No newline at end of file diff --git a/cluster/examples/kubernetes/1.5/rook-operator.yaml b/cluster/examples/kubernetes/1.5/rook-operator.yaml deleted file mode 100644 index 2fab07edd..000000000 --- a/cluster/examples/kubernetes/1.5/rook-operator.yaml +++ /dev/null @@ -1,37 +0,0 @@ -apiVersion: v1 -kind: Namespace -metadata: - name: rook-system ---- -apiVersion: extensions/v1beta1 -kind: Deployment -metadata: - name: rook-operator - namespace: rook-system -spec: - replicas: 1 - template: - metadata: - labels: - name: rook-operator - spec: - containers: - - name: rook-operator - image: rook/rook:master - args: ["operator"] - env: - # The interval to check if every mon is in the quorum. - - name: ROOK_MON_HEALTHCHECK_INTERVAL - value: "45s" - # The duration to wait before trying to failover or remove/replace the - # current mon with a new mon (useful for compensating flapping network). - - name: ROOK_MON_OUT_TIMEOUT - value: "300s" - - name: POD_NAMESPACE - valueFrom: - fieldRef: - fieldPath: metadata.namespace - - name: NODE_NAME - valueFrom: - fieldRef: - fieldPath: spec.nodeName diff --git a/cluster/examples/kubernetes/1.5/rook-pool.yaml b/cluster/examples/kubernetes/1.5/rook-pool.yaml deleted file mode 120000 index 9ceb09760..000000000 --- a/cluster/examples/kubernetes/1.5/rook-pool.yaml +++ /dev/null @@ -1 +0,0 @@ -../rook-pool.yaml \ No newline at end of file diff --git a/cluster/examples/kubernetes/1.5/rook-storageclass.yaml b/cluster/examples/kubernetes/1.5/rook-storageclass.yaml deleted file mode 100644 index a0267fb5b..000000000 --- a/cluster/examples/kubernetes/1.5/rook-storageclass.yaml +++ /dev/null @@ -1,27 +0,0 @@ -apiVersion: rook.io/v1alpha1 -kind: Pool -metadata: - name: replicapool - namespace: rook -spec: - replicated: - size: 1 - # For an erasure-coded pool, comment out the replication size above and uncomment the following settings. - # Make sure you have enough OSDs to support the replica size or erasure code chunks. - #erasureCoded: - # dataChunks: 2 - # codingChunks: 1 ---- -apiVersion: storage.k8s.io/v1beta1 -kind: StorageClass -metadata: - name: rook-block -provisioner: rook.io/block -parameters: - pool: replicapool - # Specify the Rook cluster from which to create volumes. - # If not specified, it will use `rook` as the name of the cluster. - # This is also the namespace where the cluster will be - clusterName: rook - # Specify the filesystem type of the volume. If not specified, it will use `ext4`. - # fstype: ext4 diff --git a/cluster/examples/kubernetes/1.5/rook-tools.yaml b/cluster/examples/kubernetes/1.5/rook-tools.yaml deleted file mode 120000 index 30b89a827..000000000 --- a/cluster/examples/kubernetes/1.5/rook-tools.yaml +++ /dev/null @@ -1 +0,0 @@ -../rook-tools.yaml \ No newline at end of file diff --git a/cluster/examples/kubernetes/1.5/wordpress.yaml b/cluster/examples/kubernetes/1.5/wordpress.yaml deleted file mode 100644 index dba453293..000000000 --- a/cluster/examples/kubernetes/1.5/wordpress.yaml +++ /dev/null @@ -1,62 +0,0 @@ -apiVersion: v1 -kind: Service -metadata: - name: wordpress - labels: - app: wordpress -spec: - ports: - - port: 80 - selector: - app: wordpress - tier: frontend - type: LoadBalancer ---- -apiVersion: v1 -kind: PersistentVolumeClaim -metadata: - name: wp-pv-claim - annotations: - volume.beta.kubernetes.io/storage-class: rook-block - labels: - app: wordpress -spec: - accessModes: - - ReadWriteOnce - resources: - requests: - storage: 20Gi ---- -apiVersion: extensions/v1beta1 -kind: Deployment -metadata: - name: wordpress - labels: - app: wordpress -spec: - strategy: - type: Recreate - template: - metadata: - labels: - app: wordpress - tier: frontend - spec: - containers: - - image: wordpress:4.6.1-apache - name: wordpress - env: - - name: WORDPRESS_DB_HOST - value: wordpress-mysql - - name: WORDPRESS_DB_PASSWORD - value: changeme - ports: - - containerPort: 80 - name: wordpress - volumeMounts: - - name: wordpress-persistent-storage - mountPath: /var/www/html - volumes: - - name: wordpress-persistent-storage - persistentVolumeClaim: - claimName: wp-pv-claim diff --git a/cluster/examples/kubernetes/1.6/rook-object.yaml b/cluster/examples/kubernetes/1.6/rook-object.yaml deleted file mode 100644 index 8f1350460..000000000 --- a/cluster/examples/kubernetes/1.6/rook-object.yaml +++ /dev/null @@ -1,45 +0,0 @@ -apiVersion: rook.io/v1alpha1 -# For earlier versions than K8s 1.7, the kind must be Objectstore -kind: Objectstore -metadata: - name: my-store - namespace: rook -spec: - # The pool spec used to create the metadata pools - metadataPool: - replicated: - size: 2 - # The pool spec used to create the data pool - dataPool: - erasureCoded: - dataChunks: 2 - codingChunks: 1 - # The gaeteway service configuration - gateway: - # type of the gateway (s3) - type: s3 - # A reference to the secret in the rook namespace where the ssl certificate is stored - sslCertificateRef: - # The port that RGW pods will listen on (http) - port: 80 - # The port that RGW pods will listen on (https). An ssl certificate is required. - securePort: 443 - # The number of pods in the rgw deployment (ignored if allNodes=true) - instances: 1 - # Whether the rgw pods should be deployed on all nodes as a daemonset - allNodes: false - # The affinity rules to apply to the rgw deployment or daemonset. - placement: - # nodeAffinity: - # requiredDuringSchedulingIgnoredDuringExecution: - # nodeSelectorTerms: - # - matchExpressions: - # - key: role - # operator: In - # values: - # - rgw-node - # tolerations: - # - key: rgw-node - # operator: Exists - # podAffinity: - # podAntiAffinity: diff --git a/cluster/examples/kubernetes/1.6/rook-operator.yaml b/cluster/examples/kubernetes/1.6/rook-operator.yaml deleted file mode 100644 index b6fe01752..000000000 --- a/cluster/examples/kubernetes/1.6/rook-operator.yaml +++ /dev/null @@ -1,173 +0,0 @@ ---- -apiVersion: extensions/v1beta1 -description: ThirdPartyResource for cluster -kind: ThirdPartyResource -metadata: - name: cluster.rook.io -versions: -- name: v1alpha1 ---- -apiVersion: extensions/v1beta1 -description: ThirdPartyResource for filesystem -kind: ThirdPartyResource -metadata: - name: filesystem.rook.io -versions: -- name: v1alpha1 ---- -apiVersion: extensions/v1beta1 -description: ThirdPartyResource for objectstore -kind: ThirdPartyResource -metadata: - name: objectstore.rook.io -versions: -- name: v1alpha1 ---- -apiVersion: extensions/v1beta1 -description: ThirdPartyResource for pool -kind: ThirdPartyResource -metadata: - name: pool.rook.io -versions: -- name: v1alpha1 ---- -apiVersion: extensions/v1beta1 -description: ThirdPartyResource for volumeattachment -kind: ThirdPartyResource -metadata: - name: volumeattachment.rook.io -versions: -- name: v1alpha1 ---- -apiVersion: v1 -kind: Namespace -metadata: - name: rook-system ---- -kind: ClusterRole -apiVersion: rbac.authorization.k8s.io/v1beta1 -metadata: - name: rook-operator -rules: -- apiGroups: - - "" - resources: - - namespaces - - serviceaccounts - - secrets - - pods - - services - - nodes - - nodes/proxy - - configmaps - - events - - persistentvolumes - - persistentvolumeclaims - verbs: - - get - - list - - watch - - patch - - create - - update - - delete -- apiGroups: - - rbac.authorization.k8s.io - resources: - - clusterroles - - clusterrolebindings - - roles - - rolebindings - verbs: - - get - - list - - watch - - create - - update - - delete -- apiGroups: - - storage.k8s.io - resources: - - storageclasses - verbs: - - get - - list - - watch - - delete -- apiGroups: - - rook.io - resources: - - "*" - verbs: - - "*" ---- -apiVersion: v1 -kind: ServiceAccount -metadata: - name: rook-operator - namespace: rook-system ---- -kind: ClusterRoleBinding -apiVersion: rbac.authorization.k8s.io/v1beta1 -metadata: - name: rook-operator - namespace: rook-system -roleRef: - apiGroup: rbac.authorization.k8s.io - kind: ClusterRole - name: rook-operator -subjects: -- kind: ServiceAccount - name: rook-operator - namespace: rook-system ---- -apiVersion: apps/v1beta1 -kind: Deployment -metadata: - name: rook-operator - namespace: rook-system -spec: - replicas: 1 - template: - metadata: - labels: - app: rook-operator - spec: - serviceAccountName: rook-operator - containers: - - name: rook-operator - image: rook/rook:master - args: ["operator"] - env: - # To disable RBAC, uncomment the following: - # - name: RBAC_ENABLED - # value: "false" - # Rook Agent toleration. Will tolerate all taints with all keys. - # Choose between NoSchedule, PreferNoSchedule and NoExecute: - # - name: AGENT_TOLERATION - # value: "NoSchedule" - # (Optional) Rook Agent toleration key. Set this to the key of the taint you want to tolerate - # - name: AGENT_TOLERATION_KEY - # value: "" - # Set the path where the Rook agent can find the flex volumes - # - name: FLEXVOLUME_DIR_PATH - # value: "" - # The interval to check if every mon is in the quorum. - - name: ROOK_MON_HEALTHCHECK_INTERVAL - value: "45s" - # The duration to wait before trying to failover or remove/replace the - # current mon with a new mon (useful for compensating flapping network). - - name: ROOK_MON_OUT_TIMEOUT - value: "300s" - - name: NODE_NAME - valueFrom: - fieldRef: - fieldPath: spec.nodeName - - name: POD_NAME - valueFrom: - fieldRef: - fieldPath: metadata.name - - name: POD_NAMESPACE - valueFrom: - fieldRef: - fieldPath: metadata.namespace diff --git a/cluster/examples/kubernetes/rook-object.yaml b/cluster/examples/kubernetes/rook-object.yaml index 95fb88ed7..5f75a9f87 100644 --- a/cluster/examples/kubernetes/rook-object.yaml +++ b/cluster/examples/kubernetes/rook-object.yaml @@ -1,5 +1,4 @@ apiVersion: rook.io/v1alpha1 -# For earlier versions than K8s 1.7, the kind must be Objectstore kind: ObjectStore metadata: name: my-store diff --git a/design/local-node-agent.md b/design/local-node-agent.md index b98297028..0b29280bc 100644 --- a/design/local-node-agent.md +++ b/design/local-node-agent.md @@ -99,7 +99,7 @@ Furthermore, if users want to consume a Rook volume outside of the default names Both of these issues are handled by this proposed design for the Rook agent, greatly streamlining the Rook storage experience in Kubernetes and removing the common causes for errors and failure. This is a big win for our users. -This proposed design also allows Rook to normalize on a single path for providing persistent volumes across all Kubernetes versions that it supports (1.5+). +This proposed design also allows Rook to normalize on a single path for providing persistent volumes across all Kubernetes versions that it supports. Since this design is fully out-of-tree for Kubernetes, it is not tied to Kubernetes release timelines. Updates, fixes and new features can be released through the normal Rook release process and timelines. diff --git a/pkg/apis/rook.io/v1alpha1/register.go b/pkg/apis/rook.io/v1alpha1/register.go index 9d0ba812f..13aa54c39 100644 --- a/pkg/apis/rook.io/v1alpha1/register.go +++ b/pkg/apis/rook.io/v1alpha1/register.go @@ -61,8 +61,6 @@ func addKnownTypes(scheme *runtime.Scheme) error { &FilesystemList{}, &ObjectStore{}, &ObjectStoreList{}, - // TODO: Remove ObjectstoreList after TPRs are no longer supported - &ObjectstoreList{}, &VolumeAttachment{}, &VolumeAttachmentList{}, ) diff --git a/pkg/apis/rook.io/v1alpha1/types.go b/pkg/apis/rook.io/v1alpha1/types.go index 2b4e5eb03..18fb68b87 100644 --- a/pkg/apis/rook.io/v1alpha1/types.go +++ b/pkg/apis/rook.io/v1alpha1/types.go @@ -288,15 +288,6 @@ type ObjectStoreList struct { Items []ObjectStore `json:"items"` } -// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object - -// ObjectstoreList is the definition of a list of object stores for TPRs (pre-1.7) -type ObjectstoreList struct { - metav1.TypeMeta `json:",inline"` - metav1.ListMeta `json:"metadata"` - Items []ObjectStore `json:"items"` -} - // ObjectStoreSpec represent the spec of a pool type ObjectStoreSpec struct { // The metadata pool settings diff --git a/pkg/apis/rook.io/v1alpha1/zz_generated.deepcopy.go b/pkg/apis/rook.io/v1alpha1/zz_generated.deepcopy.go index 4ce22dddf..75eb8a5cf 100644 --- a/pkg/apis/rook.io/v1alpha1/zz_generated.deepcopy.go +++ b/pkg/apis/rook.io/v1alpha1/zz_generated.deepcopy.go @@ -430,40 +430,6 @@ func (in *ObjectStoreSpec) DeepCopy() *ObjectStoreSpec { return out } -// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. -func (in *ObjectstoreList) DeepCopyInto(out *ObjectstoreList) { - *out = *in - out.TypeMeta = in.TypeMeta - out.ListMeta = in.ListMeta - if in.Items != nil { - in, out := &in.Items, &out.Items - *out = make([]ObjectStore, len(*in)) - for i := range *in { - (*in)[i].DeepCopyInto(&(*out)[i]) - } - } - return -} - -// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ObjectstoreList. -func (in *ObjectstoreList) DeepCopy() *ObjectstoreList { - if in == nil { - return nil - } - out := new(ObjectstoreList) - in.DeepCopyInto(out) - return out -} - -// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object. -func (in *ObjectstoreList) DeepCopyObject() runtime.Object { - if c := in.DeepCopy(); c != nil { - return c - } else { - return nil - } -} - // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. func (in *Placement) DeepCopyInto(out *Placement) { *out = *in diff --git a/pkg/daemon/agent/flexvolume/attachment/crd.go b/pkg/daemon/agent/flexvolume/attachment/crd.go index 9e8d0209c..cb13bb7fb 100644 --- a/pkg/daemon/agent/flexvolume/attachment/crd.go +++ b/pkg/daemon/agent/flexvolume/attachment/crd.go @@ -17,25 +17,15 @@ limitations under the License. package attachment import ( - "fmt" - "github.com/coreos/pkg/capnslog" rookalpha "github.com/rook/rook/pkg/apis/rook.io/v1alpha1" "github.com/rook/rook/pkg/clusterd" - "github.com/rook/rook/pkg/operator/k8sutil" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" - "k8s.io/kubernetes/pkg/util/version" ) var logger = capnslog.NewPackageLogger("github.com/rook/rook", "rook-agent-crd") -const ( - serverVersionV170 = "v1.7.0" -) - // Attachment handles custom resource VolumeAttachment storage operations. -// This interface goes away when there is no longer a need to support TPRs since -// we can call the RookClientset directly. type Attachment interface { Create(volumeAttachment *rookalpha.VolumeAttachment) error Get(namespace, name string) (*rookalpha.VolumeAttachment, error) @@ -51,22 +41,7 @@ type crd struct { // CreateController creates a new controller for volume attachment func New(context *clusterd.Context) (Attachment, error) { - - // CRD is available on v1.7.0. TPR became deprecated on v1.7.0 - // Remove this code when TPR is not longer supported - kubeVersion, err := k8sutil.GetK8SVersion(context.Clientset) - if err != nil { - return nil, fmt.Errorf("Error getting server version: %v", err) - } - if kubeVersion.AtLeast(version.MustParseSemantic(serverVersionV170)) { - return &crd{ - context: context, - }, nil - } - - return &tpr{ - clientset: context.Clientset, - }, nil + return &crd{context: context}, nil } // Get queries the VolumeAttachment CRD from Kubernetes diff --git a/pkg/daemon/agent/flexvolume/attachment/tpr.go b/pkg/daemon/agent/flexvolume/attachment/tpr.go deleted file mode 100644 index 15496a7ed..000000000 --- a/pkg/daemon/agent/flexvolume/attachment/tpr.go +++ /dev/null @@ -1,103 +0,0 @@ -/* -Copyright 2017 The Rook Authors. All rights reserved. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package attachment - -import ( - "encoding/json" - "fmt" - - "github.com/coreos/pkg/capnslog" - rookalpha "github.com/rook/rook/pkg/apis/rook.io/v1alpha1" - "k8s.io/client-go/kubernetes" -) - -const ( - tprKind = "Volumeattachment" -) - -var tprlogger = capnslog.NewPackageLogger("github.com/rook/rook", "rook-agent-tpr") - -// TPR is a controller to manage VolumeAttachment TPR objects -// CRD is available on v1.7.0. TPR became deprecated on v1.7.0 -// Remove this code when TPR is not longer supported -type tpr struct { - clientset kubernetes.Interface -} - -// Get queries the VolumeAttachment TPR from Kubernetes -func (t *tpr) Get(namespace, name string) (*rookalpha.VolumeAttachment, error) { - - var result rookalpha.VolumeAttachment - uri := fmt.Sprintf("apis/%s/%s/namespaces/%s/%s", rookalpha.CustomResourceGroup, rookalpha.Version, namespace, CustomResourceNamePlural) - return &result, t.clientset.CoreV1().RESTClient().Get(). - RequestURI(uri). - Name(name). - Do(). - Into(&result) -} - -// List lists all the volume attachment TPR resources in the given namespace -func (t *tpr) List(namespace string) (*rookalpha.VolumeAttachmentList, error) { - - var result rookalpha.VolumeAttachmentList - uri := fmt.Sprintf("apis/%s/%s/namespaces/%s/%s", rookalpha.CustomResourceGroup, rookalpha.Version, namespace, CustomResourceNamePlural) - return &result, t.clientset.CoreV1().RESTClient().Get(). - RequestURI(uri). - Do(). - Into(&result) -} - -// Create creates the volume attach TPR resource in Kubernetes -func (t *tpr) Create(volumeAttachment *rookalpha.VolumeAttachment) error { - volumeAttachment.APIVersion = fmt.Sprintf("%s/%s", rookalpha.CustomResourceGroup, rookalpha.Version) - volumeAttachment.Kind = tprKind - body, _ := json.Marshal(volumeAttachment) - uri := fmt.Sprintf("apis/%s/%s/namespaces/%s/%s", rookalpha.CustomResourceGroup, rookalpha.Version, volumeAttachment.Namespace, CustomResourceNamePlural) - return t.clientset.CoreV1().RESTClient().Post(). - RequestURI(uri). - Body(body). - Do().Error() -} - -// Update updates VolumeAttachment TPR resource -func (t *tpr) Update(volumeAttachment *rookalpha.VolumeAttachment) error { - volumeAttachment.APIVersion = fmt.Sprintf("%s/%s", rookalpha.CustomResourceGroup, rookalpha.Version) - volumeAttachment.Kind = tprKind - body, _ := json.Marshal(volumeAttachment) - uri := fmt.Sprintf("apis/%s/%s/namespaces/%s/%s", rookalpha.CustomResourceGroup, rookalpha.Version, volumeAttachment.Namespace, CustomResourceNamePlural) - err := t.clientset.CoreV1().RESTClient().Put(). - RequestURI(uri). - Name(volumeAttachment.Name). - Body(body). - Do().Error() - if err != nil { - tprlogger.Errorf("failed to update VolumeAttachment CRD. %+v", err) - return err - } - tprlogger.Infof("updated Volumeattach TPR %s", volumeAttachment.ObjectMeta.Name) - return nil -} - -// Delete deletes the volume attach TPR resource in Kubernetes -func (t *tpr) Delete(namespace, name string) error { - uri := fmt.Sprintf("apis/%s/%s/namespaces/%s/%s", rookalpha.CustomResourceGroup, rookalpha.Version, namespace, CustomResourceNamePlural) - return t.clientset.CoreV1().RESTClient().Delete(). - RequestURI(uri). - Name(name). - Do(). - Error() -} diff --git a/pkg/operator/cluster/controller.go b/pkg/operator/cluster/controller.go index 613e463a1..f605985a2 100644 --- a/pkg/operator/cluster/controller.go +++ b/pkg/operator/cluster/controller.go @@ -44,7 +44,6 @@ import ( "k8s.io/apimachinery/pkg/types" "k8s.io/apimachinery/pkg/util/wait" "k8s.io/client-go/tools/cache" - "k8s.io/kubernetes/pkg/util/version" ) const ( @@ -297,17 +296,9 @@ func (c *ClusterController) onDelete(obj interface{}) { } func (c *ClusterController) addFinalizer(clust *rookalpha.Cluster) error { - kubeVersion, err := k8sutil.GetK8SVersion(c.context.Clientset) - if err != nil { - return fmt.Errorf("Error getting server version: %v", err) - } - if !kubeVersion.AtLeast(version.MustParseSemantic(k8sutil.FirstCRDVersion)) { - logger.Infof("not adding finalizer for cluster TPR") - return nil - } // get the latest cluster object since we probably updated it before we got to this point (e.g. by updating its status) - clust, err = c.context.RookClientset.RookV1alpha1().Clusters(clust.Namespace).Get(clust.Name, metav1.GetOptions{}) + clust, err := c.context.RookClientset.RookV1alpha1().Clusters(clust.Namespace).Get(clust.Name, metav1.GetOptions{}) if err != nil { return err } diff --git a/pkg/operator/k8sutil/k8sutil.go b/pkg/operator/k8sutil/k8sutil.go index b9088f899..65369cfbe 100644 --- a/pkg/operator/k8sutil/k8sutil.go +++ b/pkg/operator/k8sutil/k8sutil.go @@ -45,8 +45,6 @@ const ( PodNamespaceEnvVar = "POD_NAMESPACE" // NodeNameEnvVar is the env variable for getting the node via downward api NodeNameEnvVar = "NODE_NAME" - // FirstCRDVersion is the first K8s version with CRDs to replace TPRs - FirstCRDVersion = "v1.7.0" ) // GetK8SVersion gets the version of the running K8S cluster diff --git a/pkg/operator/operator.go b/pkg/operator/operator.go index 5155a8132..1895905a1 100644 --- a/pkg/operator/operator.go +++ b/pkg/operator/operator.go @@ -97,8 +97,6 @@ func (o *Operator) Run() error { signal.Notify(signalChan, syscall.SIGINT, syscall.SIGTERM) // Run volume provisioner - // The controller needs to know what the server version is because out-of-tree - // provisioners aren't officially supported until 1.5 serverVersion, err := o.context.Clientset.Discovery().ServerVersion() if err != nil { return fmt.Errorf("Error getting server version: %v", err) diff --git a/tests/README.md b/tests/README.md index 740891eb0..2b5394196 100644 --- a/tests/README.md +++ b/tests/README.md @@ -56,11 +56,7 @@ tests/scripts/kubeadm.sh clean #### Alternate Kubernetes Versions These two scripts can install any version of Kubernetes you wish based on the `KUBE_VERSION` environment variable. To use an alternate version, simply set this variable before running the relevant `up` command from above. -For example, if you wanted to use `v1.6.7`, you would run `export KUBE_VERSION=v1.6.7` first before running `up`. - -By default, the following versions of Kubernetes are used: -* `minikube.sh`: `v1.9.1` -* `kubeadm.sh`: `v1.8.5` +For example, if you wanted to use `v1.9.6`, you would run `export KUBE_VERSION=v1.9.6` first before running `up`. ### Install Helm Use [helm.sh](/tests/scripts/helm.sh) to install helm and set up rook charts defined under `_output/charts` (generated by build). @@ -101,7 +97,7 @@ In addition to standard go tests parameters, the following custom parameters are Parameter | Description | Possible values | Default --- |--- | --- | --- rook_platform| platform rook needs to be installed on | kubernetes | kubernetes -k8s_version | version of Kubernetes to be installed | v1.6+ | v1.8.5 +k8s_version | version of Kubernetes to be installed | v1.7+ | v1.8.5 rook_image | rook image name to be installed | valid image name | rook/rook toolbox_image | toolbox image name to be installed | valid image name | rook/toolbox skip_install_rook | skips installing rook (if already installed) | true or false | false diff --git a/tests/framework/clients/object.go b/tests/framework/clients/object.go index 462900c42..5ca075171 100644 --- a/tests/framework/clients/object.go +++ b/tests/framework/clients/object.go @@ -40,13 +40,9 @@ func CreateObjectOperation(k8sh *utils.K8sHelper) *ObjectOperation { // ObjectCreate Function to create a object store in rook func (ro *ObjectOperation) Create(namespace, storeName string, replicaCount int32) error { - kind := "ObjectStore" - if !ro.k8sh.VersionAtLeast("1.7.0") { - kind = "Objectstore" - } logger.Infof("creating the object store via CRD") storeSpec := fmt.Sprintf(`apiVersion: rook.io/v1alpha1 -kind: %s +kind: ObjectStore metadata: name: %s namespace: %s @@ -64,7 +60,7 @@ spec: securePort: instances: %d allNodes: false -`, kind, storeName, namespace, rgwPort, replicaCount) +`, storeName, namespace, rgwPort, replicaCount) if _, err := ro.k8sh.ResourceOperation("create", storeSpec); err != nil { return err @@ -80,10 +76,6 @@ spec: func (ro *ObjectOperation) Delete(namespace, storeName string, replicaCount int32) error { - if !ro.k8sh.VersionAtLeast("1.7.0") { - // The operator fails to process the Objectstore TPR upon deletion. Support for 1.6 is going away, so just disable this test in that case. - return nil - } logger.Infof("Deleting the object store via CRD") storeSpec := fmt.Sprintf(`apiVersion: rook.io/v1alpha1 kind: ObjectStore diff --git a/tests/framework/installer/install_data.go b/tests/framework/installer/install_data.go index c44de0fb8..3cdea71c3 100644 --- a/tests/framework/installer/install_data.go +++ b/tests/framework/installer/install_data.go @@ -29,48 +29,6 @@ func NewK8sInstallData() *InstallData { return &InstallData{} } -func (i *InstallData) GetRookTPRs(namespace string) string { - return `apiVersion: extensions/v1beta1 -description: ThirdPartyResource for cluster -kind: ThirdPartyResource -metadata: - name: cluster.rook.io -versions: -- name: v1alpha1 ---- -apiVersion: extensions/v1beta1 -description: ThirdPartyResource for filesystem -kind: ThirdPartyResource -metadata: - name: filesystem.rook.io -versions: -- name: v1alpha1 ---- -apiVersion: extensions/v1beta1 -description: ThirdPartyResource for objectstore -kind: ThirdPartyResource -metadata: - name: objectstore.rook.io -versions: -- name: v1alpha1 ---- -apiVersion: extensions/v1beta1 -description: ThirdPartyResource for pool -kind: ThirdPartyResource -metadata: - name: pool.rook.io -versions: -- name: v1alpha1 ---- -apiVersion: extensions/v1beta1 -description: ThirdPartyResource for volumeattachment -kind: ThirdPartyResource -metadata: - name: volumeattachment.rook.io -versions: -- name: v1alpha1` -} - func (i *InstallData) GetRookCRDs(namespace string) string { return `apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition diff --git a/tests/framework/installer/install_helper.go b/tests/framework/installer/install_helper.go index 7c46a7d92..a3b212e48 100644 --- a/tests/framework/installer/install_helper.go +++ b/tests/framework/installer/install_helper.go @@ -65,15 +65,9 @@ type InstallHelper struct { func (h *InstallHelper) CreateK8sRookResources(namespace string) (err error) { var resources string - if h.k8shelper.VersionAtLeast("v1.7.0") { - logger.Info("Creating Rook CRD's") + logger.Info("Creating Rook CRD's") - resources = h.installData.GetRookCRDs(namespace) - } else { - logger.Info("Creating Rook TPR's") - - resources = h.installData.GetRookTPRs(namespace) - } + resources = h.installData.GetRookCRDs(namespace) _, err = h.k8shelper.KubectlWithStdin(resources, createArgs...) @@ -98,14 +92,8 @@ func (h *InstallHelper) CreateK8sRookOperator(namespace string) (err error) { return fmt.Errorf("Failed to create rook-operator pod : %v ", err) } - if h.k8shelper.VersionAtLeast("v1.7.0") { - if !h.k8shelper.IsCRDPresent(rookOperatorCreatedCrd) { - return fmt.Errorf("Failed to start Rook Operator; k8s CustomResourceDefinition did not appear") - } - } else { - if !h.k8shelper.IsThirdPartyResourcePresent(rookOperatorCreatedTpr) { - return fmt.Errorf("Failed to start Rook Operator; k8s thirdpartyresource did not appear") - } + if !h.k8shelper.IsCRDPresent(rookOperatorCreatedCrd) { + return fmt.Errorf("Failed to start Rook Operator; k8s CustomResourceDefinition did not appear") } logger.Infof("Rook Operator started") @@ -130,14 +118,8 @@ func (h *InstallHelper) CreateK8sRookOperatorViaHelm(namespace string) error { } - if h.k8shelper.VersionAtLeast("v1.7.0") { - if !h.k8shelper.IsCRDPresent(rookOperatorCreatedCrd) { - return fmt.Errorf("Failed to start Rook Operator; k8s CustomResourceDefinition did not appear") - } - } else { - if !h.k8shelper.IsThirdPartyResourcePresent(rookOperatorCreatedTpr) { - return fmt.Errorf("Failed to start Rook Operator; k8s thirdpartyresource did not appear") - } + if !h.k8shelper.IsCRDPresent(rookOperatorCreatedCrd) { + return fmt.Errorf("Failed to start Rook Operator; k8s CustomResourceDefinition did not appear") } return nil @@ -349,13 +331,8 @@ func (h *InstallHelper) UninstallRookFromMultipleNS(helmInstalled bool, systemNa } logger.Infof("removing the operator from namespace %s", systemNamespace) - if h.k8shelper.VersionAtLeast("v1.7.0") { - _, err = h.k8shelper.DeleteResource([]string{"crd", "clusters.rook.io", "pools.rook.io", "objectstores.rook.io", "filesystems.rook.io", "volumeattachments.rook.io"}) - h.checkError(err, "cannot delete CRDs") - } else { - _, err = h.k8shelper.DeleteResource([]string{"thirdpartyresources", "cluster.rook.io", "pool.rook.io", "objectstore.rook.io", "filesystem.rook.io", "volumeattachment.rook.io"}) - h.checkError(err, "cannot delete TPRs") - } + _, err = h.k8shelper.DeleteResource([]string{"crd", "clusters.rook.io", "pools.rook.io", "objectstores.rook.io", "filesystems.rook.io", "volumeattachments.rook.io"}) + h.checkError(err, "cannot delete CRDs") if helmInstalled { err = h.helmHelper.DeleteLocalRookHelmChart(helmDeployName) @@ -381,7 +358,6 @@ func (h *InstallHelper) checkError(err error, message string) { func (h *InstallHelper) waitForCustomResourceDeletion(namespace string) error { if !h.k8shelper.VersionAtLeast("v1.8.0") { - // v1.6 does not have finalizers for TPRs // v1.7 has an intermittent issue with long delay to delete resources so we will skip waiting return nil } diff --git a/tests/framework/utils/k8s_helper.go b/tests/framework/utils/k8s_helper.go index 9f8701535..41ab9dff5 100644 --- a/tests/framework/utils/k8s_helper.go +++ b/tests/framework/utils/k8s_helper.go @@ -453,23 +453,6 @@ func (k8sh *K8sHelper) GetService(servicename string, namespace string) (*v1.Ser return result, nil } -//IsThirdPartyResourcePresent returns true if Third party resource is present -func (k8sh *K8sHelper) IsThirdPartyResourcePresent(tprname string) bool { - args := []string{"get", "thirdpartyresources", tprname} - inc := 0 - for inc < RetryLoop { - _, err := k8sh.Kubectl(args...) - if err == nil { - k8slogger.Infof("Found the thirdparty resource: " + tprname) - return true - } - time.Sleep(RetryInterval * time.Second) - inc++ - } - - return false -} - //IsCRDPresent returns true if custom resource definition is present func (k8sh *K8sHelper) IsCRDPresent(crdName string) bool { diff --git a/tests/scripts/kubeadm-install.sh b/tests/scripts/kubeadm-install.sh index 5a4a93421..a52921932 100755 --- a/tests/scripts/kubeadm-install.sh +++ b/tests/scripts/kubeadm-install.sh @@ -10,7 +10,7 @@ KUBE_INSTALL_VERSION="${KUBE_VERSION/v/$null_str}"-00 sudo swapoff -a # init flexvolume -if [[ $KUBE_VERSION == v1.5* ]] || [[ $KUBE_VERSION == v1.6* ]] || [[ $KUBE_VERSION == v1.7* ]] ; +if [[ $KUBE_VERSION == v1.7* ]] ; then sudo mkdir -p /usr/libexec/kubernetes/kubelet-plugins/volume/exec/rook.io~rook cat << EOF | sudo tee -a /usr/libexec/kubernetes/kubelet-plugins/volume/exec/rook.io~rook/rook diff --git a/tests/scripts/kubeadm.sh b/tests/scripts/kubeadm.sh index 566dc0c2b..fc592fefd 100755 --- a/tests/scripts/kubeadm.sh +++ b/tests/scripts/kubeadm.sh @@ -20,14 +20,7 @@ usage(){ #install k8s master node install_master(){ - # This is needed on K8S 1.6 to fix a regression. https://github.com/kubernetes/kubernetes/issues/47109 - if [[ $KUBE_VERSION == v1.6* ]] ; then - cat << EOF | sudo tee -a /etc/systemd/system/kubelet.service.d/11-disable_attachdetach_controller.conf -[Service] -Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --enable-controller-attach-detach=false" -EOF - sudo systemctl daemon-reload - elif [[ $KUBE_VERSION == v1.8* ]] ; then + if [[ $KUBE_VERSION == v1.8* ]] ; then # for k8s 1.8, use a non default value for volume plugins cat << EOF | sudo tee -a /etc/systemd/system/kubelet.service.d/11-volume_plugin_dir.conf [Service] @@ -45,17 +38,19 @@ EOF kubectl taint nodes --all node-role.kubernetes.io/master- kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')" - echo "wait for K8s master node to be Ready" - kube_ready=$(kubectl get node -o jsonpath='{.items[0].status.conditions[3].status}') + echo "wait for K8s master node to be Ready" INC=0 - until [[ "${kube_ready}" == "True" || $INC -gt 20 ]]; do + while [[ $INC -lt 20 ]]; do + kube_ready=$(kubectl get node -o jsonpath='{.items[0].status.conditions[3].status}') + if [ "${kube_ready}" == "True" ]; then + break + fi echo "." sleep 10 ((++INC)) - kube_ready=$(kubectl get node -o jsonpath='{.items[0].status.conditions[3].status}') done - if [ "${kube_ready}" == "False" ]; then + if [ "${kube_ready}" != "True" ]; then echo "k8s master node never went to Ready status" exit 1 fi @@ -90,17 +85,20 @@ wait_for_ready(){ export KUBECONFIG=$HOME/admin.conf until [[ $count -eq $numberOfNode ]]; do - echo "wait for K8s node $count to be Ready" - kube_ready=$(kubectl get node -o jsonpath='{.items['$count'].status.conditions[3].status}') + echo "wait for K8s node $count to be Ready." INC=0 - until [[ "${kube_ready}" == "True" || $INC -gt 90 ]]; do + while [[ $INC -lt 90 ]]; do + kubectl get node -o jsonpath='{.items['$count'].status.conditions[?(@.reason == "KubeletReady")]}' + kube_ready=$(kubectl get node -o jsonpath='{.items['$count'].status.conditions[3].status}') + if [ "${kube_ready}" != "True" ]; then + break + fi echo -n "." sleep 10 ((++INC)) - kube_ready=$(kubectl get node -o jsonpath='{.items['$count'].status.conditions[3].status}') done echo - if [ "${kube_ready}" == "False" ]; then + if [ "${kube_ready}" != "True" ]; then echo "k8s node ${count} never went to Ready status" exit 1 fi diff --git a/tests/scripts/minikube.sh b/tests/scripts/minikube.sh index a38221f7b..9db0708c1 100755 --- a/tests/scripts/minikube.sh +++ b/tests/scripts/minikube.sh @@ -96,7 +96,7 @@ MEMORY=${MEMORY:-"3000"} case "${1:-}" in up) # Use kubeadm bootstrapper for 1.9+ since localkube was deprecated in 1.8 - if [[ $KUBE_VERSION == v1.5* ]] || [[ $KUBE_VERSION == v1.6* ]] || [[ $KUBE_VERSION == v1.7* ]] ; then + if [[ $KUBE_VERSION == v1.7* ]] ; then echo "starting minikube with localkube bootstrapper" minikube start --memory=${MEMORY} --kubernetes-version ${KUBE_VERSION} --extra-config=apiserver.Authorization.Mode=RBAC wait_for_ssh diff --git a/tests/scripts/multi-node/build-rook.sh b/tests/scripts/multi-node/build-rook.sh index 476ef1057..8731cdcd2 100755 --- a/tests/scripts/multi-node/build-rook.sh +++ b/tests/scripts/multi-node/build-rook.sh @@ -29,8 +29,7 @@ function purge_rook_pods { kubectl -n kube-system delete secret rook-admin || true kubectl delete -f kube-registry.yaml || true kubectl delete -n rook cluster rook || true - kubectl delete thirdpartyresources cluster.rook.io pool.rook.io objectstore.rook.io filesystem.rook.io volumeattachment.rook.io || true # ignore errors if on K8s 1.7+ - kubectl delete crd clusters.rook.io pools.rook.io objectstores.rook.io filesystems.rook.io volumeattachments.rook.io || true # ignore errors if on K8s 1.5 and 1.6 + kubectl delete crd clusters.rook.io pools.rook.io objectstores.rook.io filesystems.rook.io volumeattachments.rook.io || true kubectl delete -n rook-system daemonset rook-agent || true kubectl delete -f rook-operator.yaml || true kubectl delete clusterroles rook-agent || true