From ce23a986b1021509860def116fd1fc85871cdb93 Mon Sep 17 00:00:00 2001 From: travisn Date: Fri, 14 Dec 2018 11:20:46 -0700 Subject: [PATCH 1/2] reset the pending release notes since v0.9 shipped Signed-off-by: travisn --- PendingReleaseNotes.md | 44 +++--------------------------------------- 1 file changed, 3 insertions(+), 41 deletions(-) diff --git a/PendingReleaseNotes.md b/PendingReleaseNotes.md index 0cfb989f0..f3faf7ee2 100644 --- a/PendingReleaseNotes.md +++ b/PendingReleaseNotes.md @@ -1,54 +1,16 @@ # Major Themes + ## Action Required -## Notable Features -- Different versions of Ceph can be orchestrated by Rook. Both Luminous and Mimic are now supported, with Nautilus coming soon. - The version of Ceph is specified in the cluster CRD with the cephVersion.image property. For example, to run Mimic you could use image `ceph/ceph:v13.2.2-20181023` - or any other image found on the [Ceph DockerHub](https://hub.docker.com/r/ceph/ceph/tags). -- The Ceph CRDs are now v1. The operator will automatically convert the CRDs from v1beta1 to v1. -- The `fsType` default for StorageClass examples are now using XFS to bring it in line with Ceph recommendations. -- Ceph OSDs will be automatically updated by the operator when there is a change to the operator version or when the OSD configuration changes. See the [OSD upgrade notes](Documentation/upgrade-patch.md#object-storage-daemons-osds). -- Rook Ceph block storage provisioner can now correctly create erasure coded block images. See [Advanced Example: Erasure Coded Block Storage](Documentation/ceph-block.md#advanced-example-erasure-coded-block-storage) for an example usage. -- [Network File System (NFS)](https://github.com/nfs-ganesha/nfs-ganesha/wiki) is now supported by Rook with a new operator to deploy and manage this widely used server. NFS servers can be automatically deployed by creating an instance of the new `nfsservers.nfs.rook.io` custom resource. See the [NFS server user guide](Documentation/nfs.md) to get started with NFS. -- [Cassandra](http://cassandra.apache.org/) and [Scylla](https://www.scylladb.com/) are now supported by Rook with the rook-cassandra operator. Users can now deploy, configure and manage Cassandra or Scylla clusters, by creating an instance of the `clusters.cassandra.rook.io` custom resource. See the [user guide](Documentation/cassandra.md) to get started. -- Service account (`rook-ceph-mgr`) added for the mgr daemon to grant the mgr orchestrator modules access to the K8s APIs. -- The minimum version of Kubernetes supported by Rook changed from `1.7` to `1.8`. -- `reclaimPolicy` parameter of `StorageClass` definition is now supported. -- K8s client-go updated from version 1.8.2 to 1.11.3 -- The toolbox manifest now creates a deployment based on the `rook/ceph` image instead of creating a pod on a specialized `rook/ceph-toolbox` image. -- The frequency of discovering devices on a node is reduced to 60 minutes by default, and is configurable with the setting `ROOK_DISCOVER_DEVICES_INTERVAL` in operator.yaml. -- The number of mons can be changed by updating the `mon.count` in the cluster CRD. -- RBD Mirroring is enabled by Rook. By setting the number of [rbd mirroring workers](Documentation/ceph-cluster-crd.md#cluster-settings), the daemon(s) will be started by rook. To configure the pools or images to be mirrored, use the Rook toolbox to run the [rbd mirror](http://docs.ceph.com/docs/mimic/rbd/rbd-mirroring/) configuration tool. -- Object Store User creation via CRD for Ceph clusters. -- Ceph OSD, MGR, MDS, and RGW deployments (or DaemonSets) will be updated/upgraded automatically with updates to the Rook operator. -- Ceph OSDs are created with the `ceph-volume` tool when configuring devices, adding support for multiple OSDs per device. See the [OSD configuration settings](Documentation/ceph-cluster-crd.md#osd-configuration-settings) +## Notable Features ## Breaking Changes -- The Ceph CRDs are now v1. With the version change, the `kind` has been renamed for the following Ceph CRDs: - - `Cluster` --> `CephCluster` - - `Pool` --> `CephBlockPool` - - `Filesystem` --> `CephFilesystem` - - `ObjectStore` --> `CephObjectStore` - - `ObjectStoreUser` --> `CephObjectStoreUser` -- The `rook-ceph-cluster` service account was renamed to `rook-ceph-osd` as this service account only applies to OSDs. - - On upgrade from v0.8, the `rook-ceph-osd` service account must be created before starting the operator on v0.9. - - The `serviceAccount` property has been removed from the cluster CRD. -- Ceph mons are [named consistently](https://github.com/rook/rook/issues/1751) with other daemons with the letters a, b, c, etc. -- Ceph mons are now created with Deployments instead of ReplicaSets to improve the upgrade implementation. -- Ceph mon, mgr, mds, and rgw container names in pods have changed with the refactors to initialize the - daemon environments via pod **InitContainers** and run the Ceph daemons directly from the - container entrypoint. -- The Rook container images are no longer published to quay.io, they are published only to Docker Hub. All manifests have referenced Docker Hub for multiple releases now, so we do not expect any directly affected users from this change. -- Rook no longer supports kubernetes `1.7`. Users running Kubernetes `1.7` on their clusters are recommended to upgrade to Kubernetes `1.8` or higher. If you are using `kubeadm`, you can follow this [guide](https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-8/) to from Kubernetes `1.7` to `1.8`. If you are using `kops` or `kubespray` for managing your Kubernetes cluster, just follow the respective projects' `upgrade` guide. -- Minio no longer exposes a configurable port for each distributed server instance to use. - This was an internal only port that should not need to be configured by the user. - All connections from users and clients are expected to come in through the [configurable Service instance](cluster/examples/kubernetes/minio/object-store.yaml#37). ## Known Issues -- Upgrades are not supported to nautilus. Specifically, OSDs configured before the upgrade (without ceph-volume) will fail to start on nautilus. Nautilus is not officially supported until its release, but otherwise is expected to be working in test clusters. + ## Deprecations From 0230400e27237f130d59f46a9a1ff417de22d6ec Mon Sep 17 00:00:00 2001 From: travisn Date: Wed, 19 Dec 2018 23:42:01 -0700 Subject: [PATCH 2/2] build: remove 1.8 and 1.9 and add 1.13 to integration tests Signed-off-by: travisn --- Documentation/ceph-quickstart.md | 2 +- Documentation/helm-operator.md | 2 +- Documentation/k8s-pre-reqs.md | 2 +- Jenkinsfile | 9 +++--- PendingReleaseNotes.md | 1 + tests/README.md | 6 ++-- tests/framework/installer/ceph_installer.go | 2 +- tests/framework/utils/k8s_helper.go | 14 ++------- tests/integration/base_file_test.go | 8 ----- tests/integration/smoke_test.go | 34 --------------------- tests/scripts/kubeadm-install.sh | 2 +- tests/scripts/kubeadm.sh | 23 ++------------ tests/scripts/minikube.sh | 4 +-- tests/scripts/minishift.sh | 3 +- 14 files changed, 18 insertions(+), 94 deletions(-) diff --git a/Documentation/ceph-quickstart.md b/Documentation/ceph-quickstart.md index fcb455a08..2e614ae30 100644 --- a/Documentation/ceph-quickstart.md +++ b/Documentation/ceph-quickstart.md @@ -11,7 +11,7 @@ from other pods running in your cluster. ## Minimum Version -Kubernetes **v1.8** or higher is supported by Rook. +Kubernetes **v1.10** or higher is supported by Rook. ## Prerequisites diff --git a/Documentation/helm-operator.md b/Documentation/helm-operator.md index c22010807..7a10c8287 100644 --- a/Documentation/helm-operator.md +++ b/Documentation/helm-operator.md @@ -14,7 +14,7 @@ This chart bootstraps a [rook-ceph-operator](https://github.com/rook/rook) deplo ## Prerequisites -- Kubernetes 1.8+ +- Kubernetes 1.10+ ### RBAC diff --git a/Documentation/k8s-pre-reqs.md b/Documentation/k8s-pre-reqs.md index 147763ec0..6c67a44e3 100644 --- a/Documentation/k8s-pre-reqs.md +++ b/Documentation/k8s-pre-reqs.md @@ -9,7 +9,7 @@ Rook can be installed on any existing Kubernetes clusters as long as it meets th ## Minimum Version -Kubernetes v1.8 or higher is supported by Rook. +Kubernetes v1.10 or higher is supported by Rook. ## Privileges and RBAC diff --git a/Jenkinsfile b/Jenkinsfile index 43cccbdf2..1518887d6 100644 --- a/Jenkinsfile +++ b/Jenkinsfile @@ -68,11 +68,10 @@ pipeline { stash name: 'repo-amd64',includes: 'ceph-amd64.tar,cockroachdb-amd64.tar,cassandra-amd64.tar,nfs-amd64.tar,build/common.sh,_output/tests/linux_amd64/,_output/charts/,tests/scripts/' script{ def data = [ - "aws_1.8.x": "v1.8.5", - "gce_1.9.x": "v1.9.9", - "aws_1.10.x": "v1.10.5", - "aws_1.11.x": "v1.11.1", - "aws_1.12.x": "v1.12.0" + "aws_1.10.x": "v1.10.11", + "aws_1.11.x": "v1.11.5", + "aws_1.12.x": "v1.12.3", + "gce_1.13.x": "v1.13.1" ] testruns = [:] for (kv in mapToList(data)) { diff --git a/PendingReleaseNotes.md b/PendingReleaseNotes.md index f3faf7ee2..6f2096d00 100644 --- a/PendingReleaseNotes.md +++ b/PendingReleaseNotes.md @@ -9,6 +9,7 @@ ## Breaking Changes +- Rook no longer supports Kubernetes `1.8` and `1.9`. ## Known Issues diff --git a/tests/README.md b/tests/README.md index 43342b256..35fb166ea 100644 --- a/tests/README.md +++ b/tests/README.md @@ -18,8 +18,6 @@ quickly spin up a Kubernetes cluster.The Test framework is designed to install R You can choose any Kubernetes flavor of your choice. The test framework only depends on kubectl being configured. The framework also provides scripts to install Kubernetes. There are two scripts to start the cluster: 1. **Minikube** (recommended for MacOS): Run [minikube.sh](/tests/scripts/minikube.sh) to setup a single-node Minikube Kubernetes. - 1. Minikube v0.21.0 and higher is supported. Older minikube versions do not have cephfs or rbd tools installed. - 1. If using Kubernetes v1.8 or higher, then Minikube v0.23.0 or higher is required. 1. **Kubeadm** (recommended for Ubuntu): run [kubeadm.sh](/tests/scripts/kubeadm.sh) to setup a single-node K8s cluster using kubeadm #### Minikube (recommended for MacOS) @@ -55,7 +53,7 @@ tests/scripts/kubeadm.sh clean #### Alternate Kubernetes Versions These two scripts can install any version of Kubernetes you wish based on the `KUBE_VERSION` environment variable. To use an alternate version, simply set this variable before running the relevant `up` command from above. -For example, if you wanted to use `v1.9.6`, you would run `export KUBE_VERSION=v1.9.6` first before running `up`. +For example, if you wanted to use `v1.12.5`, you would run `export KUBE_VERSION=v1.12.5` first before running `up`. ### Install Helm Use [helm.sh](/tests/scripts/helm.sh) to install Helm and set up Rook charts defined under `_output/charts` (generated by build): @@ -101,7 +99,7 @@ In addition to standard go tests parameters, the following custom parameters are | Parameter | Description | Possible values | Default | | ----------------- | -------------------------------------------- | ---------------- | ----------------- | | rook_platform | platform Rook needs to be installed on | kubernetes | kubernetes | -| k8s_version | version of Kubernetes to be installed | v1.8+ | v1.8.5 | +| k8s_version | version of Kubernetes to be installed | v1.10+ | v1.13.1 | | rook_image | Rook image name to be installed | valid image name | rook/ceph | | skip_install_rook | skips installing Rook (if already installed) | true or false | false | diff --git a/tests/framework/installer/ceph_installer.go b/tests/framework/installer/ceph_installer.go index 51a3e5212..00ac1ccc1 100644 --- a/tests/framework/installer/ceph_installer.go +++ b/tests/framework/installer/ceph_installer.go @@ -458,7 +458,7 @@ func NewCephInstaller(t func() *testing.T, clientset *kubernetes.Clientset, rook helmHelper: utils.NewHelmHelper(Env.Helm), k8sVersion: version.String(), cephVersion: cephVersion, - changeHostnames: rookVersion != Version0_8 && k8shelp.VersionAtLeast("v1.11.0"), + changeHostnames: rookVersion != Version0_8 && k8shelp.VersionAtLeast("v1.13.0"), T: t, } flag.Parse() diff --git a/tests/framework/utils/k8s_helper.go b/tests/framework/utils/k8s_helper.go index 1946e498d..3710946f7 100644 --- a/tests/framework/utils/k8s_helper.go +++ b/tests/framework/utils/k8s_helper.go @@ -353,10 +353,6 @@ func (k8sh *K8sHelper) DeleteResource(args ...string) (string, error) { // WaitForCustomResourceDeletion waits for the CRD deletion func (k8sh *K8sHelper) WaitForCustomResourceDeletion(namespace string, checkerFunc func() error) error { - if !k8sh.VersionAtLeast("v1.8.0") { - // v1.7 has an intermittent issue with long delay to delete resources so we will skip waiting - return nil - } // wait for the operator to finalize and delete the CRD for i := 0; i < 10; i++ { @@ -1482,14 +1478,8 @@ func (k8sh *K8sHelper) CreateAnonSystemClusterBinding() { inc := 0 for inc < RetryLoop { var err error - if k8sh.VersionAtLeast("v1.8.0") { - if _, err = k8sh.Clientset.RbacV1().ClusterRoleBindings().Get("anon-user-access", metav1.GetOptions{}); err == nil { - break - } - } else { - if _, err = k8sh.Clientset.RbacV1beta1().ClusterRoleBindings().Get("anon-user-access", metav1.GetOptions{}); err == nil { - break - } + if _, err = k8sh.Clientset.RbacV1beta1().ClusterRoleBindings().Get("anon-user-access", metav1.GetOptions{}); err == nil { + break } logger.Warningf("failed to get anon-user-access clusterrolebinding, will try again: %+v", err) inc++ diff --git a/tests/integration/base_file_test.go b/tests/integration/base_file_test.go index ad69da184..6598f1ce9 100644 --- a/tests/integration/base_file_test.go +++ b/tests/integration/base_file_test.go @@ -19,14 +19,12 @@ package integration import ( "fmt" - "github.com/rook/rook/pkg/daemon/ceph/agent/flexvolume" "github.com/rook/rook/tests/framework/clients" "github.com/rook/rook/tests/framework/installer" "github.com/rook/rook/tests/framework/utils" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" "github.com/stretchr/testify/suite" - "k8s.io/kubernetes/pkg/util/version" ) const ( @@ -133,12 +131,6 @@ func podWithFilesystem( testPod func(podName string, namespace string, filesystemName string, driverName string) string, ) error { driverName := installer.SystemNamespace(namespace) - v := version.MustParseSemantic(k8sh.GetK8sServerVersion()) - if v.LessThan(version.MustParseSemantic("1.10.0")) { - // k8s 1.10 and newer requires the new driver name to avoid conflicts in the test - driverName = flexvolume.FlexDriverName - } - testPodManifest := testPod(podName, namespace, filesystemName, driverName) logger.Infof("creating test pod: %s", testPodManifest) _, err := k8sh.ResourceOperation(action, testPodManifest) diff --git a/tests/integration/smoke_test.go b/tests/integration/smoke_test.go index c36da119b..0bba8da48 100644 --- a/tests/integration/smoke_test.go +++ b/tests/integration/smoke_test.go @@ -18,7 +18,6 @@ package integration import ( "fmt" - "regexp" "strings" "testing" @@ -26,11 +25,7 @@ import ( "github.com/rook/rook/tests/framework/installer" "github.com/rook/rook/tests/framework/utils" "github.com/stretchr/testify/assert" - "github.com/stretchr/testify/require" "github.com/stretchr/testify/suite" - "k8s.io/api/core/v1" - metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" - "k8s.io/kubernetes/pkg/util/version" ) // ************************************************ @@ -107,35 +102,6 @@ func (suite *SmokeSuite) TestRookClusterInstallation_SmokeTest() { checkIfRookClusterIsInstalled(suite.Suite, suite.k8sh, installer.SystemNamespace(suite.namespace), suite.namespace, 3) } -func (suite *SmokeSuite) TestOperatorGetFlexvolumePath() { - v := version.MustParseSemantic(suite.k8sh.GetK8sServerVersion()) - if !v.LessThan(version.MustParseSemantic("1.9.0")) { - suite.T().Skip("Skipping test - known issues with k8s 1.9 (https://github.com/rook/rook/issues/1330)") - } - // Get the operator pod - sysNamespace := installer.SystemNamespace(suite.namespace) - listOpts := metav1.ListOptions{LabelSelector: "app=rook-ceph-operator"} - podList, err := suite.k8sh.Clientset.CoreV1().Pods(sysNamespace).List(listOpts) - require.Nil(suite.T(), err) - require.Equal(suite.T(), 1, len(podList.Items)) - - // Get the raw log for the operator pod - opPodName := podList.Items[0].Name - rawLog, err := suite.k8sh.Clientset.CoreV1().Pods(sysNamespace).GetLogs(opPodName, &v1.PodLogOptions{}).Do().Raw() - require.Nil(suite.T(), err) - - r := regexp.MustCompile(`discovered flexvolume dir path from source.*\n`) - logStmt := string(r.Find(rawLog)) - logger.Infof("flexvolume discovery log statement: %s", logStmt) - - // Verify that the volume plugin dir was discovered by the operator pod and that it did not come from - // an env var or the default - require.NotEmpty(suite.T(), logStmt) - assert.True(suite.T(), strings.Contains(logStmt, "discovered flexvolume dir path from source")) - assert.False(suite.T(), strings.Contains(logStmt, "discovered flexvolume dir path from source env var")) - assert.False(suite.T(), strings.Contains(logStmt, "discovered flexvolume dir path from source default")) -} - func checkOrderedSubstrings(t *testing.T, input string, substrings ...string) { if len(input) == 0 { // Nothing to compare. An error was likely returned, which should be checked elsewhere. diff --git a/tests/scripts/kubeadm-install.sh b/tests/scripts/kubeadm-install.sh index 0959a39f6..68a44b5d0 100755 --- a/tests/scripts/kubeadm-install.sh +++ b/tests/scripts/kubeadm-install.sh @@ -1,6 +1,6 @@ #!/bin/bash +e -KUBE_VERSION=${1:-"v1.8.5"} +KUBE_VERSION=${1:-"v1.13.1"} null_str= KUBE_INSTALL_VERSION="${KUBE_VERSION/v/$null_str}"-00 diff --git a/tests/scripts/kubeadm.sh b/tests/scripts/kubeadm.sh index c221820eb..7f31ace05 100755 --- a/tests/scripts/kubeadm.sh +++ b/tests/scripts/kubeadm.sh @@ -5,9 +5,9 @@ scriptdir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" tarname=image.tar tarfile="${WORK_DIR}/tests/${tarname}" -export KUBE_VERSION=${KUBE_VERSION:-"v1.8.5"} +export KUBE_VERSION=${KUBE_VERSION:-"v1.13.1"} -if [[ $KUBE_VERSION == v1.12* ]] ; then +if [[ $KUBE_VERSION != v1.10* && $KUBE_VERSION != v1.11* ]] ; then skippreflightcheck=--ignore-preflight-errors=all else skippreflightcheck=--skip-preflight-checks @@ -25,15 +25,6 @@ usage(){ #install k8s master node install_master(){ - if [[ $KUBE_VERSION == v1.8* ]] ; then - # for k8s 1.8, use a non default value for volume plugins - cat << EOF | sudo tee -a /etc/systemd/system/kubelet.service.d/11-volume_plugin_dir.conf -[Service] -Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins" -EOF - sudo systemctl daemon-reload - fi - sudo kubeadm init $skippreflightcheck --kubernetes-version ${KUBE_VERSION} sudo cp /etc/kubernetes/admin.conf $HOME/ @@ -66,16 +57,6 @@ EOF #install k8s node install_node(){ echo "inside install node function" - - # for k8s 1.8, use a non default value for volume plugins - if [[ $KUBE_VERSION == v1.8* ]] ; then - cat << EOF | sudo tee -a /etc/systemd/system/kubelet.service.d/11-volume_plugin_dir.conf -[Service] -Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins" -EOF - sudo systemctl daemon-reload - fi - echo "kubeadm join ${1} ${2} ${3} ${4} ${5} $skippreflightcheck" sudo kubeadm join ${1} ${2} ${3} ${4} ${5} $skippreflightcheck || true } diff --git a/tests/scripts/minikube.sh b/tests/scripts/minikube.sh index a445cf7db..30998ff9e 100755 --- a/tests/scripts/minikube.sh +++ b/tests/scripts/minikube.sh @@ -120,13 +120,11 @@ function copy_images() { } # configure minikube -# ** K8s v1.12 requires at least minikube v0.30.0 ** -KUBE_VERSION=${KUBE_VERSION:-"v1.12.2"} +KUBE_VERSION=${KUBE_VERSION:-"v1.13.1"} MEMORY=${MEMORY:-"3000"} case "${1:-}" in up) - # Use kubeadm bootstrapper for 1.8+ since localkube was deprecated in 1.8 echo "starting minikube with kubeadm bootstrapper" minikube start --memory="${MEMORY}" -b kubeadm --kubernetes-version "${KUBE_VERSION}" wait_for_ssh diff --git a/tests/scripts/minishift.sh b/tests/scripts/minishift.sh index 441da1eb9..ff3a4cb44 100755 --- a/tests/scripts/minishift.sh +++ b/tests/scripts/minishift.sh @@ -36,8 +36,7 @@ MEMORY=${MEMORY:-"3000"} case "${1:-}" in up) - # Use kubeadm bootstrapper for 1.9+ since localkube was deprecated in 1.8 - echo "starting minishift with localkube bootstrapper" + echo "starting minishift" minishift start --memory=${MEMORY} --vm-driver=virtualbox --iso-url centos wait_for_ssh