Merge pull request #2381 from travisn/add-remove-k8s-tests
Remove 1.8 and 1.9 and add 1.13 to integration tests
This commit is contained in:
14 files changed
+21
-135
No files matched your search
@@ -11,7 +11,7 @@ from other pods running in your cluster.
|
||||
|
||||
## Minimum Version
|
||||
|
||||
Kubernetes **v1.8** or higher is supported by Rook.
|
||||
Kubernetes **v1.10** or higher is supported by Rook.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
|
||||
@@ -14,7 +14,7 @@ This chart bootstraps a [rook-ceph-operator](https://github.com/rook/rook) deplo
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Kubernetes 1.8+
|
||||
- Kubernetes 1.10+
|
||||
|
||||
### RBAC
|
||||
|
||||
|
||||
@@ -9,7 +9,7 @@ Rook can be installed on any existing Kubernetes clusters as long as it meets th
|
||||
|
||||
## Minimum Version
|
||||
|
||||
Kubernetes v1.8 or higher is supported by Rook.
|
||||
Kubernetes v1.10 or higher is supported by Rook.
|
||||
|
||||
## Privileges and RBAC
|
||||
|
||||
|
||||
Vendored
+4
-5
@@ -68,11 +68,10 @@ pipeline {
|
||||
stash name: 'repo-amd64',includes: 'ceph-amd64.tar,cockroachdb-amd64.tar,cassandra-amd64.tar,nfs-amd64.tar,build/common.sh,_output/tests/linux_amd64/,_output/charts/,tests/scripts/'
|
||||
script{
|
||||
def data = [
|
||||
"aws_1.8.x": "v1.8.5",
|
||||
"gce_1.9.x": "v1.9.9",
|
||||
"aws_1.10.x": "v1.10.5",
|
||||
"aws_1.11.x": "v1.11.1",
|
||||
"aws_1.12.x": "v1.12.0"
|
||||
"aws_1.10.x": "v1.10.11",
|
||||
"aws_1.11.x": "v1.11.5",
|
||||
"aws_1.12.x": "v1.12.3",
|
||||
"gce_1.13.x": "v1.13.1"
|
||||
]
|
||||
testruns = [:]
|
||||
for (kv in mapToList(data)) {
|
||||
|
||||
+4
-41
@@ -1,54 +1,17 @@
|
||||
# Major Themes
|
||||
|
||||
|
||||
## Action Required
|
||||
|
||||
## Notable Features
|
||||
|
||||
- Different versions of Ceph can be orchestrated by Rook. Both Luminous and Mimic are now supported, with Nautilus coming soon.
|
||||
The version of Ceph is specified in the cluster CRD with the cephVersion.image property. For example, to run Mimic you could use image `ceph/ceph:v13.2.2-20181023`
|
||||
or any other image found on the [Ceph DockerHub](https://hub.docker.com/r/ceph/ceph/tags).
|
||||
- The Ceph CRDs are now v1. The operator will automatically convert the CRDs from v1beta1 to v1.
|
||||
- The `fsType` default for StorageClass examples are now using XFS to bring it in line with Ceph recommendations.
|
||||
- Ceph OSDs will be automatically updated by the operator when there is a change to the operator version or when the OSD configuration changes. See the [OSD upgrade notes](Documentation/upgrade-patch.md#object-storage-daemons-osds).
|
||||
- Rook Ceph block storage provisioner can now correctly create erasure coded block images. See [Advanced Example: Erasure Coded Block Storage](Documentation/ceph-block.md#advanced-example-erasure-coded-block-storage) for an example usage.
|
||||
- [Network File System (NFS)](https://github.com/nfs-ganesha/nfs-ganesha/wiki) is now supported by Rook with a new operator to deploy and manage this widely used server. NFS servers can be automatically deployed by creating an instance of the new `nfsservers.nfs.rook.io` custom resource. See the [NFS server user guide](Documentation/nfs.md) to get started with NFS.
|
||||
- [Cassandra](http://cassandra.apache.org/) and [Scylla](https://www.scylladb.com/) are now supported by Rook with the rook-cassandra operator. Users can now deploy, configure and manage Cassandra or Scylla clusters, by creating an instance of the `clusters.cassandra.rook.io` custom resource. See the [user guide](Documentation/cassandra.md) to get started.
|
||||
- Service account (`rook-ceph-mgr`) added for the mgr daemon to grant the mgr orchestrator modules access to the K8s APIs.
|
||||
- The minimum version of Kubernetes supported by Rook changed from `1.7` to `1.8`.
|
||||
- `reclaimPolicy` parameter of `StorageClass` definition is now supported.
|
||||
- K8s client-go updated from version 1.8.2 to 1.11.3
|
||||
- The toolbox manifest now creates a deployment based on the `rook/ceph` image instead of creating a pod on a specialized `rook/ceph-toolbox` image.
|
||||
- The frequency of discovering devices on a node is reduced to 60 minutes by default, and is configurable with the setting `ROOK_DISCOVER_DEVICES_INTERVAL` in operator.yaml.
|
||||
- The number of mons can be changed by updating the `mon.count` in the cluster CRD.
|
||||
- RBD Mirroring is enabled by Rook. By setting the number of [rbd mirroring workers](Documentation/ceph-cluster-crd.md#cluster-settings), the daemon(s) will be started by rook. To configure the pools or images to be mirrored, use the Rook toolbox to run the [rbd mirror](http://docs.ceph.com/docs/mimic/rbd/rbd-mirroring/) configuration tool.
|
||||
- Object Store User creation via CRD for Ceph clusters.
|
||||
- Ceph OSD, MGR, MDS, and RGW deployments (or DaemonSets) will be updated/upgraded automatically with updates to the Rook operator.
|
||||
- Ceph OSDs are created with the `ceph-volume` tool when configuring devices, adding support for multiple OSDs per device. See the [OSD configuration settings](Documentation/ceph-cluster-crd.md#osd-configuration-settings)
|
||||
## Notable Features
|
||||
|
||||
|
||||
## Breaking Changes
|
||||
|
||||
- The Ceph CRDs are now v1. With the version change, the `kind` has been renamed for the following Ceph CRDs:
|
||||
- `Cluster` --> `CephCluster`
|
||||
- `Pool` --> `CephBlockPool`
|
||||
- `Filesystem` --> `CephFilesystem`
|
||||
- `ObjectStore` --> `CephObjectStore`
|
||||
- `ObjectStoreUser` --> `CephObjectStoreUser`
|
||||
- The `rook-ceph-cluster` service account was renamed to `rook-ceph-osd` as this service account only applies to OSDs.
|
||||
- On upgrade from v0.8, the `rook-ceph-osd` service account must be created before starting the operator on v0.9.
|
||||
- The `serviceAccount` property has been removed from the cluster CRD.
|
||||
- Ceph mons are [named consistently](https://github.com/rook/rook/issues/1751) with other daemons with the letters a, b, c, etc.
|
||||
- Ceph mons are now created with Deployments instead of ReplicaSets to improve the upgrade implementation.
|
||||
- Ceph mon, mgr, mds, and rgw container names in pods have changed with the refactors to initialize the
|
||||
daemon environments via pod **InitContainers** and run the Ceph daemons directly from the
|
||||
container entrypoint.
|
||||
- The Rook container images are no longer published to quay.io, they are published only to Docker Hub. All manifests have referenced Docker Hub for multiple releases now, so we do not expect any directly affected users from this change.
|
||||
- Rook no longer supports kubernetes `1.7`. Users running Kubernetes `1.7` on their clusters are recommended to upgrade to Kubernetes `1.8` or higher. If you are using `kubeadm`, you can follow this [guide](https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-8/) to from Kubernetes `1.7` to `1.8`. If you are using `kops` or `kubespray` for managing your Kubernetes cluster, just follow the respective projects' `upgrade` guide.
|
||||
- Minio no longer exposes a configurable port for each distributed server instance to use.
|
||||
This was an internal only port that should not need to be configured by the user.
|
||||
All connections from users and clients are expected to come in through the [configurable Service instance](cluster/examples/kubernetes/minio/object-store.yaml#37).
|
||||
- Rook no longer supports Kubernetes `1.8` and `1.9`.
|
||||
|
||||
## Known Issues
|
||||
- Upgrades are not supported to nautilus. Specifically, OSDs configured before the upgrade (without ceph-volume) will fail to start on nautilus. Nautilus is not officially supported until its release, but otherwise is expected to be working in test clusters.
|
||||
|
||||
|
||||
## Deprecations
|
||||
+2
-4
@@ -18,8 +18,6 @@ quickly spin up a Kubernetes cluster.The Test framework is designed to install R
|
||||
You can choose any Kubernetes flavor of your choice. The test framework only depends on kubectl being configured.
|
||||
The framework also provides scripts to install Kubernetes. There are two scripts to start the cluster:
|
||||
1. **Minikube** (recommended for MacOS): Run [minikube.sh](/tests/scripts/minikube.sh) to setup a single-node Minikube Kubernetes.
|
||||
1. Minikube v0.21.0 and higher is supported. Older minikube versions do not have cephfs or rbd tools installed.
|
||||
1. If using Kubernetes v1.8 or higher, then Minikube v0.23.0 or higher is required.
|
||||
1. **Kubeadm** (recommended for Ubuntu): run [kubeadm.sh](/tests/scripts/kubeadm.sh) to setup a single-node K8s cluster using kubeadm
|
||||
|
||||
#### Minikube (recommended for MacOS)
|
||||
@@ -55,7 +53,7 @@ tests/scripts/kubeadm.sh clean
|
||||
#### Alternate Kubernetes Versions
|
||||
These two scripts can install any version of Kubernetes you wish based on the `KUBE_VERSION` environment variable.
|
||||
To use an alternate version, simply set this variable before running the relevant `up` command from above.
|
||||
For example, if you wanted to use `v1.9.6`, you would run `export KUBE_VERSION=v1.9.6` first before running `up`.
|
||||
For example, if you wanted to use `v1.12.5`, you would run `export KUBE_VERSION=v1.12.5` first before running `up`.
|
||||
|
||||
### Install Helm
|
||||
Use [helm.sh](/tests/scripts/helm.sh) to install Helm and set up Rook charts defined under `_output/charts` (generated by build):
|
||||
@@ -101,7 +99,7 @@ In addition to standard go tests parameters, the following custom parameters are
|
||||
| Parameter | Description | Possible values | Default |
|
||||
| ----------------- | -------------------------------------------- | ---------------- | ----------------- |
|
||||
| rook_platform | platform Rook needs to be installed on | kubernetes | kubernetes |
|
||||
| k8s_version | version of Kubernetes to be installed | v1.8+ | v1.8.5 |
|
||||
| k8s_version | version of Kubernetes to be installed | v1.10+ | v1.13.1 |
|
||||
| rook_image | Rook image name to be installed | valid image name | rook/ceph |
|
||||
| skip_install_rook | skips installing Rook (if already installed) | true or false | false |
|
||||
|
||||
|
||||
@@ -458,7 +458,7 @@ func NewCephInstaller(t func() *testing.T, clientset *kubernetes.Clientset, rook
|
||||
helmHelper: utils.NewHelmHelper(Env.Helm),
|
||||
k8sVersion: version.String(),
|
||||
cephVersion: cephVersion,
|
||||
changeHostnames: rookVersion != Version0_8 && k8shelp.VersionAtLeast("v1.11.0"),
|
||||
changeHostnames: rookVersion != Version0_8 && k8shelp.VersionAtLeast("v1.13.0"),
|
||||
T: t,
|
||||
}
|
||||
flag.Parse()
|
||||
|
||||
@@ -353,10 +353,6 @@ func (k8sh *K8sHelper) DeleteResource(args ...string) (string, error) {
|
||||
|
||||
// WaitForCustomResourceDeletion waits for the CRD deletion
|
||||
func (k8sh *K8sHelper) WaitForCustomResourceDeletion(namespace string, checkerFunc func() error) error {
|
||||
if !k8sh.VersionAtLeast("v1.8.0") {
|
||||
// v1.7 has an intermittent issue with long delay to delete resources so we will skip waiting
|
||||
return nil
|
||||
}
|
||||
|
||||
// wait for the operator to finalize and delete the CRD
|
||||
for i := 0; i < 10; i++ {
|
||||
@@ -1482,14 +1478,8 @@ func (k8sh *K8sHelper) CreateAnonSystemClusterBinding() {
|
||||
inc := 0
|
||||
for inc < RetryLoop {
|
||||
var err error
|
||||
if k8sh.VersionAtLeast("v1.8.0") {
|
||||
if _, err = k8sh.Clientset.RbacV1().ClusterRoleBindings().Get("anon-user-access", metav1.GetOptions{}); err == nil {
|
||||
break
|
||||
}
|
||||
} else {
|
||||
if _, err = k8sh.Clientset.RbacV1beta1().ClusterRoleBindings().Get("anon-user-access", metav1.GetOptions{}); err == nil {
|
||||
break
|
||||
}
|
||||
if _, err = k8sh.Clientset.RbacV1beta1().ClusterRoleBindings().Get("anon-user-access", metav1.GetOptions{}); err == nil {
|
||||
break
|
||||
}
|
||||
logger.Warningf("failed to get anon-user-access clusterrolebinding, will try again: %+v", err)
|
||||
inc++
|
||||
|
||||
@@ -19,14 +19,12 @@ package integration
|
||||
import (
|
||||
"fmt"
|
||||
|
||||
"github.com/rook/rook/pkg/daemon/ceph/agent/flexvolume"
|
||||
"github.com/rook/rook/tests/framework/clients"
|
||||
"github.com/rook/rook/tests/framework/installer"
|
||||
"github.com/rook/rook/tests/framework/utils"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"github.com/stretchr/testify/suite"
|
||||
"k8s.io/kubernetes/pkg/util/version"
|
||||
)
|
||||
|
||||
const (
|
||||
@@ -133,12 +131,6 @@ func podWithFilesystem(
|
||||
testPod func(podName string, namespace string, filesystemName string, driverName string) string,
|
||||
) error {
|
||||
driverName := installer.SystemNamespace(namespace)
|
||||
v := version.MustParseSemantic(k8sh.GetK8sServerVersion())
|
||||
if v.LessThan(version.MustParseSemantic("1.10.0")) {
|
||||
// k8s 1.10 and newer requires the new driver name to avoid conflicts in the test
|
||||
driverName = flexvolume.FlexDriverName
|
||||
}
|
||||
|
||||
testPodManifest := testPod(podName, namespace, filesystemName, driverName)
|
||||
logger.Infof("creating test pod: %s", testPodManifest)
|
||||
_, err := k8sh.ResourceOperation(action, testPodManifest)
|
||||
|
||||
@@ -18,7 +18,6 @@ package integration
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"regexp"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
@@ -26,11 +25,7 @@ import (
|
||||
"github.com/rook/rook/tests/framework/installer"
|
||||
"github.com/rook/rook/tests/framework/utils"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"github.com/stretchr/testify/suite"
|
||||
"k8s.io/api/core/v1"
|
||||
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
|
||||
"k8s.io/kubernetes/pkg/util/version"
|
||||
)
|
||||
|
||||
// ************************************************
|
||||
@@ -107,35 +102,6 @@ func (suite *SmokeSuite) TestRookClusterInstallation_SmokeTest() {
|
||||
checkIfRookClusterIsInstalled(suite.Suite, suite.k8sh, installer.SystemNamespace(suite.namespace), suite.namespace, 3)
|
||||
}
|
||||
|
||||
func (suite *SmokeSuite) TestOperatorGetFlexvolumePath() {
|
||||
v := version.MustParseSemantic(suite.k8sh.GetK8sServerVersion())
|
||||
if !v.LessThan(version.MustParseSemantic("1.9.0")) {
|
||||
suite.T().Skip("Skipping test - known issues with k8s 1.9 (https://github.com/rook/rook/issues/1330)")
|
||||
}
|
||||
// Get the operator pod
|
||||
sysNamespace := installer.SystemNamespace(suite.namespace)
|
||||
listOpts := metav1.ListOptions{LabelSelector: "app=rook-ceph-operator"}
|
||||
podList, err := suite.k8sh.Clientset.CoreV1().Pods(sysNamespace).List(listOpts)
|
||||
require.Nil(suite.T(), err)
|
||||
require.Equal(suite.T(), 1, len(podList.Items))
|
||||
|
||||
// Get the raw log for the operator pod
|
||||
opPodName := podList.Items[0].Name
|
||||
rawLog, err := suite.k8sh.Clientset.CoreV1().Pods(sysNamespace).GetLogs(opPodName, &v1.PodLogOptions{}).Do().Raw()
|
||||
require.Nil(suite.T(), err)
|
||||
|
||||
r := regexp.MustCompile(`discovered flexvolume dir path from source.*\n`)
|
||||
logStmt := string(r.Find(rawLog))
|
||||
logger.Infof("flexvolume discovery log statement: %s", logStmt)
|
||||
|
||||
// Verify that the volume plugin dir was discovered by the operator pod and that it did not come from
|
||||
// an env var or the default
|
||||
require.NotEmpty(suite.T(), logStmt)
|
||||
assert.True(suite.T(), strings.Contains(logStmt, "discovered flexvolume dir path from source"))
|
||||
assert.False(suite.T(), strings.Contains(logStmt, "discovered flexvolume dir path from source env var"))
|
||||
assert.False(suite.T(), strings.Contains(logStmt, "discovered flexvolume dir path from source default"))
|
||||
}
|
||||
|
||||
func checkOrderedSubstrings(t *testing.T, input string, substrings ...string) {
|
||||
if len(input) == 0 {
|
||||
// Nothing to compare. An error was likely returned, which should be checked elsewhere.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
#!/bin/bash +e
|
||||
|
||||
KUBE_VERSION=${1:-"v1.8.5"}
|
||||
KUBE_VERSION=${1:-"v1.13.1"}
|
||||
|
||||
null_str=
|
||||
KUBE_INSTALL_VERSION="${KUBE_VERSION/v/$null_str}"-00
|
||||
|
||||
@@ -5,9 +5,9 @@ scriptdir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
|
||||
tarname=image.tar
|
||||
tarfile="${WORK_DIR}/tests/${tarname}"
|
||||
|
||||
export KUBE_VERSION=${KUBE_VERSION:-"v1.8.5"}
|
||||
export KUBE_VERSION=${KUBE_VERSION:-"v1.13.1"}
|
||||
|
||||
if [[ $KUBE_VERSION == v1.12* ]] ; then
|
||||
if [[ $KUBE_VERSION != v1.10* && $KUBE_VERSION != v1.11* ]] ; then
|
||||
skippreflightcheck=--ignore-preflight-errors=all
|
||||
else
|
||||
skippreflightcheck=--skip-preflight-checks
|
||||
@@ -25,15 +25,6 @@ usage(){
|
||||
#install k8s master node
|
||||
install_master(){
|
||||
|
||||
if [[ $KUBE_VERSION == v1.8* ]] ; then
|
||||
# for k8s 1.8, use a non default value for volume plugins
|
||||
cat << EOF | sudo tee -a /etc/systemd/system/kubelet.service.d/11-volume_plugin_dir.conf
|
||||
[Service]
|
||||
Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins"
|
||||
EOF
|
||||
sudo systemctl daemon-reload
|
||||
fi
|
||||
|
||||
sudo kubeadm init $skippreflightcheck --kubernetes-version ${KUBE_VERSION}
|
||||
|
||||
sudo cp /etc/kubernetes/admin.conf $HOME/
|
||||
@@ -66,16 +57,6 @@ EOF
|
||||
#install k8s node
|
||||
install_node(){
|
||||
echo "inside install node function"
|
||||
|
||||
# for k8s 1.8, use a non default value for volume plugins
|
||||
if [[ $KUBE_VERSION == v1.8* ]] ; then
|
||||
cat << EOF | sudo tee -a /etc/systemd/system/kubelet.service.d/11-volume_plugin_dir.conf
|
||||
[Service]
|
||||
Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins"
|
||||
EOF
|
||||
sudo systemctl daemon-reload
|
||||
fi
|
||||
|
||||
echo "kubeadm join ${1} ${2} ${3} ${4} ${5} $skippreflightcheck"
|
||||
sudo kubeadm join ${1} ${2} ${3} ${4} ${5} $skippreflightcheck || true
|
||||
}
|
||||
|
||||
@@ -120,13 +120,11 @@ function copy_images() {
|
||||
}
|
||||
|
||||
# configure minikube
|
||||
# ** K8s v1.12 requires at least minikube v0.30.0 **
|
||||
KUBE_VERSION=${KUBE_VERSION:-"v1.12.2"}
|
||||
KUBE_VERSION=${KUBE_VERSION:-"v1.13.1"}
|
||||
MEMORY=${MEMORY:-"3000"}
|
||||
|
||||
case "${1:-}" in
|
||||
up)
|
||||
# Use kubeadm bootstrapper for 1.8+ since localkube was deprecated in 1.8
|
||||
echo "starting minikube with kubeadm bootstrapper"
|
||||
minikube start --memory="${MEMORY}" -b kubeadm --kubernetes-version "${KUBE_VERSION}"
|
||||
wait_for_ssh
|
||||
|
||||
@@ -36,8 +36,7 @@ MEMORY=${MEMORY:-"3000"}
|
||||
|
||||
case "${1:-}" in
|
||||
up)
|
||||
# Use kubeadm bootstrapper for 1.9+ since localkube was deprecated in 1.8
|
||||
echo "starting minishift with localkube bootstrapper"
|
||||
echo "starting minishift"
|
||||
minishift start --memory=${MEMORY} --vm-driver=virtualbox --iso-url centos
|
||||
wait_for_ssh
|
||||
|
||||
|
||||
Reference in new issue
Block a user