Commit Graph
548 Commits
Author SHA1 Message Date
Satoru Takeuchi 6a09cc0177 ceph: remove unnecessary lvremove in test
LVs in a VG are removed with vgremove. So lvremove before vgremove
is not necessary.

Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
2020-04-13 14:49:05 +09:00
Satoru Takeuchi ea5177c4a1 ceph: suppress shellcheck warnings
Suppress the following shellcheck warnings.

https://github.com/koalaman/shellcheck/wiki/SC2018
https://github.com/koalaman/shellcheck/wiki/SC2019
https://github.com/koalaman/shellcheck/wiki/SC2021

Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
2020-04-07 16:49:58 +00:00
Satoru Takeuchi 7e4f32790d ceph: prevent pre-exisiting lvms from wipe
Ceph integration test tries to wipe all lvms including
the ones that are not related to this test.

Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
2020-04-07 14:09:00 +00:00
Satoru Takeuchi 7bd17c4806 tests: fix minor bugs
- `helm_read` is not a command, but a variable
- If `helm` was installed in `helm.sh up`, the following `heml.sh clean`
  fails since `helm_rest` can't find the installed `helm`.

Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
2020-04-07 14:09:00 +00:00
Sébastien Han e17b6e11d5 ci: use exec combined output to check fs status
On Octopus, the "ceph fs status" reports on stderr for some reason.
So we need to combined the output to read the filesystem status.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-04-06 16:07:01 +02:00
Travis Nielsen a737975077 ceph: integration test for upgrade from nautilus to octopus
The upgrade test will now support upgrading from nautilus
to octopus and that files and be read and written
after the upgrade.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-04-06 16:07:01 +02:00
Travis Nielsen d3a8a397f9 tests: skip upgrade test where csi is not supported
The upgrade test requires the CSI driver to be running so we cannot
run the test on older versions than k8s 1.13.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
(cherry picked from commit f90664209b)
2020-04-03 15:57:19 -06:00
Travis Nielsen 948c0328bd Merge pull request #5162 from travisn/csi-v2.0.1
Enable the Ceph CSI v2.0.1 driver by default in Rook
2020-04-03 13:31:59 -06:00
Travis Nielsen b8e0d8902c Merge pull request #4077 from leseb/rook-multus
ceph: add multus support
2020-04-03 11:14:08 -06:00
Chu Yong Mark Lee a018d89d9c csi: Update cephcsi driver to v2.0.1
Update cephcsi driver to v2.0.1 to allow nodes with
Ceph persistent volumes to shutdown cleanly. This requires
an additional bind mount (/run/mount) in cephfsplugin
and rbdplugin templates.

Signed-off-by: Chu Yong Mark Lee <markleechuyong@gmail.com>
2020-04-03 11:05:41 -06:00
Sébastien Han 20d1543507 ceph: add multus support
You can now use Rook along with Multus. Multus must be up and running
and the right ressources must exist such as NetworkAttachmentDefinition
CR.
The Cluster CR spec has new fields to work with multus:

network:
  provider: multus (or 'host' for hostNetworking)
  selectors:
    public: NetworkAttachmentDefinition name
    cluster: NetworkAttachmentDefinition name

If only a single NetworkAttachmentDefinition is provided Rook will use
both anyway for the Ceph traffic.

Please refer to the doc to learn more.

Closes: https://github.com/rook/rook/issues/4716
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-04-03 18:00:15 +02:00
Rohan CJ ecef80eda5 Ceph: Use both "apps" and "extensions" for deployment RBAC.
Some versions of k8s can optionally enable extensions after deprecation.
This could lead to rbac denials when trying to perform operations on deployments

Signed-off-by: Rohan CJ <rohantmp@gmail.com>
2020-04-03 19:10:45 +05:30
Santosh Pillai 1673291589 ceph: Integration Tests: moving GetCleanupPod() and GetCleanupVerificationPod() methods out of CephManifests interface
The CephManifests interface is intended for manifests that install the product (RBAC, CRDs, operator, etc). These cleanup methods are more about test code. The test code needs to behave the same whether it is a normal install or an upgrade. We don't need to version it differently for upgrades between the CephManifestsMaster and CephManifestsV1_1 implementations

Signed-off-by: Santosh Pillai <sapillai@redhat.com>
2020-04-03 13:54:52 +05:30
Santosh Pillai e1a76bfb96 ceph: Cluster CleanupPolicy: Delete correct mon directories under the dataDirHostPath
In case of multiple clusters, we don't want to delete all the mon directories under the dataDirHostPath during cluster cleanup.
This PR deletes the mon directory only if the montior secret key matches.

Signed-off-by: Santosh Pillai <sapillai@redhat.com>
2020-04-03 10:01:36 +05:30
Travis Nielsen f30c8066d1 docs: update docs for Rook v1.3 release
Various updates are needed for the v1.3 release.
- The pending release notes were missing some features
- Added a section to the upgrade guide for breaking changes to OSDs
- Clarifications around Ceph prereqs and min version
- Other misc clarifications

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-04-02 10:26:00 -06:00
ShyamsundarR de97b6a1d6 ceph: Add ability to specify compression mode for pools
This commit enables configuring a compression_mode for pools
that takes the values as specified for bluestore OSD
compression_mode values,
https://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref/#inline-compression

Signed-off-by: ShyamsundarR <srangana@redhat.com>
2020-04-01 14:24:30 -04:00
Sébastien Han 6062771687 Merge pull request #4901 from travisn/octopus-tests
Run integration tests against octopus
2020-03-31 10:59:11 +02:00
Travis Nielsen ba3e601a50 ceph: update k8s versions for ceph tests on 1.18
With 1.18 in the test matrix we need to update which versions
will run the different ceph test suites.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-03-30 16:51:02 -06:00
Travis Nielsen 360e5552b6 ceph: run integration tests against octopus
With the release of Octopus we need to run the integration tests
on Octopus in order to have validation as a supported platform.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-03-30 13:32:47 -06:00
Santosh Pillai 2bfd7c42bc ceph: cleanup cluster.Spec.DataDirHostPath on cluster deletion
In order to ensure proper clean up of all the rook-ceph data when the cluster is deleted, we need to clean up the dataDirHostPath (var/lib/rook)

Signed-off-by: Santosh Pillai <sapillai@redhat.com>
2020-03-30 19:36:01 +05:30
Elise Gafford 2ca0d8f93d ceph: add cleanupPolicy to cephcluster spec
In order to perform full automation of cephcluster deletion safely
and protect against catastrophic data loss, the end user must be
able to signify that they intend to irrecoverably delete the data
in their cluster. The cleanupPolicy field of the cluster spec is
intended to communicate this. This patch does not implement
automated deletion, but only creates the spec field and the
safety feature of halting orchestration other than deletion on a
cluster with a set cleanup policy value.

Partially-fixes: #3222
Signed-off-by: Elise Gafford <egafford@redhat.com>
2020-03-30 19:11:54 +05:30
Travis Nielsen bef11804b8 flex: enable flex driver in the integration tests
The flex agent was mistakenly disabled in the CSI config change
in this PR: https://github.com/rook/rook/pull/5032. The agent must
be enabled for integration tests.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-03-27 16:26:22 -06:00
Sébastien Han 6f70c9bd5a Merge pull request #5101 from cybozu-go/check-lv-availability
ceph: check LV availability by "ceph-volume lvm list"
2020-03-27 16:43:48 +01:00
Travis Nielsen cf7a910df6 Merge pull request #5062 from travisn/test-14.2.8
Run Ceph integration tests on latest nautilus
2020-03-27 09:21:28 -06:00
Sébastien Han 9c343ae94e Merge pull request #5032 from umangachapagain/config-override
Ceph: add CSI configurations to ConfigMap
2020-03-27 15:21:24 +01:00
Travis Nielsen 9cecf9e4d5 ceph: better attempt in integration tests to wipe the disks
The integration tests are failing to run when the disks
are not properly cleaned up and the bluestore label
is still found. Now we run sgdisk to more completely
zap the disk.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-03-27 07:51:10 -06:00
Travis Nielsen eb1e4beff8 ceph: run integration tests on latest nautilus
Run the tests on latest nautilus with the tag ceph/ceph:v14 so we
are constantly testing on the latest release.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-03-27 07:48:54 -06:00
Umanga Chapagain ac9244fc1a Ceph: integration test creates ObjectStoreUser before ObjectStore
This fix is added to ensure that everything works as expected
when user tries to create ObjectStoreUser before creating
ObjectStore itself. ObjectStoreUser will wait for ObjectStore
to be up and running.

Signed-off-by: Umanga Chapagain <chapagainumanga@gmail.com>
2020-03-27 18:16:28 +05:30
morimoto-cybozu ab83738aac ceph: fix device path passed to "ceph-volume inventory"
This commit fixes the argument for "ceph-volume inventory".
When a device "/dev/mapper/foo" is being checked for its availability,
the argument should not be "/dev/foo" nor "/dev/dm-1".

Signed-off-by: morimoto-cybozu <kenji_morimoto@cybozu.co.jp>
2020-03-27 11:24:47 +00:00
Umanga Chapagain 0e932c15eb Ceph: add CSI configurations to ConfigMap
This commit adds all the CSI configurations to ConfigMap.
This configMap can be used in combination with Env Vars
to configure Ceph CSI drivers in rook.

Signed-off-by: Umanga Chapagain <chapagainumanga@gmail.com>
2020-03-27 15:17:30 +05:30
Sébastien Han a044b27650 ci: fix MockExecuteCommandWithOutput mock definition
With the recent exex update we missed this one.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-03-25 17:24:30 +01:00
Travis Nielsen da3bf49b61 Merge pull request #5043 from cybozu-go/remove-unnecessary-napespace-fields
remove unnecessary namespace fields
2020-03-20 17:18:21 -06:00
Sébastien Han ca0a30f38d ceph: convert Filesystem controller to the controller-runtime
The CRD watcher has been replaced by the new controller-runtime
framework.
This brings robustness in our operator, meaning that any resources that
are modified will be reconciled into the desired state.

Closes: https://github.com/rook/rook/issues/4940
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-03-19 23:34:58 +01:00
Travis Nielsen e8f9cfcb71 exec: always write commands to debug log
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-03-19 07:49:54 -06:00
Travis Nielsen f2ecaa2bda exec: remove the unused actionName param
The helpers for executing a process have long required an actionName
param which is not being used. Now we remove the old param
while also cleaning up various other usages of the exec
package.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-03-19 07:49:53 -06:00
Travis Nielsen 84b8cdcf75 exec: simplify the exec package from unused methods and logging
The methods and arguments to the exec methods are not all used anymore.
This cleans up the methods to only what is necessary to improve
the readability and maintainability.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-03-19 07:49:53 -06:00
Travis Nielsen 18b0e7d295 ceph: scrub ceph commands to write actions to the log
The ceph commands are now only written to the log in debug mode.
For commands that change the system state we now ensure that
a useful log entry is written. If all the details of the ceph
commands are needed, debug logging should still be enabled.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-03-19 07:49:53 -06:00
Satoru Takeuchi a5ea2858c9 manifests: remove unnecessary namespace fields
There are many namespace fields in ClusterRole{,Binding}. However,
ClusterRole{,Binding} are not namespaced. So we can remove these.

Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
2020-03-18 15:58:31 +00:00
Travis Nielsen ce1fedf884 ci: only list the images in the needed pool during cleanup
The pool cleanup only needs to happen for an individual pool.
No need to query the block images in all pools. One of the rgw
pools is periodically causing a hang when it is queried,
but there is no need to query for it when we are cleaning
up the pool tests.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-03-17 15:28:17 -06:00
Sébastien Han a8159c62c1 ci: enable DEBUG logging
Activate DEBUG logging for the integration tests.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-03-17 15:12:41 -06:00
Sébastien Han e3b3aa7702 ci: bump upgrade to 1.2.6
Let's upgrade from 1.1.9 to 1.2.6

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-03-17 15:12:41 -06:00
Sébastien Han 44a4c1826d ci: remove object store when done
Purge the object store properly, otherwise the cephobjectstore CRD won't
be deleted since a finalizer is in place for the CR.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-03-17 15:12:41 -06:00
Sébastien Han f268c897e9 ceph: Convert the Ceph ObjectStore controller to the controller-runtime
The CRD watcher has been replaced by the new controller-runtime
framework.
This brings robustness in our operator, meaning that any resources that
are modified will be reconciled into the desired state.

Closes: https://github.com/rook/rook/issues/4937
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-03-17 15:12:41 -06:00
moricho 7d0215d820 ci: switch to use go.mod.check target
This switches to use `go.mod.check` target instead of `go.vendor.check` target.

Signed-off-by: moricho <ikeda.morito@gmail.com>
2020-03-17 16:48:43 +09:00
Travis Nielsen 9c5c78cf01 Merge pull request #5025 from jmolmo/blinking_lights
Ceph: Added rook manager module rights for read pod logs
2020-03-16 13:57:13 -06:00
Travis Nielsen 017a560db3 Merge pull request #5021 from samkulkarni20/yb-mem-limits
YugabyteDB: Add resource limits to YugabyteDB pods
2020-03-16 11:12:07 -06:00
Juan Miguel Olmo Martínez c87a01bb8c Ceph: Added rook manager module rights for read pod logs
This is needed to allow the ceph manager rook module to read the output
of the <lsmcli> command used to switch disk identification light on/off on
physical disk devices

[test ceph]

Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
2020-03-16 18:03:23 +01:00
Travis Nielsen 2724791d6c Merge pull request #5019 from HaichaoLii/arm64
test: Kubeadm-install add support for arm64
2020-03-16 07:45:39 -06:00
Haichao Li 24027b969e test: Kubeadm-install add support for arm64
Only kubectl for amd64 is currently available.This modification adds
arm64 version of kubectl that can successfully deploy kubernetes on
arm64 machine.

Signed-off-by: Haichao Li <haichao.li@arm.com>
2020-03-16 16:15:59 +08:00
Sameer Kulkarni 909770a346 YugabyteDB: Add resource limits to YugabyteDB pods
Current YugabyteDB operator code creates Master and TServer pods without any resource requests/limits (specifically CPU and memory).
This causes the operator to run into soft/hard memory limit issue. The fix adds recommended resource requests and limits as defaults to the pods it creates.

Closes: https://github.com/yugabyte/yugabyte-db/issues/3884
Signed-off-by: Sameer Kulkarni <samkulkarni20@gmail.com>
2020-03-14 12:23:14 +05:30