Commit Graph
4610 Commits
Author SHA1 Message Date
Travis Nielsen 3bd9c0eca9 Merge pull request #5158 from sp98/minior-integration-test-changes
Integration Tests: moving GetCleanupPod() and GetCleanupVerificationPod() methods out of CephManifests interface
2020-04-03 07:37:46 -06:00
Santosh Pillai 1673291589 ceph: Integration Tests: moving GetCleanupPod() and GetCleanupVerificationPod() methods out of CephManifests interface
The CephManifests interface is intended for manifests that install the product (RBAC, CRDs, operator, etc). These cleanup methods are more about test code. The test code needs to behave the same whether it is a normal install or an upgrade. We don't need to version it differently for upgrades between the CephManifestsMaster and CephManifestsV1_1 implementations

Signed-off-by: Santosh Pillai <sapillai@redhat.com>
2020-04-03 13:54:52 +05:30
Travis Nielsen 5f44522146 Merge pull request #5134 from sp98/mon-dir-cleanup
Cluster CleanupPolicy: Delete correct mon directories under the dataDirHostPath
2020-04-02 23:48:53 -06:00
Santosh Pillai e1a76bfb96 ceph: Cluster CleanupPolicy: Delete correct mon directories under the dataDirHostPath
In case of multiple clusters, we don't want to delete all the mon directories under the dataDirHostPath during cluster cleanup.
This PR deletes the mon directory only if the montior secret key matches.

Signed-off-by: Santosh Pillai <sapillai@redhat.com>
2020-04-03 10:01:36 +05:30
Travis Nielsen 843390a4fc Merge pull request #5152 from cybozu-go/ceph-remove-unnecessary-whitespace-in-the-example
ceph: remove unnecessary whitespaces in the example
2020-04-02 16:10:47 -06:00
Satoru Takeuchi e4812f36d0 ceph: remove unnecessary whitespaces in the example
Fix the following error message due to unnecessary whitespaces.

```console
$ kubectl create -f cluster-on-pvc.yaml
error: error parsing cluster-on-pvc.yaml: error converting YAML to JSON: yaml: line 85: mapping values are not allowed in this context
```

Closes: #5149
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
2020-04-02 21:36:22 +00:00
Travis Nielsen c79325f92c Merge pull request #5138 from cybozu-go/support-topology-spread-constraints
ceph: support topoplogySpreadConstraints
2020-04-02 12:00:45 -06:00
Satoru Takeuchi 06b10e248c ceph: support topoplogySpreadConstraints
Support topologySpreadConstraints to spread OSD pods among failure
domains as even as possible.

https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/

Without this feature, the number of OSD pods would be different
between the failure domains very much. At worst, there is
a possibility that a failure domain has many OSDs and other
failure domains have no OSDs. It can partly be mitigated
with podAntiAffinity. However, it only works if there are at most
one OSD per node. For more detail, please refer to the following
document.

https://github.com/rook/rook/blob/master/cluster/examples/kubernetes/ceph/cluster-on-pvc.yaml#L59

Closes: https://github.com/rook/rook/issues/4387
Signed-off-by: Hiroshi Muraoka <h.muraoka714@gmail.com>
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
2020-04-02 16:40:58 +00:00
Travis Nielsen 52ef25827b Merge pull request #5133 from travisn/v1.3-doc-updates
Updated docs need for the Rook v1.3 release
2020-04-02 10:32:55 -06:00
Travis Nielsen b51291af00 Merge pull request #5140 from cloudical-io/go_114
build: allow go 1.14
2020-04-02 10:29:51 -06:00
Travis Nielsen f30c8066d1 docs: update docs for Rook v1.3 release
Various updates are needed for the v1.3 release.
- The pending release notes were missing some features
- Added a section to the upgrade guide for breaking changes to OSDs
- Clarifications around Ceph prereqs and min version
- Other misc clarifications

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-04-02 10:26:00 -06:00
Blaine Gardner 9d41b38095 Ceph: upgrade docs: update upgrade docs for v1.3
Update both documents and `upgrade-from-v1.2*.yaml` files for Rook v1.3
release.

Of note:
- note when the upgrade to CSI driver v2.0 should be updated
- note that CSI image envs can be removed to have Rook use defaults
- update Ceph version upgrade section
- update supported Ceph version info
- update CRDs before upgrade b/c new 'status' fields
- remove special CSI update info left from v1.2

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
(cherry picked from commit 2ccff83d94)
2020-04-02 10:26:00 -06:00
Travis Nielsen 6b38dd3a6a Merge pull request #4933 from leseb/rm-pid-host
ceph: do not use host PID anymore
2020-04-02 07:29:04 -06:00
Sébastien Han 58f1971172 ceph: do not use host PID anymore
Thanks to https://github.com/ceph/ceph/pull/33633, when running on
Octopus, we don't need to use `--pid=host`. This means we are not using
the host PID but use the PID namespace of the pod.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-04-01 15:42:53 -06:00
Travis Nielsen 14fcbcb151 Merge pull request #5124 from ShyamsundarR/ceph-pool-compression
ceph: Add ability to specify compression mode for pools
2020-04-01 14:29:00 -06:00
ShyamsundarR de97b6a1d6 ceph: Add ability to specify compression mode for pools
This commit enables configuring a compression_mode for pools
that takes the values as specified for bluestore OSD
compression_mode values,
https://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref/#inline-compression

Signed-off-by: ShyamsundarR <srangana@redhat.com>
2020-04-01 14:24:30 -04:00
Alexander Trost a85df12b73 build: allow go 1.14
This adds v1.14 to the supported Golang versions in the makefile.

Signed-off-by: Alexander Trost <galexrt@googlemail.com>
2020-04-01 15:11:35 +02:00
ShyamsundarR 7ce1c4fc71 build: Fix ability to build with podman
This commit corrects choosing DOCKERCMD in image.mk file to choose
the environment variable if present, else default to "docker".

Signed-off-by: ShyamsundarR <srangana@redhat.com>
2020-04-01 08:19:51 -04:00
Sébastien Han ed37a6a699 Merge pull request #5128 from leseb/liveness-other-daemons
ceph: add liveness probe to mon, mds and osd daemons
2020-04-01 10:53:12 +02:00
Sébastien Han 040193bb5a ceph: remove DaemonType type
This type was a string already and was just making us doing string()
calls all the time to it's not worth it.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-04-01 09:08:18 +02:00
Sébastien Han 7788901a88 ceph: add liveness probe to mon, mds and osd daemons
Now Kubernetes will perform liveness checks on mon, mds and osd daemons.
The command will:

* call the socket (check for existence)
* execute a command and check the return code (success if 0)

This handles the case where the daemon is stuck locally and
unresponsive. It's unlikely but not impossible.
These checks bring more robustness to the implementation.

rbd-mirror and nfs have been leftover for the following reason. The
rbd-mirror socket name is different from other daemons (could be fixed
though): /run/ceph/ceph-client.rbd-mirror.a.1.94362516231272.asok also,
the command to call would need to be changed from "status" to "rbd
mirror status" so we can keep this for a later.
The nfs ganesha has no socket only a PID file which doesn't mean much.
No PID means the process does not run so Kubernetes will already handle
this and the pod will crash loop.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-03-31 15:20:42 +02:00
Sébastien Han 6062771687 Merge pull request #4901 from travisn/octopus-tests
Run integration tests against octopus
2020-03-31 10:59:11 +02:00
Sébastien Han 04c78135bb Merge pull request #5125 from travisn/ceph-test-versions
Update K8s versions for ceph tests with K8s 1.18
2020-03-31 09:13:28 +02:00
Travis Nielsen ba3e601a50 ceph: update k8s versions for ceph tests on 1.18
With 1.18 in the test matrix we need to update which versions
will run the different ceph test suites.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-03-30 16:51:02 -06:00
Travis Nielsen dcc1e2a291 Merge pull request #5117 from travisn/test-k8s-1.18
Test rook on K8s 1.18
v1.3.0-beta.0
2020-03-30 16:48:04 -06:00
Travis Nielsen 9039c674ce Merge pull request #5123 from travisn/dirty-build
Tidy the go modules before the release build
2020-03-30 16:05:24 -06:00
Travis Nielsen 0a5a17cf51 build: tidy the go modules before the release build
The go modules are changed by the install of the junit
tool for the unit tests. Before the release build we need
to ensure that the modules are tidied.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-03-30 15:33:28 -06:00
Travis Nielsen 32d6c78e12 Merge pull request #5121 from travisn/git-restore-cmd
Use git checkout instead of git restore
2020-03-30 13:37:12 -06:00
Travis Nielsen 0e55aa348b tests: reduce unhelpful debug logging
Removed some extremely verbose debug logging that makes
the logs difficult to read when debug is enabled.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-03-30 13:32:47 -06:00
Travis Nielsen 360e5552b6 ceph: run integration tests against octopus
With the release of Octopus we need to run the integration tests
on Octopus in order to have validation as a supported platform.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-03-30 13:32:47 -06:00
Travis Nielsen 928e901ec5 Merge pull request #5113 from leseb/rgw-port-sdn
ceph: use a different port for rgw on sdn
2020-03-30 13:32:30 -06:00
Dmitry Yusupov c2f175e266 Merge pull request #5120 from dyusupov/master
edgefs: TZ needs to be exposed
2020-03-30 12:30:35 -07:00
Travis Nielsen defa687b3d Merge pull request #4959 from sp98/hostPath-cleanUp
Cleanup cluster.Spec.DataDirHostPath on cluster deletion
2020-03-30 13:07:08 -06:00
Travis Nielsen 0e1ccce61a build: use git checkout instead of git restore
The CI is using an older version of git that doesn't
recognize git restore.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-03-30 13:05:25 -06:00
Dmitry Yusupov 080510bb6f edgefs: TZ needs to be exposed
/etc/timezone now exposed by operator

Signed-off-by: Dmitry Yusupov <dmitry.yusupov@nexenta.com>
2020-03-30 11:42:45 -07:00
Travis Nielsen f3e1a3407b Merge pull request #5115 from travisn/dirty-build
Check for modified files in build
2020-03-30 12:23:39 -06:00
Travis Nielsen 9a615eef99 build: check for modified files in build
The master build is publishing with the dirty tag so we
need to ensure that the files do not remain modified.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-03-30 12:15:53 -06:00
Sébastien Han f136105951 ceph: use a different port for rgw on sdn
When running on SDN, the container is not privileged and the network
stack is not exposed so running the gw on port 80 will not work (not
enough privileged).

Closes: https://github.com/rook/rook/issues/5106
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-03-30 19:37:11 +02:00
Travis Nielsen a3a7a10236 build: test rook on k8s 1.18
With the release of k8s 1.18 we now will run the integration
tests on 1.14 through 1.18 in master.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-03-30 11:14:14 -06:00
Santosh Pillai 2bfd7c42bc ceph: cleanup cluster.Spec.DataDirHostPath on cluster deletion
In order to ensure proper clean up of all the rook-ceph data when the cluster is deleted, we need to clean up the dataDirHostPath (var/lib/rook)

Signed-off-by: Santosh Pillai <sapillai@redhat.com>
2020-03-30 19:36:01 +05:30
Elise Gafford 2ca0d8f93d ceph: add cleanupPolicy to cephcluster spec
In order to perform full automation of cephcluster deletion safely
and protect against catastrophic data loss, the end user must be
able to signify that they intend to irrecoverably delete the data
in their cluster. The cleanupPolicy field of the cluster spec is
intended to communicate this. This patch does not implement
automated deletion, but only creates the spec field and the
safety feature of halting orchestration other than deletion on a
cluster with a set cleanup policy value.

Partially-fixes: #3222
Signed-off-by: Elise Gafford <egafford@redhat.com>
2020-03-30 19:11:54 +05:30
Sébastien Han bd850252e5 Merge pull request #5110 from moricho/fix/codegen
Fix `codegen` to find boilerplate outside of the $GOPATH
2020-03-30 15:02:34 +02:00
Sébastien Han bc6186dabe Merge pull request #5108 from AdlerFleurant/operator-openshift-env-fix
ceph: Removed 2nd env variable ROOK_HOSTPATH_REQUIRES_PRIVILEGED
2020-03-30 14:53:40 +02:00
Adler Fleurant 70e7f39db0 ceph: Removed 2nd env variable ROOK_HOSTPATH_REQUIRES_PRIVILEGED
It is not necessary to have 2 same environment variables and not desired as
changing only one may lead to the second variable being applied with unexpected
value. It is better to have only 1 ROOK_HOSTPATH_REQUIRES_PRIVILEGED environment
variable.

Closes: https://github.com/rook/rook/issues/5107
Signed-off-by: Adler Fleurant <Adler.Fleurant@PicoChange.com>
2020-03-30 07:50:30 -04:00
moricho f556add260 codegen: use the bopilerplate for the header of the generated codes
This specifies `--output-base` and `--go-header-file` option at executing `code-generator`.

Signed-off-by: moricho <ikeda.morito@gmail.com>
2020-03-30 17:54:56 +09:00
Travis Nielsen a50bab382a Merge pull request #5105 from travisn/flex-test-enable-agent
Reenable flex driver in the integration tests
2020-03-27 19:59:46 -06:00
Travis Nielsen bef11804b8 flex: enable flex driver in the integration tests
The flex agent was mistakenly disabled in the CSI config change
in this PR: https://github.com/rook/rook/pull/5032. The agent must
be enabled for integration tests.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-03-27 16:26:22 -06:00
Travis Nielsen 90e0c44e72 Merge pull request #5102 from travisn/test-dirty-build
Ensure files are not modified during build
2020-03-27 14:32:26 -06:00
Travis Nielsen 816d8286e4 build: ensure files are not modified during build
The recent change to go modules is causing a file to change
during the build, which results in a dirty tag to be
generated.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-03-27 14:14:15 -06:00
Travis Nielsen 7bbc559e86 Merge pull request #5098 from travisn/metadata-pool-pgs-master
Set pg_num for rgw metadata pools to lower default
2020-03-27 13:02:24 -06:00