Commit Graph
3078 Commits
Author SHA1 Message Date
Travis Nielsen 87fd4e44bf Merge pull request #3467 from travisn/provider-integration-tests
Allow executing integration tests for specific storage providers
2019-07-18 13:29:12 -06:00
Sébastien Han 6ce0532768 Merge pull request #2985 from invidian/agent-discover-tolerations
ceph: allow to set multiple tolerations for agent and discover pods
2019-07-18 09:56:45 +02:00
travisn 74a3eb6180 Rename the ceph integration test entry points
With the storage providers growing, we should make it
clear in the source which tests belong to which storage
provider. This change renames the ceph integration tests.

Signed-off-by: travisn <tnielsen@redhat.com>
2019-07-17 23:29:52 -06:00
travisn 807326049e fix the links in the PR checklist
The relative links in the PR template are broken so we
need to specify full paths.

Signed-off-by: travisn <tnielsen@redhat.com>
2019-07-17 23:29:52 -06:00
travisn d40e490444 allow executing tests specific to a storage provider
Since most PRs are focused on a single storage provider,
the tests for other storage providers do not need to be
run. The CI will allow skipping tests for other storage
providers by specifying keywords in the PR description.
Similar to [skip ci], you can now specify [test storage-provider].
For example, [test ceph] will only trigger the test
suites for ceph and none of the other storage providers.

Signed-off-by: travisn <tnielsen@redhat.com>
2019-07-17 23:29:52 -06:00
Mateusz Gozdek f936394764 ceph: allow to set multiple tolerations for agent and discover pods
Closes #2896

Signed-off-by: Mateusz Gozdek <mgozdekof@gmail.com>
2019-07-17 23:11:18 +02:00
Dmitry Yusupov 01ba50ea3c Merge pull request #3463 from sabbot/node-add-remove
Add/remove nodes via EdgeFS cluster' CRD
2019-07-17 14:01:03 -07:00
Travis Nielsen 41005b53a6 Merge pull request #3456 from travisn/remove-legacy-conversion
Remove legacy Ceph CRD conversion from v1beta1 to v1
2019-07-17 11:52:41 -06:00
Anton Skriptsov aea098b8b5 Add/remove nodes via EdgeFS cluster' CRD
Signed-off-by: Anton Skriptsov <sabbotagge@gmail.com>
2019-07-17 09:09:52 -07:00
Sébastien Han 50ef726ad3 Merge pull request #2901 from leseb/improve-upgrade
ceph: improve upgrade procedure
2019-07-17 11:54:22 +02:00
Sébastien Han 05d8d23f18 ceph: upgrade: pass daemonType and Name directly
We do not determine the daemon type and name by string parsing anymore
but passing that info directly in the update call, it's more reliable.
Also removing a bunch of unused functions.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-07-17 00:58:31 +02:00
Sébastien Han a17c47d36a ceph: upgrade: preserve operator state
This commit makes OnAdd() and OnUpdate() consistent in behavior. They
will do their best detecting if we can upgrade, continue an upgrade or
refuse to do one. It also refuses to downgrade to any version.
It also load ClusterInfo earlier in the controller so that we can
compare the spec version versus the cluster running one.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-07-17 00:50:42 +02:00
Alexander Trost 38c3e6f5bb Fix mgr prometheus annotations (#3453)
Fix mgr prometheus annotations
2019-07-17 00:46:21 +02:00
Sébastien Han c24ae45eca ceph: upgrade, don't check osd if aio
If we detect all OSDs are running on the same host, we don't perform any
checks for upgrade.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-07-17 00:41:40 +02:00
Sébastien Han 40d37a165f test framework: collect logs earlier
Logs will be lost if we collect them after pods are uninstalled.
So we need to collect them before purging.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-07-17 00:39:18 +02:00
Sébastien Han 25f4f889c5 ceph: upgrade: display success when upgrade is done
If we successfully changed all the Ceph version of all the daemon we
consider the upgrade complete and successfully, thus we display a
message.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-07-17 00:39:18 +02:00
Sébastien Han 7eb7e8a192 ceph: upgrade: always retrieve cluster version
Do not consider clusterInfo Ceph version since it's the one picked by
the spec image.
The CephVersion provided here needs to be the current running version
of Ceph for when it is later passed to OkToStop and checked with
IsAtLeast. If you are upgrading from a pre 14.2.1 version to a
version 14.2.1 or newer, IsAtLeast will incorrectly return true and
then fail trying to run mon ok-to-stop on a mon pod running a pre 14.2.1 version.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-07-17 00:39:18 +02:00
Sébastien Han 7d86069f6e doc: ceph: enhance upgrade section
Add more details on how upgrades are performed, as well as updating the
release note.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-07-17 00:39:18 +02:00
Sébastien Han bd7df53b18 ceph: refuse to upgrade an un-healthy cluster
If the cluster shows a state of HEALTH_ERR, Rook will refuse to upgrade
it if the image version changed.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-07-17 00:39:15 +02:00
Sébastien Han f317cfdd1c ceph: osd: remove leftover info logs
These were used on a previous patch as a debug, let's remove them. They
are not intended to be displayed.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-07-17 00:37:43 +02:00
Sébastien Han 32a9435c29 ceph: rgw: rename startDeployment()
This method does not start anything but creates the deployment. So let's
rename it to createDeployment() instead.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-07-17 00:37:43 +02:00
Sébastien Han ae9d33061c rgw: implement UpdateDeploymentAndWait()
rgw wasn't using UpdateDeploymentAndWait() to update its deployment
configuration. We now use it so that it's consistent with the other
daemons configuration.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-07-17 00:37:43 +02:00
Sébastien Han 9824bd2ff9 ceph: improve upgrade procedure
When a cluster is updated with a different image version, this triggers
a serialized restart of all the pods. Prior to this commit, no safety
check were performed and rook was hoping for the best outcome.

Now before doing restarting a daemon we check it can be restarted. Once
it's restarted we also check we can pursue with the rest of the
platform. For instance, with monitors we check that they are in quorum,
for OSD we check that PGs are clean and for MDS we make sure they are
 all active.

Fixes: https://github.com/rook/rook/issues/2889
Signed-off-by: Sébastien Han <seb@redhat.com>
2019-07-17 00:37:43 +02:00
Sébastien Han 487ded50f3 Merge pull request #3355 from noahdesu/mon-sched2
ceph: schedule mons across failure domains
2019-07-16 22:47:07 +02:00
Dinar Valeev 38c4e5ed7b Mgr: fix prometheus annotations
This fixes https://github.com/rook/rook/issues/3204

Move prometheus Annoation logic before Deployment is initialized,
otherwise, podSpec is not uptodate

Signed-off-by: Dinar Valeev <k0da@opensuse.org>
2019-07-16 17:01:27 +02:00
Sébastien Han 22e457e57a Merge pull request #3454 from SUSE/minor-error-fix
cmdreporter: fix minor error bug
2019-07-16 16:34:37 +02:00
Sébastien Han 6d40c000c5 Merge pull request #3435 from leseb/backport-script
contrib: add an helper script to perform backports
2019-07-16 15:57:14 +02:00
travisn 9e073857c5 ceph: remove legacy conversion from v1beta1 to v1
In the 0.9 release rook converted the v1beta1 CRD resources
to v1 resources. This conversion code is no longer necessary
as we will be using v1 resources going forward.
The code paths will not be triggered anymore, thus
removing the dead code.

Signed-off-by: travisn <tnielsen@redhat.com>
2019-07-15 16:58:37 -06:00
Blaine Gardner aa6925ae79 cmdreporter: fix minor error bug
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2019-07-15 10:08:51 -06:00
Alexander Trost dfb9714257 github: re-add default template for new issues (#3440)
github: re-add default template for new issues
2019-07-14 20:08:52 +02:00
Blaine Gardner 823018b1c8 Merge pull request #3429 from SUSE/upgrade-from-manifests
ceph manifests: update upgrade-from yaml files
2019-07-12 14:31:25 -06:00
Travis Nielsen 2317ce6ecc Merge pull request #3441 from krig/k8sutil-cleanup
k8sutil: 1. Use label max length constant from k8s, 2. Add missing docstrings in pod.go
2019-07-12 11:45:31 -06:00
Kristoffer Grönlund f79fe5fa28 k8sutil: Docstrings for some funcs in pod.go
Add missing docstrings for container image
functions in pod.go, noting the behavior
when getting a container image from a pod
which only has a single container defined.

Signed-off-by: Kristoffer Grönlund <kgronlund@suse.com>
2019-07-12 14:12:57 +02:00
Kristoffer Grönlund 3873251ce6 k8sutil: Use label max length constant from k8s
Follow-up to #3436:

The validation module has a defined constant for
label value max length. It makes sense to use it
here.

Signed-off-by: Kristoffer Grönlund <kgronlund@suse.com>
2019-07-12 13:37:58 +02:00
Sébastien Han 571eaf2308 github: re-add default template for new issues
This is not a recommended workflow but useful if people click on "Don’t
see your issue here? Open a regular issue."

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-07-12 12:18:32 +02:00
Sébastien Han 35d6f73d5e Merge pull request #3432 from leseb/checklist-pr
PR checklist and Issue template update
2019-07-12 12:06:25 +02:00
Sébastien Han 562633ce99 Merge pull request #3436 from krig/label-validation
k8sutil: Ensure rook version label has valid syntax
2019-07-12 11:35:52 +02:00
Kristoffer Grönlund 6ff69a4e72 k8sutil: Ensure rook version label has valid syntax
Replace any invalid characters and ensure length
and format for the rook version label, to avoid
problems with version strings that contain
illegal characters (common characters that aren't
valid in labels would be + and ~, for example).

Kubernetes label syntax:
https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#syntax-and-character-set

Fixes: #3434
Signed-off-by: Kristoffer Grönlund <kgronlund@suse.com>
2019-07-12 10:20:30 +02:00
Sébastien Han b25b3c68c8 contrib: add an helper script to perform backports
Run the script like this:

./contrib/backport_to_stable_branch.sh STABLE_BRANCH_NAME COMMIT_SHA1 BACKPORT_BRANCH_NAME

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-07-12 09:15:07 +02:00
Sébastien Han fdaecc263b Merge pull request #3406 from travisn/template-skip-ci
Add documentation for skipping the CI
2019-07-12 08:47:02 +02:00
Blaine Gardner db09e58c4e ceph manifests: update upgrade-from yaml files
Update the upgrade-from-*.yaml files to be appropriate for upgrade from
v1.0 to master. Currently no creations or deletions are necessary, but
the files are kept as a placeholder for the upcoming v1.1 upgrade docs.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2019-07-11 17:04:32 -06:00
Blaine Gardner 582fe014e4 Merge pull request #3438 from SUSE/fix-merge-bug-in-master
Ceph: Fix crit bug in call to detect Ceph version
2019-07-11 17:02:17 -06:00
travisn 9af20e8dc0 Add documentation for skipping the CI
The CI only needs to be run for PRs that have code changes.
PRs that are only documentation, yaml, or other files
that are not built should skip the long and tedious
CI process. This makes the [skip ci] flag more discoverable.

Signed-off-by: travisn <tnielsen@redhat.com>
2019-07-11 16:40:40 -06:00
Blaine Gardner cc359b6d6b Ceph: Fix crit bug in call to detect Ceph version
For all calls to detect the Ceph version, ensure they are consistent.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2019-07-11 16:37:45 -06:00
Blaine Gardner c2225ce453 Merge pull request #3420 from rohan47/safe_to_destroy
Updated condition check to avoid panic
2019-07-11 16:10:16 -06:00
Blaine Gardner 5fc99a6266 Merge pull request #3369 from SUSE/cmd-reporter
Add cmd-reporter daemon; use it to get ceph version
2019-07-11 16:05:14 -06:00
Noah Watkins f034f000e1 ceph: schedule mons across failure domains
this patch introduces failure-domain aware scheduling for monitor pods.
previously the only policy was to avoid scheduling monitors on the same
nodes. the new policy tries to spread monitors across zones and nodes.

   NOTE: this patch only enforces the well-known k8s "zone"
   failure-domain label (i.e. it does not look for region labels). this
   decision is made for two reasons. first, eliminating explicit
   scheduling is an important goal and will be done as part of upcoming
   work. this will enable all supported failure-domain labels to be
   handled. second, the "zone" domain is likely more appropriate for a
   larger portion of users (as opposed to the geographical distinction
   made with region), and supporting multiple failure domain types would
   add significant complexity for a temporary resolution to failure
   domain scheduling.

at a high-level this patch introduces a data structure `NodeUsage` that
is a pairing of a v1.Node and metadata relevant to scheduling: number of
monitor pods on the node, and its status w.r.t. to being scheduable for
new monitor pods.

each scheduling event computes a unified data structure that organizes
for each node a `NodeUsage` structure into per-failure-zone groups.
subsequent algorithms rely soley on this unified structure to make
scheduling decisions.

there are two entry points to the changes made:

1) during orchestration mon:Cluster:assignMons is invoked to make a
place new monitor pods onto nodes. the core scheduling logic is now
self-contained in mon:scheduleMonitor which implements node and zone
aware scheduling policies.

2) periodic health checks resolve any conflicts that may occur due to
changes to the cluster (e.g. new nodes, configuration changes, etc...).
the two existing checks: mons on invalid nodes, and overloaded nodes are
both handled.

fixes: #2603

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
2019-07-11 14:23:15 -07:00
Noah Watkins 60cace8f7b util: add ValidNode variant without sched check
this adds a variant of ValidNode that doesn't check for no-sched taint
(e.g. kubectl cordon).

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
2019-07-11 14:21:40 -07:00
Sébastien Han 928c4a01da Merge pull request #3418 from InfuseAI/feature/add-linear-type-to-supported-disks
Add linear to support diskType
2019-07-11 17:35:46 +02:00
Sébastien Han 6e91054422 Merge pull request #3179 from Miouge1/rgw-tutorial-with-region
Add a --region in the s3cmd tutorial
2019-07-11 16:17:50 +02:00