Commit Graph
520 Commits
Author SHA1 Message Date
Travis Nielsen f80ae4b292 Merge pull request #4692 from leseb/fix-4633
ceph: reset devicesInUse on orchestration error
2020-01-16 13:03:07 -07:00
Travis Nielsen 0033e9713b Merge pull request #4559 from ashishranjan738/monfixwq
ceph: Skipping quorum check if no. of ready mons is 0
2020-01-16 12:53:01 -07:00
Travis Nielsen 6ad62bebf4 Merge pull request #4688 from leseb/external-cluster-crash
ceph: ability to disable crash controller
2020-01-16 11:13:48 -07:00
Ashish Ranjan 44a4722155 ceph: Skipping quorum check if no. of ready mons is 0
If all mons are down (e.g. set the replicas to 0 on all the mon deployments) and the operator is restarted, the operator waits indefinitely for a single mon to start. This PR fixes this issue by skipping the quorum check of mons if the number of ready mon pods is equal to 0.

Signed-off-by: Ashish Ranjan <ashishranjan738@gmail.com>
2020-01-16 23:26:32 +05:30
Sébastien Han ed21121f87 ceph: remove devicesInUse
When the OSDs are being provisioned,
if one cluster has already provisioned the OSDs,
the new cluster will skip them since it will see they are already in use.
So we don't need this check anymore.
Coordinating the flag across multiple cluster instances is more complicated than it's worth.
Someone also might want useAllDevices: true in multiple clusters
and they would keep them on separate nodes by using the node labels and selectors.

Closes: https://github.com/rook/rook/issues/4633
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-01-16 17:09:22 +01:00
Sébastien Han bb214b2f27 ceph: ability to disable crash controller
A new CR setting has been introduced to disable the crash controller.
To disable it, add `disable: true` on the crashCollector section.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-01-16 16:05:22 +01:00
Travis Nielsen 9ce01c50c1 Merge pull request #4689 from leseb/rm-crush-dead-code
ceph: remove dead code for osd removal
2020-01-15 11:28:14 -07:00
Sébastien Han 81dd279062 Merge pull request #4653 from leseb/pick-14.2.6
ceph: bump to 14.2.6
2020-01-15 17:44:52 +01:00
Sébastien Han 260ef519d7 ceph: remove dead code for osd removal
This code is not used anymore, also fixing go-lint issues.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-01-15 12:41:34 +01:00
Sébastien Han dec0c7d4b0 ceph: bump to 14.2.6
14.2.6 is released, let's use it!

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-01-15 11:02:44 +01:00
Sébastien Han 4de61b01d4 ceph: fix external cluster crash k8s secret
The external cluster now creates the crash key when appropriate.

Closes: https://github.com/rook/rook/issues/4553
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-01-15 10:56:23 +01:00
rohan47 fda2b5d96c osd: Added check for --crush-location
Added a check for --crush-location, if --crush-location is not present,
the node topology is not applied and the operator will need to determine
the value based on the node labels instead of skipping this setting.

Signed-off-by: rohan47 <rohgupta@redhat.com>
2020-01-15 02:47:34 +05:30
Travis Nielsen 078e87a722 ceph: fix non-portable osd crush host name
The host name of an OSD in the CRUSH map should be the real
host name for non-portable OSDs. It was incorrectly being set
to the PVC name. Now the non-portable OSDs based on PVCs will
corretly have the host name set to the node name.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-01-13 12:17:49 -07:00
Travis Nielsen aacec0dcda ceph: simplify arguments to getConfigEnvVars method
The method to construct osd environment variables is simplified
by passing a struct param instead of individual params

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-01-13 12:15:15 -07:00
Sébastien Han 05aa639317 ceph: add option to continue on unclean PGs
A new CRD option `continueUpgradeAfterChecksEvenIfNotHealthy` is added.
When upgrading, Rook goes OSD by OSD and then waits for PGs to be clean
before proceeding to the next OSD. Currently, Rook waits for 5 hours but
there might be circumstances where PGs need more time to settle.
Thus setting `continueUpgradeAfterChecksEvenIfNotHealthy` to true will
pursue the upgrade process, even if PGs are not 100% active+clean.

Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1786029
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-01-07 12:18:54 -07:00
Travis Nielsen 43a66f2a8b Merge pull request #4621 from travisn/operator-kit-dependency
Remove operator kit dependency
2020-01-07 11:03:28 -07:00
Travis Nielsen 7fa0032027 Merge pull request #4585 from jmolmo/issue_4567
ceph: Avoid invalid execution state
2020-01-07 10:05:15 -07:00
Travis Nielsen d0c7d28a63 build: remove operator kit dependency
The operator kit had more utility originally when the operator
was creating and managing the TPRs and CRDs directly. Since
the CRDs are now created from a manifest and no longer by the
operators, the utility of operator kit is limited to the
controller watcher. Since we are moving to the controller runtime
we simplify the code to make the transition smoother. Now
there is only a simple WatchCR method that will need to be
replaced as we maek that transition.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-01-07 08:30:14 -07:00
Nizamudeen 729b883dc8 ceph: Generating a strong ceph dashboard password
Replaced `math/rand` with more secure `crypto/rand` which will generate CSPRNG.
Changed the length of dashboard password from 10 to 20 and included more special characters that can strengthen password.

Passwords before this change: QoMmqLkaOf,8tqTyWF1sZ
Passwords after this change: L{QZ@_c'Rl7$uvW:UHU1, fP-OjX`UVw0J0Ka/scS(

Resolves: https://github.com/rook/rook/issues/4571

Signed-off-by: Nizamudeen <nia@redhat.com>
2020-01-06 15:44:23 +05:30
Sébastien Han 4224f84d7c ceph: osd add missing ROOK_CEPH_MON_HOST env
The ROOK_CEPH_MON_HOST env var was missing from the pod env so the
CEPH_ARGS won't be populated correctly.

Closes: https://github.com/rook/rook/issues/4558
Signed-off-by: Sébastien Han <seb@redhat.com>
2019-12-23 16:15:46 +01:00
Juan Miguel Olmo Martínez 5f1c95002b ceph: Avoid invalid execution state
security issues identified by TrailOfBits:

Improperly handled type assertion error could result in invalid
execution state.
If we obtain an error trying to get a Node object is better not
continue executing this function.

Resolves #4567

Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
2019-12-23 11:44:18 +01:00
Sébastien Han 7d11524643 ceph: add missing vol mount for encrypted osds
"ceph-volume activate" on encrypted osds needs a ceph.conf as well as
connectivity to the mons. This commit adds that to that "activate-osd"
init container.

Closes: https://github.com/rook/rook/issues/4558
Signed-off-by: Sébastien Han <seb@redhat.com>
2019-12-23 09:59:04 +01:00
Travis Nielsen dfabccc2ee ceph: rehydrate the osd CRUSH location for PVCs
The OSDs on PVCs were changing location in the CRUSH
hierarchy after an operator restart due to the location
property not being re-discovered on the OSD daemon.
Since OSDs on PVCs do not run the prepare pod again
to discover this property, the operator must read it
off the existing osd deployment. The operator was
reading other properties, but now will read the location
as well.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2019-12-18 15:10:24 -07:00
Travis Nielsen b79c223986 ceph: only set isUpgrade when the image changes
The isUpgrade flag was being set every time the operator
was started. This caused the upgrade checks to be run every
time the operator restarted. Thus, if any mons were down,
you likely would get stuck on the upgrade checks since another
mon could not be stopped. We need to rely on the upgrade being
detected either by a different image name from the cluster CR
update, or from different versions of the daemons running
in case the operator was restarted in the middle of an upgrade.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2019-12-17 10:57:26 -07:00
Juan Miguel Olmo Martínez e8ccef854d Ceph: Toleration seconds for <NotReady> nodes is now configurable
Reduce from 5 minutes to less than 50 seconds the time needed to restart a
mgr and osds(PVCs based),rgw,mds,rbd,nfs and toolbox pod that was running
in a k8s "NotReady" Node.

A new Rook Operator setting has been added to allow the user to set the
time used in pod's toleration <node.kubernetes.io/unreachable>:
ROOK_UNREACHABLE_NODE_TOLERATION_SECONDS = <5>

New setting added in release notes

Addressed @travisn,@leseb, and @blaineEXE suggestions

Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
2019-12-16 21:38:46 -07:00
Sébastien Han ad95c7296f ceph: do not print extended format for loggers.
When using the "errors" package, using `%+v` (extended format),
each Frame of the error's StackTrace will be printed in detail.
Let's only print `%v` to print the error.
If the error has a Cause it will be printed recursively.

Basically `%+v` has been replaced with `%v` for all `error` type
interfaces, whether the logger is Info, Warning or Error.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-12-16 18:35:54 +01:00
Sébastien Han 0a1d700927 ceph: osd tune startup flags when running on GP2
gp2 devices present themselves as SSDs,
but have much more variable performance.
Ceph attempts to adjust some settings based on the drive type,
HDD/SSD
In this case client I/O can be impacted too much by assuming gp2 devices are like other SSDs
(an example is https://bugzilla.redhat.com/show_bug.cgi?id=1765923#c6)

In order to prevent this, rook can set the following for OSDs
if the new CR setting `tuneSlowDeviceClass` is set to `true`:

```
osd_recovery_sleep = 0.1
osd_snap_trim_sleep = 2
osd_delete_sleep = 2
```

Closes: https://github.com/rook/rook/issues/4298 and https://bugzilla.redhat.com/show_bug.cgi?id=1771047
Signed-off-by: Sébastien Han <seb@redhat.com>
2019-12-16 16:37:02 +01:00
Travis Nielsen fbf3837f13 ceph: stop removing osds for any cephcluster CR update
Removing an OSD unintentionally can result in data loss,
which is to be prevented at all costs. Rook's functionality
to remove OSDs automatically when a node was removed from the
cluster CR has led on several occasions to users having OSDs
unintentionally removed from their cluster.

If the admin decides they really need to remove OSDs they should
do it as a one-time action that they intentionally run
rather than the operator attempting to manage the deletion
with desired state. The risk of misinterpreting the desired
state, even with a typo, is too great.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2019-12-13 08:26:23 -07:00
rohan47 6dfc2c139f Ceph: Added version check for ceph osd safe-to-destroy command
It's not 100% safe since before Nautilus osd safe-to-destroy command
didn't have any json output. So added a version check.

Signed-off-by: rohan47 <rohgupta@redhat.com>
2019-12-13 00:07:39 +05:30
Sébastien Han 440a68f57e ceph: add crash key
As of 14.2.5, the crash collector pod will use the client.crash user
instead of the client.admin keyring to post crash dumps.

Closes: https://github.com/rook/rook/issues/4388
Signed-off-by: Sébastien Han <seb@redhat.com>
2019-12-12 09:05:38 +01:00
Sébastien Han 6051a3ca26 ceph: controller accept multiple changes at once
If editing the cluster CR we must update the spec to reflect its new
content. Now the spec is updated so we can update with an unsupported
image as well as setting allowUnsupported: true.
It was not possible before we had to restart the operator, now we can do
it as an update.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-12-11 10:35:53 +01:00
Sébastien Han b26c614ecf ceph: crash do not use selector = label
We must not default the selectors to the labels since labels can be
changed and selector cannot until the deployment is updated.
This way the deployment was failing with:

2019-12-10 10:48:13.414438 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Deployment.apps "rook-ceph-crashcollector-minikube" is invalid: spec.template.metadata.labels: Invalid value: map[string]string{"app":"rook-ceph-crashcollector", "ceph-version":"15.0.0", "ceph_daemon_id":"crash", "crashcollector":"crash", "kubernetes.io/hostname":"minikube", "node_name":"minikube", "rook-version":"v1.2.0-3.g8d3b85306.dirty"}: `selector` does not match template `labels`

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-12-11 10:35:53 +01:00
Sébastien Han e91cf1ac1d ceph: crash retry in seconds
The loop wasn't using seconds for its retry so the crash controller
would simply crash since the version is used later in the pod spec.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-12-11 10:35:53 +01:00
Travis Nielsen 3294384da7 Merge pull request #4415 from leseb/use-minor-version
rook-ceph: stop using ceph version with timestamp and bump to 14.2.5
2019-12-10 08:14:51 -07:00
Sébastien Han 68fd4f4df9 rook: stop using ceph version with timestamp
With https://github.com/ceph/ceph-container/pull/1529, we now build
vX.X.X version so we don't need to have the timestamp. Also we will
automatically get the last changes from the OS/packages too more
frequently.

Closes: https://github.com/rook/rook/issues/4414
Signed-off-by: Sébastien Han <seb@redhat.com>
2019-12-10 11:39:35 +01:00
Blaine Gardner caba125b84 Merge pull request #4370 from SUSE/osd-refactor-legacy
Ceph: OSD: refactor config for filestore OSDs
2019-12-09 16:20:17 -07:00
Blaine Gardner 003b870d8f ceph: osd: disable creation of new filestore osds
Creation of new filestore OSDs is being deprecated as per
https://github.com/rook/rook/issues/3379

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2019-12-09 14:15:13 -07:00
Blaine Gardner 64bad24712 ceph: osd: do not init ceph.conf for filestore osd
Just as with the mon, mgr, mds, rgw, and rbd-mirror daemons, do not
generate a ceph.conf in an init container for filestore OSDs.
Instead use the commandline to supply all the needed arguments for
running these OSDs.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2019-12-09 14:15:12 -07:00
Elise Gafford c379e116a9 ceph: add rook-version and ceph-version tags to crash deployment
This change adds rook-version and ceph-version tags to crash
collector deployments, in keeping with other Ceph deployments.

Resolves: #4357
Signed-off-by: Elise Gafford <egafford@redhat.com>
2019-12-09 16:11:08 -05:00
Blaine Gardner af822018e3 ceph: osd: filestore: fix filename too long error
Add osd-max-object-{name,namespace}-len flags to filestore OSDs to avoid
`ERROR: osd init failed: (36) File name too long` errors.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2019-12-09 10:41:54 -07:00
Sébastien Han 5ce2ed220e ceph: use "github.com/pkg/errors"
We now use the error package.
Kubernetes errors have been renamed kerrors since they are lower than
'errors'.

Closes: https://github.com/rook/rook/issues/4054
Signed-off-by: Sébastien Han <seb@redhat.com>
2019-12-09 16:58:32 +01:00
Travis Nielsen 6a66e010ad Merge pull request #4445 from travisn/continue-on-node-missing
Continue orchestration on missing node for osd
2019-12-09 08:05:36 -07:00
Mateusz Los 50f3d29cf4 ceph: client controller tests and docs
ceph client CRD refactoring
Add info about client crd to documentation
Add tests for client controller CRD

Signed-off-by: Mateusz Los <los.mateusz@gmail.com>
2019-12-09 12:48:44 +01:00
Mateusz Gozdek 65fd8ece2a ceph: Create initial client controller code
Create initial client controller code

Signed-off-by: Mateusz Los <los.mateusz@gmail.com>
2019-12-08 16:47:57 +01:00
Travis Nielsen 618028d2cd ceph: continue orchestration on missing node for osd
When a node has been removed, cordoned, or is otherwise
missing from the filters that rook uses to evaluate deploying
osds on nodes, the orchestration should continue instead
of failing. Otherwise, the orchestration will stay stuck
in a retry loop if any nodes are cordoned or removed in
the middle of an orchestration.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2019-12-06 15:48:19 -07:00
dulltzandSatoru Takeuchi dfe45ac6b0 ceph: support OSD on PVC backed by LV
"OSD on PVC" doesn't work for PV backed by LV. Fixing this problem
by the following changes.

- Rook accepts LVM disk type.
- If a LV-backed device is passed, Rook/Ceph invokes
  "ceph-volume lvm prepare" with "--data vg/lv"
  instead of "--data /path/to/device".
- If a LV-backed device is passed, Rook/Ceph suppresses
  activation/deactivation of VG that owns this LV.

Fixes: https://github.com/rook/rook/issues/4185
Signed-off-by: dulltz <isrgnoe@gmail.com>
Co-authored-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
2019-12-05 17:26:27 +00:00
Sébastien Han e108f87b06 Merge pull request #4386 from leseb/location-prepare
ceph: do not restart the osds when upgrading the operator
2019-12-05 16:59:33 +01:00
Travis Nielsen 0a352f64d9 Merge pull request #4285 from cybozu-go/use-mapper-name
ceph: support device selection for OSDs by device path name patterns
2019-12-05 07:54:07 -07:00
Sébastien Han 470798c570 ceph: osd stop using rook binary and tini
This patch removes the need of using both tini and rook binary to run
the main osd container.
By removing this, we know can upgrade the operator image without
restarting the OSD since the osd pod does not use the rook image
anymore.
This patch only works when not running OSD on PVC since they are certain
actions that we cannot achieve with an init container.
A new init container has been introduced to do the activation of the
OSD, the activate files are placed in a empty dir mount and shared with
the main osd process.
Now the main osd process runs the ceph-osd daemon binary directly.

This currently has a few limitation:

* only works with bluestore OSDs
* only works when not running OSD on PVC

Closes: https://github.com/rook/rook/issues/4363
Signed-off-by: Sébastien Han <seb@redhat.com>
2019-12-05 14:39:51 +01:00
Kazuhito MATSUDA 40d868ebd7 ceph: support device selection for OSDs by device path name patterns
Add devicePathFilter to Storage Selection Settings which enables
device selection with a regular expression for device path names.

Fixes: https://github.com/rook/rook/issues/4275
Signed-off-by: Kazuhito MATSUDA <kazuto.jinnai@gmail.com>
2019-12-05 07:02:44 +00:00