Commit Graph
566 Commits
Author SHA1 Message Date
Sébastien Han 227d2d527a ceph: osd store refactor
Multiple things:

1. We removed all the function/methods/tests that were used to
create and manage rook legacy OSDS as well as bringing support to
Bluestore OSD only.
It also fixes various go-lint issues in the respectives files.

2. use c-v inventory to detect available devices:
Now we rely on the 'ceph-volume inventory' command to tell us if a
device is available or not.

3. implement raw mode for osd on pvc
When an OSD will be bootstrap on a PVC, the new c-v raw mode will be
used. It consists of putting block, db and wal under the same device.
Here LVM is out of the picture and the raw device is used as is. The
implementation is backward compatible so existing OSD on PVC will LVM
will continue to operate.

Closes: https://github.com/rook/rook/issues/4363
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-01-23 19:13:09 +01:00
Sébastien Han 6fc6ea7a66 Merge pull request #4729 from ashangit/not_rr_osds_on_operator_restart
ceph: ensure crush-location osd command args is always the same to avoid useless osds restart
2020-01-21 18:08:38 +01:00
n.fraison 7bac027796 ceph: ensure crush-location osd command args is always the same
Currently each time the ceph operator restart, osds also restart.
This is due to a change in the crush-location args with location not ordered
At each restart the location order can change and lead to that kind of change in the osd deployment
104c105
<                             "--crush-location=root=default host=hostname datacenter=PAR pod=2 rack=2",
---
>                             "--crush-location=root=default host=hostname pod=2 rack=2 datacenter=PAR",
Sorting the topology to ensure stability of this command arg

Signed-off-by: n.fraison <n.fraison@criteo.com>
2020-01-21 15:04:16 +01:00
Sébastien Han ef8ef454c3 ceph: fix resources for prepare job
Now, we purely rely on what the user puts in the CR and apply it.
Prior to this commit we were putting 0 if not declared, this was not the
intentional.

Closes: https://github.com/rook/rook/issues/4713
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-01-21 14:37:28 +01:00
Sébastien Han 260ef519d7 ceph: remove dead code for osd removal
This code is not used anymore, also fixing go-lint issues.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-01-15 12:41:34 +01:00
rohan47 fda2b5d96c osd: Added check for --crush-location
Added a check for --crush-location, if --crush-location is not present,
the node topology is not applied and the operator will need to determine
the value based on the node labels instead of skipping this setting.

Signed-off-by: rohan47 <rohgupta@redhat.com>
2020-01-15 02:47:34 +05:30
Travis Nielsen 078e87a722 ceph: fix non-portable osd crush host name
The host name of an OSD in the CRUSH map should be the real
host name for non-portable OSDs. It was incorrectly being set
to the PVC name. Now the non-portable OSDs based on PVCs will
corretly have the host name set to the node name.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-01-13 12:17:49 -07:00
Travis Nielsen aacec0dcda ceph: simplify arguments to getConfigEnvVars method
The method to construct osd environment variables is simplified
by passing a struct param instead of individual params

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-01-13 12:15:15 -07:00
Juan Miguel Olmo Martínez e27dcdd419 ceph: Remove ineffectual assignments
Description of your changes:
Fix security issues identified by TrailOfBits

Modifications in osd/status.go:
I opted by remove the offending variables because "c.checkNodesCompleted"
provides this variables always, so no need to initialize them.
var was not used, so i have replaced it by the blank identifier.

Modifications in machinelabel/add.go:
I have followed the same criteria used in other disruption packages,
and return the error if it is produced when we try to "watch" machines.

Closes: https://github.com/rook/rook/issues/4565

[test ceph]

Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
2020-01-10 10:12:49 +01:00
Sébastien Han 05aa639317 ceph: add option to continue on unclean PGs
A new CRD option `continueUpgradeAfterChecksEvenIfNotHealthy` is added.
When upgrading, Rook goes OSD by OSD and then waits for PGs to be clean
before proceeding to the next OSD. Currently, Rook waits for 5 hours but
there might be circumstances where PGs need more time to settle.
Thus setting `continueUpgradeAfterChecksEvenIfNotHealthy` to true will
pursue the upgrade process, even if PGs are not 100% active+clean.

Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1786029
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-01-07 12:18:54 -07:00
Sébastien Han 4224f84d7c ceph: osd add missing ROOK_CEPH_MON_HOST env
The ROOK_CEPH_MON_HOST env var was missing from the pod env so the
CEPH_ARGS won't be populated correctly.

Closes: https://github.com/rook/rook/issues/4558
Signed-off-by: Sébastien Han <seb@redhat.com>
2019-12-23 16:15:46 +01:00
Sébastien Han 7d11524643 ceph: add missing vol mount for encrypted osds
"ceph-volume activate" on encrypted osds needs a ceph.conf as well as
connectivity to the mons. This commit adds that to that "activate-osd"
init container.

Closes: https://github.com/rook/rook/issues/4558
Signed-off-by: Sébastien Han <seb@redhat.com>
2019-12-23 09:59:04 +01:00
Travis Nielsen dfabccc2ee ceph: rehydrate the osd CRUSH location for PVCs
The OSDs on PVCs were changing location in the CRUSH
hierarchy after an operator restart due to the location
property not being re-discovered on the OSD daemon.
Since OSDs on PVCs do not run the prepare pod again
to discover this property, the operator must read it
off the existing osd deployment. The operator was
reading other properties, but now will read the location
as well.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2019-12-18 15:10:24 -07:00
Juan Miguel Olmo Martínez e8ccef854d Ceph: Toleration seconds for <NotReady> nodes is now configurable
Reduce from 5 minutes to less than 50 seconds the time needed to restart a
mgr and osds(PVCs based),rgw,mds,rbd,nfs and toolbox pod that was running
in a k8s "NotReady" Node.

A new Rook Operator setting has been added to allow the user to set the
time used in pod's toleration <node.kubernetes.io/unreachable>:
ROOK_UNREACHABLE_NODE_TOLERATION_SECONDS = <5>

New setting added in release notes

Addressed @travisn,@leseb, and @blaineEXE suggestions

Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
2019-12-16 21:38:46 -07:00
Sébastien Han ad95c7296f ceph: do not print extended format for loggers.
When using the "errors" package, using `%+v` (extended format),
each Frame of the error's StackTrace will be printed in detail.
Let's only print `%v` to print the error.
If the error has a Cause it will be printed recursively.

Basically `%+v` has been replaced with `%v` for all `error` type
interfaces, whether the logger is Info, Warning or Error.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-12-16 18:35:54 +01:00
Sébastien Han 0a1d700927 ceph: osd tune startup flags when running on GP2
gp2 devices present themselves as SSDs,
but have much more variable performance.
Ceph attempts to adjust some settings based on the drive type,
HDD/SSD
In this case client I/O can be impacted too much by assuming gp2 devices are like other SSDs
(an example is https://bugzilla.redhat.com/show_bug.cgi?id=1765923#c6)

In order to prevent this, rook can set the following for OSDs
if the new CR setting `tuneSlowDeviceClass` is set to `true`:

```
osd_recovery_sleep = 0.1
osd_snap_trim_sleep = 2
osd_delete_sleep = 2
```

Closes: https://github.com/rook/rook/issues/4298 and https://bugzilla.redhat.com/show_bug.cgi?id=1771047
Signed-off-by: Sébastien Han <seb@redhat.com>
2019-12-16 16:37:02 +01:00
Travis Nielsen fbf3837f13 ceph: stop removing osds for any cephcluster CR update
Removing an OSD unintentionally can result in data loss,
which is to be prevented at all costs. Rook's functionality
to remove OSDs automatically when a node was removed from the
cluster CR has led on several occasions to users having OSDs
unintentionally removed from their cluster.

If the admin decides they really need to remove OSDs they should
do it as a one-time action that they intentionally run
rather than the operator attempting to manage the deletion
with desired state. The risk of misinterpreting the desired
state, even with a typo, is too great.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2019-12-13 08:26:23 -07:00
rohan47 6dfc2c139f Ceph: Added version check for ceph osd safe-to-destroy command
It's not 100% safe since before Nautilus osd safe-to-destroy command
didn't have any json output. So added a version check.

Signed-off-by: rohan47 <rohgupta@redhat.com>
2019-12-13 00:07:39 +05:30
Blaine Gardner caba125b84 Merge pull request #4370 from SUSE/osd-refactor-legacy
Ceph: OSD: refactor config for filestore OSDs
2019-12-09 16:20:17 -07:00
Blaine Gardner 003b870d8f ceph: osd: disable creation of new filestore osds
Creation of new filestore OSDs is being deprecated as per
https://github.com/rook/rook/issues/3379

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2019-12-09 14:15:13 -07:00
Blaine Gardner 64bad24712 ceph: osd: do not init ceph.conf for filestore osd
Just as with the mon, mgr, mds, rgw, and rbd-mirror daemons, do not
generate a ceph.conf in an init container for filestore OSDs.
Instead use the commandline to supply all the needed arguments for
running these OSDs.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2019-12-09 14:15:12 -07:00
Blaine Gardner af822018e3 ceph: osd: filestore: fix filename too long error
Add osd-max-object-{name,namespace}-len flags to filestore OSDs to avoid
`ERROR: osd init failed: (36) File name too long` errors.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2019-12-09 10:41:54 -07:00
Sébastien Han 5ce2ed220e ceph: use "github.com/pkg/errors"
We now use the error package.
Kubernetes errors have been renamed kerrors since they are lower than
'errors'.

Closes: https://github.com/rook/rook/issues/4054
Signed-off-by: Sébastien Han <seb@redhat.com>
2019-12-09 16:58:32 +01:00
Travis Nielsen 618028d2cd ceph: continue orchestration on missing node for osd
When a node has been removed, cordoned, or is otherwise
missing from the filters that rook uses to evaluate deploying
osds on nodes, the orchestration should continue instead
of failing. Otherwise, the orchestration will stay stuck
in a retry loop if any nodes are cordoned or removed in
the middle of an orchestration.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2019-12-06 15:48:19 -07:00
dulltzandSatoru Takeuchi dfe45ac6b0 ceph: support OSD on PVC backed by LV
"OSD on PVC" doesn't work for PV backed by LV. Fixing this problem
by the following changes.

- Rook accepts LVM disk type.
- If a LV-backed device is passed, Rook/Ceph invokes
  "ceph-volume lvm prepare" with "--data vg/lv"
  instead of "--data /path/to/device".
- If a LV-backed device is passed, Rook/Ceph suppresses
  activation/deactivation of VG that owns this LV.

Fixes: https://github.com/rook/rook/issues/4185
Signed-off-by: dulltz <isrgnoe@gmail.com>
Co-authored-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
2019-12-05 17:26:27 +00:00
Sébastien Han e108f87b06 Merge pull request #4386 from leseb/location-prepare
ceph: do not restart the osds when upgrading the operator
2019-12-05 16:59:33 +01:00
Sébastien Han 470798c570 ceph: osd stop using rook binary and tini
This patch removes the need of using both tini and rook binary to run
the main osd container.
By removing this, we know can upgrade the operator image without
restarting the OSD since the osd pod does not use the rook image
anymore.
This patch only works when not running OSD on PVC since they are certain
actions that we cannot achieve with an init container.
A new init container has been introduced to do the activation of the
OSD, the activate files are placed in a empty dir mount and shared with
the main osd process.
Now the main osd process runs the ceph-osd daemon binary directly.

This currently has a few limitation:

* only works with bluestore OSDs
* only works when not running OSD on PVC

Closes: https://github.com/rook/rook/issues/4363
Signed-off-by: Sébastien Han <seb@redhat.com>
2019-12-05 14:39:51 +01:00
Kazuhito MATSUDA 40d868ebd7 ceph: support device selection for OSDs by device path name patterns
Add devicePathFilter to Storage Selection Settings which enables
device selection with a regular expression for device path names.

Fixes: https://github.com/rook/rook/issues/4275
Signed-off-by: Kazuhito MATSUDA <kazuto.jinnai@gmail.com>
2019-12-05 07:02:44 +00:00
Sébastien Han 4a7258eb0d ceph: osd pass crush location from prepare pod
We shouldn't need to compute the location from the main OSD pod, since
the prepare pod runs on the same node we can populate the crush location
information from that pod and pass it along to the main osd container.

This is an initial commit that aims to eventually remove the need of
using the rook binary when running the main osd container.

Closes: https://github.com/rook/rook/issues/4362
Signed-off-by: Sébastien Han <seb@redhat.com>
2019-12-04 23:03:02 +01:00
Travis Nielsen add3129c12 ceph: continue orchestration on osd update errors
During an orchestration if there were failures to update any OSD deployment,
the orchestration would fail and continue trying again. If the OSD(s)
continued failing, the operator would be stuck in a loop and never
respond to other CR updates.

Now the operator will log the OSD errors, if any, and complete the
orchestration. We will rely on K8s to continue to attempt to start
the OSD pods instead of the operator being stuck in the loop.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2019-12-03 19:16:48 -07:00
Travis Nielsen 87e2b09453 ceph: clarify log message for Ceph upgrades
Some log messages indicate when the upgrade checks will be
performed. If the Ceph image hasn't changed, the upgrade checks
are skipped. The messages were confusing if running a Rook
upgrade.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2019-11-22 09:03:03 -07:00
Blaine Gardner f32cecb701 Merge pull request #3206 from d-luu/ceph_priority_classes
ceph: added priority classes to components
2019-11-18 11:27:57 -07:00
d-luu 9c755fac66 ceph: added priority classes to components
Adds priority class support to Ceph components
  to influence scheduler's pod preemption

Signed-off-by: d-luu <david@davidluu.info>
2019-11-15 16:28:41 -06:00
Sébastien Han 3e82a0f4a6 ceph: osd add env var to skip uded on lvm
With DM_DISABLE_UDEV, LVM will avoid interaction with udev.
LVM will manage the relevant nodes in /dev directly.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-11-15 22:54:03 +01:00
Sébastien Han 314394f464 Merge pull request #4281 from rohantmp/consolidated
Ceph: Fix topology logic for drains.
2019-11-13 16:37:09 +01:00
Rohan CJ 1ba2633c27 Ceph: Clean up canaries on nodes that no longer have OSDs or when the node is deleted.
This prevents long-lasting false positives that prevent the ordered progression
of drains.

Signed-off-by: Rohan CJ <rohantmp@gmail.com>
2019-11-13 16:04:45 +05:30
Rohan CJ dd3f0ca63c Ceph: Set the drain-canaries deployment ownerReference to the rook operator.
The drain canary was not being garbage collected properly.

Signed-off-by: Rohan CJ <rohantmp@gmail.com>
2019-11-13 16:04:45 +05:30
Sébastien Han d9679e5cf4 Merge pull request #3316 from leseb/ceph-crash
ceph: add ceph-crash controller
2019-11-12 23:16:02 +01:00
Sébastien Han a0fdf1bb5f ceph: add CEPH_VOLUME_SKIP_RESTORECON env var
This will tell ceph-volume not to run any restorecon commands.

This relies on https://github.com/ceph/ceph/pull/31421

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-11-12 16:27:20 +01:00
Sébastien HanandRohan CJ 90cd8224f4 ceph: add ceph-crash controller
A new controller to bootstrap ceph-crash pod on Ceph nodes running Ceph
pods only.

This implementation is unfortunately a mix of 'hacks' to make the daemon
working correclty. The ceph-crash script faced different issues:

* it's not a daemon
* it does not have any key in cephx
* it runs ceph commands under the hood which need an admin key and a
ceph.conf

On upstream Ceph, ceph-crash needs to grow better, once that happens we
will improve our implementation.

This also enhances osd provisionConfig struct with DataPathMap

Pod volumes and volume mount needs DataPathMap to perform the right
actions on log and crash dir. Exposing DataPathMap makes that possible.

Obviously, this is exposing ceph crash reports on the host as
well as pushing them into the mgr.
Basically, if a daemon fails, it'll put its core dump into
/var/lib/ceph/crash, bindmounting this dir on the host, ensures that the
crashes don't get lost when the pod dies.

Example:

```
[leseb@tarox~/go/src/github.com/rook/rook][rgw-liveprobe !] kubectl -n rook-ceph exec -ti rook-ceph-crashcollector-minikube-574858b99c-zvg4z bash
bash: warning: setlocale: LC_CTYPE: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_COLLATE: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_MESSAGES: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_NUMERIC: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_TIME: cannot change locale (en_US.UTF-8): No such file or directory
[root@rook-ceph-crashcollector-minikube-574858b99c-zvg4z /]# ceph crash ls

[root@rook-ceph-osd-2-7644f99695-cljzh /]# pidof ceph-osd
13258 5415 5397
[root@rook-ceph-osd-2-7644f99695-cljzh /]# kill -SIGABRT 13258
[root@rook-ceph-osd-2-7644f99695-cljzh /]# ls /var/lib/ceph/crash/
2019-11-12_12:56:51.404109Z_39f060e1-776d-4605-8f28-85c97e53de96     posted

... wait maximum 10 min (ceph-crash scraps every 10 minutes)
... the container will exit

[root@rook-ceph-osd-2-7644f99695-cljzh /]# exit
[leseb@tarox~/go/src/github.com/rook/rook][rgw-liveprobe !]
[leseb@tarox~/go/src/github.com/rook/rook][rgw-liveprobe !] kubectl -n rook-ceph exec -ti rook-ceph-crashcollector-minikube-574858b99c-zvg4z bash
bash: warning: setlocale: LC_CTYPE: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_COLLATE: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_MESSAGES: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_NUMERIC: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_TIME: cannot change locale (en_US.UTF-8): No such file or directory
[root@rook-ceph-crashcollector-minikube-574858b99c-zvg4z /]#
[root@rook-ceph-crashcollector-minikube-574858b99c-zvg4z /]#
[root@rook-ceph-crashcollector-minikube-574858b99c-zvg4z /]# ceph crash ls
2019-11-12_12:56:51.404109Z_39f060e1-776d-4605-8f28-85c97e53de96 osd.1
[root@rook-ceph-crashcollector-minikube-574858b99c-zvg4z /]# ls /var/lib/ceph/crash/
posted
[root@rook-ceph-crashcollector-minikube-574858b99c-zvg4z /]# ls /var/lib/ceph/crash/posted/
2019-11-12_12:56:51.404109Z_39f060e1-776d-4605-8f28-85c97e53de96
[root@rook-ceph-crashcollector-minikube-574858b99c-zvg4z /]# tail /var/lib/ceph/crash/posted/2019-11-12_12\:56\:51.404109Z_39f060e1-776d-4605-8f28-85c97e53de96/log
   1/ 5 mgr
   1/ 5 mgrc
   1/ 5 dpdk
   1/ 5 eventtrace
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent     10000
  max_new         1000
  log_file /var/lib/ceph/crash/2019-11-12_12:56:51.404109Z_39f060e1-776d-4605-8f28-85c97e53de96/log
--- end dump of recent events ---
```

Signed-off-by: Sébastien Han <seb@redhat.com>
Co-authored-by: Rohan CJ <rohantmp@gmail.com>
2019-11-12 16:06:07 +01:00
Sébastien Han 76c33e0f85 Revert "ceph: osd hide 'restorecon' when necessary"
This reverts commit d224fbdef1.
Since https://github.com/ceph/ceph/pull/31421, the monkey patching of
'restorecon' is not needed anymore.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-11-12 15:34:12 +01:00
Travis Nielsen 0bb8000825 Merge pull request #4273 from egafford/associate-keyrings-with-deployments
ceph: Standardize association of keyrings with deployments
2019-11-11 17:06:24 -07:00
Rohan CJ 4d7c6df034 Ceph: Fix topologyAware in clusterdisruption package does not respect the rook topolgy prefix.
Allow the clusterdisruption package to recognize node topologies other than zone.

Signed-off-by: Rohan CJ <rohantmp@gmail.com>
2019-11-11 09:38:33 +05:30
egafford bb7aac4b62 ceph: Standardize association of keyrings with deployments
This change standardizes the ownerRefs of Ceph keyrings to
k8s Deployment resources across all daemons and refactors a
bit of duplicate code on keyring association for ease of
maintenance.

Fixes: #4089

Signed-off-by: egafford <egafford@redhat.com>
2019-11-08 09:50:37 -05:00
Santosh Pillai 26360689f5 ceph: skip osd prepare job creation if osd daemon exists for the pvc
When an orchestration is restarted or retriggered and starts new osd prepare pods, they are unable to start because the device is already in use by the osd daemon pods.
This fix simple skips the osd prepare pod creation if the daemon is already running.

Signed-off-by: Santosh Pillai <sapillai@redhat.com>
2019-11-08 20:11:49 +05:30
Travis Nielsen 47764715fe ceph: remove the location from the cluster CR
The topology of the cluster should be based on the node labels rather
than a setting in the cluster CR. This allows a much richer and more
dynamic topology to be configured. The location will now be ignored
if specified in the cluster CR.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2019-11-04 23:07:13 -07:00
Sébastien Han ef6815ebfa ceph: remove topologyAware CRD option
We removed the topologyAware CRD option since it was redundant with the
use of an OSD being backed by a PVC.
So now, if an OSD is backed by a PVC we assume the topology aware
decision and will discover zone and region labels on that host.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-11-04 09:53:14 -07:00
Travis Nielsen 831c15086a ceph: support all layers of CRUSH map with node labels
The node labels were already supported for zones and regions to add to
the CRUSH map. Now all layers of the CRUSH map will be supported
with the new labels such as topology.rook.io/rack.
The labels will be detected at the osd startup time
similar to the zone and region labels already being detected.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2019-10-31 14:47:18 -06:00
Sébastien Han 09e29d01b1 Merge pull request #4214 from leseb/skip-restorecon
ceph: osd hide 'restorecon' when necessary
2019-10-30 14:44:30 +01:00
Sébastien Han d224fbdef1 ceph: osd hide 'restorecon' when necessary
Some testing on OpenShift has shown that enabling Hostnetworking causes
issues with ceph-volume creating OSD.
This is a pure Red Hat OpenShift downstream issue.

This is meant to be temporary while we are waiting for the Selinux team
to get back to us.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-10-30 11:32:15 +01:00