Commit Graph
532 Commits
Author SHA1 Message Date
Sébastien Han caa8382e25 ceph: do not update deployment if no changes
We know use k8s-objectmatcher to verify whether a deployment is going to
change or not.
This avoids calling Update() from k8s utils which takes time to run.

Closes: https://github.com/rook/rook/issues/4519 and https://github.com/rook/rook/issues/4642
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-01-24 17:02:40 +01:00
Sébastien Han 92c1696bfc ceph: cleanup/trim isUpgrade variable a bit
We now always perform ok-to-stop change when the deployment is going to
be updated. This could be for various reasons like change in the spec or
a new Ceph image.
In any case, if Ceph resources are going to be restarted this must be
done carefully.

Also trimmed the isUpgrade variable, the upcoming commit will remove the
need of that variable.

Keeping isUpgrade from the controller code since we need it to detect
whether an orchestration upgrade should be triggered or not (based on
the ceph cluster status).

Also, we keep it on the child controllers so we don't go into an
orchestration phase if the cluster image hasn't changed. The child does
not need to be updated/go into its orchestration phase because the
cluster CR changed.

Closes: https://github.com/rook/rook/issues/4379
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-01-24 17:02:37 +01:00
Sébastien Han a8cd184aa1 Merge pull request #4629 from egafford/add-ceph-version-to-cluster-status
ceph: add ceph version to cluster status
2020-01-24 16:42:11 +01:00
Elise Gafford b19f8d4989 ceph: add ceph version to cluster status
In order for deployments managed outside the immediate scope of
the ClusterController to accurately detect the Ceph version of
their images, the ClusterController must store the detected
version. This change records the Ceph version as part of the
cluster status and modifies the crashcollector deployment to
consume this field to record its own Ceph version. This replaces
an earlier solution to this problem in which the Ceph version
for a given image to version mapping was stored in a global
in-memory map by the operator.

Resolves: rook#4357
Signed-off-by: Elise Gafford <egafford@redhat.com>
2020-01-24 09:22:48 -05:00
Sébastien Han 227d2d527a ceph: osd store refactor
Multiple things:

1. We removed all the function/methods/tests that were used to
create and manage rook legacy OSDS as well as bringing support to
Bluestore OSD only.
It also fixes various go-lint issues in the respectives files.

2. use c-v inventory to detect available devices:
Now we rely on the 'ceph-volume inventory' command to tell us if a
device is available or not.

3. implement raw mode for osd on pvc
When an OSD will be bootstrap on a PVC, the new c-v raw mode will be
used. It consists of putting block, db and wal under the same device.
Here LVM is out of the picture and the raw device is used as is. The
implementation is backward compatible so existing OSD on PVC will LVM
will continue to operate.

Closes: https://github.com/rook/rook/issues/4363
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-01-23 19:13:09 +01:00
Sébastien Han 19350d6268 Merge pull request #4725 from eliaswimmer/fix_4717
ceph: fixed #4717 Ceph external cluster, repeatedly saving mon endpoi…
2020-01-22 16:49:25 +01:00
Elias Wimmer 70f312e35d ceph: fixed #4717 Ceph external cluster, repeatedly saving mon endpoints to config
Before, every few seconds the config files for the cluster where written to disk.
Even when there were no changes in the clusters monitors.

Fixes: https://github.com/rook/rook/issues/4717
Signed-off-by: Elias Wimmer <elias.wimmer@tuwien.ac.at>
2020-01-22 15:15:56 +01:00
Sébastien Han 2292b67127 Merge pull request #4734 from leseb/fix-4686
ceph: fix external cluster config override cm creation
2020-01-22 14:42:40 +01:00
Sébastien Han f59bdd626a ceph: fix external cluster config override cm creation
If an image is specified in the external cluster CR, the
'rook-ceph-config' config map will be created and then child CRs like
rgw/mds/nfs can be created.

Also, calling config.GetStore() was a mistake since it's calling by the
health check if new mons are being added to clusterInfo.

Closes: https://github.com/rook/rook/issues/4686
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-01-22 14:01:20 +01:00
Sébastien Han 6fc6ea7a66 Merge pull request #4729 from ashangit/not_rr_osds_on_operator_restart
ceph: ensure crush-location osd command args is always the same to avoid useless osds restart
2020-01-21 18:08:38 +01:00
n.fraison 7bac027796 ceph: ensure crush-location osd command args is always the same
Currently each time the ceph operator restart, osds also restart.
This is due to a change in the crush-location args with location not ordered
At each restart the location order can change and lead to that kind of change in the osd deployment
104c105
<                             "--crush-location=root=default host=hostname datacenter=PAR pod=2 rack=2",
---
>                             "--crush-location=root=default host=hostname pod=2 rack=2 datacenter=PAR",
Sorting the topology to ensure stability of this command arg

Signed-off-by: n.fraison <n.fraison@criteo.com>
2020-01-21 15:04:16 +01:00
Sébastien Han ef8ef454c3 ceph: fix resources for prepare job
Now, we purely rely on what the user puts in the CR and apply it.
Prior to this commit we were putting 0 if not declared, this was not the
intentional.

Closes: https://github.com/rook/rook/issues/4713
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-01-21 14:37:28 +01:00
Travis Nielsen f80ae4b292 Merge pull request #4692 from leseb/fix-4633
ceph: reset devicesInUse on orchestration error
2020-01-16 13:03:07 -07:00
Travis Nielsen 0033e9713b Merge pull request #4559 from ashishranjan738/monfixwq
ceph: Skipping quorum check if no. of ready mons is 0
2020-01-16 12:53:01 -07:00
Travis Nielsen 6ad62bebf4 Merge pull request #4688 from leseb/external-cluster-crash
ceph: ability to disable crash controller
2020-01-16 11:13:48 -07:00
Ashish Ranjan 44a4722155 ceph: Skipping quorum check if no. of ready mons is 0
If all mons are down (e.g. set the replicas to 0 on all the mon deployments) and the operator is restarted, the operator waits indefinitely for a single mon to start. This PR fixes this issue by skipping the quorum check of mons if the number of ready mon pods is equal to 0.

Signed-off-by: Ashish Ranjan <ashishranjan738@gmail.com>
2020-01-16 23:26:32 +05:30
Sébastien Han ed21121f87 ceph: remove devicesInUse
When the OSDs are being provisioned,
if one cluster has already provisioned the OSDs,
the new cluster will skip them since it will see they are already in use.
So we don't need this check anymore.
Coordinating the flag across multiple cluster instances is more complicated than it's worth.
Someone also might want useAllDevices: true in multiple clusters
and they would keep them on separate nodes by using the node labels and selectors.

Closes: https://github.com/rook/rook/issues/4633
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-01-16 17:09:22 +01:00
Sébastien Han bb214b2f27 ceph: ability to disable crash controller
A new CR setting has been introduced to disable the crash controller.
To disable it, add `disable: true` on the crashCollector section.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-01-16 16:05:22 +01:00
Travis Nielsen 9ce01c50c1 Merge pull request #4689 from leseb/rm-crush-dead-code
ceph: remove dead code for osd removal
2020-01-15 11:28:14 -07:00
Sébastien Han 81dd279062 Merge pull request #4653 from leseb/pick-14.2.6
ceph: bump to 14.2.6
2020-01-15 17:44:52 +01:00
Sébastien Han 260ef519d7 ceph: remove dead code for osd removal
This code is not used anymore, also fixing go-lint issues.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-01-15 12:41:34 +01:00
Sébastien Han dec0c7d4b0 ceph: bump to 14.2.6
14.2.6 is released, let's use it!

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-01-15 11:02:44 +01:00
Sébastien Han 4de61b01d4 ceph: fix external cluster crash k8s secret
The external cluster now creates the crash key when appropriate.

Closes: https://github.com/rook/rook/issues/4553
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-01-15 10:56:23 +01:00
rohan47 fda2b5d96c osd: Added check for --crush-location
Added a check for --crush-location, if --crush-location is not present,
the node topology is not applied and the operator will need to determine
the value based on the node labels instead of skipping this setting.

Signed-off-by: rohan47 <rohgupta@redhat.com>
2020-01-15 02:47:34 +05:30
Travis Nielsen 078e87a722 ceph: fix non-portable osd crush host name
The host name of an OSD in the CRUSH map should be the real
host name for non-portable OSDs. It was incorrectly being set
to the PVC name. Now the non-portable OSDs based on PVCs will
corretly have the host name set to the node name.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-01-13 12:17:49 -07:00
Travis Nielsen aacec0dcda ceph: simplify arguments to getConfigEnvVars method
The method to construct osd environment variables is simplified
by passing a struct param instead of individual params

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-01-13 12:15:15 -07:00
Sébastien Han 05aa639317 ceph: add option to continue on unclean PGs
A new CRD option `continueUpgradeAfterChecksEvenIfNotHealthy` is added.
When upgrading, Rook goes OSD by OSD and then waits for PGs to be clean
before proceeding to the next OSD. Currently, Rook waits for 5 hours but
there might be circumstances where PGs need more time to settle.
Thus setting `continueUpgradeAfterChecksEvenIfNotHealthy` to true will
pursue the upgrade process, even if PGs are not 100% active+clean.

Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1786029
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-01-07 12:18:54 -07:00
Travis Nielsen 43a66f2a8b Merge pull request #4621 from travisn/operator-kit-dependency
Remove operator kit dependency
2020-01-07 11:03:28 -07:00
Travis Nielsen 7fa0032027 Merge pull request #4585 from jmolmo/issue_4567
ceph: Avoid invalid execution state
2020-01-07 10:05:15 -07:00
Travis Nielsen d0c7d28a63 build: remove operator kit dependency
The operator kit had more utility originally when the operator
was creating and managing the TPRs and CRDs directly. Since
the CRDs are now created from a manifest and no longer by the
operators, the utility of operator kit is limited to the
controller watcher. Since we are moving to the controller runtime
we simplify the code to make the transition smoother. Now
there is only a simple WatchCR method that will need to be
replaced as we maek that transition.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-01-07 08:30:14 -07:00
Nizamudeen 729b883dc8 ceph: Generating a strong ceph dashboard password
Replaced `math/rand` with more secure `crypto/rand` which will generate CSPRNG.
Changed the length of dashboard password from 10 to 20 and included more special characters that can strengthen password.

Passwords before this change: QoMmqLkaOf,8tqTyWF1sZ
Passwords after this change: L{QZ@_c'Rl7$uvW:UHU1, fP-OjX`UVw0J0Ka/scS(

Resolves: https://github.com/rook/rook/issues/4571

Signed-off-by: Nizamudeen <nia@redhat.com>
2020-01-06 15:44:23 +05:30
Sébastien Han 4224f84d7c ceph: osd add missing ROOK_CEPH_MON_HOST env
The ROOK_CEPH_MON_HOST env var was missing from the pod env so the
CEPH_ARGS won't be populated correctly.

Closes: https://github.com/rook/rook/issues/4558
Signed-off-by: Sébastien Han <seb@redhat.com>
2019-12-23 16:15:46 +01:00
Juan Miguel Olmo Martínez 5f1c95002b ceph: Avoid invalid execution state
security issues identified by TrailOfBits:

Improperly handled type assertion error could result in invalid
execution state.
If we obtain an error trying to get a Node object is better not
continue executing this function.

Resolves #4567

Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
2019-12-23 11:44:18 +01:00
Sébastien Han 7d11524643 ceph: add missing vol mount for encrypted osds
"ceph-volume activate" on encrypted osds needs a ceph.conf as well as
connectivity to the mons. This commit adds that to that "activate-osd"
init container.

Closes: https://github.com/rook/rook/issues/4558
Signed-off-by: Sébastien Han <seb@redhat.com>
2019-12-23 09:59:04 +01:00
Travis Nielsen dfabccc2ee ceph: rehydrate the osd CRUSH location for PVCs
The OSDs on PVCs were changing location in the CRUSH
hierarchy after an operator restart due to the location
property not being re-discovered on the OSD daemon.
Since OSDs on PVCs do not run the prepare pod again
to discover this property, the operator must read it
off the existing osd deployment. The operator was
reading other properties, but now will read the location
as well.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2019-12-18 15:10:24 -07:00
Travis Nielsen b79c223986 ceph: only set isUpgrade when the image changes
The isUpgrade flag was being set every time the operator
was started. This caused the upgrade checks to be run every
time the operator restarted. Thus, if any mons were down,
you likely would get stuck on the upgrade checks since another
mon could not be stopped. We need to rely on the upgrade being
detected either by a different image name from the cluster CR
update, or from different versions of the daemons running
in case the operator was restarted in the middle of an upgrade.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2019-12-17 10:57:26 -07:00
Juan Miguel Olmo Martínez e8ccef854d Ceph: Toleration seconds for <NotReady> nodes is now configurable
Reduce from 5 minutes to less than 50 seconds the time needed to restart a
mgr and osds(PVCs based),rgw,mds,rbd,nfs and toolbox pod that was running
in a k8s "NotReady" Node.

A new Rook Operator setting has been added to allow the user to set the
time used in pod's toleration <node.kubernetes.io/unreachable>:
ROOK_UNREACHABLE_NODE_TOLERATION_SECONDS = <5>

New setting added in release notes

Addressed @travisn,@leseb, and @blaineEXE suggestions

Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
2019-12-16 21:38:46 -07:00
Sébastien Han ad95c7296f ceph: do not print extended format for loggers.
When using the "errors" package, using `%+v` (extended format),
each Frame of the error's StackTrace will be printed in detail.
Let's only print `%v` to print the error.
If the error has a Cause it will be printed recursively.

Basically `%+v` has been replaced with `%v` for all `error` type
interfaces, whether the logger is Info, Warning or Error.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-12-16 18:35:54 +01:00
Sébastien Han 0a1d700927 ceph: osd tune startup flags when running on GP2
gp2 devices present themselves as SSDs,
but have much more variable performance.
Ceph attempts to adjust some settings based on the drive type,
HDD/SSD
In this case client I/O can be impacted too much by assuming gp2 devices are like other SSDs
(an example is https://bugzilla.redhat.com/show_bug.cgi?id=1765923#c6)

In order to prevent this, rook can set the following for OSDs
if the new CR setting `tuneSlowDeviceClass` is set to `true`:

```
osd_recovery_sleep = 0.1
osd_snap_trim_sleep = 2
osd_delete_sleep = 2
```

Closes: https://github.com/rook/rook/issues/4298 and https://bugzilla.redhat.com/show_bug.cgi?id=1771047
Signed-off-by: Sébastien Han <seb@redhat.com>
2019-12-16 16:37:02 +01:00
Travis Nielsen fbf3837f13 ceph: stop removing osds for any cephcluster CR update
Removing an OSD unintentionally can result in data loss,
which is to be prevented at all costs. Rook's functionality
to remove OSDs automatically when a node was removed from the
cluster CR has led on several occasions to users having OSDs
unintentionally removed from their cluster.

If the admin decides they really need to remove OSDs they should
do it as a one-time action that they intentionally run
rather than the operator attempting to manage the deletion
with desired state. The risk of misinterpreting the desired
state, even with a typo, is too great.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2019-12-13 08:26:23 -07:00
rohan47 6dfc2c139f Ceph: Added version check for ceph osd safe-to-destroy command
It's not 100% safe since before Nautilus osd safe-to-destroy command
didn't have any json output. So added a version check.

Signed-off-by: rohan47 <rohgupta@redhat.com>
2019-12-13 00:07:39 +05:30
Sébastien Han 440a68f57e ceph: add crash key
As of 14.2.5, the crash collector pod will use the client.crash user
instead of the client.admin keyring to post crash dumps.

Closes: https://github.com/rook/rook/issues/4388
Signed-off-by: Sébastien Han <seb@redhat.com>
2019-12-12 09:05:38 +01:00
Sébastien Han 6051a3ca26 ceph: controller accept multiple changes at once
If editing the cluster CR we must update the spec to reflect its new
content. Now the spec is updated so we can update with an unsupported
image as well as setting allowUnsupported: true.
It was not possible before we had to restart the operator, now we can do
it as an update.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-12-11 10:35:53 +01:00
Sébastien Han b26c614ecf ceph: crash do not use selector = label
We must not default the selectors to the labels since labels can be
changed and selector cannot until the deployment is updated.
This way the deployment was failing with:

2019-12-10 10:48:13.414438 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Deployment.apps "rook-ceph-crashcollector-minikube" is invalid: spec.template.metadata.labels: Invalid value: map[string]string{"app":"rook-ceph-crashcollector", "ceph-version":"15.0.0", "ceph_daemon_id":"crash", "crashcollector":"crash", "kubernetes.io/hostname":"minikube", "node_name":"minikube", "rook-version":"v1.2.0-3.g8d3b85306.dirty"}: `selector` does not match template `labels`

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-12-11 10:35:53 +01:00
Sébastien Han e91cf1ac1d ceph: crash retry in seconds
The loop wasn't using seconds for its retry so the crash controller
would simply crash since the version is used later in the pod spec.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-12-11 10:35:53 +01:00
Travis Nielsen 3294384da7 Merge pull request #4415 from leseb/use-minor-version
rook-ceph: stop using ceph version with timestamp and bump to 14.2.5
2019-12-10 08:14:51 -07:00
Sébastien Han 68fd4f4df9 rook: stop using ceph version with timestamp
With https://github.com/ceph/ceph-container/pull/1529, we now build
vX.X.X version so we don't need to have the timestamp. Also we will
automatically get the last changes from the OS/packages too more
frequently.

Closes: https://github.com/rook/rook/issues/4414
Signed-off-by: Sébastien Han <seb@redhat.com>
2019-12-10 11:39:35 +01:00
Blaine Gardner caba125b84 Merge pull request #4370 from SUSE/osd-refactor-legacy
Ceph: OSD: refactor config for filestore OSDs
2019-12-09 16:20:17 -07:00
Blaine Gardner 003b870d8f ceph: osd: disable creation of new filestore osds
Creation of new filestore OSDs is being deprecated as per
https://github.com/rook/rook/issues/3379

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2019-12-09 14:15:13 -07:00
Blaine Gardner 64bad24712 ceph: osd: do not init ceph.conf for filestore osd
Just as with the mon, mgr, mds, rgw, and rbd-mirror daemons, do not
generate a ceph.conf in an init container for filestore OSDs.
Instead use the commandline to supply all the needed arguments for
running these OSDs.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2019-12-09 14:15:12 -07:00