Commit Graph
193 Commits
Author SHA1 Message Date
Elise Gafford 2ca0d8f93d ceph: add cleanupPolicy to cephcluster spec
In order to perform full automation of cephcluster deletion safely
and protect against catastrophic data loss, the end user must be
able to signify that they intend to irrecoverably delete the data
in their cluster. The cleanupPolicy field of the cluster spec is
intended to communicate this. This patch does not implement
automated deletion, but only creates the spec field and the
safety feature of halting orchestration other than deletion on a
cluster with a set cleanup policy value.

Partially-fixes: #3222
Signed-off-by: Elise Gafford <egafford@redhat.com>
2020-03-30 19:11:54 +05:30
Sébastien Han f268c897e9 ceph: Convert the Ceph ObjectStore controller to the controller-runtime
The CRD watcher has been replaced by the new controller-runtime
framework.
This brings robustness in our operator, meaning that any resources that
are modified will be reconciled into the desired state.

Closes: https://github.com/rook/rook/issues/4937
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-03-17 15:12:41 -06:00
Juan Miguel Olmo Martínez c87a01bb8c Ceph: Added rook manager module rights for read pod logs
This is needed to allow the ceph manager rook module to read the output
of the <lsmcli> command used to switch disk identification light on/off on
physical disk devices

[test ceph]

Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
2020-03-16 18:03:23 +01:00
Travis Nielsen 7de388ce92 ceph: grant access to the mgr to delete pods
The Ceph mgr modules need access to restart the daemon pods
when the configuration changes for the daemons.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-03-13 17:08:27 -06:00
Stefan Haas b3cc4aeb44 ceph: ceph-csi version detection #3824
Checks the version of the configured ceph-csi image while starting the operator. The operator will fail if the image is not supported.
Added an additional parameter to operator to disable the check e.g. to test not yet supported csi images.

Signed-off-by: Stefan Haas <shaas@suse.com>
2020-02-28 14:14:40 +01:00
Sébastien Han dab9233c8b ceph: add CRD setting for pool size 1
As of Octopus, Ceph will prevent you from creating a pool with a
replica size of 1. Allowing such pool could lead to data loss, so enable
the new option: requireSafeReplicaSize: false if you are **ABSOLUTELY**
certain that is what you want.

Closes: https://github.com/rook/rook/issues/4889
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-02-25 16:28:58 +01:00
desprert 712873641a helm: update charts for cephcsi v2.0.0
Updating clusterroles for cephcsi v2.0.0 permissions

Signed-off-by: Ryan Despres <rdespres117@gmail.com>
2020-02-07 22:45:25 -08:00
Travis Nielsen dbfcef8c19 ceph: set min size to 0 on pool replication
The replication and erasure code settings are mutually exclusive
so must both be treated as optional by the schema validation.
The minimum must be set to 0 to treat them as optional.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-02-06 13:48:52 -07:00
Sébastien Han cface95317 ceph: ability to set target_size_ratio
target_size_ratio is used the tell the Ceph cluster the excepted
occupancy of a pool in the cluster.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-02-06 09:13:47 +01:00
Madhu Rajanna 48a6e493e8 CSI: Add LogLevel for csi templates
Currently there is no option to turn on and off the
log level verbosity in csi containers, This PR adds
the functionality to select logging levels for csi containers.

Fixes: #4690

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-01-31 15:04:11 +05:30
Madhu Rajanna cf16184899 CSI: Add support for CSI volume expansion
This PR makes the necessary changes required
for volume expansion support for both cephfs and
rbd.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-01-28 23:50:23 +05:30
Madhu Rajanna 8c1a90ca01 CSI: upgrade node-driver-registrar from v1.1.0 to v1.2.0
latest released upstream version is v1.2.0,this
PR updates it to latest releasd version.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-01-28 23:50:15 +05:30
Madhu Rajanna 06d2dd2994 CSI: update csi-attacher from v1.2.0 to v2.1.0
external-attacher v2.x is not compatible with v1.x
This PR makes the require changes in csi templates
and upgrade documentation for easily upgrade

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-01-28 23:46:24 +05:30
Madhu Rajanna 0abb3b9e1b CSI: update ceph-csi image tag to v2.0.0
ceph-csi v2.0.0 is released. This commit updates
the image tag to latest released ceph-csi version

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-01-28 12:20:13 +05:30
Travis Nielsen d40cb217ae Merge pull request #4694 from egafford/secure-port-for-ceph-object-manifests
ceph: add securePort values for objectstore integration tests
2020-01-27 14:49:47 -07:00
Elise Gafford d7906266e1 ceph: remove validation on SecurePort for k8s <= 1.13
Versions of k8s prior to 1.14 do not support the nullable field in
CRD properties. As k8s 1.13 remains part of our support matrix and
as this field must remain nullable, we must remove CRD validation
of SecurePort. Range validation of this field has been moved into
the logic of rgw.validateStore.

Fixes: #4693
Signed-off-by: Elise Gafford <egafford@redhat.com>
2020-01-27 13:49:35 -05:00
Sébastien Han 227d2d527a ceph: osd store refactor
Multiple things:

1. We removed all the function/methods/tests that were used to
create and manage rook legacy OSDS as well as bringing support to
Bluestore OSD only.
It also fixes various go-lint issues in the respectives files.

2. use c-v inventory to detect available devices:
Now we rely on the 'ceph-volume inventory' command to tell us if a
device is available or not.

3. implement raw mode for osd on pvc
When an OSD will be bootstrap on a PVC, the new c-v raw mode will be
used. It consists of putting block, db and wal under the same device.
Here LVM is out of the picture and the raw device is used as is. The
implementation is backward compatible so existing OSD on PVC will LVM
will continue to operate.

Closes: https://github.com/rook/rook/issues/4363
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-01-23 19:13:09 +01:00
Yu Chih 1f73d59dea ceph: Add validation to cluster spec rgw port range
When creating a CephObjectStore object, operator will
use .Gateway.Port or .Gateway.SecurePort to create k8s
Service object, if the port is invalid, the Service
will not be created.

Note .Gateway.Port/.Gateway.SecurePort should in 1~65535,
however .Dashboard.Port is 0~65535 because 0 will use
default setting for .Dashboard.Port and there is no
such mechanism for .Gateway.Port/.Gateway.SecurePort

Closes: https://github.com/rook/rook/issues/4574
Signed-off-by: Yu Chih <secret104278@gmail.com>
2020-01-13 18:00:34 +08:00
Sébastien Han 05aa639317 ceph: add option to continue on unclean PGs
A new CRD option `continueUpgradeAfterChecksEvenIfNotHealthy` is added.
When upgrading, Rook goes OSD by OSD and then waits for PGs to be clean
before proceeding to the next OSD. Currently, Rook waits for 5 hours but
there might be circumstances where PGs need more time to settle.
Thus setting `continueUpgradeAfterChecksEvenIfNotHealthy` to true will
pursue the upgrade process, even if PGs are not 100% active+clean.

Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1786029
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-01-07 12:18:54 -07:00
Jeff Billimek 127e9d289a ceph: bumping ceph-operator memory limit to 256Mi
* bumping ceph-operator memory limit from 128Mi to 256Mi in order to
avoid OOMkills of the rook-ceph-operator pod
* updating documentation example for operator to denote a limit of 256Mi
instead of 128Mi

When upgrading from rook v1.1.7 to v1.2.0, the rook-ceph-operator pod
continued to get OOMKilled by the kubernetes scheduler.  Manually
bumping this limit to something higher than 128Mi (I tried 256Mi and it
was good) prevented the OOMKills.

Signed-off-by: Jeff Billimek <jeff@billimek.com>
2019-12-20 11:41:40 -05:00
Madhu Rajanna 8be3cdd10e CSI: cephfs and rbd daemonset upgrade strategy
CSI nodeplugins, specifically when using cephfs FUSE or
rbd-nbd as the mounters, when upgraded, will
cause existing mounts to become stale/not-rechable
(usually connection timeout errors).

This is due to losing the mount processes running within the
CSI nodeplugin pods.

This PR add updated the Daemonset update strategy
based on the ENV variable to take care of above issue
with some manual steps

Moreinfo: https://github.com/ceph/ceph-csi/issues/703

Resolves: https://github.com/rook/rook/issues/4248

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-12-16 23:15:56 -07:00
Juan Miguel Olmo Martínez e8ccef854d Ceph: Toleration seconds for <NotReady> nodes is now configurable
Reduce from 5 minutes to less than 50 seconds the time needed to restart a
mgr and osds(PVCs based),rgw,mds,rbd,nfs and toolbox pod that was running
in a k8s "NotReady" Node.

A new Rook Operator setting has been added to allow the user to set the
time used in pod's toleration <node.kubernetes.io/unreachable>:
ROOK_UNREACHABLE_NODE_TOLERATION_SECONDS = <5>

New setting added in release notes

Addressed @travisn,@leseb, and @blaineEXE suggestions

Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
2019-12-16 21:38:46 -07:00
Travis Nielsen 804e41132d ceph: force enable the cephfs kernel driver by default
There are two issues at odds with the cephfs driver. The
first is that older kernels less than 4.17 do not support
quotas so PVC sizes will not be enforced with the kernel
driver in that case. If the FUSE client is used instead, there
are issues during upgrade because the client mount will be
disrupted and the client pods will have to restart.

To prevent client pods from breaking during upgrade, we enable
the kernel driver by default. If the quotas are critical,
the admin will need to take steps during upgrade to prevent
breaking the applications.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2019-12-16 13:04:16 -07:00
Madhu Rajanna 85d622dc2f CSI: Add ENV variable to use ceph kernel client
In ceph-csi v1.2.2 we have add a flag `forcecephkernelclient`
to force use Ceph Kernel clients on kernel < 4.17 which
support quotas for Cephfs

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-12-16 12:36:31 -07:00
Travis Nielsen 32a69b8afc ceph: clarifications to v1.2 upgrade guide
Removed some obsolete instructions from the 1.1 upgrade
and made a few minor fixes to make the upgrade to 1.2 smoother.
With the addition of the CephClient CRD, the CRD needs to be
created before the upgrade or else the operator log will be
filled with errors until it is created. The CSI driver also
had an error when registering after the upgrade due to an RBAC
change.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2019-12-16 11:31:52 -07:00
Nizamudeen 910fe6f980 ceph: make the udevfilter into configurable
Added default regex values.
Created an environment variable and using that variable the user can give custom regex values
Updated the PendingReleaseNotes.md
Signed-off-by: Nizamudeen <nia@redhat.com>
2019-12-10 10:07:30 +05:30
Mateusz Los 50f3d29cf4 ceph: client controller tests and docs
ceph client CRD refactoring
Add info about client crd to documentation
Add tests for client controller CRD

Signed-off-by: Mateusz Los <los.mateusz@gmail.com>
2019-12-09 12:48:44 +01:00
Mateusz Gozdek 65fd8ece2a ceph: Create initial client controller code
Create initial client controller code

Signed-off-by: Mateusz Los <los.mateusz@gmail.com>
2019-12-08 16:47:57 +01:00
ruslanloman 59e17dcdf5 ceph: Allow to run rook operator on host network
There is use case when need to run rook operator on host network since
  ceph mon is not accessible from kubernetes pod network.

Signed-off-by: ruslanloman <ruslanloman@gmail.com>
2019-12-05 13:25:08 +02:00
Madhu Rajanna cc418bce3b CSI: Ability to disable snapshotter from ceph-csi rbd
we need to set a ENV `CSI_RBD_ENABLE_SNAPSHOTTER`
variable to "false" in operator template to disable
deployment of snapshotter sidecar in ceph-csi rbd
provisioner.

Fixes: #4401
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-12-03 13:27:29 +05:30
Madhu Rajanna 7f14b52ba6 CSI: update ceph-csi from v1.2.1 to v1.2.2
cephcsi v1.2.2 release is out and the image is available
for use, updating rook to use latest available release
image

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-11-21 10:59:20 +05:30
Travis Nielsen 8e3a620bc5 Merge pull request #4335 from Madhu-1/update-sidecar
CSI: update kubernetes sidecar images
2019-11-18 22:57:06 -08:00
Madhu Rajanna 49381b322d CSI: update kubernetes sidecar images
update csi sidecar images to fix CVE-2019-11255

more info:
https://github.com/kubernetes/kubernetes/issues/85233
https://github.com/kubernetes-csi/external-snapshotter/issues/193
https://github.com/kubernetes-csi/external-provisioner/issues/380

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-11-19 10:20:12 +05:30
Blaine Gardner f32cecb701 Merge pull request #3206 from d-luu/ceph_priority_classes
ceph: added priority classes to components
2019-11-18 11:27:57 -07:00
d-luu 9c755fac66 ceph: added priority classes to components
Adds priority class support to Ceph components
  to influence scheduler's pod preemption

Signed-off-by: d-luu <david@davidluu.info>
2019-11-15 16:28:41 -06:00
ShyamsundarR 48a0decc49 CSI: Make attach mandatory in ceph CSIDriver object
A recent PR #4172 added the ability for rook to create
the CSIDriver object on a kubernetes cluster. The change also
made it so that attachRequired value in the ceph CSI drivers
were set to false.

This leads to the entire attach skipping the AD controller
that deals with RWO attachment enforcement. Thus, if a
CephFS volume is marked RWO, 2 pods on different hosts can
end up mounting the same and consuming it at the same time.

Further for RBD, currently the image watchers are the only ones
that prevent a double mount of an image, other than the AD controller
enforcement of single attach for RWO RBD PVs. This has proven
not to be reliable in the face of MON failures, as the watcher
information is lost, and a second mount is allowed.

Thus, we need to bring back the attach requirement to leverage
kubernetes checks and balances for RWO volume type, and
prevent unwanted multi-attach use of the same.

This PR hence undoes some of the changes introduced by the
PR #4172, to bring the attacher side car back, and also declare
the attachRequired for both RBD and CephFS CSI plugins as true.

NOTE: Initially the attachRequired was set to false as this
improved attach times considerably on kubernetes, as the
entire phase was being avoided.

Signed-off-by: ShyamsundarR <srangana@redhat.com>
2019-11-15 16:15:53 -05:00
Rohan CJ dd3f0ca63c Ceph: Set the drain-canaries deployment ownerReference to the rook operator.
The drain canary was not being garbage collected properly.

Signed-off-by: Rohan CJ <rohantmp@gmail.com>
2019-11-13 16:04:45 +05:30
Sébastien Han f78ed36539 Merge pull request #4131 from leseb/osd-fs-flag
ceph: remove topologyAware CRD option
2019-11-06 15:08:21 +01:00
Sébastien Han 546271c179 Merge pull request #4172 from Madhu-1/csi-driver-object
CSI: Add support for CSIDriver object creation
2019-11-06 09:40:21 +01:00
Madhu Rajanna 9b368ef15c CSI: Create CSIDriver Object
This will be helpful to list the CSI drivers
deployed in the kubernetes cluster. if we have
N number of CSI drivers deployed in the cluster,
the user can query kubectl get csidriver and get
the list of registered CSI drivers and it requires
kube 1.14+

This removes the deployment of Attacher sidecar container,
as we are setting the attacher required to false during
the CSIDriver object creation.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-11-06 11:18:03 +05:30
Travis Nielsen 47764715fe ceph: remove the location from the cluster CR
The topology of the cluster should be based on the node labels rather
than a setting in the cluster CR. This allows a much richer and more
dynamic topology to be configured. The location will now be ignored
if specified in the cluster CR.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2019-11-04 23:07:13 -07:00
Sébastien Han ef6815ebfa ceph: remove topologyAware CRD option
We removed the topologyAware CRD option since it was redundant with the
use of an OSD being backed by a PVC.
So now, if an OSD is backed by a PVC we assume the topology aware
decision and will discover zone and region labels on that host.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-11-04 09:53:14 -07:00
Alexander Trost 9fe5e2ad3e ceph: helm: fix operator chart rbdGrpcMetricsPort
This fixes the conditional for the `rbdGrpcMetricsPort` option.
Previously the `rbdGrpcMetricsPort` was useless as it was not used,
instead the Helm chart looked for `cephfsrbdGrpcMetricsPort` option.

Signed-off-by: Alexander Trost <galexrt@googlemail.com>
2019-11-03 13:06:38 +01:00
Juan Miguel Olmo Martínez 9c152e74af ceph: Included missing rolebindings for PodSecurityPolicies in Helm chart
**Description of your changes:**
Include missing rollebindings present in common.yaml file and not in the
rolebindings file for the Ceph Helm chart.

Resolves #

Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
2019-10-30 17:11:40 +01:00
Sebastian Wagner b2fe1b951d ceph: Add removeOSDsIfOutAndSafeToRemove to CRD
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
2019-10-25 15:57:32 +02:00
Sébastien Han 6839b66b83 Merge pull request #4092 from S-Corkum/chart-image-update
update helm chart image
2019-10-25 10:17:33 +02:00
Travis NielsenandSean Corkum 8787a86b3c ceph: update the image in the helm chart
The image in the helm chart was broken. The url
needs to be updated to the new location for the rook
image.

Co-authored-by: Sean Corkum <seancorkum81@gmail.com>
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2019-10-24 11:36:36 -06:00
Travis Nielsen 000392533c Merge pull request #4022 from sebastian-philipp/validation-reservePoolsOnDelete
Ceph CRD Validation: Add reservePoolsOnDelete
2019-10-17 09:43:27 -06:00
Travis Nielsen b40321fa1f Merge pull request #4032 from travisn/no-hyperkube-helm
Remove obsolete hyperkube settings from the helm chart
2019-10-02 09:16:37 -06:00
travisn 66e7562045 ceph: no hyperkube settings in the helm chart
In the original help chart hyperkube was used in the prototype.
The usage of hyperkube has long since been removed from the
chart. This is cleanup to remove the reference to hyperkube
in the chart settings.

Signed-off-by: travisn <tnielsen@redhat.com>
2019-10-01 17:45:57 -06:00