Commit Graph
79 Commits
Author SHA1 Message Date
Umanga Chapagain 0e932c15eb Ceph: add CSI configurations to ConfigMap
This commit adds all the CSI configurations to ConfigMap.
This configMap can be used in combination with Env Vars
to configure Ceph CSI drivers in rook.

Signed-off-by: Umanga Chapagain <chapagainumanga@gmail.com>
2020-03-27 15:17:30 +05:30
Stefan Haas b3cc4aeb44 ceph: ceph-csi version detection #3824
Checks the version of the configured ceph-csi image while starting the operator. The operator will fail if the image is not supported.
Added an additional parameter to operator to disable the check e.g. to test not yet supported csi images.

Signed-off-by: Stefan Haas <shaas@suse.com>
2020-02-28 14:14:40 +01:00
Umanga Chapagain 7b201b8dae Ceph: adds "rook-ceph-operator-config" configmap
"rook-ceph-operator-config" can be used to override Env Var
initialized during operator deployment.

This commit makes use of the ConfigMap to determine NodeAffinity
and Tolerations for ceph-csi drivers.

Closes #3239
Signed-off-by: Umanga Chapagain <chapagainumanga@gmail.com>
2020-02-12 14:07:03 +05:30
Madhu Rajanna 48a6e493e8 CSI: Add LogLevel for csi templates
Currently there is no option to turn on and off the
log level verbosity in csi containers, This PR adds
the functionality to select logging levels for csi containers.

Fixes: #4690

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-01-31 15:04:11 +05:30
Madhu Rajanna cf16184899 CSI: Add support for CSI volume expansion
This PR makes the necessary changes required
for volume expansion support for both cephfs and
rbd.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-01-28 23:50:23 +05:30
Madhu Rajanna 8c1a90ca01 CSI: upgrade node-driver-registrar from v1.1.0 to v1.2.0
latest released upstream version is v1.2.0,this
PR updates it to latest releasd version.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-01-28 23:50:15 +05:30
Madhu Rajanna 06d2dd2994 CSI: update csi-attacher from v1.2.0 to v2.1.0
external-attacher v2.x is not compatible with v1.x
This PR makes the require changes in csi templates
and upgrade documentation for easily upgrade

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-01-28 23:46:24 +05:30
Madhu Rajanna 0abb3b9e1b CSI: update ceph-csi image tag to v2.0.0
ceph-csi v2.0.0 is released. This commit updates
the image tag to latest released ceph-csi version

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-01-28 12:20:13 +05:30
Madhu Rajanna 8be3cdd10e CSI: cephfs and rbd daemonset upgrade strategy
CSI nodeplugins, specifically when using cephfs FUSE or
rbd-nbd as the mounters, when upgraded, will
cause existing mounts to become stale/not-rechable
(usually connection timeout errors).

This is due to losing the mount processes running within the
CSI nodeplugin pods.

This PR add updated the Daemonset update strategy
based on the ENV variable to take care of above issue
with some manual steps

Moreinfo: https://github.com/ceph/ceph-csi/issues/703

Resolves: https://github.com/rook/rook/issues/4248

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-12-16 23:15:56 -07:00
Juan Miguel Olmo Martínez e8ccef854d Ceph: Toleration seconds for <NotReady> nodes is now configurable
Reduce from 5 minutes to less than 50 seconds the time needed to restart a
mgr and osds(PVCs based),rgw,mds,rbd,nfs and toolbox pod that was running
in a k8s "NotReady" Node.

A new Rook Operator setting has been added to allow the user to set the
time used in pod's toleration <node.kubernetes.io/unreachable>:
ROOK_UNREACHABLE_NODE_TOLERATION_SECONDS = <5>

New setting added in release notes

Addressed @travisn,@leseb, and @blaineEXE suggestions

Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
2019-12-16 21:38:46 -07:00
Travis Nielsen 804e41132d ceph: force enable the cephfs kernel driver by default
There are two issues at odds with the cephfs driver. The
first is that older kernels less than 4.17 do not support
quotas so PVC sizes will not be enforced with the kernel
driver in that case. If the FUSE client is used instead, there
are issues during upgrade because the client mount will be
disrupted and the client pods will have to restart.

To prevent client pods from breaking during upgrade, we enable
the kernel driver by default. If the quotas are critical,
the admin will need to take steps during upgrade to prevent
breaking the applications.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2019-12-16 13:04:16 -07:00
Madhu Rajanna 85d622dc2f CSI: Add ENV variable to use ceph kernel client
In ceph-csi v1.2.2 we have add a flag `forcecephkernelclient`
to force use Ceph Kernel clients on kernel < 4.17 which
support quotas for Cephfs

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-12-16 12:36:31 -07:00
Nizamudeen 910fe6f980 ceph: make the udevfilter into configurable
Added default regex values.
Created an environment variable and using that variable the user can give custom regex values
Updated the PendingReleaseNotes.md
Signed-off-by: Nizamudeen <nia@redhat.com>
2019-12-10 10:07:30 +05:30
ruslanloman 59e17dcdf5 ceph: Allow to run rook operator on host network
There is use case when need to run rook operator on host network since
  ceph mon is not accessible from kubernetes pod network.

Signed-off-by: ruslanloman <ruslanloman@gmail.com>
2019-12-05 13:25:08 +02:00
Madhu Rajanna cc418bce3b CSI: Ability to disable snapshotter from ceph-csi rbd
we need to set a ENV `CSI_RBD_ENABLE_SNAPSHOTTER`
variable to "false" in operator template to disable
deployment of snapshotter sidecar in ceph-csi rbd
provisioner.

Fixes: #4401
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-12-03 13:27:29 +05:30
Madhu Rajanna 7f14b52ba6 CSI: update ceph-csi from v1.2.1 to v1.2.2
cephcsi v1.2.2 release is out and the image is available
for use, updating rook to use latest available release
image

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-11-21 10:59:20 +05:30
Travis Nielsen 8e3a620bc5 Merge pull request #4335 from Madhu-1/update-sidecar
CSI: update kubernetes sidecar images
2019-11-18 22:57:06 -08:00
Madhu Rajanna 49381b322d CSI: update kubernetes sidecar images
update csi sidecar images to fix CVE-2019-11255

more info:
https://github.com/kubernetes/kubernetes/issues/85233
https://github.com/kubernetes-csi/external-snapshotter/issues/193
https://github.com/kubernetes-csi/external-provisioner/issues/380

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-11-19 10:20:12 +05:30
Blaine Gardner f32cecb701 Merge pull request #3206 from d-luu/ceph_priority_classes
ceph: added priority classes to components
2019-11-18 11:27:57 -07:00
d-luu 9c755fac66 ceph: added priority classes to components
Adds priority class support to Ceph components
  to influence scheduler's pod preemption

Signed-off-by: d-luu <david@davidluu.info>
2019-11-15 16:28:41 -06:00
ShyamsundarR 48a0decc49 CSI: Make attach mandatory in ceph CSIDriver object
A recent PR #4172 added the ability for rook to create
the CSIDriver object on a kubernetes cluster. The change also
made it so that attachRequired value in the ceph CSI drivers
were set to false.

This leads to the entire attach skipping the AD controller
that deals with RWO attachment enforcement. Thus, if a
CephFS volume is marked RWO, 2 pods on different hosts can
end up mounting the same and consuming it at the same time.

Further for RBD, currently the image watchers are the only ones
that prevent a double mount of an image, other than the AD controller
enforcement of single attach for RWO RBD PVs. This has proven
not to be reliable in the face of MON failures, as the watcher
information is lost, and a second mount is allowed.

Thus, we need to bring back the attach requirement to leverage
kubernetes checks and balances for RWO volume type, and
prevent unwanted multi-attach use of the same.

This PR hence undoes some of the changes introduced by the
PR #4172, to bring the attacher side car back, and also declare
the attachRequired for both RBD and CephFS CSI plugins as true.

NOTE: Initially the attachRequired was set to false as this
improved attach times considerably on kubernetes, as the
entire phase was being avoided.

Signed-off-by: ShyamsundarR <srangana@redhat.com>
2019-11-15 16:15:53 -05:00
Madhu Rajanna 9b368ef15c CSI: Create CSIDriver Object
This will be helpful to list the CSI drivers
deployed in the kubernetes cluster. if we have
N number of CSI drivers deployed in the cluster,
the user can query kubectl get csidriver and get
the list of registered CSI drivers and it requires
kube 1.14+

This removes the deployment of Attacher sidecar container,
as we are setting the attacher required to false during
the CSIDriver object creation.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-11-06 11:18:03 +05:30
Sébastien Han 98bc8e4ca0 ceph: revert "CSI: Add support for CSIDriver object creation"
I merged https://github.com/rook/rook/pull/4169 too quickly and didn't
noticed we were supposed to wait for csi 1.2.2.

This reverts commit 520ad44d71.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-10-24 12:30:11 +02:00
Madhu Rajanna 520ad44d71 CSI: Add support for CSIDriver object creation
This will be helpful to list the CSI drivers deployed
in the kubernetes cluster. if we have N number of
CSI drivers deployed in the cluster, the user can
query kubectl get csidriver and get  the list of
registered CSI drivers and it requires kube 1.14+
and also the functionality is disabled by default

More info here: https://github.com/ceph/ceph-csi/pull/698

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-10-24 15:08:26 +05:30
Madhu Rajanna f54099df0c CSI: Make metrics and liveness port configurable
provided ENV variables to configure the grpc
and liveness metrics port for both cephfs and
rbd CSI drivers.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-09-27 17:31:45 +05:30
Madhu Rajanna 9d298fe4f0 Add Toleration and NodeAffinity to CSI
Add support for setting Toleration and NodeAffinity
to CSI Provisioner deployment or statefulset and
plugin daemonset through ENV variables.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-09-27 10:23:12 +05:30
Madhu Rajanna 0a516d5f32 use v1.2.1 cephcsi release
update CSI deployment templates and
doc to use new cephcsi v1.2.1 release

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-09-25 16:08:55 +05:30
Madhu Rajanna 345f099bbc Make kubelet path configurable in operator
currently CSI templates have hardcoded kubelet
path which will  be used for  mounting PVC,this
will not work if kubelet is configured to use
different path, with this PR  kubelet path
can be configured for CSI.

Fixes: #3921

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-09-20 18:51:44 +05:30
Sébastien Han 7f359ba41b operator: ceph disable flexdriver
Since Ceph CSI is stable and deployed by default, we don't need to
deploy the rook-ceph-agent daemonset.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-09-18 00:28:25 +02:00
Madhu Rajanna b64768ee39 Use release ceph-csi version
replace canary tag with release ceph-csi
v1.2.0 tag

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-09-05 16:10:30 +05:30
Madhu Rajanna a8589af337 Implement grpc metrics for cephcsi
grpc metrics is made option in rook
this can be disabled/enabled with ROOK_CSI_ENABLE_GRPC_METRCS
variable in operator.yaml
this will help admin who want to check the grpc metrics
of the ceph-csi plugins

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-08-30 15:01:45 +05:30
Madhu Rajanna 78fb35a93d Use Deployment with leader election instead of StatefulSet
Deployment behaves better when a node gets disconnected
from the rest of the cluster - new provisioner leader
is elected in ~15 seconds, while it may take up to
5 minutes for StatefulSet to start a new replica.

if kube version is 1.13.x deploy provisioner as statefulset.
if kube version is higher than 1.14+ deploy provisioner
as deployment.

Refer: kubernetes-csi/external-provisioner@52d1fbc
Refer: ceph/ceph-csi#497

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-08-26 19:29:37 +05:30
Sébastien Han 40c5bc7d4e ceph: let operator watch all namespace by default
Let's configure the operator so that it watches for all the cluster in
all namespaces.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-08-22 17:19:33 +02:00
travisn 7b0a7fbfeb allow the discovery daemon to be optional
The discovery daemon is only necessary to be run when
OSDs are being created on raw devices and detect when
new devices are added to the cluster. If OSDs do not need
to be configured on devices, the discovery daemonset
has no need to be started.

Signed-off-by: travisn <tnielsen@redhat.com>
2019-08-07 10:01:29 -06:00
travisn 63e6029e82 remove csi default settings from yaml
The version of the csi driver to be installed that is officially
supported is the default baked into rook. If a different version
needs to be specified, it is officially unsupported. Therefore,
we leave the csi driver version settings unspecified in operator.yaml
so we can officially support the defaults.

Signed-off-by: travisn <tnielsen@redhat.com>
2019-08-07 10:00:55 -06:00
travisn 0a3fbf62f3 enable the ceph-csi driver by default
CSI is now the preferred storage driver for Rook.
By default both the CSI and flex drivers will be started
by the operator. In a future release the flex driver
will be deprecated, but for now is still supported.

The drivers can be disabled with an environment
variable in the operator deployment in operator.yaml.
If users are only using one driver or the other there is
no need to enable both drivers.

Signed-off-by: travisn <tnielsen@redhat.com>
2019-08-07 10:00:41 -06:00
Mateusz Gozdek f936394764 ceph: allow to set multiple tolerations for agent and discover pods
Closes #2896

Signed-off-by: Mateusz Gozdek <mgozdekof@gmail.com>
2019-07-17 23:11:18 +02:00
Umanga Chapagain 25e4275082 Rook: Added NodeAffinity to agent and discovery daemon
Previously, Rook Agent and Discovery DaemonSet deployment didn't allow
adding nodeAffinity. This commit adds nodeAffinity spec to daemonSet
deployment, which can be configured through environment variables in
operator deployment yaml.

+ Support multiple LabelKey, each with multiple LabelValue
+ Support multiple LabelKey with no value

Signed-off-by: Umanga Chapagain <chapagainumanga@gmail.com>
2019-07-05 13:48:09 +05:30
Mateusz Gozdek 04c70bfe3f ceph/operator.yaml: fix alignment and add newline
This commit fixes misaligned example values in operator.yaml and adds a
newline at the end of the file, which is considered a good practice.

Signed-off-by: Mateusz Gozdek <mgozdekof@gmail.com>
2019-05-31 11:02:48 +02:00
Sébastien Han 0fc3775b99 deployment: OLM integration
Add necessary script to auto-generate a CSV file for OLM.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-05-16 22:53:30 +02:00
Noah Watkins f2fc644190 ceph: add operator option to disable hotplug
this adds a new operator environment variable that allows automatic
orchestration triggered by device discovery to be disabled.

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
2019-04-23 13:44:30 -07:00
travisn 5ebb651e37 ceph: update the cephcluster custom resource with ceph health
The CephCluster resource will now expose the health of the ceph cluster
so admins can query the status with k8s api or kubectl instead of
needing to run the rook toolbox. The operator will query the
ceph status periodically and update the status in the custom resource.

Signed-off-by: travisn <tnielsen@redhat.com>
2019-04-10 14:41:18 -06:00
travisn eb74bf5547 Configure a cluster in the same namespace as the operator
To simplify the deployment of Rook, both the operator and cluster crd are now created in the
same namespace in the examples. By default the operator will only manage a cluster crd
in the same namespace instead of watching for clusters in all namespaces. To manage a cluster
in another namespace, the roles in latter half of operator-common.yaml need to point to the new
namespace and the operator env var ROOK_CURRENT_NAMESPACE_ONLY should be set to false.

Signed-off-by: travisn <tnielsen@redhat.com>
2019-04-04 12:42:08 -06:00
travisn ceb2be1aca refactor the ceph cluster crd and related example yamls
The RBAC is tedious to copy everywhere with the cluster.yaml and is confusing to newcomers.
The common resources are now factored into the operator-common.yaml for the common scenario
of creating a single cluster. Examples are now added for configuration of the operator
and cluster crds to provide a more complete set.

Signed-off-by: travisn <tnielsen@redhat.com>
2019-04-04 12:42:07 -06:00
travisn 7548e76bab Increase the number of mons when nodes are added
The desired number of mons could change depending on the number of nodes in a cluster.
For example, three mons would be the min number of mons in a production cluster with at
least three nodes. If there are five or more nodes, the number of mons could increase
to five in order to increase the failure tolerance to two nodes.

This is accomplished by a new setting in the cluster CRD preferredCount. If the number
of hosts exceeds preferredCount, more mons are added to quorum. If the number
of hosts drops below the preferred count, the operator would reduce the quorum size to
the smaller desired count.

Signed-off-by: travisn <tnielsen@redhat.com>
2019-03-19 13:00:37 -06:00
Travis Nielsen 3b14e51c16 Merge pull request #2678 from jtlayton/wip-bootstrap
Add a way to start up a minimal ceph cluster
2019-02-21 13:31:05 -07:00
Jeff Layton 20de4920ba examples/ceph: fix permissions on the files
The yaml files are not executable.

Signed-off-by: Jeff Layton <jlayton@redhat.com>
2019-02-21 15:20:18 -05:00
Madhu Rajanna 2a80d505ae Fix misspelled words
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-02-20 19:07:29 +05:30
Huamin Chen 1b21e42a90 convert extensions to apps
Signed-off-by: Huamin Chen <hchen@redhat.com>
2019-02-14 16:06:05 -05:00
travisn b72c6436ef handle ceph mon failover more gracefully
Signed-off-by: travisn <tnielsen@redhat.com>
2019-02-11 17:35:28 -07:00