Commit Graph
74 Commits
Author SHA1 Message Date
Sébastien Han dab9233c8b ceph: add CRD setting for pool size 1
As of Octopus, Ceph will prevent you from creating a pool with a
replica size of 1. Allowing such pool could lead to data loss, so enable
the new option: requireSafeReplicaSize: false if you are **ABSOLUTELY**
certain that is what you want.

Closes: https://github.com/rook/rook/issues/4889
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-02-25 16:28:58 +01:00
Travis Nielsen 57fb7cda2e Merge pull request #4714 from Madhu-1/log-level
CSI: Add LogLevel for csi templates
2020-01-31 11:54:32 -07:00
Sébastien Han 720c06cfec Merge pull request #4791 from galexrt/pickup_3689
doc: fix CephFS CSI StorageClass name
2020-01-31 14:18:13 +01:00
Alexander Trost a4f9981d83 doc: fix CephFS CSI StorageClass name
This fixes the StorageClass name example and doc to be `rook-cephfs`
instead of just `csi-cephfs`.

Closes #4130.

Signed-off-by: Alexander Trost <galexrt@googlemail.com>
2020-01-31 11:49:35 +01:00
Madhu Rajanna 48a6e493e8 CSI: Add LogLevel for csi templates
Currently there is no option to turn on and off the
log level verbosity in csi containers, This PR adds
the functionality to select logging levels for csi containers.

Fixes: #4690

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-01-31 15:04:11 +05:30
Madhu Rajanna 5a685fba77 CSI: Add support for rbd erasure coded pool
In ceph-csi v2.0.0 added a support for specifing
the erasure coded pool in storageclass which will be
used to store data.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-01-28 23:50:23 +05:30
Madhu Rajanna cf16184899 CSI: Add support for CSI volume expansion
This PR makes the necessary changes required
for volume expansion support for both cephfs and
rbd.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-01-28 23:50:23 +05:30
Madhu Rajanna 06d2dd2994 CSI: update csi-attacher from v1.2.0 to v2.1.0
external-attacher v2.x is not compatible with v1.x
This PR makes the require changes in csi templates
and upgrade documentation for easily upgrade

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-01-28 23:46:24 +05:30
Madhu Rajanna 29f8749ded CSI: Remove containerized flag from rbd daemonset template
In ceph-csi release v2.0.0, deprecated `containerized` flag
is removed.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-01-28 12:20:13 +05:30
Madhu Rajanna 47cc1adf62 CSI: run containers as non-privileged in provisioner pod
currently we are running plugin container as privileged
in the provisioner pod for both cephfs and rbd.

we are also running snapshotter container as privileged
in the provisioner pod for rbd.

as we are executing the ceph and rbd commands in provisioner
pod we dont need to be privileged, This PR removed the
privileged capability from the templates.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-01-13 09:07:21 +05:30
Madhu Rajanna 7a7e65a9b2 CSI: use empty dir to store provisioner socket file
currently we are making use of hostpath
directory to store the provisioner socket, as this
socket is not needed by anyone else other than containers
inside the provisioner pod using empty directory
to store this socket is  the best option.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-01-10 08:31:18 +05:30
Madhu Rajanna d9dd91d9fb CSI: Add privileged securityContext to all daemonset containers
On systems with SELinux enabled, non-privileged containers
can't access data of privileged containers.
Since the socket is exposed by privileged containers,
all sidecars must be privileged too.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-01-10 08:31:12 +05:30
Travis Nielsen 92e7f12451 cephcsi: add upgrade documentation for csi driver
If using ceph-fuse or rbd-nbd the application pods will need
to be restarted during upgrade. This adds a description of
the steps necessary to ensure the applications remain healthy
during upgrade.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2019-12-16 23:16:05 -07:00
Madhu Rajanna 8be3cdd10e CSI: cephfs and rbd daemonset upgrade strategy
CSI nodeplugins, specifically when using cephfs FUSE or
rbd-nbd as the mounters, when upgraded, will
cause existing mounts to become stale/not-rechable
(usually connection timeout errors).

This is due to losing the mount processes running within the
CSI nodeplugin pods.

This PR add updated the Daemonset update strategy
based on the ENV variable to take care of above issue
with some manual steps

Moreinfo: https://github.com/ceph/ceph-csi/issues/703

Resolves: https://github.com/rook/rook/issues/4248

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-12-16 23:15:56 -07:00
Madhu Rajanna 85d622dc2f CSI: Add ENV variable to use ceph kernel client
In ceph-csi v1.2.2 we have add a flag `forcecephkernelclient`
to force use Ceph Kernel clients on kernel < 4.17 which
support quotas for Cephfs

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-12-16 12:36:31 -07:00
Madhu Rajanna cc418bce3b CSI: Ability to disable snapshotter from ceph-csi rbd
we need to set a ENV `CSI_RBD_ENABLE_SNAPSHOTTER`
variable to "false" in operator template to disable
deployment of snapshotter sidecar in ceph-csi rbd
provisioner.

Fixes: #4401
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-12-03 13:27:29 +05:30
ShyamsundarR 48a0decc49 CSI: Make attach mandatory in ceph CSIDriver object
A recent PR #4172 added the ability for rook to create
the CSIDriver object on a kubernetes cluster. The change also
made it so that attachRequired value in the ceph CSI drivers
were set to false.

This leads to the entire attach skipping the AD controller
that deals with RWO attachment enforcement. Thus, if a
CephFS volume is marked RWO, 2 pods on different hosts can
end up mounting the same and consuming it at the same time.

Further for RBD, currently the image watchers are the only ones
that prevent a double mount of an image, other than the AD controller
enforcement of single attach for RWO RBD PVs. This has proven
not to be reliable in the face of MON failures, as the watcher
information is lost, and a second mount is allowed.

Thus, we need to bring back the attach requirement to leverage
kubernetes checks and balances for RWO volume type, and
prevent unwanted multi-attach use of the same.

This PR hence undoes some of the changes introduced by the
PR #4172, to bring the attacher side car back, and also declare
the attachRequired for both RBD and CephFS CSI plugins as true.

NOTE: Initially the attachRequired was set to false as this
improved attach times considerably on kubernetes, as the
entire phase was being avoided.

Signed-off-by: ShyamsundarR <srangana@redhat.com>
2019-11-15 16:15:53 -05:00
Madhu Rajanna 9b368ef15c CSI: Create CSIDriver Object
This will be helpful to list the CSI drivers
deployed in the kubernetes cluster. if we have
N number of CSI drivers deployed in the cluster,
the user can query kubectl get csidriver and get
the list of registered CSI drivers and it requires
kube 1.14+

This removes the deployment of Attacher sidecar container,
as we are setting the attacher required to false during
the CSIDriver object creation.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-11-06 11:18:03 +05:30
Sébastien Han 98bc8e4ca0 ceph: revert "CSI: Add support for CSIDriver object creation"
I merged https://github.com/rook/rook/pull/4169 too quickly and didn't
noticed we were supposed to wait for csi 1.2.2.

This reverts commit 520ad44d71.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-10-24 12:30:11 +02:00
Madhu Rajanna 520ad44d71 CSI: Add support for CSIDriver object creation
This will be helpful to list the CSI drivers deployed
in the kubernetes cluster. if we have N number of
CSI drivers deployed in the cluster, the user can
query kubectl get csidriver and get  the list of
registered CSI drivers and it requires kube 1.14+
and also the functionality is disabled by default

More info here: https://github.com/ceph/ceph-csi/pull/698

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-10-24 15:08:26 +05:30
Sébastien Han fcc2e21faf ceph: rework csi keys and secrets
We know create 4 secrets containing 4 ceph keys where each of them have
limited permissions to access the cluster.

Closes: https://github.com/rook/rook/issues/4074
Signed-off-by: Sébastien Han <seb@redhat.com>
2019-10-22 15:14:28 +02:00
awalker60 0345468b6b ceph: fix for csi-snapshotter always pulling
This switches csi-snapshotter to IfNotPresent which works better in production and offline environments.

Signed-off-by: awalker60 <awalker125@users.noreply.github.com>
2019-10-15 10:41:03 +01:00
Madhu Rajanna 0056e58ba6 csi: update cephfs registration directory name
when multiple CSI drivers are running in a cluster,
each should use unique socket for registration.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-10-10 10:52:27 +05:30
Madhu Rajanna f54099df0c CSI: Make metrics and liveness port configurable
provided ENV variables to configure the grpc
and liveness metrics port for both cephfs and
rbd CSI drivers.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-09-27 17:31:45 +05:30
Humble Chirammal 3da9514191 We were defaulting to ext4 at first and then moved to
`xfs`. However further testing shows that, ext4 should be
the default or preferred fs for RBD devices.

This patch bring that change

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2019-09-24 14:28:16 +05:30
Travis Nielsen 4cbe8b5a3e Merge pull request #3931 from Madhu-1/fix-umount
Fix umount issue when rbd-nodeplugin restarts
2019-09-23 22:21:52 -06:00
Madhu Rajanna 345f099bbc Make kubelet path configurable in operator
currently CSI templates have hardcoded kubelet
path which will  be used for  mounting PVC,this
will not work if kubelet is configured to use
different path, with this PR  kubelet path
can be configured for CSI.

Fixes: #3921

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-09-20 18:51:44 +05:30
Madhu Rajanna 915bed0c3c remove / host mount from pod template
set containerized flag as false

This is not required as we are setting
containerized=false and we have deprecated
this flag in ceph-csi.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-09-20 17:16:51 +05:30
Madhu Rajanna ac295a57a5 Remove containerized flag from provisioner
containerized is only useful in nodeplugin
not the provisioner pod

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-09-20 17:03:34 +05:30
Madhu Rajanna 3c7dc1efe4 corrent liveness container socket file
liveness container is deployed as part of
the provisioner pod so it has to connect to
the provisioner container and check the liveness.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-09-17 15:43:20 +05:30
Madhu Rajanna 1830c68947 Increase sidecar timeout from 60s to 150s
At present, the request timeout of sidecars
are at 60s and this is a request to increase
this time out value to 150s or higher. The
higher timeout value can help to reduce the
load of our backend ceph cluster and also can
avoid throttling issues at sidecars to an extend.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-09-09 08:37:14 +05:30
Madhu Rajanna f776671e3d configure cephfs and rbd servers
https://github.com/ceph/ceph-csi/pull/550 made
the servers configurable to avoid noise
on the GRPC metrics and also to avoid
unnecessary server starting

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-09-06 09:48:29 +05:30
Madhu Rajanna a8589af337 Implement grpc metrics for cephcsi
grpc metrics is made option in rook
this can be disabled/enabled with ROOK_CSI_ENABLE_GRPC_METRCS
variable in operator.yaml
this will help admin who want to check the grpc metrics
of the ceph-csi plugins

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-08-30 15:01:45 +05:30
Madhu Rajanna 8d5d9b9117 Fix liveness service creation
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-08-30 13:16:55 +05:30
Madhu Rajanna 53e33d4e08 Fix invalid port name in service monitoring
http-metrics1 port was not the service port name
defined for liveness

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-08-30 13:16:55 +05:30
Madhu Rajanna 77715a1096 change liveness port of cephfs and rbd
earlier 8080 was used by rbd for liveness and 8081
was used by cephfs for liveness. in most of the system
8080 will be sued for some other service, this will
cause  the rbd service to fail,  choosing 9080 and 9081
as ports for rbd and cephfs to avoid this  kind of
suitation.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-08-30 13:16:55 +05:30
Travis Nielsen a53503100a Merge pull request #3723 from Madhu-1/fix-pvc-res
update storageclass name in  pvc-restore
2019-08-29 11:36:27 -06:00
Madhu Rajanna ba66e53d10 Add pidlimit flag for cephfs and rbd plugin
pidlimit flag was  added only for cephfs
and rbd provisioner earlier, this PR adds
these flags to the rbd and cephfs plugin
to configure the pidlimit to max

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-08-29 22:01:12 +05:30
Madhu Rajanna b447e720f0 update storageclass name in pvc-restore
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-08-29 12:02:14 +05:30
Madhu Rajanna 78fb35a93d Use Deployment with leader election instead of StatefulSet
Deployment behaves better when a node gets disconnected
from the rest of the cluster - new provisioner leader
is elected in ~15 seconds, while it may take up to
5 minutes for StatefulSet to start a new replica.

if kube version is 1.13.x deploy provisioner as statefulset.
if kube version is higher than 1.14+ deploy provisioner
as deployment.

Refer: kubernetes-csi/external-provisioner@52d1fbc
Refer: ceph/ceph-csi#497

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-08-26 19:29:37 +05:30
Madhu Rajanna d4befa79db update plugin directory name in csi
This is required if you are running multiple
csi drivers, if multiple node plugins are
listening on the same socket they request
may go to any node plugin. to avoid this we
need to have a unique socket per csi driver.
It is recommended to have socket inside the
driver name folder.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-08-26 15:45:23 +05:30
Daniel-Pivonka 665172a9ea ceph: deploy liveness sidecar with csi
Signed-off-by: Daniel-Pivonka <dpivonka@redhat.com>
2019-08-19 14:26:47 -04:00
Travis Nielsen dd7a82fdc2 Merge pull request #3565 from nixpanic/ceph-csi/pidlimit
ceph-csi: pass --pidlimit option to provisioners
2019-08-13 22:58:26 -06:00
David Vossel da072708c3 Update example pvc.yaml to reflect name of example rdb storageclass
Signed-off-by: David Vossel <davidvossel@gmail.com>
2019-08-13 17:29:01 -04:00
Sébastien Han d7716aba47 Merge pull request #3373 from phlogistonjohn/jjm-csi-one-per-operator
Ceph-CSI: maintain one csi configuration per operator
2019-08-13 14:04:21 +02:00
John Mulligan 6ea24c8b39 ceph csi: use namespace prefix in the storageclass test yaml
Signed-off-by: John Mulligan <jmulligan@redhat.com>
2019-08-13 06:42:22 -04:00
John Mulligan 9f6c0ba2c5 csi: prevent trivial conflicts between ceph csi instances with a prefix
Add a prefix to the csi drivername when ceph csi is deployed by rook.
This aims to prevent trivial conflict between csi instances when two
rook operators are deployed into the same cluster. By default the
prefix is the "<rook-namespace>." but can by customized on the cli
and via environment vars.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2019-08-12 16:41:26 -04:00
John Mulligan 4f9333bff7 ceph csi: switch templates to new cross-cluster config map
Signed-off-by: John Mulligan <jmulligan@redhat.com>
2019-08-12 16:41:26 -04:00
Madhu Rajanna 0abacc8738 Add xfs fstype as default type in storageclass
we have seen better performance in device format and
mounting by setting the fstype to xfs
from default ext4.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2019-08-12 13:55:25 +05:30
travisn da8332a223 automatically create the csi secret with the cluster
The CSI driver requires the ceph key for administrating
the cluster. The key is stored in a secret and specified in the
storage class. The operator will now automatically generate
a secret named rook-ceph-csi that can be specified by
default in the storage class instead of requiring the
admin to create it.

Signed-off-by: travisn <tnielsen@redhat.com>
2019-08-07 10:01:29 -06:00