Commit Graph
295 Commits
Author SHA1 Message Date
Travis Nielsen 3a6deb8775 Merge pull request #7008 from TomHellier/master
#7007 Ability to configure ROOK_ALLOW_MULTIPLE_FILESYSTEMS environment to rook operator pod in helm chart
2021-01-21 11:15:34 -07:00
Madhu Rajanna 05c75b7084 ceph: add support to disable snapshotter sidecar
In some cases the user dont want to run snapshotter
container either for CephFS or RBD. In that case the
user wont install the required snapshot CRD's due
to that the snapshotter sidecar container produces
lot of noisy logs.
Snapshotter will be enabled by default for both
CephFS and RBD, but with this PR we are providing
an option to disable snapshotter sidecar deployment
either for CephFS or RBD.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-01-21 15:29:55 +05:30
Tom Hellier 5d6f4ef4d4 ceph: allowMultipleFilesystems in helm-operator
This change allows the setting of the ROOK_ALLOW_MULTIPLE_FILESYSTEMS in the helm operator

Signed-off-by: Tom Hellier <me@tomhellier.com>
2021-01-21 01:17:49 +00:00
Travis Nielsen 1705c64ced ceph: add devices to schema at overall storage level
With the 1.5 schema added to the CRDs, the devices at the root
level of the storage element were missed. Now the ability to
specify devices at the root storage level to apply to all
nodes is restored.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-01-11 11:46:27 -07:00
Blaine Gardner b8dc2a3214 ceph: update lib bucket provisioner
Update to the latest lib bucket provisioner code.
Fixes issue 6650

Modifies CRD for objectbucketclaims to fix an additional bug where an
ObjectBucket's 'ClaimRef' is lost due to the CRD validation being
specified incorrectly.

Changes OBC deletion/cleanup to delete the bucket before the user. A
user cannot be deleted without an unsafe purge option if the user has
buckets associated to it.

Does not reintroduce bug 6767 from previous fix for 6650

Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
2020-12-18 12:48:54 -07:00
Madhu Rajanna 5392bc0d07 ceph: update csi to v3.2.0
updating cephcsi to latest release version

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-12-11 20:22:36 +05:30
Sébastien Han 8aaff235bb ceph: ability to set external services ports
When the cluster is external we want to expose manager service port
along with the endpoints.
This allows us to connect but Rook will keep on using 9283 as a facing
port for Prometheus and more.

```
monitoring:
  enabled: true
  ...
  ...
  externalMgrPrometheusPort: 9283
```

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-12-04 09:52:09 +01:00
Blaine Gardner 3b4ee6c8e3 ceph: update object bucket provisioner library
The library for object bucket provisioning is updated to fix errors
during provisioning.

Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
2020-12-03 09:43:12 -07:00
Blaine Gardner 87ee8dce29 Merge pull request #6679 from leseb/add-log-collector
ceph: add log collector
2020-12-01 12:08:50 -07:00
Sébastien Han c6a87203ca ceph: add log collector
We can now collect logs directly into a side-car container.
A new CRD spec has been added:

spec:
  logCollector:
    enabled: true
    periodicity: 24h

Every 24h we will rotate log files for each Ceph daemon.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-12-01 16:30:17 +01:00
Satoru Takeuchi a17cec45a6 Merge pull request #6709 from leseb/fix-6708
ceph: fix RBAC for cron cash pruner
2020-11-27 15:18:36 +09:00
Sébastien Han 13da4a1521 ceph: fix RBAC for cron cash pruner
The rook-ceph-system service account was lacking delete permission and
the operator was complaining.

Closes: https://github.com/rook/rook/issues/6708
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-11-26 11:29:47 +01:00
Renan Campos ad9dcb0c05 ceph: periodically prune crash entries older than user-provided days
Rook's crashcollector pod posts entries to the ceph cluster when a crash occurs.
Over time the number cluster may hold crash entries needlessly.
To clean up old crash entries, this PR adds a field to the ceph cluster CR for the user to specify the number of days a crash entry should be kept for.
Providing a value for the field keepXDays creates a cronjob that runs every day at midnight, calling "ceph crash prune <keepXDays>".

Closes: https://github.com/rook/rook/issues/6332
Signed-off-by: Renan Campos <rcampos@redhat.com>
2020-11-24 17:06:18 -05:00
Madhu Rajanna ca3e23853e ceph: update cephcsi to latest release
updating cephcsi to v3.1.2 which is a latest
bugfix release.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-11-19 12:20:11 +05:30
Travis Nielsen 54560250e9 Merge pull request #6497 from sp98/osd-pdb-reconciler
ceph: OSD PDB reconciler changes
2020-11-18 17:30:24 -07:00
Santosh Pillai 8602b9c116 ceph: osd pdb reconciler changes
-creates a single PDB (max-unavailable=1) for all OSDs.  This PDB allows one OSD to go down at a given time.
-When a drain is detected, blocking PDBs (max-unavailable=0) will be created for each failure domain that is not being drained and the main PDB (max-unavilable=1) will be deleted. This will allow all the OSDs in the currently drained failure domain to be removed while blocking the deletion  of OSDs in other failure domains.
-Once the PGs are healthy again, the blocking PDBs will be deleted and the main PDB will be restored.
-Add PG healthcheck timeout
-Delete any legacy node drain pods and blocking OSD PDBs

Signed-off-by: Santosh Pillai <sapillai@redhat.com>
2020-11-18 18:25:07 +05:30
Sébastien Han afc7ecff31 ceph: add snapshot scheduling for mirrored pools
Now, we can schedule snapshots on pools from the CephBlockPool CR when
the pool is mirrored.
It can be enabled like this:

```
mirroring:
  enabled: true
  mode: pool
  snapshotSchedules:
    - interval: 24h # daily snapshots
      startTime: 14:00:00-05:00
```

Multiple schedules are supported since snapshotSchedules is a list.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-11-18 09:50:09 +01:00
Travis Nielsen ba534ce285 docs: clarify helm warning that could delete cluster
In the helm chart the CRDs are installed if crds.enabled is set to
true. If false, the helm chart will not install them. If changed
to false during an upgrade, the CRDs will be removed and the cluster
is destroyed. There is no way to prevent this while still being flexible
about CRD management, so we make the warnings as clear as possible.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-11-17 15:45:33 -07:00
Alexander Trost fa62f4ac5d ceph: allow custom labels to be added to the discover daemonset
Signed-off-by: Alexander Trost <galexrt@googlemail.com>
2020-11-12 16:20:01 +01:00
Alexander Trost 34930020ab ceph: allow pod labels to be added to CSI components
This adds the functionality to add custom pod labels to the CSI
components through the operator configuration way of env vars or config
map.

Resolves #6593

Signed-off-by: Alexander Trost <galexrt@googlemail.com>
2020-11-12 13:06:24 +01:00
Pete Birley 152a05c85e ceph: update to helm 3 for the rook chart
This updates the chart to make use of helm3 which has been released
for some time, and also permits CRDs to be installed pror to other objects
allowing the chart to be deployed at the same time as CRs for rook objects.

Co-authored-by: Pete Birley <pete@port.direct>
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-11-11 14:41:24 -07:00
Travis Nielsen 6142efe14b Merge pull request #6579 from subhamkrai/nfs-log
ceph: change nfs-ganesha log level
2020-11-11 10:09:52 -07:00
subhamkrai 7d1b5f7416 ceph: change nfs-ganesha log level
this commit allows changing the default
log level from nfs.yaml file by adding
new command.

Signed-off-by: subhamkrai <srai@redhat.com>
2020-11-11 20:34:07 +05:30
Travis Nielsen b9f692a56e ceph: disable the discovery daemon by default
The discovery daemon is not needed in most scenarios, therefore we disable it
by default. More and more clusters are moving to the cluster-on-pvc scenario
which certainly does not need the local discovery. Even where clusters are not
running on PVCs, the discovery is not needed since the device discovery is again
performed in the osd prepare job.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-11-10 13:16:52 -07:00
Satoru Takeuchi c7990777d2 ceph: make the placement of admission controller configurable
If users want to restrict the nodes where Ceph daemons should exist,
it's better to make the placement of admission controller configurable
as other daemons.

Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
2020-11-09 11:54:52 +00:00
Travis Nielsen c029b1513d ceph: remove nullable attribute from pre-1.16 schema
The nullable attribute was causing the integration tests to fail
on 1.13 and earlier where it is not supported in the schema.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
(cherry picked from commit cc8f406ce9)
2020-11-06 10:47:26 -07:00
Sébastien Han e3d032dbc1 Merge pull request #6489 from travisn/stretch-mons
ceph: Stretch cluster configuration
2020-11-06 15:16:39 +01:00
Mateusz Gozdek 8ba3762fa4 docs: fix bunch of typos
Found by running the following command:

codespell -S .git,*.png,*.jpg -L \
aks,keyserver,atleast,dne,ser,ist,files\',ba,dum,iam,te -f -H

Signed-off-by: Mateusz Gozdek <mgozdekof@gmail.com>
2020-11-06 10:01:04 +01:00
Travis Nielsen b91f4211c9 ceph: configure a stretched cluster
In clusters where only two datacenters (or similar failure domains)
are available, a different mon and osd approach is needed to deal
with the network partitions or some other reason for one of the failure
domains going down. The Ceph stretched cluster makes the mons aware
of the failure domains by configuring one as the arbiter in a third
zone, while keeping two replicas of the data in each of the data
zoens.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-11-05 09:17:52 -07:00
subhamkrai b80d83fcfa ceph: some fields in FS CRD are missing
this commit add the missing `deviceClass` field
in filesystem CRD with replication of pool spec.

Signed-off-by: subhamkrai <srai@redhat.com>
2020-11-05 20:14:11 +05:30
Travis Nielsen bea2a7bcc5 Merge pull request #6526 from Madhu-1/csi-rbac
ceph: update required rbac for csi
2020-11-04 15:17:09 -07:00
subhamkrai 4f9b8a2561 ceph: helm test for k8s 1.15 failing
helm test for k8s 1.15 failing showing errors
CRDs not found, after some latest changes to CRDS.

Signed-off-by: subhamkrai <srai@redhat.com>
2020-11-04 20:01:37 +05:30
Madhu Rajanna 5de2ded83c ceph: update required rbac for csi
as we are approaching the rook 1.5 release, we
are making a required RBAC changes for cephcsi ahead
of cephcsi release to provide smooth upgrade experience
for the users without any RBAC changes in rook minor release.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-11-04 12:32:08 +05:30
Travis Nielsen 44bf443dca Merge pull request #6495 from LalitMaganti/preserve-fs
ceph: add option to preserve filesystem on CRD deletion
2020-10-30 16:56:38 -06:00
Lalit Maganti c6aec79c4f ceph: add option to preserve filesystem on CRD deletion
Due to #6492, preservePoolsOnDelete is not useful at all for CephFS;
after the filesystem is deleted, the leftover pools cannot be
reassocaited with a newly created filesystem without wiping all
metadata. The only way we can actually preserve data is keeping around
the entire filesystem.

This commit implements a `preserveFilesystemOnDelete` option which work
similar to the existing pool preservation option but instead keeps the
whole CephFS while taking it down and removing all MDSes.

This commit also changes all documentation to refer to this new option
with the intent of essentially deprecating `preservePoolsOnDelete`. IMO,
keeping around `preservePoolsOnDelete` is actively harmful because it
lulls users into thinking their data will be safe but, in reality,
recovering from this situation is highly complex and has large potential
for data loss.

Signed-off-by: Lalit Maganti <lalitm@google.com>
2020-10-30 17:04:18 +00:00
Sébastien Han ea1d71cbfb ceph: add vault kms support for osd encryption
When the Ceph cluster runs on PVC and the OSDs are encrypted we can
store LUKS's Key Encryption Key inside a Key Management System. Today,
Rook only supports HashiCorp Vault: https://www.vaultproject.io/

The CephCluster has now a new "security" field which will plug onto the
KMS. Here is an example:

security:
  kms:
    tokenSecretName: <name of the secret containing a Vault token, used
    to authenticate>
    connectionDetails: < a map of strings containing connection
    information>

Refer to the ceph-cluster-crd documentation to lear more.

Closes: https://github.com/rook/rook/issues/6105
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-10-30 16:16:33 +01:00
subhamkraiandtravisn 1b2c15041f ceph: update deprecated CRD apiextensions.k8s.io/v1beta1 to v1
the apiextensions.k8s.io/v1beta1 version of CustomResourceDefinition
is deprecated in Kubernetes v1.16 and will no longer be supported from
v1.19. For now, we changing only for ceph and it's related documented.

Signed-off-by: subhamkrai <srai@redhat.com>
Co-authored-by: travisn <tnielsen@redhat.com>
Signed-off-by: subhamkrai <srai@redhat.com>
2020-10-29 06:26:10 +05:30
Sébastien Han b70b098405 ceph: add support for stretched cluster crush rule
The pool spec has now a new property called "replicasPerFailureDomain"
which essentially represents the number of replicas to store in each
failure domain.
Assuming the failure domain is a datacenter (if the cluster is
stretched) then you will have 2 replicas per datacenter where each
replica ends up on a different host. This gives you a total of 4
replicas and for this, the "size" must be set to 4.

Closes: https://github.com/rook/rook/issues/5591
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-10-13 16:33:34 +02:00
Daniel Chaplin 231317c5f1 core: rbac: allow manipulations on monitoring.coreos.com resources
Before it wasn't allowed but required to successfully deploy Ceph cluster with monitoring enabled.
Fixed by adding more rules to rook-ceph-system rbac role

Closes: https://github.com/rook/rook/issues/6255
Signed-off-by: Daniel Chaplin <black.danek@gmail.com>
2020-10-01 12:11:57 +03:00
Madhu Rajanna 2fa3e71733 ceph: update cephcsi to v3.1.1
as cephcsi v3.1.1 is released today
with many bug fixes and enhancements,
more details can be found at
https://github.com/ceph/ceph-csi/releases/tag/v3.1.1

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-09-29 14:03:44 +05:30
Mudit Agarwal c547b36aa0 ceph: update csi sidecar images
This PR makes the changes in csi templates and
upgrade documentation required for updating
csi sidecar images.

Signed-off-by: Mudit Agarwal <muagarwa@redhat.com>
2020-09-28 18:58:52 +05:30
Travis Nielsen 02fa0eb921 Merge pull request #6228 from Madhu-1/fix-ownerref
ceph: remove owner reference set by rook on csidriver
2020-09-18 10:09:36 -06:00
Sébastien Han 451622a955 ceph: add rbd-mirror configuration
Rook is now capable of configuring mirroring between sites. The
implementation works at different levels:

* CephBlockPool: which introduces a new `mirroring` configuration as well
as `statusCheck`. When turned on, Rook will enable mirroring on the
pool. It will also create a bootstrap peer token and store it in a
Kubernetes Secret. The name of that Secret can be found in the Status
field of the CephBlockPool CRD. This token can be fetched and used by
other clusters to configure the site as a peer. Mirroring can be
configured either at the pool or the image level.

* CephRBDMirror: which introduces a new `peers` configuration allowing
Rook to connect to peers by passing a Secret name. The administrator will
create a Kubernetes Secret with 2 keys: 'token' for the bootstrap peer
token and 'pool' for the name of pool. Once detected the rbd-mirror
controller will go ahead and import the peer configuration.

Pool mirroring status example:

```
status:
  info:
    rbdMirrorBootstrapPeerSecretName: pool-peer-token-test
  mirroringInfo:
    lastChanged: "2020-09-17T14:47:27Z"
    lastChecked: "2020-09-17T14:48:27Z"
    summary:
      summary:
        mode: image
        peers:
        - client_name: client.rbd-mirror-peer
          direction: rx-tx
          mirror_uuid: ""
          site_name: rhcs
          uuid: c50522a4-28a4-4bd3-ba68-e11780308882
        site_name: 91eae0dd-06b1-4d2c-91f3-1311c9df382b-rook-ceph
  mirroringStatus:
    lastChecked: "2020-09-17T14:48:27Z"
    summary:
      summary:
        daemon_health: OK
        health: OK
        image_health: OK
        states:
          replaying: 1
```

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-09-17 19:05:04 +02:00
Madhu Rajanna a1024ebc75 ceph: remove owner reference set by rook on csidriver
currently, we are setting the rook operator as owner
on the csidriver objects,  as csidriver is cluster
scoped object we should not set namespaced object as
owner on cluster scoped object

fixes #6162

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-09-15 11:39:27 +05:30
Alexander Trost be16bc455f docs: operator use rbac.authorization.k8s.io/v1
This replaces any uses of `rbac.authorization.k8s.io/v1beta1` with
`rbac.authorization.k8s.io/v1` affecting RBAC objects like ClusterRole
and ClusterRoleBindings, etc. to make it consistent.
This is done as some RBAC objects used the `v1beta1` and others already
using `v1`.

Signed-off-by: Alexander Trost <galexrt@googlemail.com>
2020-08-20 12:08:32 +02:00
Madhu Rajanna 244b94dbc6 ceph: pass csi resources as string in helm
Currently the CSI resources are getting passed as
the map instead of string, due to this the yaml
unmarshal is getting failed. This commit adds
the missing '|' to pass the resources as the
string instead of map

fixes: #5501

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-08-18 18:41:48 +05:30
Travis Nielsen 7481993959 Merge pull request #6017 from humblec/ceph-csi-snapshotter
Add/update ceph csi snapshotter rbac and sidecar deployments
2020-08-13 12:32:35 -06:00
Humble Chirammal 97a3283099 ceph: add snapshot create/delete and restore doc for cephfs
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2020-08-13 22:17:01 +05:30
subhamkrai e21854145a ceph: disable object store port when secureport specified
currently, the object store schema requires the
port to be non-zero. This does not allow http
access to the object store to be disabled.
so setting the port minimum to 0 to from 1 in the cr.

Signed-off-by: subhamkrai <subhamkumarrai03@gmail.com>
2020-08-12 21:57:01 +05:30
Travis Nielsen d5672595ae ceph: fix psp clusterrole name for helm chart
With the removal of aggregate roles from the ClusterRole specs, the
psp:rook clusterrole was mistakenly removed. Now we rename the clusterrole
properly so the operator will be granted privileges as expected for psp.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-08-10 15:10:01 -06:00