In some cases the user dont want to run snapshotter
container either for CephFS or RBD. In that case the
user wont install the required snapshot CRD's due
to that the snapshotter sidecar container produces
lot of noisy logs.
Snapshotter will be enabled by default for both
CephFS and RBD, but with this PR we are providing
an option to disable snapshotter sidecar deployment
either for CephFS or RBD.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
With the 1.5 schema added to the CRDs, the devices at the root
level of the storage element were missed. Now the ability to
specify devices at the root storage level to apply to all
nodes is restored.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Update to the latest lib bucket provisioner code.
Fixes issue 6650
Modifies CRD for objectbucketclaims to fix an additional bug where an
ObjectBucket's 'ClaimRef' is lost due to the CRD validation being
specified incorrectly.
Changes OBC deletion/cleanup to delete the bucket before the user. A
user cannot be deleted without an unsafe purge option if the user has
buckets associated to it.
Does not reintroduce bug 6767 from previous fix for 6650
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
When the cluster is external we want to expose manager service port
along with the endpoints.
This allows us to connect but Rook will keep on using 9283 as a facing
port for Prometheus and more.
```
monitoring:
enabled: true
...
...
externalMgrPrometheusPort: 9283
```
Signed-off-by: Sébastien Han <seb@redhat.com>
We can now collect logs directly into a side-car container.
A new CRD spec has been added:
spec:
logCollector:
enabled: true
periodicity: 24h
Every 24h we will rotate log files for each Ceph daemon.
Signed-off-by: Sébastien Han <seb@redhat.com>
Rook's crashcollector pod posts entries to the ceph cluster when a crash occurs.
Over time the number cluster may hold crash entries needlessly.
To clean up old crash entries, this PR adds a field to the ceph cluster CR for the user to specify the number of days a crash entry should be kept for.
Providing a value for the field keepXDays creates a cronjob that runs every day at midnight, calling "ceph crash prune <keepXDays>".
Closes: https://github.com/rook/rook/issues/6332
Signed-off-by: Renan Campos <rcampos@redhat.com>
-creates a single PDB (max-unavailable=1) for all OSDs. This PDB allows one OSD to go down at a given time.
-When a drain is detected, blocking PDBs (max-unavailable=0) will be created for each failure domain that is not being drained and the main PDB (max-unavilable=1) will be deleted. This will allow all the OSDs in the currently drained failure domain to be removed while blocking the deletion of OSDs in other failure domains.
-Once the PGs are healthy again, the blocking PDBs will be deleted and the main PDB will be restored.
-Add PG healthcheck timeout
-Delete any legacy node drain pods and blocking OSD PDBs
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
Now, we can schedule snapshots on pools from the CephBlockPool CR when
the pool is mirrored.
It can be enabled like this:
```
mirroring:
enabled: true
mode: pool
snapshotSchedules:
- interval: 24h # daily snapshots
startTime: 14:00:00-05:00
```
Multiple schedules are supported since snapshotSchedules is a list.
Signed-off-by: Sébastien Han <seb@redhat.com>
In the helm chart the CRDs are installed if crds.enabled is set to
true. If false, the helm chart will not install them. If changed
to false during an upgrade, the CRDs will be removed and the cluster
is destroyed. There is no way to prevent this while still being flexible
about CRD management, so we make the warnings as clear as possible.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
This adds the functionality to add custom pod labels to the CSI
components through the operator configuration way of env vars or config
map.
Resolves#6593
Signed-off-by: Alexander Trost <galexrt@googlemail.com>
This updates the chart to make use of helm3 which has been released
for some time, and also permits CRDs to be installed pror to other objects
allowing the chart to be deployed at the same time as CRs for rook objects.
Co-authored-by: Pete Birley <pete@port.direct>
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The discovery daemon is not needed in most scenarios, therefore we disable it
by default. More and more clusters are moving to the cluster-on-pvc scenario
which certainly does not need the local discovery. Even where clusters are not
running on PVCs, the discovery is not needed since the device discovery is again
performed in the osd prepare job.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
If users want to restrict the nodes where Ceph daemons should exist,
it's better to make the placement of admission controller configurable
as other daemons.
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
The nullable attribute was causing the integration tests to fail
on 1.13 and earlier where it is not supported in the schema.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
(cherry picked from commit cc8f406ce9)
Found by running the following command:
codespell -S .git,*.png,*.jpg -L \
aks,keyserver,atleast,dne,ser,ist,files\',ba,dum,iam,te -f -H
Signed-off-by: Mateusz Gozdek <mgozdekof@gmail.com>
In clusters where only two datacenters (or similar failure domains)
are available, a different mon and osd approach is needed to deal
with the network partitions or some other reason for one of the failure
domains going down. The Ceph stretched cluster makes the mons aware
of the failure domains by configuring one as the arbiter in a third
zone, while keeping two replicas of the data in each of the data
zoens.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
as we are approaching the rook 1.5 release, we
are making a required RBAC changes for cephcsi ahead
of cephcsi release to provide smooth upgrade experience
for the users without any RBAC changes in rook minor release.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
Due to #6492, preservePoolsOnDelete is not useful at all for CephFS;
after the filesystem is deleted, the leftover pools cannot be
reassocaited with a newly created filesystem without wiping all
metadata. The only way we can actually preserve data is keeping around
the entire filesystem.
This commit implements a `preserveFilesystemOnDelete` option which work
similar to the existing pool preservation option but instead keeps the
whole CephFS while taking it down and removing all MDSes.
This commit also changes all documentation to refer to this new option
with the intent of essentially deprecating `preservePoolsOnDelete`. IMO,
keeping around `preservePoolsOnDelete` is actively harmful because it
lulls users into thinking their data will be safe but, in reality,
recovering from this situation is highly complex and has large potential
for data loss.
Signed-off-by: Lalit Maganti <lalitm@google.com>
When the Ceph cluster runs on PVC and the OSDs are encrypted we can
store LUKS's Key Encryption Key inside a Key Management System. Today,
Rook only supports HashiCorp Vault: https://www.vaultproject.io/
The CephCluster has now a new "security" field which will plug onto the
KMS. Here is an example:
security:
kms:
tokenSecretName: <name of the secret containing a Vault token, used
to authenticate>
connectionDetails: < a map of strings containing connection
information>
Refer to the ceph-cluster-crd documentation to lear more.
Closes: https://github.com/rook/rook/issues/6105
Signed-off-by: Sébastien Han <seb@redhat.com>
the apiextensions.k8s.io/v1beta1 version of CustomResourceDefinition
is deprecated in Kubernetes v1.16 and will no longer be supported from
v1.19. For now, we changing only for ceph and it's related documented.
Signed-off-by: subhamkrai <srai@redhat.com>
Co-authored-by: travisn <tnielsen@redhat.com>
Signed-off-by: subhamkrai <srai@redhat.com>
The pool spec has now a new property called "replicasPerFailureDomain"
which essentially represents the number of replicas to store in each
failure domain.
Assuming the failure domain is a datacenter (if the cluster is
stretched) then you will have 2 replicas per datacenter where each
replica ends up on a different host. This gives you a total of 4
replicas and for this, the "size" must be set to 4.
Closes: https://github.com/rook/rook/issues/5591
Signed-off-by: Sébastien Han <seb@redhat.com>
Before it wasn't allowed but required to successfully deploy Ceph cluster with monitoring enabled.
Fixed by adding more rules to rook-ceph-system rbac role
Closes: https://github.com/rook/rook/issues/6255
Signed-off-by: Daniel Chaplin <black.danek@gmail.com>
This PR makes the changes in csi templates and
upgrade documentation required for updating
csi sidecar images.
Signed-off-by: Mudit Agarwal <muagarwa@redhat.com>
Rook is now capable of configuring mirroring between sites. The
implementation works at different levels:
* CephBlockPool: which introduces a new `mirroring` configuration as well
as `statusCheck`. When turned on, Rook will enable mirroring on the
pool. It will also create a bootstrap peer token and store it in a
Kubernetes Secret. The name of that Secret can be found in the Status
field of the CephBlockPool CRD. This token can be fetched and used by
other clusters to configure the site as a peer. Mirroring can be
configured either at the pool or the image level.
* CephRBDMirror: which introduces a new `peers` configuration allowing
Rook to connect to peers by passing a Secret name. The administrator will
create a Kubernetes Secret with 2 keys: 'token' for the bootstrap peer
token and 'pool' for the name of pool. Once detected the rbd-mirror
controller will go ahead and import the peer configuration.
Pool mirroring status example:
```
status:
info:
rbdMirrorBootstrapPeerSecretName: pool-peer-token-test
mirroringInfo:
lastChanged: "2020-09-17T14:47:27Z"
lastChecked: "2020-09-17T14:48:27Z"
summary:
summary:
mode: image
peers:
- client_name: client.rbd-mirror-peer
direction: rx-tx
mirror_uuid: ""
site_name: rhcs
uuid: c50522a4-28a4-4bd3-ba68-e11780308882
site_name: 91eae0dd-06b1-4d2c-91f3-1311c9df382b-rook-ceph
mirroringStatus:
lastChecked: "2020-09-17T14:48:27Z"
summary:
summary:
daemon_health: OK
health: OK
image_health: OK
states:
replaying: 1
```
Signed-off-by: Sébastien Han <seb@redhat.com>
currently, we are setting the rook operator as owner
on the csidriver objects, as csidriver is cluster
scoped object we should not set namespaced object as
owner on cluster scoped object
fixes#6162
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
This replaces any uses of `rbac.authorization.k8s.io/v1beta1` with
`rbac.authorization.k8s.io/v1` affecting RBAC objects like ClusterRole
and ClusterRoleBindings, etc. to make it consistent.
This is done as some RBAC objects used the `v1beta1` and others already
using `v1`.
Signed-off-by: Alexander Trost <galexrt@googlemail.com>
Currently the CSI resources are getting passed as
the map instead of string, due to this the yaml
unmarshal is getting failed. This commit adds
the missing '|' to pass the resources as the
string instead of map
fixes: #5501
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
currently, the object store schema requires the
port to be non-zero. This does not allow http
access to the object store to be disabled.
so setting the port minimum to 0 to from 1 in the cr.
Signed-off-by: subhamkrai <subhamkumarrai03@gmail.com>
With the removal of aggregate roles from the ClusterRole specs, the
psp:rook clusterrole was mistakenly removed. Now we rename the clusterrole
properly so the operator will be granted privileges as expected for psp.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>