`cluster-on-local-pvc.yaml` doesn't work due to missing topologyKey.
In addition, it's better to add the example of `topologySpreadConstraints`
for K8s >= 1.18 users.
Closes: https://github.com/rook/rook/issues/8478
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
earlier, we had the issue for osd placement regarding
whether we should merge the placements or not.
Now, we have option `skipApplyAllPlacement` to
enable/disable osd placement. By default it is false
which means it will merge both placement.All() and
storageClassDeviceSets.Placement, when true
it will only apply storageClassDeviceSets.Placement.
Also, adding unit test.
Closes: https://github.com/rook/rook/issues/8135
Signed-off-by: subhamkrai <srai@redhat.com>
The `None` value Indicates that volumes will be mounted
with no modifications, as the CSI volume driver does not
support these operations. While volumes are provisioned
by the CephFS CSI driver the global permissions are set
on the volume and we dont expect the Fsgroup policy or
check from CO side to play a role here.
Ref #ceph/ceph-csi/../internal/cephfs/nodeserver.go#L190
```
!csicommon.MountOptionContains(fuseMountOptions, readOnly) {
// #nosec - allow anyone to write inside the stagingtarget path
err = os.Chmod(stagingTargetPath, 0o777)
```
The current default value ie `ReadWriteOnceWithFSType` cause
volumes to be examined to determine if volume ownership and permissions
should be modified to match the pod's security policy. Changes could occur
if the fsType is defined and the persistent volume's accessModes contains
ReadWriteOnce.
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
This PR updates the required RBAC, templates,
CSI image version and examples for new
cephcsi v3.4.0 release.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
Adds volumes and mounts for /var/lib/rook and /etc/ceph to the operator deployment in the Helm
chart. This is consistent with the example operator manifest and is sometimes even required on
clusters with stricter permissions on the local filesystem.
Signed-off-by: Dominik Münch <d.muench@celonis.com>
This commit adds an ingress resource to the rook-ceph-cluster helm chart, allowing
ingress to the ceph-dashboard service. It also adds the ability to define the
various storage types that you can run on ceph inside kubernetes.
Closes https://github.com/rook/rook/issues/8384
Signed-off-by: Tom Hellier <me@tomhellier.com>
References to luminous and mimic, and unnecessary references to nautilus
were found in the docs that have been cleaned up and clarified. At the same
time, the tests are updated to use more recent versions.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Instead of having the RBDMirroringPeerSpec in the RBDMirror CRD we want
to move it on the CephBlockPool CRD so that each pool can have its own peer.
This enables pool to re-use an existing peer secret if it points to the
same cluster peer.So we don't need to duplicate secret peer anymore.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
If a user were to `make build` Rook, and after the Ceph CSV was
generated and before make had finished, the Rook repo would be left with
changes caused by the build modifying files temporarily in-place.
All CSV template generation in the Ceph makefile is now self-contained
and can be configured to operate outside of the main repo so that files
are not modified in-place.
In-place modification of files was also incorrect. On Mac some files
with `-e` trailing were left, and in-place backups were also present in
the form `<filename>''`. Therefore, also fix the generation to fix
in-place modifications.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
Sometimes the default 15s is not enough for timeout of ceph commands. For examples,
I encountered that `radosgw-admin` command took dozens of seconds under heavy load.
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
Since we changed the Rook orchestrator module for Ceph, it now has
to access Storage Classes and Persistent Volumes in the cluster to
gather inventory and create OSDs so we have to make changes to the
rook-ceph-mgr-cluster role so the orchestrator has permission to
access these resources.
Signed-off-by: Joseph Sawaya <jsawaya@redhat.com>
Multiple filesystems are supported as of the Ceph Pacific release. So we
remove the flag from the operator configuration and just have a ceph
version check instead.
The Operator configuration option `ROOK_ALLOW_MULTIPLE_FILESYSTEMS`
has been removed in favor of simply verifying the Ceph version is
at least Pacific.
Multiple filesystems are stable since Ceph Pacific.
So users who had `ROOK_ALLOW_MULTIPLE_FILESYSTEMS` enabled will
need to update their Ceph version to Pacific.
Closes: https://github.com/rook/rook/issues/7183
Signed-off-by: Sébastien Han <seb@redhat.com>
Recently, the builds of `ceph/ceph` image moved to quay.io, see
https://github.com/ceph/ceph-build/pull/1883 for more details.
Current images will remain but new builds will happen on quay.io only.
This means that tags such as `v14.2`, `v15.2`,`v16.2` will need to
switch to quay.io to get updates.
Signed-off-by: Sébastien Han <seb@redhat.com>
Currently ceph osd purge job rely on rook-ceph-system service account which is only created
on the rook operator namespace
When multiple clusters are deployed on distinct namespace the ceph osd purge failed to run
as the service account is not available
Creating a specific service account rook-ceph-purge-osd with appropriate rights on each cluster
namespace and operator namespace to still manage case where cluster is installed
along side the operator
Signed-off-by: n.fraison <n.fraison@criteo.com>
The CRDs v1 requires the full schema for all settings, so we now
generate the CRDs for nfs for full fidelity of all settings.
Co-authored-by: Nicolaj Græsholt <figaw@hotmail.com>
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The CRDs v1 requires the full schema for all settings, so we now
generate the CRDs for cassandra for full fidelity of all the
settings.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Operator can run on a namespace not named rook-ceph
The ClusterRoleBinding muste rely on .Release.Namespace helm value
Also move the ClusterRole and ClusterRoleBinding in appropriate yaml file
Signed-off-by: n.fraison <n.fraison@criteo.com>
When the CephCluster is configured with Multus and multiple networks are
used to deploy Ceph some commands are failing to be executed from the
Operator. These commands, in particular, `radosgw-admin` ones need access
to the "ceph public network" to talk to OSDs. Unfortunately, the
Rook-Ceph Operator does not have the network annotations and thus
doesn't have the networks available and cannot reach OSDs. So the commands end
up hanging and eventually time out.
Applying the annotations to the Operator pod is possible but will result
in restarting the operator too and this should be avoided at all costs.
Also, applying the annotations beforehand is not possible since the
Multus declaration is in the CephCluster specification. So we would have
no idea what to do.
So the current approach runs a new sidecar container in the mgr pod to
act as a proxy for "some" ceph commands, only the `radosgw-admin` ones
for multi-site setup. This is a small container with admin access
running idle waiting for commands to be executed. In a sense, it is
similar to the toolbox but we didn't want to clearly expose it, so
running as a sidecar is quite nice.
Proxying command is obviously not always recommended since we add an
extra hop in the network path. Now each request has to go from the
operator pod to the API server to the remote pod to Ceph. Previously,
the command only goes from the operator to Ceph.
It's worth noting that external mode is not impacted since no rgw pod
is configured. This scenario is flexible and allows us to scale
pretty well since any CephCluster with Multus will see its mgr sidecar
deployed and can then talk to Ceph. We are not limited.
Signed-off-by: Sébastien Han <seb@redhat.com>
ci is complaining due to empty field in
values.yaml with error
```
coalesce.go:199: warning: destination for config is a table. Ignoring non-table value <nil>
```
this commit make those field empty map/list.
Closes: https://github.com/rook/rook/issues/8239
Signed-off-by: subhamkrai <srai@redhat.com>
adds a default empty dict for the network when deciding whether toolbox
should also be on the host network to catch the case when the entire
`network` block is empty
Signed-off-by: Henry Zhang <me@henry.dev>
Similarly to block volume replication, Ceph is capable of replicating the
content of a Ceph Filesystem from one cluster to another.
For this, during the 1.6 cycle, we introduced a new CRD called
CephFilesystemMirror which effectively deploys a cephfs-mirror daemon.
However, configuring peers to enable replication between two clusters
had to be done manually.
Also various bug fix made it in Ceph eventually and the minimum required
version for this to work is to run on Ceph Pacific 16.2.5 at least.
So the automatic configuration of Ceph Filesystem peers is now possible.
By editing the CephFilesystem CRD, you can now turn on mirroring:
```yaml
mirroring:
enabled: false
# list of Kubernetes Secrets containing the peer token
# for more details see: https://docs.ceph.com/en/latest/dev/cephfs-mirroring/#bootstrap-peers
peers:
secretNames:
- secondary-cluster-peer
```
Also, the mirroring status is displayed in the CR status:
```
status:
info:
fsMirrorBootstrapPeerSecretName: fs-peer-token-myfs
mirroringStatus:
daemonsStatus:
- daemon_id: 4186
filesystems:
- filesystem_id: 2
name: myfs
lastChecked: "2021-07-01T14:16:29Z"
phase: Ready
snapshotScheduleStatus:
lastChecked: "2021-07-01T14:16:29Z"
snapshotSchedules:
- fs: myfs
path: /
rel_path: /
retention: {}
schedule: 24h
```
Closes: https://github.com/rook/rook/issues/7063
Signed-off-by: Sébastien Han <seb@redhat.com>
Creates hybrid crush rule for choosing Primary OSD for
high performing SSD devices and remaining OSD for low performance HDD devices.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
When the cluster is external we don't have any instances of rgw running
so let's not have this as a required field in the spec.
Signed-off-by: Sébastien Han <seb@redhat.com>
The topology spread constraint example allowed scheduling even
if the TSC could not be satisfied when scheduling the osd prepare
jobs. This could lead to unevenly spread OSDs especially when
the cluster was not using the zone labels. Now the topology will be
required for scheduling, to ensure proper spread of the OSDs.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Denis Egorenko
Satoru Takeuchi
Travis Nielsen
Sébastien Han
subhamkrai
Humble Chirammal
Madhu Rajanna
Dominik Münch
Blaine Gardner
Tom Hellier
Santosh Pillai
Blaine Gardner
Joseph Sawaya
Timothy Asir Jeyasingh
n.fraison
Henry Zhang
Rakshith R