Similarly to block volume replication, Ceph is capable of replicating the
content of a Ceph Filesystem from one cluster to another.
For this, during the 1.6 cycle, we introduced a new CRD called
CephFilesystemMirror which effectively deploys a cephfs-mirror daemon.
However, configuring peers to enable replication between two clusters
had to be done manually.
Also various bug fix made it in Ceph eventually and the minimum required
version for this to work is to run on Ceph Pacific 16.2.5 at least.
So the automatic configuration of Ceph Filesystem peers is now possible.
By editing the CephFilesystem CRD, you can now turn on mirroring:
```yaml
mirroring:
enabled: false
# list of Kubernetes Secrets containing the peer token
# for more details see: https://docs.ceph.com/en/latest/dev/cephfs-mirroring/#bootstrap-peers
peers:
secretNames:
- secondary-cluster-peer
```
Also, the mirroring status is displayed in the CR status:
```
status:
info:
fsMirrorBootstrapPeerSecretName: fs-peer-token-myfs
mirroringStatus:
daemonsStatus:
- daemon_id: 4186
filesystems:
- filesystem_id: 2
name: myfs
lastChecked: "2021-07-01T14:16:29Z"
phase: Ready
snapshotScheduleStatus:
lastChecked: "2021-07-01T14:16:29Z"
snapshotSchedules:
- fs: myfs
path: /
rel_path: /
retention: {}
schedule: 24h
```
Closes: https://github.com/rook/rook/issues/7063
Signed-off-by: Sébastien Han <seb@redhat.com>
Creates hybrid crush rule for choosing Primary OSD for
high performing SSD devices and remaining OSD for low performance HDD devices.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
When the cluster is external we don't have any instances of rgw running
so let's not have this as a required field in the spec.
Signed-off-by: Sébastien Han <seb@redhat.com>
The topology spread constraint example allowed scheduling even
if the TSC could not be satisfied when scheduling the osd prepare
jobs. This could lead to unevenly spread OSDs especially when
the cluster was not using the zone labels. Now the topology will be
required for scheduling, to ensure proper spread of the OSDs.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The operator has very bursty behavior while it is reconciling, then
has much lower resource requirements. We reduce the memory requests to 128Mi
while the limits remain at 256Mi. When not reconciling, the background
health operations are much less resource demanding.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
When there are two mgr daemons, a sidecar of the mgr will update the
service monitor depending on the active mgr daemon. When there is only
a single mgr, the operator will create the service monitor. Now, we
add rbac for the mgr to create these resources.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Implement the first step of `design/ceph/resource-dependencies.md` to
add dependency checking when deleting a CephCluster.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
We have been having many issues with external mode with Ceph version
mismatching. The operator would have a Ceph version different than the
external cluster. The `radosgw-admin` was used to interact with S3
users, even a small version delta would cause the command to coredump.
After checking with the rgw core team it appears Rook was misusing the
CLI and the admin ops API should be used instead.
So this patch is the first introduction of go-ceph in Rook to consume
the rgw admin ops API instead of the `radosgw-admin` CLI, **only** for
user management in this initial commit.
Later we can do more such as bucket operation, zone management etc.
Closes: https://github.com/rook/rook/issues/7924
Signed-off-by: Sébastien Han <seb@redhat.com>
Currently it is not possible to set resource limits based on device classes
for different OSDs. Now adding an ability to use predefined keys for main
cluster spec Resource section to reflect resource limits for different OSDs
with different device classes.
Closes: https://github.com/rook/rook/issues/8007
Signed-off-by: Denis Egorenko <degorenko@mirantis.com>
Both MonSpec.Count and CephVersionSpec.Image should be optional for the
external cluster. So marking them as so.
Signed-off-by: Sébastien Han <seb@redhat.com>
This commit adds support for adding tolerations and affinities for
cephfs and rbd (provisioner & nodeplugin) pods separately.
Closes: #7060
Signed-off-by: Rakshith R <rar@redhat.com>
earlier, to change the operator log level we need
to change the deployment which requires a restart
the operator pod.
Now, we are changing the log level dynamically
by reading the configmap. For, backward compatibility
we still load from env var.
Signed-off-by: subhamkrai <srai@redhat.com>
Ceph v16 will be the default in Rook v1.7. Update references in the
master branch from v15 (currently v15.2.11) to v16 (currently v16.2.4).
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
added volume replication operator release-v0.1
branch to the go.mod as we are doing enhancements to the
volume replicaiton CRDS. Once we have a release will
change this to release-v0.1 branch.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
corrected the component in mirroring yaml and added
appropriate mirroring related affinity example.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
The ingress kind is found in apiversion networking.k8s.io/v1 since K8s
1.19. We now update the examples to use v1 instead of v1beta1.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
`CephObjectStore->gateway->type` is not used. Rook has only supported s3-like
interface and hasn't had no code which handles `type` field.
In addition, this field was removed from CRD in the following commit.
ceph: auto-gen crds
31db03fece
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
Enabling rbac is a legacy setting from way back when the operator
was creating the rbac internally, instead of declaring it explicitly
in the common.yaml. There is no need for this setting now.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The preparation script does not require any manual inputs anymore. The
cluster/examples/kubernetes/ceph/create-external-cluster-resources.sh
will create and set the env variable needed for the import script.
Hopefully this will reduce the friction in some installation, especially
for the mon ports.
Signed-off-by: Sébastien Han <seb@redhat.com>
This commit removes `# apiVersion: snapshot.storage.k8s.io/v1alpha1`
line in cephfs snapshotclass.yaml which was introduced in PR 7711 by mistake.
Signed-off-by: Rakshith R <rar@redhat.com>
In some clusters, the schema validation is failing for the default
object.yaml example that the value must be of a type string null.
While the null value shouldnt be strictly needed, we need working
examples by default.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The upgrade of the csv is failing to the latest schema when
the crushDeviceClass is specified and when the volumeClaimTemplate
annotations do not preserve the unknown fields. To fix the upgrade,
we preserve the fields even though it should not be necessary
from the strict schema.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Ceph v16.2.2 is released with dated tag v16.2.2-20210505. Update Ceph
images to match with or without date as appropriate.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
This commit updates external-snapshotter version to
v4.0.0 which supports snapshots v1.
Rook now defaults to enabling RBD and CephFS snapshotter
for K8s >= v1.17 and disabling it for K8s <= v1.16.
Supporting changes in documents and examples yaml files
are made.
Signed-off-by: Rakshith R <rar@redhat.com>
The VolumeSource type was only used internally when loading
the storageClassDeviceSets. The volumeSource element is never
expected to be specified in the CephCluster CR, so it is now
removed from the schema. The VolumeSource type is renamed to
deviceSet for clarity.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Because of the recent CRD changes made, the mon count had a minimum of
1, making the configuration of the external cluster impossible. The
operator would fail to add the finalizer:
```
2021-04-29 16:57:38.429451 E | ceph-cluster-controller: failed to reconcile. failed to add finalizer: failed to add finalizer "cephcluster.ceph.rook.io" on "test-external": CephCluster.ceph.rook.io "test-external" is invalid: spec.mon.count: Invalid value: 0: spec.mon.count in body should be greater than or equal to 1
```
We now fixed the CRD as well as adding the CR status.
Signed-off-by: Sébastien Han <seb@redhat.com>
The schema in v1.6 is more strict and we are seeing errors in the csv
upgrades where nullable values are not allowed for the pool settings.
We now allow the nullable settings again to fix the upgrade scenario.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Add `SecuritySpec` to specify kms details for rgw in `ObjectStoreSpec` than using existing
`SecuritySpec` used for `ClusterSpec`. The vault secret engines conflict with OSD and RGW
kms encrytpion
Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
Sébastien Han
Santosh Pillai
Blaine Gardner
Travis Nielsen
Blaine Gardner
Rakshith R
Madhu Rajanna
subhamkrai
Arun Kumar Mohan
Denis Egorenko
rohan47
Satoru Takeuchi
Jiffin Tony Thottan