Commit Graph
435 Commits
Author SHA1 Message Date
Travis Nielsen 0591753673 docs: update master doc links to latest
With the renaming of master docs to latest, all the links also
need to be updated.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-09-22 12:41:32 -06:00
Madhu Rajanna ed5f281a74 ceph: make provisioner replicas configurable
added new option to set the provisioner replicas.
with this new option the user/admin can choose
how many replicas he want for provisioner pod if
number of nodes is greater than 1.

fixes #8153

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-09-22 11:24:53 +05:30
Travis Nielsen 6bcb569a88 Merge pull request #8771 from kubealex/patch-1
helm: Add possibility to default filesystem storageclass in rook-ceph-cluster chart
2021-09-21 13:05:22 -06:00
kubealex d5f42aa4a1 ceph: corrected placement of values
I fixed the placement of the modification, from object storage to filesystem

Signed-off-by: kubealex <al.rossi87@gmail.com>
2021-09-21 20:07:27 +02:00
kubealex eead60457b ceph: add default field to filesystem-sc helm chart
I Added the chance to default filesystem storageclass in helm chart

Signed-off-by: kubealex <al.rossi87@gmail.com>
2021-09-21 14:49:03 +02:00
Sébastien Han 0c33493f27 ceph: bump manifests to ceph pacific 16.2.6
New version is out so let's use it.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-09-21 09:16:32 +02:00
Blaine Gardner 83c05324ce ceph: remove unused crd permission from roles
Remove unused ability to modify CRDs from Rook cluster roles. Rook (as
an operator) should not have the ability to modify CRDs used by the
components it controls.

Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
2021-09-14 12:52:06 -06:00
Bryton Hall 5e4ed30bc9 ceph: add networking.k8s.io/v1 Ingress chart compatability
Kubernetes cluster versions 1.22+ must use the v1 Ingress version.

Signed-off-by: Bryton Hall <email@bryton.io>
2021-09-09 09:11:49 -04:00
Jiffin Tony Thottan ca43800119 ceph: add options for cephobjectstore user
Adding options for quota, bucket limit, caps for the
`cephobjectstoreuser`.

Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
2021-09-07 22:43:09 +05:30
Travis Nielsen 849f434a79 ceph: only allow up to 9 mons to be specified
Typical clusters only require 3 or maybe 5 mons. If someone really wants
to go crazy they can add more mons, but over 9 mons really should not be
used.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-09-03 07:43:41 -06:00
Travis Nielsen 1cb97574ea ceph: allow an even number of mons
While an even number of mons can cause lower availability of
mon quorum, it also can provide higher durability for the cluster.
Mon quorum can be restored from a single mon according to the
disaster recovery guide, so there may be scenarios where
an even number of mons may be preferable.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-09-03 07:43:32 -06:00
Sébastien Han ab166742c5 ceph: avoid double reconcile with default value
When setting a default values with kubebuilder flags it seems that the
kube operator intercepts the API call, mutate the CR and inject it.
This results in two events, one for creation and one for edition and we
end up with 2 reconciles. Let's avoid this.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-09-02 13:49:52 +02:00
Sébastien Han 656dd0f334 ceph: move the admission webhook to the operator
Our admission webhooks will now run as part of the Operator container
and not an additional deployment. This has the advantage of consuming
fewer resources in the cluster and not having to manage affinities and
tolerations. This only drawback is that the Secret containing the
certificates is not mounted anymore and the content needs to be written
inside the Operator. This is not practical since we also need to watch
for the Secret content to change. Meaning that the certificates have
been renewed and the webhook server needs to use them.
A new approach is on its way to hopefully simplify this last issue and
implement a watcher for the Secret.
In the meantime, users need to use the cert-manager or renew
certificates manually. Additionally, they must update the
ValidatingWebhookConfiguration object with the new CA bundle.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-08-24 19:07:04 +02:00
Joseph Sawaya 331e06659e ceph: add permissions to rook-ceph-mgr role for osd removal
This commit adds permissions to the rook-ceph-mgr role
to delete PVCs and deployments and patch deployment scale
to remove OSDs using `ceph orch osd rm` in the rook orchestrator.

Signed-off-by: Joseph Sawaya <jsawaya@redhat.com>
2021-08-19 16:43:01 -04:00
subhamkrai 1681cfe89d ceph: add -0 for helm k8s version check
to use prerelease of kubernetes it requires
to have `-0` added in kubernetes version check
ex:
```{{- if semverCompare ">=1.16.0-0" .Capabilities.KubeVersion.GitVersion }}```

Closes: https://github.com/rook/rook/issues/8548
Signed-off-by: subhamkrai <srai@redhat.com>
2021-08-18 19:15:48 +05:30
Blaine Gardner 36c53fe6f3 ceph: make storage device config nullable
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
2021-08-17 12:21:32 -06:00
Ali Maredia 7a804ac27e ceph: tweak to types.go and additions after make crds
This commit will be squashed.

Signed-off-by: Ali Maredia <amaredia@redhat.com>
2021-08-11 13:25:44 -04:00
Denis Egorenko fb04908315 ceph: add ability to specify ca bundle for rgw
Specify ca Bundle for RGW spec and mount inside pods.

Related-Issue: https://github.com/rook/rook/issues/8490
Signed-off-by: Denis Egorenko <degorenko@mirantis.com>
2021-08-10 23:18:26 +04:00
Travis Nielsen cc16b60f5f Merge pull request #8381 from subhamkrai/osd-placement-merge
ceph: add an option to enable/disable merge all placement
2021-08-04 07:23:33 -06:00
Sébastien Han 1ddf28144a Merge pull request #8459 from humblec/cephfs-fsgroup
ceph: set default FsGroupChangePolicy value to 'None'
2021-08-04 11:03:41 +02:00
subhamkrai 9838313398 ceph: add an option to enable/disable merge all placement
earlier, we had the issue for osd placement regarding
whether we should merge the placements or not.

Now, we have option `skipApplyAllPlacement` to
enable/disable osd placement. By default it is false
which means it will merge both placement.All() and
storageClassDeviceSets.Placement, when true
it will only apply storageClassDeviceSets.Placement.

Also, adding unit test.

Closes: https://github.com/rook/rook/issues/8135
Signed-off-by: subhamkrai <srai@redhat.com>
2021-08-03 19:47:40 +05:30
Travis Nielsen f9042f0fb9 Merge pull request #8440 from muenchdo/patch-1
ceph: add empty dirs for /var/lib/rook and /etc/ceph
2021-08-03 07:32:34 -06:00
Humble Chirammal 8034a4ef98 ceph: set default FsGroupChangePolicy value to 'None'
The `None` value Indicates that volumes will be mounted
with no modifications, as the CSI volume driver does not
support these operations. While volumes are provisioned
by the CephFS CSI driver the global permissions are set
on the volume and we dont expect the Fsgroup policy or
check from CO side  to play a role here.

Ref #ceph/ceph-csi/../internal/cephfs/nodeserver.go#L190

```
!csicommon.MountOptionContains(fuseMountOptions, readOnly) {
		// #nosec - allow anyone to write inside the stagingtarget path
		err = os.Chmod(stagingTargetPath, 0o777)
```

The current default value ie `ReadWriteOnceWithFSType` cause
volumes to be examined to determine if volume ownership and permissions
should be modified to match the pod's security policy. Changes could occur
if the fsType is defined and the persistent volume's accessModes contains
ReadWriteOnce.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2021-08-03 16:54:10 +05:30
Madhu Rajanna b556dbf109 ceph: update cephcsi to v3.4.0
This PR updates the required RBAC, templates,
CSI image version and examples for new
cephcsi v3.4.0 release.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2021-08-02 10:22:54 +05:30
Dominik Münch c980dc2d4d ceph: add empty dirs for /var/lib/rook and /etc/ceph
Adds volumes and mounts for /var/lib/rook and /etc/ceph to the operator deployment in the Helm
chart. This is consistent with the example operator manifest and is sometimes even required on
clusters with stricter permissions on the local filesystem.

Signed-off-by: Dominik Münch <d.muench@celonis.com>
2021-07-30 23:03:50 +02:00
Travis Nielsen 990d92790b Merge pull request #8420 from travisn/cleanup-old-versions
ceph: Remove old references to luminous and mimic versions
2021-07-29 07:40:46 -06:00
Travis Nielsen c2a551123f Merge pull request #8401 from TomHellier/add-additional-rook-ceph-cluster-helm-features
ceph: Add additional helm chart functionality for ingresses and defining the ceph storage crds
2021-07-28 16:14:48 -06:00
Tom Hellier 89ab4f90ec ceph: adds helm functionality for ingress, and ceph storage crds
This commit adds an ingress resource to the rook-ceph-cluster helm chart, allowing
ingress to the ceph-dashboard service. It also adds the ability to define the
various storage types that you can run on ceph inside kubernetes.

Closes https://github.com/rook/rook/issues/8384

Signed-off-by: Tom Hellier <me@tomhellier.com>
2021-07-28 21:51:56 +01:00
Travis Nielsen f5030f0560 ceph: remove old references to mimic version
References to luminous and mimic, and unnecessary references to nautilus
were found in the docs that have been cleaned up and clarified. At the same
time, the tests are updated to use more recent versions.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-07-28 14:00:35 -06:00
Sébastien Han a62e7e75e7 Merge pull request #8380 from sp98/update-cephblockpool-cr
ceph: add mirroring peer specification to CephBlockPool CRD
2021-07-28 17:34:52 +02:00
Santosh Pillai 3b50f5fd00 ceph: add MirroringPeerSpec to CephBlockPool.Spec.Mirroring
Instead of having the RBDMirroringPeerSpec in the RBDMirror CRD we want
to move it on the CephBlockPool CRD so that each pool can have its own peer.
This enables pool to re-use an existing peer secret if it points to the
same cluster peer.So we don't need to duplicate secret peer anymore.

Signed-off-by: Santosh Pillai <sapillai@redhat.com>
2021-07-28 17:47:36 +05:30
Travis Nielsen 6dcc60173a Merge pull request #8347 from josephsawaya/rook-ceph-mgr-role-changes
ceph: update rook-ceph-mgr-cluster role rules to include PV and SC
2021-07-27 11:22:31 -06:00
Satoru Takeuchi 30e4fbb01f ceph: make the timeout of ceph commands cofigurable
Sometimes the default 15s is not enough for timeout of ceph commands. For examples,
I encountered that `radosgw-admin` command took dozens of seconds under heavy load.

Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
2021-07-27 12:42:16 +00:00
Joseph Sawaya 1ddc390d1e ceph: update rook-ceph-mgr-cluster role rules to include PV and SC
Since we changed the Rook orchestrator module for Ceph, it now has
to access Storage Classes and Persistent Volumes in the cluster to
gather inventory and create OSDs so we have to make changes to the
rook-ceph-mgr-cluster role so the orchestrator has permission to
access these resources.

Signed-off-by: Joseph Sawaya <jsawaya@redhat.com>
2021-07-26 13:42:21 -04:00
Sébastien Han 0811359d28 Merge pull request #8358 from leseb/move-to-quay
ceph: move all of our docker.io reference to quay.io
2021-07-23 18:03:53 +02:00
Travis Nielsen 16403ac901 Merge pull request #8350 from leseb/fix-7183
ceph: remove operator config `ROOK_ALLOW_MULTIPLE_FILESYSTEMS`
2021-07-22 11:33:40 -06:00
Sébastien Han 7c5a86cb85 ceph: remove operator config ROOK_ALLOW_MULTIPLE_FILESYSTEMS
Multiple filesystems are supported as of the Ceph Pacific release. So we
remove the flag from the operator configuration and just have a ceph
version check instead.

The Operator configuration option `ROOK_ALLOW_MULTIPLE_FILESYSTEMS`
has been removed in favor of simply verifying the Ceph version is
at least Pacific.
Multiple filesystems are stable since Ceph Pacific.
So users who had `ROOK_ALLOW_MULTIPLE_FILESYSTEMS` enabled will
need to update their Ceph version to Pacific.

Closes: https://github.com/rook/rook/issues/7183
Signed-off-by: Sébastien Han <seb@redhat.com>
2021-07-22 11:42:33 +02:00
Sébastien Han 6bce1ff3e9 ceph: move all of our docker.io reference to quay.io
Recently, the builds of `ceph/ceph` image moved to quay.io, see
https://github.com/ceph/ceph-build/pull/1883 for more details.
Current images will remain but new builds will happen on quay.io only.

This means that tags such as `v14.2`, `v15.2`,`v16.2` will need to
switch to quay.io to get updates.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-07-22 11:17:05 +02:00
n.fraison f7bc76abad ceph: define specific ServceAccount/RBAC for ceph osd purge job
Currently ceph osd purge job rely on rook-ceph-system service account which is only created
on the rook operator namespace
When multiple clusters are deployed on distinct namespace the ceph osd purge failed to run
as the service account is not available
Creating a specific service account rook-ceph-purge-osd with appropriate rights on each cluster
namespace and operator namespace to still manage case where cluster is installed
along side the operator

Signed-off-by: n.fraison <n.fraison@criteo.com>
2021-07-22 00:25:32 +02:00
n.fraison db2c8792b0 ceph: create rook-ceph-system ClusterRoleBinding in operator namespace
Operator can run on a namespace not named rook-ceph
The ClusterRoleBinding muste rely on .Release.Namespace helm value
Also move the ClusterRole and ClusterRoleBinding in appropriate yaml file

Signed-off-by: n.fraison <n.fraison@criteo.com>
2021-07-20 16:02:23 +02:00
Travis Nielsen 64df4cdbb6 ceph: update default ceph image to latest v16.2.5
The latest pacific release is 16.2.5 so we set this as the default
for Rook to deploy.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-07-16 10:21:48 -06:00
n.fraison 50c8c7a7f7 ceph: add image pull secrets option on new helm cluster
Allow to specify imagePullSecrets to pull operator/ceph images from private registry

Signed-off-by: n.fraison <n.fraison@criteo.com>
2021-07-12 10:15:06 +02:00
Sébastien Han eaa6e7732c Merge pull request #8272 from leseb/exec-in-pod
ceph: proxy ceph command when multus is configured
2021-07-07 21:36:54 +02:00
Sébastien Han bd58790c31 ceph: proxy ceph commands when multus is configured
When the CephCluster is configured with Multus and multiple networks are
used to deploy Ceph some commands are failing to be executed from the
Operator. These commands, in particular, `radosgw-admin` ones need access
to the "ceph public network" to talk to OSDs. Unfortunately, the
Rook-Ceph Operator does not have the network annotations and thus
doesn't have the networks available and cannot reach OSDs. So the commands end
up hanging and eventually time out.
Applying the annotations to the Operator pod is possible but will result
in restarting the operator too and this should be avoided at all costs.
Also, applying the annotations beforehand is not possible since the
Multus declaration is in the CephCluster specification. So we would have
no idea what to do.

So the current approach runs a new sidecar container in the mgr pod to
act as a proxy for "some" ceph commands, only the `radosgw-admin` ones
for multi-site setup. This is a small container with admin access
running idle waiting for commands to be executed. In a sense, it is
similar to the toolbox but we didn't want to clearly expose it, so
running as a sidecar is quite nice.

Proxying command is obviously not always recommended since we add an
extra hop in the network path. Now each request has to go from the
operator pod to the API server to the remote pod to Ceph. Previously,
the command only goes from the operator to Ceph.

It's worth noting that external mode is not impacted since no rgw pod
is configured. This scenario is flexible and allows us to scale
pretty well since any CephCluster with Multus will see its mgr sidecar
deployed and can then talk to Ceph. We are not limited.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-07-07 19:08:32 +02:00
subhamkrai 34313c9dc9 ci: in helm chart field should not be empty
ci is complaining due to empty field in
values.yaml with error
```
coalesce.go:199: warning: destination for config is a table. Ignoring non-table value <nil>
```
this commit make those field empty map/list.

Closes: https://github.com/rook/rook/issues/8239
Signed-off-by: subhamkrai <srai@redhat.com>
2021-07-06 14:28:58 +05:30
Sébastien Han c4d15f4de2 Merge pull request #8254 from henryzhangsta/helm-network-optional
ceph: network configuration should be optional in Helm chart
2021-07-05 11:23:09 +02:00
Henry Zhang a5227bf126 ceph: network configuration should be optional in Helm chart
adds a default empty dict for the network when deciding whether toolbox
should also be on the host network to catch the case when the entire
`network` block is empty

Signed-off-by: Henry Zhang <me@henry.dev>
2021-07-04 01:05:08 -07:00
Sébastien Han b578f916e7 ceph: add fs mirror config
Similarly to block volume replication, Ceph is capable of replicating the
content of a Ceph Filesystem from one cluster to another.
For this, during the 1.6 cycle, we introduced a new CRD called
CephFilesystemMirror which effectively deploys a cephfs-mirror daemon.
However, configuring peers to enable replication between two clusters
had to be done manually.
Also various bug fix made it in Ceph eventually and the minimum required
version for this to work is to run on Ceph Pacific 16.2.5 at least.

So the automatic configuration of Ceph Filesystem peers is now possible.

By editing the CephFilesystem CRD, you can now turn on mirroring:

```yaml
  mirroring:
    enabled: false
    # list of Kubernetes Secrets containing the peer token
    # for more details see: https://docs.ceph.com/en/latest/dev/cephfs-mirroring/#bootstrap-peers
    peers:
      secretNames:
        - secondary-cluster-peer
```

Also, the mirroring status is displayed in the CR status:

```
status:
  info:
    fsMirrorBootstrapPeerSecretName: fs-peer-token-myfs
  mirroringStatus:
    daemonsStatus:
    - daemon_id: 4186
      filesystems:
      - filesystem_id: 2
        name: myfs
    lastChecked: "2021-07-01T14:16:29Z"
  phase: Ready
  snapshotScheduleStatus:
    lastChecked: "2021-07-01T14:16:29Z"
    snapshotSchedules:
    - fs: myfs
      path: /
      rel_path: /
      retention: {}
      schedule: 24h
```

Closes: https://github.com/rook/rook/issues/7063
Signed-off-by: Sébastien Han <seb@redhat.com>
2021-07-01 17:35:19 +02:00
Santosh Pillai 4041868e2e ceph: hybrid Storage Pools
Creates hybrid crush rule for choosing Primary OSD for
high performing SSD devices and remaining OSD for low performance HDD devices.

Signed-off-by: Santosh Pillai <sapillai@redhat.com>
2021-07-01 11:16:54 +05:30
Blaine Gardner 037945aabc Merge pull request #8112 from BlaineEXE/dependencies-2-son-of-dependencies
ceph: block delete object store when buckets and/or users exist
2021-06-30 12:45:58 -06:00