Commit Graph
182 Commits
Author SHA1 Message Date
Jiffin Tony Thottan 01649a8cd2 docs: update vault kms with RGW details
The KMS encryption via HashiCorp Vault can be consumed for RGW, adding those details
in the doc.

Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
2021-01-22 16:04:28 +05:30
Travis Nielsen b71c3ac293 ceph: add mon design to the official docs
The mon design and handling of failover is a topic that frequently
needs to be understood by rook users, thus moving this design doc
to the documentation and making some updates based on the latest
behavior.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-01-14 13:13:56 -07:00
Travis Nielsen 40c4923410 docs: fix quickstart and cluster cr broken links
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-01-04 07:44:38 -07:00
Travis Nielsen 3beb625f08 ceph: update examples and operator to Ceph v15.2.8
The latest octopus release is now out with v15.2.8. We update the
operator base image and the examples to run with this version.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-12-17 18:21:15 -07:00
Travis Nielsen eebb367e42 ceph: allow arbiter to have separate placement from other mons
The arbiter may have unique requirements for placement settings. Thus, if the arbiter
placement is specified in the cluster CR, that placement will be applied to the arbiter
mon and the other mons will retain their placement from the other mon or all
sections of the placement.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-12-07 10:58:20 -07:00
Sébastien Han 8aaff235bb ceph: ability to set external services ports
When the cluster is external we want to expose manager service port
along with the endpoints.
This allows us to connect but Rook will keep on using 9283 as a facing
port for Prometheus and more.

```
monitoring:
  enabled: true
  ...
  ...
  externalMgrPrometheusPort: 9283
```

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-12-04 09:52:09 +01:00
Jiffin Tony Thottan a92a58e23e docs: update vault kms integration doc
correct the name policy mentioned

Signed-off-by: Jiffin Tony Thottan <jthottan@redhat.com>
2020-12-02 11:52:07 +05:30
Travis Nielsen 75af038309 Merge pull request #6726 from travisn/ceph-15.2.7
ceph: Update examples and base image to v15.2.7
2020-12-01 14:05:37 -07:00
Travis Nielsen 0a35a08f2f ceph: update examples and base image to v15.2.7
With the release of the latest octopus v15.2.7 we update the base
of the operator image and set the examples to use the same release
to pick up the security and other bug fixes.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-12-01 12:05:39 -07:00
Sébastien Han c6a87203ca ceph: add log collector
We can now collect logs directly into a side-car container.
A new CRD spec has been added:

spec:
  logCollector:
    enabled: true
    periodicity: 24h

Every 24h we will rotate log files for each Ceph daemon.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-12-01 16:30:17 +01:00
Renan Campos ad9dcb0c05 ceph: periodically prune crash entries older than user-provided days
Rook's crashcollector pod posts entries to the ceph cluster when a crash occurs.
Over time the number cluster may hold crash entries needlessly.
To clean up old crash entries, this PR adds a field to the ceph cluster CR for the user to specify the number of days a crash entry should be kept for.
Providing a value for the field keepXDays creates a cronjob that runs every day at midnight, calling "ceph crash prune <keepXDays>".

Closes: https://github.com/rook/rook/issues/6332
Signed-off-by: Renan Campos <rcampos@redhat.com>
2020-11-24 17:06:18 -05:00
rohan47 5b5059ec2f ceph: support NAD from diffrent namespaces
Added the support of NAD from diffrent namespaces.
they can be referrenced as <namespace>/<name-of-nad> e.g.,
default/public-nw.
Updated the multus doc to explain the same.

Signed-off-by: rohan47 <rohgupta@redhat.com>
2020-11-17 20:34:58 +05:30
Satoru Takeuchi 21cef906f7 docs: clarify the meaning of cleanup policy of ceph cluster crd
The description of cleanupPolicy is a bit ambiguous. First, we should clarify
that both skipping orchestration and data cleanup will happen iff
"yes-really-destroy-data" is set to `confirmation`. Second, We should also
describe wiping devices will happen even if user don't specify `sanitizeDisks`
explicitly.

Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
2020-11-11 05:15:41 +00:00
Sébastien Han e3d032dbc1 Merge pull request #6489 from travisn/stretch-mons
ceph: Stretch cluster configuration
2020-11-06 15:16:39 +01:00
Travis Nielsen a5b9fbc994 ceph: update base image and examples to v15.2.5
For the v1.5 release we use the latest version of ceph by default,
which is now v15.2.5

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-11-05 15:50:23 -07:00
Travis Nielsen b91f4211c9 ceph: configure a stretched cluster
In clusters where only two datacenters (or similar failure domains)
are available, a different mon and osd approach is needed to deal
with the network partitions or some other reason for one of the failure
domains going down. The Ceph stretched cluster makes the mons aware
of the failure domains by configuring one as the arbiter in a third
zone, while keeping two replicas of the data in each of the data
zoens.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-11-05 09:17:52 -07:00
Sébastien Han ea1d71cbfb ceph: add vault kms support for osd encryption
When the Ceph cluster runs on PVC and the OSDs are encrypted we can
store LUKS's Key Encryption Key inside a Key Management System. Today,
Rook only supports HashiCorp Vault: https://www.vaultproject.io/

The CephCluster has now a new "security" field which will plug onto the
KMS. Here is an example:

security:
  kms:
    tokenSecretName: <name of the secret containing a Vault token, used
    to authenticate>
    connectionDetails: < a map of strings containing connection
    information>

Refer to the ceph-cluster-crd documentation to lear more.

Closes: https://github.com/rook/rook/issues/6105
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-10-30 16:16:33 +01:00
Jonas Schäfer 7ace6ac255 ceph: make root= CRUSH label value configurable
The CRUSH map does not allow duplicate values (nor keys) for the
labels. That means that if the currently hardcoded label value
`default` occurs anywhere else in the tree (for example, because
some of the topology labels provided by the cloud provider use it),
the cluster cannot sucessfully spawn.

By giving the user control over the label value used for the root
CRUSH map label, it is possible to adapt the cluster to this
situation.

To be able to actually use the `default` value, the root=default
node which is created by default by ceph itself has to be removed;
since that node is accompanied by a replicated crush rule, we have
to remove that rule, too.

The integration tests are modified to sometimes use the custom
crushRoot (based on an arbitrary criterium) to get coverage across
the various suites. Separate tests could be added, but were not
deemed necessary at this point.

Fixes #4993.

Signed-off-by: Jonas Schäfer <jonas.schaefer@cloudandheat.com>
2020-10-29 09:22:41 +01:00
Travis Nielsen a4b1ca3faa docs: clarify cluster-on-pvc scenario for ceph
A frequently asked question is how to configure the cluster in dynamic
environments, or basically to use the cluster-on-pvc configuration.
This adds more details to the examples in the docs.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-10-12 15:21:28 -06:00
Travis Nielsen c7e0338d5e ceph: require an odd number of mons and sufficient nodes
The ceph mons should never be started with an even number in quorum.
The desired state should always be an odd number of mons to ensure
a healthy majority quorum. The operator now rejects a request for
an even number of mons.

If there are not a sufficient number of nodes for mons to be on a
unique host, the operator also returns an early error instead of
getting stuck waiting for a pending mon. This is not foolproof since
there might be fewer available nodes for the mons, but at least it
is a quick check for the common case.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-10-12 15:21:27 -06:00
Travis Nielsen 3eddcf5477 Merge pull request #6283 from sp98/rook-IPv6-support
ceph: support IPv6 single-stack
2020-10-06 07:46:25 -06:00
subhamkrai c2c59ec47b ceph: tune fast device class for OSD on PVC in the Azure
on Azure, all SSDs are reporting as a rotational disk.
This commit will add `tuneFastDeviceClass` flag which will
tune for an SSD even if the cloud provider is reporting
rotational. We have a flag `tuneDeviceClass`
which works vice-versa.

Signed-off-by: subhamkrai <srai@redhat.com>
2020-10-05 20:19:55 +05:30
Santosh Pillai 70c5f70701 ceph: support IPv6 single-stack for ceph
Add --ms-bind-ipv6 true args to mom, osd, mgr, object, mds, rbd daemons.

Signed-off-by: Santosh Pillai <sapillai@redhat.com>
2020-10-02 01:03:03 +05:30
Josh McSavaney cc7b07c840 ceph: tweak removeOSDsIfOutAndSafeToRemove wording
- Corrects what I think was a typo ("when then" -> "when they")
- Picks a single tense
- capitalizes OSD

Signed-off-by: Josh McSavaney <me@mcsau.cc>
2020-09-14 18:17:47 -04:00
Sébastien Han a3328564bd Merge pull request #6187 from greenstatic/docs-formatting-fix
docs: fix markdown code block formatting issue
2020-09-01 09:33:43 +02:00
Raghavendra TalurandNitin Goyal f8a721dbb0 ceph: doc additions for the uninstall mode
Add documentation for the setting AllowUninstallWithVolumes to the
following files:
Documentation/ceph-cluster-crd.md
cluster/examples/kubernetes/ceph/cluster.yaml

Co-authored-by: Nitin Goyal <nigoyal@redhat.com>
Signed-off-by: Raghavendra Talur <raghavendra.talur@gmail.com>
2020-08-31 14:07:32 -04:00
Gregor Krmelj 256b284b35 docs: fix markdown code block formatting issue
this fixes the issue of the code block not being rendered
correctly from markdown into html.

Signed-off-by: Gregor Krmelj <gregor@krmelj.xyz>
2020-08-31 16:40:48 +02:00
Travis Nielsen e40b462216 Merge pull request #6084 from cloudical-io/additional_user_labels_ceph
ceph: allow user to add labels to pods
2020-08-25 15:58:40 -06:00
Travis Nielsen d786b7f547 docs: fix link to ceph upgrade guide
There was a broken link in the cluster CR documentation that was pointing
to /Documentation instead of the .md in the same dir.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-08-25 13:43:03 -06:00
Alexander Trost a87c00ddee ceph: allow user to add labels to pods
This allows users to specify additional labels to be added to the components
created by the Rook Ceph operator.
Additionally the rbd mirrors are now getting their annotations and
labels set.

Signed-off-by: Alexander Trost <galexrt@googlemail.com>
2020-08-25 19:38:51 +02:00
rohan47 8c4ede1edc ceph: added support for multus for csi
CSI pods now utilize multus networking and connect to public
network specified in the CephCluster CR.

Closes: https://github.com/rook/rook/issues/5356
Signed-off-by: rohan47 <rohgupta@redhat.com>
2020-08-20 18:25:09 +05:30
Travis Nielsen ce92249725 ceph: continue with memory limits below min settings
The operator will now allow the resource limits to be applied below the
recommended minimums. In small clusters, even the recommended minimums
may not be necessary. A warning is still printed to the operator log,
but we allow the configuration to continue.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-08-19 11:07:16 -06:00
Travis Nielsen 1111f7b3ca Merge pull request #6079 from cybozu-go/ceph-add-a-note-of-OSD-on-LV-backed-PVC
docs: add a note of OSD on LV-backed PVC's known problem
2020-08-19 09:14:34 -06:00
Yuji Ito ce1951469d ceph: reflect review
Signed-off-by: Yuji Ito <llamerada.jp@gmail.com>
2020-08-19 02:55:37 +00:00
Travis Nielsen 0ffa0348b5 ceph: separate placement for osd prepare jobs from osd daemons
The osd prepare jobs and osd daemons may have different placement
requirements. In particular, the topology spread constraints
or pod anti-affinity should only work on either the prepare job or
osd daemon, but does not work effectively on both types at the same
time. Therefore, we split out the osd prepare placement to be separate
from the osd daemon placement. If the osd prepare placement is not set
the operator will fall back to the osd daemon placement for
backward compatibility.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-08-18 15:24:04 -06:00
Satoru Takeuchi 1a3c175e63 ceph: add a note of OSD on PVC-backed PVC's known problem
There is a critical flaw in OSD on LV-backed PVC. So it's necessary
to let users to know what they should do.

Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
2020-08-14 00:10:47 +00:00
Sébastien Han d91f55e8ed Merge pull request #5580 from silverhandy/master
ceph: add support for wal device for OSD on PVC
2020-08-11 13:57:56 +02:00
Takashi IIGUNI 0e86a5a0c4 docs: fix clean policy confirmation of ceph-cluster-crd
Make the document of cleanup policy clearer.

Signed-off-by: Takashi IIGUNI <iiguni.tks@gmail.com>
2020-08-11 01:59:56 +00:00
Chen, Tingjie 0e87348093 ceph: add support for wal device for OSD on PVC
Based on the feature of metadata PVC for OSD: #4818, add the wal device.

For this, you need to create a new volumeClaimTemplates, its name must be "wal", if "wal" is
set, "metadata" device will refers specially for block.db device, and "wal" refers for block.wal
device in BlueStore.

A template will look like this:

```
volumeClaimTemplates:
- metadata:
    name: data
  spec:
    resources:
      requests:
        storage: 10Gi
    # IMPORTANT: Change the storage class depending on your environment (e.g. local-storage, gp2)
    storageClassName: gp2
    volumeMode: Block
    accessModes:
      - ReadWriteOnce
- metadata:
    name: metadata
  spec:
    resources:
      requests:
        storage: 6Gi
    # IMPORTANT: Change the storage class depending on your environment (e.g. local-storage, io1)
    storageClassName: io1
    volumeMode: Block
    accessModes:
      - ReadWriteOnce
- metadata:
    name: wal
  spec:
    resources:
      requests:
        storage: 6Gi
    # IMPORTANT: Change the storage class depending on your environment (e.g. local-storage, io1)
    storageClassName: io1
    volumeMode: Block
    accessModes:
      - ReadWriteOnce
```

We now map block, block.db and block.wal directly inside the container instead of
running ceph-volume activate.

Signed-off-by: Chen, Tingjie <tingjie.chen@intel.com>
2020-08-10 08:48:20 +08:00
rohan47 f5bbc8c5c5 ceph: fix networkAttachmentConfig to be compatible with whereabouts cni
In the whereabouts config, the network range is determined by range
option unlike other cni plugins that use Subnet option.

Signed-off-by: rohan47 <rohgupta@redhat.com>
2020-08-05 21:17:13 +05:30
Sébastien Han 89b3225441 ceph: add encryption support for osd pvc
We can now encrypted OSD device that were provisioned via a storage
class using the PV interface.
The encryption works at the storageClassDeviceSets level, which means we
can have encrypted and non-encrypted sets.
Using the new key `encrypted` we can turn it on.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-08-05 10:06:11 +02:00
Madhu Rajanna 3ebeb5a7ca ceph: add expand secret to examples in md file
add CSI expand secret details in example storageclass
templates.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-08-04 11:13:11 +05:30
Alex PunnenandSébastien Han de6a8c0782 ceph: docs
Added a StorageClass example for better document usability to test CephExternal cluster

Solves: None
Co-authored-by: Sébastien Han <seb@redhat.com>
Signed-off-by: Alex Punnen <alexcpn@gmail.com>
2020-08-03 20:03:32 +05:30
Sébastien Han 10ed662bdb ceph: enhancement disk sanitize options
Now, we can not only cleanup monitor data, logs and crashes but the
disks too. As part of the cleanupPolicy CR spec, we have a new setting
called sanitizeDisk which holds more details:

* method: indicates if the entire disk should be sanitized or simply ceph's metadata.
Possible choices are 'complete' or 'quick' (default)
* dataSource: indicate where to get random bytes from to write on the disk.
Possible choices are 'zero' (default) or 'random'
Using random sources will consume entropy from the system and will take much more time then the zero source
* iteration: overwrite N times instead of the default (1). Takes an integer value

See the documentation for more details.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-07-21 14:07:11 +02:00
Sébastien Han 34ccb84aa3 ceph: add external prometheus endpoint
We can now connect an external prometheus exporter to Rook to collect
metrics and generate alerts from Prometheus.
Just enable this in the CephCluster CR:

```
spec:
  external:
    enable: true
   monitoring:
    enabled: true
    rulesNamespace: rook-ceph
    externalMgrEndpoints:
    - ip: 192.168.39.182
```

Closes: https://github.com/rook/rook/issues/5516
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-07-17 11:29:48 +02:00
Hiroshi Muraoka e2b1a2230d ceph: ignore PodAntiAffinity for detect-version placement
When a PodAntiAffinity is set to mon, detect-version may not be scheduled because detect-version has the same PodAntiAffinity with mon.
This commit fixes this issue.

Closes #5810

Signed-off-by: Hiroshi Muraoka <h.muraoka714@gmail.com>
2020-07-16 04:27:06 +00:00
Blaine Gardner 7117fc12b7 ceph: osd: add drive groups spec to cluster CR
Add the ability to provision Ceph OSDs with Drive Groups.
This adds Drive Groups to the CephCluster CRD, and it sets code
in place for propagating this config to the OSD provisioning pod.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2020-07-14 09:43:03 -06:00
Travis Nielsen 215758bfea Merge pull request #5795 from cybozu-go/ceph-fix-the-wrong-description-of-mon-placement
ceph: remove an outdated document of mon placement
2020-07-14 07:30:43 -06:00
Satoru Takeuchi 63d077ffde ceph: remove an outdated document of mon placement
Although the document describes we can't set Pod{,Anti}Affinity
to mon's placement, the current implementation permits to do so.

Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
2020-07-14 07:42:16 +00:00
Sébastien Han ccb52b84e6 ceph: configurable status checks and livenessprobe
This commit allows us to configure status check for each daemon:

* "mon": health check on the ceph monitors (quorum)
* "osd": health check on the ceph osds
* "status": ceph health status check

Each check is controlled by the following settings:

* disabled: whether to disable the check (default: false)
* internal: interval to run the check
* timeout: only valid for mons, is the timeout for unresponsive mon
before failling over.

Example to disable the status health check:

```yaml
healthCheck:
  daemonHealth:
    status:
      disabled: true
```

As part of that, pod's livenessprobe can now be configured via the
following settings:

* disabled: whether to enable or not
* probe: override the current probe in place by a new one

```yaml
healthCheck:
  livenessProbe:
    mon:
      disabled: true
```

Closes: https://github.com/rook/rook/issues/5772
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-07-10 19:21:59 +02:00