From 8a19c730367b4d4cc4e172692cc4dcbed83d0bf3 Mon Sep 17 00:00:00 2001 From: Travis Nielsen Date: Wed, 12 Feb 2020 17:29:21 -0700 Subject: [PATCH 1/6] ceph: clarify docs for erasure coding Setting up the topology for the CRUSH map was outdated in the EC doc. Now a link to the node labels is provided for the configuration. Signed-off-by: Travis Nielsen --- Documentation/ceph-pool-crd.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Documentation/ceph-pool-crd.md b/Documentation/ceph-pool-crd.md index ec4e0edd5..50cb9f1d0 100644 --- a/Documentation/ceph-pool-crd.md +++ b/Documentation/ceph-pool-crd.md @@ -106,6 +106,6 @@ The `failureDomain` must be also be taken into account when determining the numb * `host`: All chunks will be placed on unique hosts * `osd`: All chunks will be placed on unique OSDs -If you do not have a sufficient number of hosts or OSDs for unique placement the pool can be created, although a PUT to the pool will hang. +If you do not have a sufficient number of hosts or OSDs for unique placement the pool can be created, a PUT to the pool will hang. -Rook currently only configures two levels in the CRUSH map. It is also possible to configure other levels such as `rack` with the [Ceph tools](http://docs.ceph.com/docs/master/rados/operations/crush-map/). +Rook currently only configures two levels in the CRUSH map. It is also possible to configure other levels such as `rack` with by adding [topology labels](ceph-cluster-crd.md#osd-topology) to the nodes. From 735292e8e3846079c1a35fa9c19656168b87791c Mon Sep 17 00:00:00 2001 From: Travis Nielsen Date: Thu, 27 Feb 2020 09:11:17 -0700 Subject: [PATCH 2/6] ceph: flex driver is not always a prerequisite This configuration is required when using the flex CSI driver (required for K8s 1.12 or earlier). Going forward for K8s 1.13 and newer it is recommended to use the CSI RBD driver, making Flexvolume configuration redundant. Co-authored-by: Ran Rubin Signed-off-by: Travis Nielsen --- Documentation/k8s-pre-reqs.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/Documentation/k8s-pre-reqs.md b/Documentation/k8s-pre-reqs.md index e4093cf2f..31006617e 100644 --- a/Documentation/k8s-pre-reqs.md +++ b/Documentation/k8s-pre-reqs.md @@ -20,6 +20,8 @@ setting up Rook in a Kubernetes cluster with Pod Security Policies enabled. ## Flexvolume Configuration +**NOTE** This configuration is only needed when using the FlexVolume driver (required for Kubernetes 1.12 or earlier). The Ceph-CSI RBD driver or the Ceph-CSI CephFS driver are recommended for Kubernetes 1.13 and newer, making FlexVolume configuration redundant. + The Rook agent requires setup as a Flex volume plugin to manage the storage attachments in your cluster. See the [Flex Volume Configuration](flexvolume.md) topic to configure your Kubernetes deployment to load the Rook volume plugin. From 575a73daa90c0b3a9043f7defa7b074c886fc581 Mon Sep 17 00:00:00 2001 From: Travis Nielsen Date: Wed, 26 Feb 2020 17:30:08 -0700 Subject: [PATCH 3/6] ceph: updates from a documentation review The documentation for Ceph configuration needed some clarification and cleanup from options that are no longer needed or apply. Signed-off-by: Travis Nielsen --- Documentation/ceph-advanced-configuration.md | 191 +------------------ Documentation/ceph-block.md | 8 +- Documentation/ceph-client-crd.md | 2 +- Documentation/ceph-common-issues.md | 35 +++- Documentation/ceph-monitoring.md | 4 +- 5 files changed, 39 insertions(+), 201 deletions(-) diff --git a/Documentation/ceph-advanced-configuration.md b/Documentation/ceph-advanced-configuration.md index 6a7d6a90b..f5b529947 100644 --- a/Documentation/ceph-advanced-configuration.md +++ b/Documentation/ceph-advanced-configuration.md @@ -20,7 +20,6 @@ storage cluster. * [OSD Dedicated Network](#osd-dedicated-network) * [Phantom OSD Removal](#phantom-osd-removal) * [Change Failure Domain](#change-failure-domain) -* [Monitor placement](#monitor-placement) ## Prerequisites @@ -155,12 +154,6 @@ do echo "END logs from pod: ${p} ${c}" done done -for i in $(kubectl -n rook-ceph-system get pods -o jsonpath='{.items[*].metadata.name}') -do - echo "BEGIN logs from pod: ${i}" - kubectl -n rook-ceph-system logs ${i} - echo "END logs from pod: ${i}" -done) | gzip > /tmp/rook-logs.gz ``` This gets the logs for every container in every Rook pod and then compresses them into a `.gz` archive @@ -241,112 +234,6 @@ In the following example we will separate SSD drives from spindle-based drives, a common practice for those looking to target certain workloads onto faster (database) or slower (file archive) storage. -### CRUSH Hierarchy - -To see the CRUSH hierarchy of all your hosts and OSDs run: - -```console -ceph osd tree -``` - -Before we separate our disks into groups, our example cluster looks like this: - -```console -ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY --1 7.21828 root default --2 0.94529 host node1 - 0 0.55730 osd.0 up 1.00000 1.00000 - 1 0.11020 osd.1 up 1.00000 1.00000 - 2 0.27779 osd.2 up 1.00000 1.00000 --3 1.22480 host node2 - 3 0.55730 osd.3 up 1.00000 1.00000 - 4 0.11020 osd.4 up 1.00000 1.00000 - 5 0.55730 osd.5 up 1.00000 1.00000 --4 1.22480 host node3 - 6 0.55730 osd.6 up 1.00000 1.00000 - 7 0.11020 osd.7 up 1.00000 1.00000 - 8 0.06670 osd.8 up 1.00000 1.00000 -``` - -We have one root bucket `default` that every host and OSD is under, so all of -these storage locations get pooled together for reads/writes/replication. - -Let's say that `osd.1`, `osd.3`, and `osd.7` are our small SSD drives that we -want to use separately. - -First we will create a new `root` bucket called `ssd` in our CRUSH map. Under -this new bucket we will add new `host` buckets for each node that contains an -SSD drive so data can be replicated and used separately from the default HDD -group. - -```console -# Create a new tree in the CRUSH Map for SSD hosts and OSDs -ceph osd crush add-bucket ssd root -ceph osd crush add-bucket node1-ssd host -ceph osd crush add-bucket node2-ssd host -ceph osd crush add-bucket node3-ssd host -ceph osd crush move node1-ssd root=ssd -ceph osd crush move node2-ssd root=ssd -ceph osd crush move node3-ssd root=ssd - -# Create a new rule for replication using the new tree -ceph osd crush rule create-simple ssd ssd host firstn -``` - -Secondly we will move the SSD OSDs into the new `ssd` tree, under their -respective `host` buckets: - -```console -ceph osd crush set osd.1 .1102 root=ssd host=node1-ssd -ceph osd crush set osd.3 .1102 root=ssd host=node2-ssd -ceph osd crush set osd.7 .1102 root=ssd host=node3-ssd -``` - -It's important to note that the `ceph osd crush set` command requires a weight -to be specified (our example uses `.1102`). If you'd like to change their -weight you can do that here, otherwise be sure to specify their original weight -seen in the `ceph osd tree` output. - -So let's look at our CRUSH tree again with these changes: - -```console -ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY --8 0.22040 root ssd --5 0.11020 host node1-ssd - 1 0.11020 osd.1 up 1.00000 1.00000 --6 0.11020 host node2-ssd - 4 0.11020 osd.4 up 1.00000 1.00000 --7 0.11020 host node3-ssd - 7 0.11020 osd.7 up 1.00000 1.00000 --1 7.21828 root default --2 0.83509 host node1 - 0 0.55730 osd.0 up 1.00000 1.00000 - 2 0.27779 osd.2 up 1.00000 1.00000 --3 1.11460 host node2 - 3 0.55730 osd.3 up 1.00000 1.00000 - 5 0.55730 osd.5 up 1.00000 1.00000 --4 1.11460 host node3 - 6 0.55730 osd.6 up 1.00000 1.00000 - 8 0.55730 osd.8 up 1.00000 1.00000 -``` - -### Using Disk Groups With Pools - -Now we have a separate storage group for our SSDs, but we can't use that storage -until we associate a pool with it. The default group already has a pool called -`rbd` in many cases. If you [created a pool via CustomResourceDefinition](ceph-pool-crd.md), -it will use the default storage group as well. - -Here's how to create new pools: - -```console -# SSD backed pool with 128 (total) PGs -ceph osd pool create ssd 128 128 replicated ssd -``` - -Now all you need to do is create RBD images or Kubernetes `StorageClass`es that -specify the `ssd` pool to put it to use. - ## Configuring Pools ### Placement Group Sizing @@ -377,57 +264,6 @@ it](#creating-a-pool). With backups you can try a few potentially unsafe tricks for live pools, documented [here](http://cephnotes.ksperis.com/blog/2015/04/15/ceph-pool-migration). -### Deleting A Pool - -Be warned that this deletes all data from the pool, so Ceph by default makes it -somewhat difficult to do. - -First you must inject arguments to the Mon daemons to tell them to allow the -deletion of pools. In Rook Tools you can do this: - -```console -ceph tell mon.\* injectargs '--mon-allow-pool-delete=true' -``` - -Then to delete a pool, `rbd` in this example, run: - -```console -ceph osd pool rm rbd rbd --yes-i-really-really-mean-it -``` - -### Creating A Pool - -```console -# Create a pool called rbd with 1024 total PGs, using the default -# replication ruleset -ceph osd pool create rbd 1024 1024 replicated replicated_ruleset -``` - -`replicated_ruleset` is the default CRUSH rule that replicates between the hosts -and OSDs in the `default` root hierarchy. - -### Setting The Number Of Replicas - -The `size` setting of a pool tells the cluster how many copies of the data -should be kept for redundancy. By default the cluster will distribute these -copies between `host` buckets in the CRUSH Map This can be set when [creating a -pool via CustomResourceDefinition](ceph-pool-crd.md) or after creation with `ceph`. - -So for example let's change the `size` of the `rbd` pool to three: - -```console -ceph osd pool set rbd size 3 -``` - -Now if you run `ceph -s` you may see "recovery" operations and -PGs in "undersized" and other "unclean" states. The cluster is essentially -fixing itself since the number of replicas has been increased, and should go -back to "active/clean" state shortly, after data has been replicated between -hosts. When that's done you will be able to lose two of your storage nodes and -still have access to all your data in that pool, since the CRUSH algorithm will -guarantee that at least one replica will still be available on another storage node. -Of course you will only have 1/3 the capacity as a tradeoff. - ### Setting PG Count Be sure to read the [placement group sizing](#placement-group-sizing) section @@ -592,6 +428,9 @@ For example, hostNetwork: true ``` +> IMPORTANT: This setting cannot be changed in a running Rook cluster. Host networking +> should be configured when the cluster is first created. + ### Define the subnets to use for public and private OSD networks Edit the `rook-config-override` configmap to define the custom network @@ -651,7 +490,7 @@ ceph auth del osd. ceph osd rm ``` -To recheck that the Phantom OSD got removed, re-run the following command and check if the OSD with the ID doesn't show up anymore: +To recheck that the Phantom OSD was removed, re-run the following command and check if the OSD with the ID doesn't show up anymore: ```console ceph osd tree @@ -698,25 +537,3 @@ crush_rule: replicapool_host_rule If the cluster's health was `HEALTH_OK` when we performed this change, immediately, the new rule is applied to the cluster transparently without service disruption. Exactly the same approach can be used to change from `host` back to `osd`. - -## Monitor placement - -Rook will try to schedule Ceph monitor pods on different physical nodes by -default. When available, Rook will also consider failure domain information in -the form of Kubernetes node labels when scheduling Ceph monitors. - -Currently Rook supports the node label -`topology.kubernetes.io/zone=` which can be applied to a node -to specify its failure domain using the command: - -```console -kubectl label node topology.kubernetes.io/zone= -``` - -> For versions previous to K8s 1.17, use the topology key: failure-domain.beta.kubernetes.io/zone - -Rook uses failure domain labels by trying to schedule monitor pods on different -failure domains. And all nodes without failure domain labels are treated as a single -failure domain from a scheduling point of view. When placing multiple monitor -pods within a single failure domain Rook will try to run the pods on different -physical nodes. diff --git a/Documentation/ceph-block.md b/Documentation/ceph-block.md index 0d2d6c661..66c6ac70d 100644 --- a/Documentation/ceph-block.md +++ b/Documentation/ceph-block.md @@ -74,10 +74,10 @@ parameters: reclaimPolicy: Delete ``` -If you've deployed the Rook operator in a namespace other than "rook-ceph" -as is common change the prefix in the provisioner to match the namespace -you used. For example, if the Rook operator is running in "rook-op" the -provisioner value should be "rook-op.rbd.csi.ceph.com". +If you've deployed the Rook operator in a namespace other than "rook-ceph", +change the prefix in the provisioner to match the namespace +you used. For example, if the Rook operator is running in the namespace "ilovestorage" the +provisioner value should be "ilovestorage.rbd.csi.ceph.com". Create the storage class. diff --git a/Documentation/ceph-client-crd.md b/Documentation/ceph-client-crd.md index f264601e0..6cae6ffcd 100644 --- a/Documentation/ceph-client-crd.md +++ b/Documentation/ceph-client-crd.md @@ -14,7 +14,7 @@ For more information about user management and capabilities see the [Ceph docs]( Use Client CRD in case you want to integrate Rook with with applications that are using LibRBD directly. For example for OpenStack deployment with Ceph backend use Client CRD to create OpenStack services users. -Don't use Client CRD for Flex or CSI driver users, they are created automatically. +The Client CRD is not needed for Flex or CSI driver users. The drivers create the needed users automatically. ## Creating Ceph User diff --git a/Documentation/ceph-common-issues.md b/Documentation/ceph-common-issues.md index 05593f238..a5e51dd46 100644 --- a/Documentation/ceph-common-issues.md +++ b/Documentation/ceph-common-issues.md @@ -8,7 +8,7 @@ indent: true Many of these problem cases are hard to summarize down to a short phrase that adequately describes the problem. Each problem will start with a bulleted list of symptoms. Keep in mind that all symptoms may not apply depending on the configuration of Rook. If the majority of the symptoms are seen there is a fair chance you are experiencing that problem. -If after trying the suggestions found on this page and the problem is not resolved, the Rook team is very happy to help you troubleshoot the issues in their Slack channel. Once you have [registered for the Rook Slack](https://slack.rook.io), proceed to the General channel to ask for assistance. +If after trying the suggestions found on this page and the problem is not resolved, the Rook team is very happy to help you troubleshoot the issues in their Slack channel. Once you have [registered for the Rook Slack](https://slack.rook.io), proceed to the `#ceph` channel to ask for assistance. ## Table of Contents @@ -70,6 +70,9 @@ There are many Ceph sub-commands to look at and manipulate Ceph objects, well be ## Pod Using Ceph Storage Is Not Running +> This topic is specific to creating PVCs based on Rook's **Flex** driver, which is no longer the default option. +> By default, Rook deploys the CSI driver for binding the PVCs to the storage. + ### Symptoms * The pod that is configured to use Rook storage is stuck in the `ContainerCreating` status @@ -78,6 +81,8 @@ There are many Ceph sub-commands to look at and manipulate Ceph objects, well be * `timeout expired waiting for volumes to attach/mount` * `kubectl -n rook-ceph get pod` shows the rook-ceph-agent pods in a `CrashLoopBackOff` status +If you see that the PVC remains in **pending** state, see the topic [PVCs stay in pending state](#pvcs-stay-in-pending-state). + ### Possible Solutions Summary * `rook-ceph-agent` pod is in a `CrashLoopBackOff` status because it cannot deploy its driver on a read-only filesystem: [Flexvolume configuration pre-reqs](./k8s-pre-reqs.md#flexvolume-configuration) @@ -144,7 +149,7 @@ Once the `rook-ceph-agent` pods are gone, **follow the instructions in the [Flex After that has been configured, and the Kubelet has been restarted, start the agent pods up again by restarting `rook-operator`: ```console -kubectl -n rook-ceph delete pod -l app=rook-operator +kubectl -n rook-ceph delete pod -l app=rook-ceph-operator ``` #### Volume Creation @@ -166,7 +171,7 @@ Both your volume and its claim should be in the `Bound` status. If one or neither of them is not in the `Bound` status, then look for details of the issue in the `rook-operator` logs: ```console -kubectl -n rook-ceph logs `kubectl -n rook-ceph -l app=rook-operator get pods -o jsonpath='{.items[*].metadata.name}'` +kubectl -n rook-ceph logs `kubectl -n rook-ceph -l app=rook-ceph-operator get pods -o jsonpath='{.items[*].metadata.name}'` ``` If the volume is failing to be created, there should be details in the `rook-operator` log output, especially those tagged with `op-provisioner`. @@ -181,7 +186,7 @@ E0328 18:58:32.060893 5 controller.go:801] Failed to provision volume for . output: ``` -The solution is to ensure that the [`clusterNamespace`](https://github.com/rook/rook/blob/master/cluster/examples/kubernetes/storageclass.yaml#L20) field matches the **namespace** of the Rook cluster when creating the `StorageClass`. +The solution is to ensure that the [`clusterNamespace`](https://github.com/rook/rook/blob/master/cluster/examples/kubernetes/ceph/flex/storageclass.yaml#L28) field matches the **namespace** of the Rook cluster when creating the `StorageClass`. #### Volume Mounting @@ -247,7 +252,7 @@ We want to help you get your storage working and learn from those lessons to pre ### Investigation -Create a [rook-ceph-tools pod](./ceph-toolbox.md) to investigate the current state of CEPH. Here is an example of what one might see. In this case the `ceph status` command would just hang so a CTRL-C needed to be sent. +Create a [rook-ceph-tools pod](ceph-toolbox.md) to investigate the current state of CEPH. Here is an example of what one might see. In this case the `ceph status` command would just hang so a CTRL-C needed to be sent. ```console $ kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') bash @@ -299,7 +304,7 @@ The likely causes for the mon health not being detected: First look at the logs of the operator to confirm if it is able to connect to the mons. ```console -kubectl -n rook-ceph logs -l app=rook-operator +kubectl -n rook-ceph logs -l app=rook-ceph-operator ``` Likely you will see an error similar to the following that the operator is timing out when connecting to the mon. The last command is `ceph mon_status`, @@ -390,7 +395,10 @@ wp-pv-claim Pending rook-ceph-block There are two common causes for the PVCs staying in pending state: 1. There are no OSDs in the cluster -2. The operator is not running or is otherwise not responding to the request to create the block image +2. The CSI provisioner pod is not running or is not responding to the request to provision the storage + +If you are still using the Rook flex driver for the volumes (the CSI driver is the default since Rook v1.1), +another cause could be that the operator is not running or is otherwise not responding to the request to provision the storage. #### Confirm if there are OSDs @@ -424,6 +432,19 @@ rook-ceph-osd-prepare-minikube-9twvk 0/2 Completed 0 30m See the section on [why OSDs are not getting created](#osd-pods-are-not-created-on-my-devices) to investigate the logs. +#### CSI Driver + +The CSI driver may not be responding to the requests. Look in the logs of the CSI provisioner pod to see if there are any errors +during the provisioning. + +``` +# There are two provisioner pods +kubectl -n rook-ceph get pod -l app=csi-rbdplugin-provisioner + +# Get the logs of each of the pods. One of them should be the "leader" and be responding to requests. +kubectl -n rook-ceph logs csi-cephfsplugin-provisioner-d77bb49c6-q9hwq csi-provisioner +``` + #### Operator unresponsiveness Lastly, if you have OSDs `up` and `in`, the next step is to confirm the operator is responding to the requests. diff --git a/Documentation/ceph-monitoring.md b/Documentation/ceph-monitoring.md index 9cc02da80..51947f51a 100644 --- a/Documentation/ceph-monitoring.md +++ b/Documentation/ceph-monitoring.md @@ -1,11 +1,11 @@ --- -title: Monitoring +title: Prometheus Monitoring weight: 2500 indent: true --- {% include_relative branch.liquid %} -# Monitoring +# Prometheus Monitoring Each Rook Ceph cluster has some built in metrics collectors/exporters for monitoring with [Prometheus](https://prometheus.io/). From 2c70e10bd66c43c21802586b1390d2597f04eee8 Mon Sep 17 00:00:00 2001 From: Travis Nielsen Date: Mon, 9 Mar 2020 18:21:13 -0600 Subject: [PATCH 4/6] docs: add cluster-on-pvc and remove cluster-minimal examples The cluster-minimal.yaml example is more confusing than helpful. Most users seem to think that minimal includes a fully working cluster with OSDs. The cluster-on-pvc.yaml example is frequently used and was missing from the examples doc. Signed-off-by: Travis Nielsen --- Documentation/ceph-examples.md | 11 +++---- .../kubernetes/ceph/cluster-minimal.yaml | 32 ------------------- 2 files changed, 5 insertions(+), 38 deletions(-) delete mode 100644 cluster/examples/kubernetes/ceph/cluster-minimal.yaml diff --git a/Documentation/ceph-examples.md b/Documentation/ceph-examples.md index 243e34111..2c7774a0c 100644 --- a/Documentation/ceph-examples.md +++ b/Documentation/ceph-examples.md @@ -41,15 +41,14 @@ Settings for the operator are configured through environment variables on the op ## Cluster CRD -Now that your operator is running, let's create your Ceph storage cluster: +Now that your operator is running, let's create your Ceph storage cluster. This CR contains the most critical settings +that will influence how the operator configures the storage. It is important to understand the various ways to configure +the cluster. These examples represent a very small set of the different ways to configure the storage. * `cluster.yaml`: This file contains common settings for a production storage cluster. Requires at least three nodes. -* `cluster-minimal.yaml`: Brings up a cluster with only one [ceph-mon](http://docs.ceph.com/docs/nautilus/man/8/ceph-mon/) and a [ceph-mgr](http://docs.ceph.com/docs/nautilus/mgr/) so the Ceph dashboard can be used for the remaining cluster configuration. +* `cluster-on-pvc.yaml`: This file contains common settings for backing the Ceph Mons and OSDs by PVs. Useful when running in cloud environments or where local PVs have been created for Ceph to consume. -See the [Cluster CRD](ceph-cluster-crd.md) topic for more details on the settings. - -Monitors may be configured to run on PVC storage. Details on [how to set this up -and some minor restrctions are described here](ceph-cluster-crd#mon-settings). +See the [Cluster CRD](ceph-cluster-crd.md) topic for more details and more examples for the settings. ## Setting up consumable storage diff --git a/cluster/examples/kubernetes/ceph/cluster-minimal.yaml b/cluster/examples/kubernetes/ceph/cluster-minimal.yaml deleted file mode 100644 index 3fbf2d6cd..000000000 --- a/cluster/examples/kubernetes/ceph/cluster-minimal.yaml +++ /dev/null @@ -1,32 +0,0 @@ -################################################################################################################# -# Define the settings for the rook-ceph cluster where only a single mon and mgr are started. -# Other daemons are expected to be set through the Ceph dashboard or by otherwise modifying the cluster crd -# after the initial cluster is created. -# See the cluster CRD documentation for more details: https://rook.io/docs/rook/master/ceph-cluster-crd.html -# For example, to create the cluster: -# kubectl create -f common.yaml -# kubectl create -f operator.yaml -# kubectl create -f cluster-minimal.yaml -################################################################################################################# - -apiVersion: ceph.rook.io/v1 -kind: CephCluster -metadata: - name: rook-ceph - namespace: rook-ceph -spec: - cephVersion: - image: ceph/ceph:v14.2.8 - allowUnsupported: false - dataDirHostPath: /var/lib/rook - mon: - count: 1 - allowMultiplePerNode: false - dashboard: - enabled: true - ssl: true - monitoring: - enabled: false # requires Prometheus to be pre-installed - rulesNamespace: rook-ceph - network: - hostNetwork: false From a251959061e20c5d3438d53b4a7c650d6b161ea0 Mon Sep 17 00:00:00 2001 From: Travis Nielsen Date: Mon, 9 Mar 2020 18:27:11 -0600 Subject: [PATCH 5/6] docs: restructure prerequisites and clarify ceph docs The docs were in need of some restructuring wrt prerequisites. They were mingled with general k8s and Ceph requirements. For clarity, we need to separate the ceph prereqs from general k8s requirements. There is now only a single prereqs page. All other topics should be found under other sections, such as specific to Ceph. Having subtopics under the general prereqs caused users to frequently not even notice the main prereqs topic. Signed-off-by: Travis Nielsen --- Documentation/ceph-advanced-configuration.md | 2 +- Documentation/ceph-block.md | 4 +- Documentation/ceph-common-issues.md | 16 +- Documentation/ceph-nfs-crd.md | 4 +- Documentation/ceph-openshift.md | 6 +- Documentation/ceph-pool-crd.md | 2 +- Documentation/ceph-prerequisites.md | 66 +++++++ Documentation/ceph-quickstart.md | 45 +++-- Documentation/flexvolume.md | 55 +++--- Documentation/k8s-pre-reqs.md | 172 ++++++++++--------- Documentation/psp.md | 81 --------- Documentation/tectonic.md | 6 +- 12 files changed, 234 insertions(+), 225 deletions(-) create mode 100644 Documentation/ceph-prerequisites.md delete mode 100644 Documentation/psp.md diff --git a/Documentation/ceph-advanced-configuration.md b/Documentation/ceph-advanced-configuration.md index f5b529947..dd80a1490 100644 --- a/Documentation/ceph-advanced-configuration.md +++ b/Documentation/ceph-advanced-configuration.md @@ -428,7 +428,7 @@ For example, hostNetwork: true ``` -> IMPORTANT: This setting cannot be changed in a running Rook cluster. Host networking +> IMPORTANT: Changing this setting is not supported in a running Rook cluster. Host networking > should be configured when the cluster is first created. ### Define the subnets to use for public and private OSD networks diff --git a/Documentation/ceph-block.md b/Documentation/ceph-block.md index 66c6ac70d..30f788b0a 100644 --- a/Documentation/ceph-block.md +++ b/Documentation/ceph-block.md @@ -76,8 +76,8 @@ reclaimPolicy: Delete If you've deployed the Rook operator in a namespace other than "rook-ceph", change the prefix in the provisioner to match the namespace -you used. For example, if the Rook operator is running in the namespace "ilovestorage" the -provisioner value should be "ilovestorage.rbd.csi.ceph.com". +you used. For example, if the Rook operator is running in the namespace "my-namespace" the +provisioner value should be "my-namespace.rbd.csi.ceph.com". Create the storage class. diff --git a/Documentation/ceph-common-issues.md b/Documentation/ceph-common-issues.md index a5e51dd46..573917f8f 100644 --- a/Documentation/ceph-common-issues.md +++ b/Documentation/ceph-common-issues.md @@ -252,7 +252,7 @@ We want to help you get your storage working and learn from those lessons to pre ### Investigation -Create a [rook-ceph-tools pod](ceph-toolbox.md) to investigate the current state of CEPH. Here is an example of what one might see. In this case the `ceph status` command would just hang so a CTRL-C needed to be sent. +Create a [rook-ceph-tools pod](ceph-toolbox.md) to investigate the current state of Ceph. Here is an example of what one might see. In this case the `ceph status` command would just hang so a CTRL-C needed to be sent. ```console $ kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') bash @@ -277,7 +277,7 @@ rook-ceph-osd-mwxdm 1/1 Running 0 2d 19 What is happening here is that the MON pods are restarting and one or more of the Ceph daemons are not getting configured with the proper cluster information. This is commonly the result of not specifying a value for `dataDirHostPath` in your Cluster CRD. -The `dataDirHostPath` setting specifies a path on the local host for the CEPH daemons to store configuration and data. Setting this to a path like `/var/lib/rook`, reapplying your Cluster CRD and restarting all the CEPH daemons (MON, MGR, OSD, RGW) should solve this problem. After the CEPH daemons have been restarted, it is advisable to restart the [rook-tool pod](./toolbox.md). +The `dataDirHostPath` setting specifies a path on the local host for the Ceph daemons to store configuration and data. Setting this to a path like `/var/lib/rook`, reapplying your Cluster CRD and restarting all the Ceph daemons (MON, MGR, OSD, RGW) should solve this problem. After the Ceph daemons have been restarted, it is advisable to restart the [rook-tool pod](./toolbox.md). ## Monitors are the only pods running @@ -437,11 +437,15 @@ See the section on [why OSDs are not getting created](#osd-pods-are-not-created- The CSI driver may not be responding to the requests. Look in the logs of the CSI provisioner pod to see if there are any errors during the provisioning. -``` -# There are two provisioner pods -kubectl -n rook-ceph get pod -l app=csi-rbdplugin-provisioner +There are two provisioner pods: -# Get the logs of each of the pods. One of them should be the "leader" and be responding to requests. +``` +kubectl -n rook-ceph get pod -l app=csi-rbdplugin-provisioner +``` + +Get the logs of each of the pods. One of them should be the "leader" and be responding to requests. + +``` kubectl -n rook-ceph logs csi-cephfsplugin-provisioner-d77bb49c6-q9hwq csi-provisioner ``` diff --git a/Documentation/ceph-nfs-crd.md b/Documentation/ceph-nfs-crd.md index 60a65e4ae..a321491c3 100644 --- a/Documentation/ceph-nfs-crd.md +++ b/Documentation/ceph-nfs-crd.md @@ -14,7 +14,9 @@ Rook allows exporting NFS shares of the filesystem or object store through the C The following sample will create a two-node active-active cluster of NFS Ganesha gateways. A CephFS named `myfs` is used, and the recovery objects are stored in a RADOS pool named `myfs-data0` with a RADOS namespace of `nfs-ns`. -> **NOTE**: For an RGW object store, a data pool of `my-store.rgw.buckets.data` can be used. +This example requires the filesystem to first be configured by the [Filesystem](ceph-filesystem-crd.md). + +> **NOTE**: For an RGW object store, a data pool of `my-store.rgw.buckets.data` can be used after configuring the [Object Store](ceph-object-store-crd.md). ```yaml apiVersion: ceph.rook.io/v1 diff --git a/Documentation/ceph-openshift.md b/Documentation/ceph-openshift.md index c67f0f450..91089cc64 100644 --- a/Documentation/ceph-openshift.md +++ b/Documentation/ceph-openshift.md @@ -1,6 +1,6 @@ --- title: OpenShift -weight: 1700 +weight: 2060 indent: true --- {% assign url = page.url | split: '/' %} @@ -144,7 +144,3 @@ A sample object store can be created with these settings: ```console oc create -f object-openshift.yaml ``` - -## MiniShift - -There is a known issue in MiniShift that does not allow Rook to be tested in some common end-to-end scenarios. Flex drivers are not currently supported, which means that block and file volumes cannot be mounted. See this [tracking issue](https://github.com/minishift/minishift/issues/2387). diff --git a/Documentation/ceph-pool-crd.md b/Documentation/ceph-pool-crd.md index 50cb9f1d0..52a3c858b 100644 --- a/Documentation/ceph-pool-crd.md +++ b/Documentation/ceph-pool-crd.md @@ -106,6 +106,6 @@ The `failureDomain` must be also be taken into account when determining the numb * `host`: All chunks will be placed on unique hosts * `osd`: All chunks will be placed on unique OSDs -If you do not have a sufficient number of hosts or OSDs for unique placement the pool can be created, a PUT to the pool will hang. +If you do not have a sufficient number of hosts or OSDs for unique placement the pool can be created, writing to the pool will hang. Rook currently only configures two levels in the CRUSH map. It is also possible to configure other levels such as `rack` with by adding [topology labels](ceph-cluster-crd.md#osd-topology) to the nodes. diff --git a/Documentation/ceph-prerequisites.md b/Documentation/ceph-prerequisites.md new file mode 100644 index 000000000..9025255dc --- /dev/null +++ b/Documentation/ceph-prerequisites.md @@ -0,0 +1,66 @@ +--- +title: Prerequisites +weight: 2010 +indent: true +--- + +# Ceph Prerequisites + +## LVM package + +Ceph OSDs have a dependency on LVM in the following scenarios: +- OSDs are created on raw devices or partitions +- If encryption is enabled (`encryptedDevice: true` in the cluster CR) +- A `metadata` device is specified + +LVM is not required for OSDs in these scenarios: +- Creating OSDs on PVCs using the `storageClassDeviceSets` + +If LVM is required for your scenario, LVM needs to be available on the hosts where OSDs will be running. +Some Linux distributions do not ship with the `lvm2` package. This package is required on all storage nodes in your k8s cluster to run Ceph OSDs. +Without this package even though Rook will be able to successfully create the Ceph OSDs, when a node is rebooted the OSD pods +running on the restarted node will **fail to start**. Please install LVM using your Linux distribution's package manager. For example: + +CentOS: + +```console +sudo yum install -y lvm2 +``` + +Ubuntu: + +``` +sudo apt-get install -y lvm2 +``` + +## Ceph Flexvolume Configuration + +**NOTE** This configuration is only needed when using the FlexVolume driver (required for Kubernetes 1.12 or earlier). The Ceph-CSI RBD driver or the Ceph-CSI CephFS driver are recommended for Kubernetes 1.13 and newer, making FlexVolume configuration redundant. + +If you want to configure volumes with the Flex driver instead of CSI, the Rook agent requires setup as a Flex volume plugin to manage the storage attachments in your cluster. +See the [Flex Volume Configuration](flexvolume.md) topic to configure your Kubernetes deployment to load the Rook volume plugin. + +### Extra agent mounts + +On certain distributions it may be necessary to mount additional directories into the agent container. That is what the environment variable `AGENT_MOUNTS` is for. Also see the documentation in [helm-operator](helm-operator.md) on the parameter `agent.mounts`. The format of the variable content should be `mountname1=/host/path1:/container/path1,mountname2=/host/path2:/container/path2`. + +## Kernel + +### RBD + +Ceph requires a Linux kernel built with the RBD module. Many Linux distributions have this module, but not all distributions. +For example, the GKE Container-Optimised OS (COS) does not have RBD. + +You can test your Kubernetes nodes by running `modprobe rbd`. +If it says 'not found', you may have to [rebuild your kernel](https://rook.io/docs/rook/master/common-issues.html#rook-agent-rbd-module-missing-error) +or choose a different Linux distribution. + +### CephFS + +If you will be creating volumes from a Ceph shared file system (CephFS), the recommended minimum kernel version is **4.17**. +If you have a kernel version less than 4.17, the requested PVC sizes will not be enforced. Storage quotas will only be +enforced on newer kernels. + +## Kernel modules directory configuration + +Normally, on Linux, kernel modules can be found in `/lib/modules`. However, there are some distributions that put them elsewhere. In that case the environment variable `LIB_MODULES_DIR_PATH` can be used to override the default. Also see the documentation in [helm-operator](helm-operator.md) on the parameter `agent.libModulesDirPath`. One notable distribution where this setting is useful would be [NixOS](https://nixos.org). diff --git a/Documentation/ceph-quickstart.md b/Documentation/ceph-quickstart.md index 44836b024..95ed9abfd 100644 --- a/Documentation/ceph-quickstart.md +++ b/Documentation/ceph-quickstart.md @@ -20,7 +20,10 @@ Kubernetes **v1.10** or higher is supported by Rook. To make sure you have a Kubernetes cluster that is ready for `Rook`, you can [follow these instructions](k8s-pre-reqs.md). -If you are using `dataDirHostPath` to persist rook data on kubernetes hosts, make sure your host has at least 5GB of space available on the specified path. +In order to configure the storage cluster, at least one of these local storage options are required: +- Raw devices (no partitions or formatted filesystems) +- Raw partitions (no formatted filsystem) +- PVs available from a storage class in `block` mode ## TL;DR @@ -71,24 +74,34 @@ kubectl create -f cluster.yaml Use `kubectl` to list pods in the `rook-ceph` namespace. You should be able to see the following pods once they are all running. The number of osd pods will depend on the number of nodes in the cluster and the number of devices configured. If you did not modify the `cluster.yaml` above, it is expected that one OSD will be created per node. -The `rook-ceph-agent` and `rook-discover` pods are also optional depending on your settings. +The CSI, `rook-ceph-agent`, and `rook-discover` pods are also optional depending on your settings. + +> If the `rook-ceph-mon`, `rook-ceph-mgr`, or `rook-ceph-osd` pods are not created, please refer to the +> [Ceph common issues](ceph-common-issues.md) for more details and potential solutions. ```console $ kubectl -n rook-ceph get pod -NAME READY STATUS RESTARTS AGE -rook-ceph-agent-4zkg8 1/1 Running 0 140s -rook-ceph-mgr-a-d9dcf5748-5s9ft 1/1 Running 0 77s -rook-ceph-mon-a-7d8f675889-nw5pl 1/1 Running 0 105s -rook-ceph-mon-b-856fdd5cb9-5h2qk 1/1 Running 0 94s -rook-ceph-mon-c-57545897fc-j576h 1/1 Running 0 85s -rook-ceph-operator-6c49994c4f-9csfz 1/1 Running 0 141s -rook-ceph-osd-0-7cbbbf749f-j8fsd 1/1 Running 0 23s -rook-ceph-osd-1-7f67f9646d-44p7v 1/1 Running 0 24s -rook-ceph-osd-2-6cd4b776ff-v4d68 1/1 Running 0 25s -rook-ceph-osd-prepare-node1-vx2rz 0/2 Completed 0 60s -rook-ceph-osd-prepare-node2-ab3fd 0/2 Completed 0 60s -rook-ceph-osd-prepare-node3-w4xyz 0/2 Completed 0 60s -rook-discover-dhkb8 1/1 Running 0 140s +NAME READY STATUS RESTARTS AGE +csi-cephfsplugin-provisioner-d77bb49c6-n5tgs 5/5 Running 0 140s +csi-cephfsplugin-provisioner-d77bb49c6-v9rvn 5/5 Running 0 140s +csi-cephfsplugin-rthrp 3/3 Running 0 140s +csi-rbdplugin-hbsm7 3/3 Running 0 140s +csi-rbdplugin-provisioner-5b5cd64fd-nvk6c 6/6 Running 0 140s +csi-rbdplugin-provisioner-5b5cd64fd-q7bxl 6/6 Running 0 140s +rook-ceph-agent-4zkg8 1/1 Running 0 140s +rook-ceph-crashcollector-minikube-5b57b7c5d4-hfldl 1/1 Running 0 105s +rook-ceph-mgr-a-64cd7cdf54-j8b5p 1/1 Running 0 77s +rook-ceph-mon-a-694bb7987d-fp9w7 1/1 Running 0 105s +rook-ceph-mon-b-856fdd5cb9-5h2qk 1/1 Running 0 94s +rook-ceph-mon-c-57545897fc-j576h 1/1 Running 0 85s +rook-ceph-operator-85f5b946bd-s8grz 1/1 Running 0 92m +rook-ceph-osd-0-6bb747b6c5-lnvb6 1/1 Running 0 23s +rook-ceph-osd-1-7f67f9646d-44p7v 1/1 Running 0 24s +rook-ceph-osd-2-6cd4b776ff-v4d68 1/1 Running 0 25s +rook-ceph-osd-prepare-node1-vx2rz 0/2 Completed 0 60s +rook-ceph-osd-prepare-node2-ab3fd 0/2 Completed 0 60s +rook-ceph-osd-prepare-node3-w4xyz 0/2 Completed 0 60s +rook-discover-dhkb8 1/1 Running 0 140s ``` To verify that the cluster is in a healthy state, connect to the [Rook toolbox](ceph-toolbox.md) and run the diff --git a/Documentation/flexvolume.md b/Documentation/flexvolume.md index 4f2c3c39c..58af53e38 100644 --- a/Documentation/flexvolume.md +++ b/Documentation/flexvolume.md @@ -4,7 +4,7 @@ weight: 1200 indent: true --- -# FlexVolume Configuration +# Ceph FlexVolume Configuration FlexVolume is not enabled by default since Rook v1.1. This documentation applies only if you have enabled FlexVolume. @@ -14,19 +14,20 @@ Some Kubernetes deployments require you to configure kubelet with a FlexVolume p Platform-specific instructions for the following Kubernetes deployment platforms are linked below -* [Atomic](#atomic) - see also [Atomic home page](https://www.projectatomic.io/) -* [Azure AKS](#azure-aks) - see also [Azure AKS docs](https://docs.microsoft.com/en-us/azure/aks/) -* [Container Linux](#containerlinux) (previously named CoreOS) - see also [Container Linux docs](https://coreos.com/os/docs/latest/) -* [Google Kubernetes Engine (GKE)](#google-kubernetes-engine-gke) - see also [GKE home page](https://cloud.google.com/kubernetes-engine/) -* [Kubespray](#kubespray) prior to v2.11.0 - see also [Kubespray home page](https://kubespray.io/) -* [OpenShift](#openshift) - see also [OpenShift home page](https://www.openshift.com/) -* [OpenStack Magnum](#openstack-magnum) - see also [OpenStack Magnum docs](https://docs.openstack.org/magnum/latest/) -* [Rancher](#rancher) - see also [Rancher home page](http://rancher.com/) -* [Tectonic](#tectonic) - see also [Tectonic docs](https://coreos.com/tectonic/docs/latest/) +* [Default FlexVolume path](#default-flexvolume-path) +* [Atomic](#atomic) +* [Azure AKS](#azure-aks) +* [ContainerLinux](#containerlinux) +* [Google Kubernetes Engine (GKE)](#google-kubernetes-engine-gke) +* [Kubespray](#kubespray) +* [OpenShift](#openshift) +* [OpenStack Magnum](#openstack-magnum) +* [Rancher](#rancher) +* [Tectonic](#tectonic) +* [Custom containerized kubelet](#custom-containerized-kubelet) +* [Configuring the FlexVolume path](#configuring-the-flexvolume-path) -## Platform specific FlexVolume path - -### Most common read/write FlexVolume path +## Default FlexVolume path If you are not using a platform that is listed above and the path `/usr/libexec/kubernetes/kubelet-plugins/volume/exec/` is read/write, you don't need to configure anything. @@ -34,27 +35,25 @@ That is because `/usr/libexec/kubernetes/kubelet-plugins/volume/exec/` is the ku If running `mkdir -p /usr/libexec/kubernetes/kubelet-plugins/volume/exec/` should give you an error about read-only filesystem, you need to use [another read/write FlexVolume path](#other-common-readwrite-flexvolume-paths) and configure it on the Rook operator and kubelet. -### Other common read/write FlexVolume paths - -Because `/var/lib/kubelet` is read write on most systems, these are the other commonly used paths: +These are the other commonly used paths: * `/var/lib/kubelet/volumeplugins` * `/var/lib/kubelet/volume-plugins` Continue with [configuring the FlexVolume path](#configuring-the-flexvolume-path) to configure Rook to use the FlexVolume path. -### Atomic +## Atomic See the [OpenShift](#openshift) section, unless running with OpenStack Magnum, then see [OpenStack Magnum](#openstack-magnum) section. -### Azure AKS +## Azure AKS AKS uses a non-standard FlexVolume plugin directory: `/etc/kubernetes/volumeplugins` The kubelet on AKS is already configured to use that directory. Continue with [configuring the FlexVolume path](#configuring-the-flexvolume-path) to configure Rook to use the FlexVolume path. -### ContainerLinux +## ContainerLinux Use the [Most common read/write FlexVolume path](#most-common-readwrite-flexvolume-path) for the next steps. @@ -62,16 +61,16 @@ The kubelet's systemD unit file can be located at: `/etc/systemd/system/kubelet. Continue with [configuring the FlexVolume path](#configuring-the-flexvolume-path) to configure Rook to use the FlexVolume path. -### Google Kubernetes Engine (GKE) +## Google Kubernetes Engine (GKE) Google's Kubernetes Engine uses a non-standard FlexVolume plugin directory: `/home/kubernetes/flexvolume` The kubelet on GKE is already configured to use that directory. Continue with [configuring the FlexVolume path](#configuring-the-flexvolume-path) to configure Rook to use the FlexVolume path. -### Kubespray +## Kubespray -#### Prior to v2.11.0 +### Prior to v2.11.0 Kubespray uses the [kubelet_flexvolumes_plugins_dir](https://github.com/kubernetes-sigs/kubespray/blob/v2.11.0/inventory/sample/group_vars/k8s-cluster/k8s-cluster.yml#L206) variable to define where it sets the plugin directory. @@ -80,7 +79,7 @@ The Kubespray configured kubelet is already configured to use that directory. If you are using kubespray v2.10.x or older, continue with [configuring the FlexVolume path](#configuring-the-flexvolume-path) to configure Rook to use the FlexVolume path. -#### As of v2.11.0 and newer +### As of v2.11.0 and newer Kubespray v2.11.0 included https://github.com/kubernetes-sigs/kubespray/pull/4752 which sets the same plugin directory assumed by rook by default: `/usr/libexec/kubernetes/kubelet-plugins/volume/exec`. @@ -93,19 +92,19 @@ If you have not defined one, and the default path is not writable, the [alternat If needed, continue with [configuring the FlexVolume path](#configuring-the-flexvolume-path) to configure Rook to use the FlexVolume path. -### OpenShift +## OpenShift To find out which FlexVolume directory path you need to set on the Rook operator, please look at the OpenShift docs of the version you are using, [latest OpenShift Flexvolume docs](https://docs.openshift.org/latest/install_config/persistent_storage/persistent_storage_flex_volume.html#flexvolume-installation) (they also contain the FlexVolume path for Atomic). Continue with [configuring the FlexVolume path](#configuring-the-flexvolume-path) to configure Rook to use the FlexVolume path. -### OpenStack Magnum +## OpenStack Magnum OpenStack Magnum is using Atomic, which uses a non-standard FlexVolume plugin directory at: `/var/lib/kubelet/volumeplugins` The kubelet in OpenStack Magnum is already configured to use that directory. You will need to use this value when [configuring the Rook operator](#configuring-the-rook-operator) -### Rancher +## Rancher Rancher provides an easy way to configure kubelet. The FlexVolume flag will be shown later on in the [configuring the FlexVolume path](#configuring-the-flexvolume-path). It can be provided to the kubelet configuration template at deployment time or by using the `up to date` feature if Kubernetes is already deployed. @@ -133,14 +132,14 @@ FlexVolume path for the Rook operator. If the default path as above is used no further configuration is required, otherwise if a different path is used the Rook operator will need to be reconfigured, to do this continue with [configuring the FlexVolume path](#configuring-the-flexvolume-path) to configure Rook to use the FlexVolume path. -### Tectonic +## Tectonic Follow [these instructions](tectonic.md) to configure the Flexvolume plugin for Rook on Tectonic during ContainerLinux node ignition file provisioning. If you want to use Rook with an already provisioned Tectonic cluster, please refer to the [ContainerLinux](#containerlinux) section. Continue with [configuring the FlexVolume path](#configuring-the-flexvolume-path) to configure Rook to use the FlexVolume path. -### Custom containerized kubelet +## Custom containerized kubelet Use the [most common read/write FlexVolume path](#most-common-readwrite-flexvolume-path) for the next steps. diff --git a/Documentation/k8s-pre-reqs.md b/Documentation/k8s-pre-reqs.md index 31006617e..61531ed53 100644 --- a/Documentation/k8s-pre-reqs.md +++ b/Documentation/k8s-pre-reqs.md @@ -2,6 +2,13 @@ title: Prerequisites weight: 1000 --- +{% assign url = page.url | split: '/' %} +{% assign currentVersion = url[3] %} +{% if currentVersion != 'master' %} +{% assign branchName = currentVersion | replace: 'v', '' | prepend: 'release-' %} +{% else %} +{% assign branchName = currentVersion %} +{% endif %} # Prerequisites @@ -11,105 +18,87 @@ you can quickly set one up using [Minikube](#minikube), [Kubeadm](#kubeadm) or [ ## Minimum Version -Kubernetes v1.13 or higher is supported by Rook. +Kubernetes v1.11 or higher is supported by Rook. -## Privileges and RBAC +## Ceph Prerequisites -Rook requires privileges to manage the storage in your cluster. See the details [here](psp.md) for -setting up Rook in a Kubernetes cluster with Pod Security Policies enabled. +See also **[Ceph Prerequisites](ceph-prerequisites.md)**. -## Flexvolume Configuration +## Pod Security Policies -**NOTE** This configuration is only needed when using the FlexVolume driver (required for Kubernetes 1.12 or earlier). The Ceph-CSI RBD driver or the Ceph-CSI CephFS driver are recommended for Kubernetes 1.13 and newer, making FlexVolume configuration redundant. +Rook requires privileges to manage the storage in your cluster. If you have Pod Security Policies enabled +please review this section. By default, Kubernetes clusters do not have PSPs enabled so you may +be able to skip this section. -The Rook agent requires setup as a Flex volume plugin to manage the storage attachments in your cluster. -See the [Flex Volume Configuration](flexvolume.md) topic to configure your Kubernetes deployment to load the Rook volume plugin. +If you are configuring Ceph on OpenShift, the Ceph walkthrough will configure the PSPs as well +when you start the operator with [operator-openshift.yaml]((https://github.com/rook/rook/blob/{{ branchName }}/cluster/examples/kubernetes/ceph/operator-openshift.yaml). -## Kernel +### Cluster Role -### RBD +> **NOTE**: Cluster role configuration is only needed when you are not already `cluster-admin` in your Kubernetes cluster! -Rook Ceph requires a Linux kernel built with the RBD module. Many distributions of Linux have this module but some don't, -e.g. the GKE Container-Optimised OS (COS) does not have RBD. You can test your Kubernetes nodes by running `modprobe rbd`. -If it says 'not found', you may have to [rebuild your kernel](https://rook.io/docs/rook/master/common-issues.html#rook-agent-rbd-module-missing-error) -or choose a different Linux distribution. +Creating the Rook operator requires privileges for setting up RBAC. To launch the operator you need to have created your user certificate that is bound to ClusterRole `cluster-admin`. -### CephFS - -If you will be creating volumes from a Ceph shared file system (CephFS), the recommended minimum kernel version is 4.17. -If you have a kernel version less than 4.17, the requested PVC sizes will not be enforced. Storage quotas will only be -enforced on newer kernels. - -## Kernel modules directory configuration - -Normally, on Linux, kernel modules can be found in `/lib/modules`. However, there are some distributions that put them elsewhere. In that case the environment variable `LIB_MODULES_DIR_PATH` can be used to override the default. Also see the documentation in [helm-operator](helm-operator.md) on the parameter `agent.libModulesDirPath`. One notable distribution where this setting is useful would be [NixOS](https://nixos.org). - -## Extra agent mounts - -On certain distributions it may be necessary to mount additional directories into the agent container. That is what the environment variable `AGENT_MOUNTS` is for. Also see the documentation in [helm-operator](helm-operator.md) on the parameter `agent.mounts`. The format of the variable content should be `mountname1=/host/path1:/container/path1,mountname2=/host/path2:/container/path2`. - -## LVM package - -Some Linux distributions do not ship with the `lvm2` package. This package is required on all storage nodes in your k8s cluster. Please install it using your Linux distribution's package manager; for example: +One simple way to achieve it is to assign your certificate with the `system:masters` group: ```console -# Centos -sudo yum install -y lvm2 - -# Ubuntu -sudo apt-get install -y lvm2 +-subj "/CN=admin/O=system:masters" ``` -## Bootstrapping Kubernetes +`system:masters` is a special group that is bound to `cluster-admin` ClusterRole, but it can't be easily revoked so be careful with taking that route in a production setting. +Binding individual certificate to ClusterRole `cluster-admin` is revocable by deleting the ClusterRoleBinding. -Rook will run wherever Kubernetes is running. Here are some simple environments to help you get started with Rook. +### RBAC for PodSecurityPolicies -### Minikube +If you have activated the [PodSecurityPolicy Admission Controller](https://kubernetes.io/docs/admin/admission-controllers/#podsecuritypolicy) and thus are +using [PodSecurityPolicies](https://kubernetes.io/docs/concepts/policy/pod-security-policy/), you will require additional `(Cluster)RoleBindings` +for the different `ServiceAccounts` Rook uses to start the Rook Storage Pods. -To install `minikube`, refer to this [page](https://github.com/kubernetes/minikube/releases). Once you have `minikube` installed, start a cluster by doing the following: +Security policies will differ for different backends. See Ceph's Pod Security Policies set up in +[common.yaml](https://github.com/rook/rook/blob/{{ branchName }}/cluster/examples/kubernetes/ceph/common.yaml) +for an example of how this is done in practice. -```console -$ minikube start -Starting local Kubernetes cluster... -Starting VM... -SSH-ing files into VM... -Setting up certs... -Starting cluster components... -Connecting to cluster... -Setting up kubeconfig... -Kubectl is now configured to use the cluster. +### PodSecurityPolicy + +You need at least one `PodSecurityPolicy` that allows privileged `Pod` execution. Here is an example +which should be more permissive than is needed for any backend: + +```yaml +apiVersion: policy/v1beta1 +kind: PodSecurityPolicy +metadata: + name: privileged +spec: + fsGroup: + rule: RunAsAny + privileged: true + runAsUser: + rule: RunAsAny + seLinux: + rule: RunAsAny + supplementalGroups: + rule: RunAsAny + volumes: + - '*' + allowedCapabilities: + - '*' + hostPID: true + # hostNetwork is required for using host networking + hostNetwork: false ``` -After these steps, your minikube cluster is ready to install Rook on. +**Hint**: Allowing `hostNetwork` usage is required when using `hostNetwork: true` in a Cluster `CustomResourceDefinition`! +You are then also required to allow the usage of `hostPorts` in the `PodSecurityPolicy`. The given +port range will allow all ports: -### Kubeadm - -You can easily spin up Rook on top of a `kubeadm` cluster. -You can find the instructions on how to install kubeadm in the [Install `kubeadm`](https://kubernetes.io/docs/setup/independent/install-kubeadm/) page. - -By using `kubeadm`, you can use Rook in just a few minutes! - -### New local Kubernetes cluster with Vagrant - -For a quick start with a new local cluster, use the Rook fork of [coreos-kubernetes](https://github.com/rook/coreos-kubernetes). This will bring up a multi-node Kubernetes cluster with `vagrant` and CoreOS virtual machines ready to use Rook immediately. - -```console -git clone https://github.com/rook/coreos-kubernetes.git -cd coreos-kubernetes/multi-node/vagrant -vagrant up -export KUBECONFIG="$(pwd)/kubeconfig" -kubectl config use-context vagrant-multi +```yaml + hostPorts: + # Ceph msgr2 port + - min: 1 + max: 65535 ``` -Then wait for the cluster to come up and verify that kubernetes is done initializing (be patient, it takes a bit): - -```console -kubectl cluster-info -``` - -Once you see a url response, your cluster is [ready for use by Rook](ceph-quickstart.md#deploy-rook). - -## Support for authenticated docker registries +## Authenticated docker registries If you want to use an image from authenticated docker registry (e.g. for image cache/mirror), you'll need to add an `imagePullSecret` to all relevant service accounts. This way all pods created by the operator (for service account: @@ -169,10 +158,31 @@ imagePullSecrets: # here are the new After doing this for all service accounts all pods should be able to pull the image from your registry. -## Using Rook in Kubernetes +## Bootstrapping Kubernetes -Now that you have a Kubernetes cluster running, you can start using Rook with [these steps](ceph-quickstart.md#deploy-rook). +Rook will run wherever Kubernetes is running. Here are some simple environments to help you get started with Rook. -## Using Rook on Tectonic Bare Metal +### Minikube -Follow [these instructions](tectonic.md) to run Rook on Tectonic Kubernetes +To install `minikube`, refer to this [page](https://github.com/kubernetes/minikube/releases). Once you have `minikube` installed, start a cluster by doing the following: + +```console +$ minikube start +Starting local Kubernetes cluster... +Starting VM... +SSH-ing files into VM... +Setting up certs... +Starting cluster components... +Connecting to cluster... +Setting up kubeconfig... +Kubectl is now configured to use the cluster. +``` + +After these steps, your minikube cluster is ready to install Rook on. + +### Kubeadm + +You can easily spin up Rook on top of a `kubeadm` cluster. +You can find the instructions on how to install kubeadm in the [Install `kubeadm`](https://kubernetes.io/docs/setup/independent/install-kubeadm/) page. + +By using `kubeadm`, you can use Rook in just a few minutes! diff --git a/Documentation/psp.md b/Documentation/psp.md deleted file mode 100644 index 15331c958..000000000 --- a/Documentation/psp.md +++ /dev/null @@ -1,81 +0,0 @@ ---- -title: Pod Security Policies -weight: 1400 -indent: true ---- -{% assign url = page.url | split: '/' %} -{% assign currentVersion = url[3] %} -{% if currentVersion != 'master' %} -{% assign branchName = currentVersion | replace: 'v', '' | prepend: 'release-' %} -{% else %} -{% assign branchName = currentVersion %} -{% endif %} - -# Using Rook with Pod Security Policies - -## Cluster Role - -> **NOTE**: Cluster role configuration is only needed when you are not already `cluster-admin` in your Kubernetes cluster! - -Creating the Rook operator requires privileges for setting up RBAC. To launch the operator you need to have created your user certificate that is bound to ClusterRole `cluster-admin`. - -One simple way to achieve it is to assign your certificate with the `system:masters` group: - -```console --subj "/CN=admin/O=system:masters" -``` - -`system:masters` is a special group that is bound to `cluster-admin` ClusterRole, but it can't be easily revoked so be careful with taking that route in a production setting. -Binding individual certificate to ClusterRole `cluster-admin` is revocable by deleting the ClusterRoleBinding. - -## RBAC for PodSecurityPolicies - -If you have activated the [PodSecurityPolicy Admission Controller](https://kubernetes.io/docs/admin/admission-controllers/#podsecuritypolicy) and thus are -using [PodSecurityPolicies](https://kubernetes.io/docs/concepts/policy/pod-security-policy/), you will require additional `(Cluster)RoleBindings` -for the different `ServiceAccounts` Rook uses to start the Rook Storage Pods. - -Security policies will differ for different backends. See Ceph's Pod Security Policies set up in its -[common.yaml](https://github.com/rook/rook/blob/{{ branchName }}/cluster/examples/kubernetes/ceph/common.yaml) -for an example of how this is done in practice. - -> **NOTE**: You do not have to perform these steps if you do not have the `PodSecurityPolicy` Admission Controller activated! - -### PodSecurityPolicy - -You need at least one `PodSecurityPolicy` that allows privileged `Pod` execution. Here is an example -which should be more permissive than is needed for any backend: - -```yaml -apiVersion: policy/v1beta1 -kind: PodSecurityPolicy -metadata: - name: privileged -spec: - fsGroup: - rule: RunAsAny - privileged: true - runAsUser: - rule: RunAsAny - seLinux: - rule: RunAsAny - supplementalGroups: - rule: RunAsAny - volumes: - - '*' - allowedCapabilities: - - '*' - hostPID: true - # hostNetwork is required for using host networking - hostNetwork: false -``` - -**Hint**: Allowing `hostNetwork` usage is required when using `hostNetwork: true` in a Cluster `CustomResourceDefinition`! -You are then also required to allow the usage of `hostPorts` in the `PodSecurityPolicy`. The given -port range will allow all ports: - -```yaml - hostPorts: - # Ceph msgr2 port - - min: 1 - max: 65535 -``` diff --git a/Documentation/tectonic.md b/Documentation/tectonic.md index 09a337d58..23ec6440d 100644 --- a/Documentation/tectonic.md +++ b/Documentation/tectonic.md @@ -1,10 +1,10 @@ --- -title: Tectonic Bare Metal -weight: 1600 +title: Tectonic Configuration +weight: 11800 indent: true --- -# Tectonic Bare Metal +# Tectonic Configuration Here is a running guide on how to implement Rook on Tectonic. A complete guide on how to install Tectonic is out of the scope of the Rook project. More info can be found on the [Tectonic website](https://coreos.com/tectonic/docs/latest/) From cd4c77fa7d366f1cde535786c9a7de22b30f3c26 Mon Sep 17 00:00:00 2001 From: Travis Nielsen Date: Mon, 9 Mar 2020 18:28:20 -0600 Subject: [PATCH 6/6] docs: update ceph toolbox to match example yaml The ceph toolbox no longer needs mounts to /dev and /udev. The docs had not previously been updated with this change. Signed-off-by: Travis Nielsen --- Documentation/ceph-toolbox.md | 30 ++++++++++-------------------- 1 file changed, 10 insertions(+), 20 deletions(-) diff --git a/Documentation/ceph-toolbox.md b/Documentation/ceph-toolbox.md index ab5db2745..8330f10a2 100644 --- a/Documentation/ceph-toolbox.md +++ b/Documentation/ceph-toolbox.md @@ -47,35 +47,25 @@ spec: secretKeyRef: name: rook-ceph-mon key: admin-secret - securityContext: - privileged: true volumeMounts: - - mountPath: /dev - name: dev - - mountPath: /sys/bus - name: sysbus - - mountPath: /lib/modules - name: libmodules + - mountPath: /etc/ceph + name: ceph-config - name: mon-endpoint-volume mountPath: /etc/rook - # if hostNetwork: false, the "rbd map" command hangs, see https://github.com/rook/rook/issues/2021 - hostNetwork: true volumes: - - name: dev - hostPath: - path: /dev - - name: sysbus - hostPath: - path: /sys/bus - - name: libmodules - hostPath: - path: /lib/modules - name: mon-endpoint-volume configMap: name: rook-ceph-mon-endpoints items: - key: data path: mon-endpoints + - name: ceph-config + emptyDir: {} + tolerations: + - key: "node.kubernetes.io/unreachable" + operator: "Exists" + effect: "NoExecute" + tolerationSeconds: 5 ``` Launch the rook-ceph-tools pod: @@ -96,7 +86,7 @@ Once the rook-ceph-tools pod is running, you can connect to it with: kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') bash ``` -All available tools in the toolbox are ready for your troubleshooting needs. +All available tools in the toolbox are ready for your troubleshooting needs. **Example**: