Files
my-rook-config/Documentation/ceph-quickstart.md
T

180 lines
9.8 KiB
Markdown
Raw Normal View History

2018-01-12 16:55:23 -08:00
---
2018-07-13 14:29:57 -06:00
title: Ceph Storage
2019-02-04 09:46:00 -08:00
weight: 300
2018-07-13 14:29:57 -06:00
indent: true
2018-01-12 16:55:23 -08:00
---
{% include_relative branch.liquid %}
2018-01-12 16:55:23 -08:00
2018-07-13 14:29:57 -06:00
# Ceph Storage Quickstart
2018-01-12 16:55:23 -08:00
2018-07-13 14:29:57 -06:00
This guide will walk you through the basic setup of a Ceph cluster and enable you to consume block, object, and file storage
2018-04-01 18:31:47 +02:00
from other pods running in your cluster.
2018-01-12 16:55:23 -08:00
## Minimum Version
2020-04-01 15:21:03 -06:00
Kubernetes **v1.11** or higher is supported by Rook.
2018-01-12 16:55:23 -08:00
**Important** If you are using K8s 1.15 or older, you will need to create a different version of the Rook CRDs. Create the `crds.yaml` found in the [pre-k8s-1.16](https://github.com/rook/rook/blob/{{ branchName }}/cluster/examples/kubernetes/ceph/pre-k8s-1.16) subfolder of the example manifests.
2018-01-12 16:55:23 -08:00
## Prerequisites
To make sure you have a Kubernetes cluster that is ready for `Rook`, you can [follow these instructions](k8s-pre-reqs.md).
2020-04-01 15:21:03 -06:00
In order to configure the Ceph storage cluster, at least one of these local storage options are required:
- Raw devices (no partitions or formatted filesystems)
2020-04-01 15:21:03 -06:00
- Raw partitions (no formatted filesystem)
- PVs available from a storage class in `block` mode
2018-01-12 16:55:23 -08:00
You can confirm whether your partitions or devices are formatted filesystems with the following command.
```console
lsblk -f
NAME FSTYPE LABEL UUID MOUNTPOINT
vda
└─vda1 LVM2_member eSO50t-GkUV-YKTH-WsGq-hNJY-eKNf-3i07IB
├─ubuntu--vg-root ext4 c2366f76-6e21-4f10-a8f3-6776212e2fe4 /
└─ubuntu--vg-swap_1 swap 9492a3dc-ad75-47cd-9596-678e8cf17ff9 [SWAP]
vdb
```
If the `FSTYPE` field is not empty, there is a filesystem on top of the corresponding device. In this case, you can use vdb for Ceph and can't use vda and its partitions.
## TL;DR
2018-01-12 16:55:23 -08:00
If you're feeling lucky, a simple Rook cluster can be created with the following kubectl commands and [example yaml files](https://github.com/rook/rook/blob/{{ branchName }}/cluster/examples/kubernetes/ceph). For the more detailed install, skip to the next section to [deploy the Rook operator](#deploy-the-rook-operator).
2019-10-24 10:22:08 +02:00
```console
git clone --single-branch --branch {{ branchName }} https://github.com/rook/rook.git
cd rook/cluster/examples/kubernetes/ceph
kubectl create -f crds.yaml -f common.yaml -f operator.yaml
2020-01-13 16:58:51 +01:00
kubectl create -f cluster.yaml
```
After the cluster is running, you can create [block, object, or file](#storage) storage to be consumed by other applications in your cluster.
2020-04-21 15:14:08 -06:00
### Cluster Environments
2020-04-21 15:14:08 -06:00
The Rook documentation is focused around starting Rook in a production environment. Examples are also
provided to relax some settings for test environments. When creating the cluster later in this guide, consider these example cluster manifests:
2020-11-06 12:22:33 +01:00
- [cluster.yaml](https://github.com/rook/rook/blob/{{ branchName }}/cluster/examples/kubernetes/ceph/cluster.yaml): Cluster settings for a production cluster running on bare metal. Requires at least three worker nodes.
- [cluster-on-pvc.yaml](https://github.com/rook/rook/blob/{{ branchName }}/cluster/examples/kubernetes/ceph/cluster-on-pvc.yaml): Cluster settings for a production cluster running in a dynamic cloud environment.
2020-04-21 15:14:08 -06:00
- [cluster-test.yaml](https://github.com/rook/rook/blob/{{ branchName }}/cluster/examples/kubernetes/ceph/cluster-test.yaml): Cluster settings for a test environment such as minikube.
See the [Ceph examples](ceph-examples.md) for more details.
## Deploy the Rook Operator
2019-08-01 16:31:50 -06:00
The first step is to deploy the Rook operator. Check that you are using the [example yaml files](https://github.com/rook/rook/blob/{{ branchName }}/cluster/examples/kubernetes/ceph) that correspond to your release of Rook. For more options, see the [examples documentation](ceph-examples.md).
2018-01-12 16:55:23 -08:00
2019-10-24 10:22:08 +02:00
```console
cd cluster/examples/kubernetes/ceph
kubectl create -f crds.yaml -f common.yaml -f operator.yaml
2018-01-12 16:55:23 -08:00
2019-10-24 10:22:08 +02:00
## verify the rook-ceph-operator is in the `Running` state before proceeding
kubectl -n rook-ceph get pod
2018-01-12 16:55:23 -08:00
```
You can also deploy the operator with the [Rook Helm Chart](helm-operator.md).
Before you start the operator in production, there are some settings that you may want to consider:
1. If you are using kubernetes v1.15 or older you need to create CRDs found here `/cluster/examples/kubernetes/ceph/pre-k8s-1.16/crd.yaml`.
The apiextension v1beta1 version of CustomResourceDefinition was deprecated in Kubernetes v1.16.
2. Consider if you want to enable certain Rook features that are disabled by default. See the [operator.yaml](https://github.com/rook/rook/blob/{{ branchName }}/cluster/examples/kubernetes/ceph/operator.yaml) for these and other advanced settings.
1. Device discovery: Rook will watch for new devices to configure if the `ROOK_ENABLE_DISCOVERY_DAEMON` setting is enabled, commonly used in bare metal clusters.
2. Flex driver: The flex driver is deprecated in favor of the CSI driver, but can still be enabled with the `ROOK_ENABLE_FLEX_DRIVER` setting.
3. Node affinity and tolerations: The CSI driver by default will run on any node in the cluster. To configure the CSI driver affinity, several settings are available.
If you wish to deploy into a namespace other than the default `rook-ceph`, see the
[Ceph advanced configuration section](ceph-advanced-configuration.md#using-alternate-namespaces) on the topic.
## Create a Rook Ceph Cluster
2019-08-01 16:31:50 -06:00
Now that the Rook operator is running we can create the Ceph cluster. For the cluster to survive reboots,
2018-12-03 23:48:40 -07:00
make sure you set the `dataDirHostPath` property that is valid for your hosts. For more settings, see the documentation on [configuring the cluster](ceph-cluster-crd.md).
2018-01-12 16:55:23 -08:00
Create the cluster:
2019-10-24 10:22:08 +02:00
```console
2020-01-13 16:58:51 +01:00
kubectl create -f cluster.yaml
2018-01-12 16:55:23 -08:00
```
Use `kubectl` to list pods in the `rook-ceph` namespace. You should be able to see the following pods once they are all running.
2020-01-13 16:58:51 +01:00
The number of osd pods will depend on the number of nodes in the cluster and the number of devices configured.
If you did not modify the `cluster.yaml` above, it is expected that one OSD will be created per node.
The CSI, `rook-ceph-agent` (flex driver), and `rook-discover` pods are also optional depending on your settings.
> If the `rook-ceph-mon`, `rook-ceph-mgr`, or `rook-ceph-osd` pods are not created, please refer to the
> [Ceph common issues](ceph-common-issues.md) for more details and potential solutions.
2018-01-12 16:55:23 -08:00
2019-10-24 10:22:08 +02:00
```console
$ kubectl -n rook-ceph get pod
NAME READY STATUS RESTARTS AGE
csi-cephfsplugin-provisioner-d77bb49c6-n5tgs 5/5 Running 0 140s
csi-cephfsplugin-provisioner-d77bb49c6-v9rvn 5/5 Running 0 140s
csi-cephfsplugin-rthrp 3/3 Running 0 140s
csi-rbdplugin-hbsm7 3/3 Running 0 140s
csi-rbdplugin-provisioner-5b5cd64fd-nvk6c 6/6 Running 0 140s
csi-rbdplugin-provisioner-5b5cd64fd-q7bxl 6/6 Running 0 140s
rook-ceph-crashcollector-minikube-5b57b7c5d4-hfldl 1/1 Running 0 105s
rook-ceph-mgr-a-64cd7cdf54-j8b5p 1/1 Running 0 77s
rook-ceph-mon-a-694bb7987d-fp9w7 1/1 Running 0 105s
rook-ceph-mon-b-856fdd5cb9-5h2qk 1/1 Running 0 94s
rook-ceph-mon-c-57545897fc-j576h 1/1 Running 0 85s
rook-ceph-operator-85f5b946bd-s8grz 1/1 Running 0 92m
rook-ceph-osd-0-6bb747b6c5-lnvb6 1/1 Running 0 23s
rook-ceph-osd-1-7f67f9646d-44p7v 1/1 Running 0 24s
rook-ceph-osd-2-6cd4b776ff-v4d68 1/1 Running 0 25s
rook-ceph-osd-prepare-node1-vx2rz 0/2 Completed 0 60s
rook-ceph-osd-prepare-node2-ab3fd 0/2 Completed 0 60s
rook-ceph-osd-prepare-node3-w4xyz 0/2 Completed 0 60s
2018-01-12 16:55:23 -08:00
```
To verify that the cluster is in a healthy state, connect to the [Rook toolbox](ceph-toolbox.md) and run the
`ceph status` command.
2019-10-24 10:22:08 +02:00
* All mons should be in quorum
* A mgr should be active
* At least one OSD should be active
* If the health is not `HEALTH_OK`, the warnings or errors should be investigated
```console
$ ceph status
cluster:
id: a0452c76-30d9-4c1a-a948-5d8405f19a7c
health: HEALTH_OK
services:
mon: 3 daemons, quorum a,b,c (age 3m)
mgr: a(active, since 2m)
osd: 3 osds: 3 up (since 1m), 3 in (since 1m)
...
```
If the cluster is not healthy, please refer to the [Ceph common issues](ceph-common-issues.md) for more details and potential solutions.
2019-10-24 10:22:08 +02:00
## Storage
2018-01-12 16:55:23 -08:00
For a walkthrough of the three types of storage exposed by Rook, see the guides for:
2019-10-24 10:22:08 +02:00
* **[Block](ceph-block.md)**: Create block storage to be consumed by a pod
* **[Object](ceph-object.md)**: Create an object store that is accessible inside or outside the Kubernetes cluster
* **[Shared Filesystem](ceph-filesystem.md)**: Create a filesystem to be shared across multiple pods
## Ceph Dashboard
2018-06-07 15:08:58 -07:00
Ceph has a dashboard in which you can view the status of your cluster. Please see the [dashboard guide](ceph-dashboard.md) for more details.
2019-10-24 10:22:08 +02:00
## Tools
2018-01-12 16:55:23 -08:00
2020-07-06 00:34:23 +00:00
We have created a toolbox container that contains the full suite of Ceph clients for debugging and troubleshooting your Rook cluster. Please see the [toolbox readme](ceph-toolbox.md) for setup and usage information. Also see our [advanced configuration](ceph-advanced-configuration.md) document for helpful maintenance and tuning examples.
2018-01-12 16:55:23 -08:00
2019-10-24 10:22:08 +02:00
## Monitoring
2018-01-12 16:55:23 -08:00
Each Rook cluster has some built in metrics collectors/exporters for monitoring with [Prometheus](https://prometheus.io/).
2018-12-07 12:42:04 +01:00
To learn how to set up monitoring for your Rook cluster, you can follow the steps in the [monitoring guide](./ceph-monitoring.md).
2018-01-12 16:55:23 -08:00
2019-10-24 10:22:08 +02:00
## Teardown
2018-01-12 16:55:23 -08:00
2018-07-13 14:29:57 -06:00
When you are done with the test cluster, see [these instructions](ceph-teardown.md) to clean up the cluster.