docs: update the table of contents

This commit is contained in:
Travis Nielsen committed 2018-01-12 16:55:34 -08:00
1 parent ec6ddcf048
commit 873751349a
27 files changed
+495 -401

No files matched your search

+40 -5
View File
@@ -1,8 +1,43 @@
# Rook
## Table of Contents
Rook is an open source orchestrator for distributed storage systems running in Kubernetes.
- [Rook on Kubernetes](kubernetes.md)
- [Rook Client](client.md)
- [Contributing](development-flow.md)
- [Toolbox](toolbox.md)
Rook turns distributed storage software into a self-managing, self-scaling, and self-healing storage services. It does this by automating deployment, bootstrapping, configuration, provisioning, scaling, upgrading, migration, disaster recovery, monitoring, and resource management. Rook uses the facilities provided by the underlying cloud-native container management, scheduling and orchestration platform to perform its duties.
Rook integrates deeply into cloud native environments leveraging extension points and providing a seamless experience for scheduling, lifecycle management, resource management, security, monitoring, and user experience.
Rook is currently in alpha state and has focused initially on orchestrating Ceph on top of Kubernetes. Ceph is a distributed storage system that provides file, block and object storage and is deployed in large scale production clusters. Rook plans to add support for other storage systems beyond Ceph in future releases.
## Design
Rook enables storage software systems to run on Kubernetes using Kubernetes primitives. Although Rook's reference storage system is Ceph, support for other storage systems can be added. The following image illustrates how Rook integrates with Kubernetes:
![Rook Architecture on Kubernetes](media/rook-architecture.png)
With Rook running in the Kubernetes cluster, Kubernetes applications can
mount block devices and filesystems managed by Rook, or can use the S3/Swift API for object storage. The Rook operator
automates configuration of storage components and monitors the cluster to ensure the storage remains available
and healthy.
The Rook operator is a simple container that has all that is needed to bootstrap
and monitor the storage cluster. The operator will start and monitor [ceph monitor pods](https://github.com/rook/rook/blob/master/design/mon-health.md) and a daemonset for the OSDs, which provides basic
RADOS storage. The operator manages CRDs for pools, object stores (S3/Swift), and file systems by initializing the pods and other artifacts necessary to
run the services.
The operator will monitor the storage daemons to ensure the cluster is healthy. Ceph mons will be started or failed over when necessary, and
other adjustments are made as the cluster grows or shrinks. The operator will also watch for desired state changes
requested by the api service and apply the changes.
The Rook operator also creates the Rook agents. These agents are pods deployed on every Kubernetes node. Each agent configures a Flexvolume plugin that integrates with Kubernetes' volume controller framework. All storage operations required on the node are handled such as attaching network storage devices, mounting volumes, and formating the filesystem.
![Rook Components on Kubernetes](media/kubernetes.png)
The Rook daemons (Mons, OSDs, MGR, RGW, and MDS) are compiled to a single binary `rook`, and included in a minimal container.
The `rook` container includes Ceph daemons and tools to manage and store all data -- there are no changes to the data path.
Rook does not attempt to maintain full fidelity with Ceph. Many of the Ceph concepts like placement groups and crush maps
are hidden so you don't have to worry about them. Instead Rook creates a much simplified UX for admins that is in terms
of physical resources, pools, volumes, filesystems, and buckets.
Rook is implemented in golang. Ceph is implemented in C++ where the data path is highly optimized. We believe
this combination offers the best of both worlds.
For more detailed design documentation, see the [design docs](https://github.com/rook/rook/tree/master/design).
+1 -1
View File
@@ -1,6 +1,6 @@
---
title: Advanced Configuration
weight: 61
weight: 76
indent: true
---
@@ -1,16 +1,16 @@
---
title: Block Storage
weight: 12
weight: 22
indent: true
---
# Block Storage Quickstart
# Block Storage
Block storage allows you to mount storage to a single pod.
## Prerequisites
This guide assumes you have created a Rook cluster as explained in the main [Kubernetes guide](kubernetes.md)
This guide assumes you have created a Rook cluster as explained in the main [Kubernetes guide](quickstart.md)
## Provision Storage
+5 -2
View File
@@ -1,9 +1,12 @@
---
title: Rook Client
weight: 40
weight: 74
indent: true
---
# Using Rook
# Rook Client
** **rookctl has been deprecated. [CRDs](crds.md) are the recommended way to configure the cluster.** **
The `rookctl` client tool can be used to manage your Rook cluster once it is running as well as manage block, file and object storage. See the sections below for details on how to configure each type of storage.
+3 -3
View File
@@ -1,10 +1,10 @@
---
title: Cluster CRD
weight: 17
title: Cluster
weight: 32
indent: true
---
# Creating Rook Clusters
# Cluster CRD
Rook allows creation and customization of storage clusters through the custom resource definitions (CRDs). The following settings are
available for a cluster.
+5 -4
View File
@@ -1,6 +1,7 @@
---
title: Common Problems
weight: 70
title: Common Issues
weight: 78
indent: true
---
# Common Problems
@@ -45,7 +46,7 @@ There is literally a ton of Ceph sub-commands to look at and manipulate Ceph obj
* Persistent Volume and/or Claim are failing to be created and bound: [Volume Creation](#volume-creation)
* `rook-agent` pod is failing to mount and format the volume: [Rook Agent Mounting](#volume-mounting)
* You are using Kubernetes 1.7.x or earlier and the Kubelet has not been restarted after `rook-agent` is in the `Running` status: [Restart Kubelet](#kubelet-restart)
* You are using Kubernetes 1.6.x and the attach-detach controller has not been disabled: [Disable attach-detach controller](./kubernetes.md#disable-attacher-detacher-controller)
* You are using Kubernetes 1.6.x and the attach-detach controller has not been disabled: [Disable attach-detach controller](./quickstart.md#disable-attacher-detacher-controller)
## Investigation Details
If you see some of the symptoms above, it's because the requested Rook storage for your pod is not being created and mounted successfully.
@@ -106,7 +107,7 @@ A symptom of this can be found in the Kubelet's log/journal, with the following
```console
Oct 30 22:23:03 core-02 kubelet-wrapper[31926]: E1030 22:23:03.524159 31926 desired_state_of_world_populator.go:285] Failed to add volume "mysql-persistent-storage" (specName: "pvc-9f273fbc") for pod "9f2ff89a-bdbf" to desiredStateOfWorld. err=failed to get Plugin from volumeSpec for volume "pvc-9f273fbc" err=no volume plugin matched
```
If you encounter this, just **restart the Kubelet process**, as described in the **[Restart Kubelet](./kubernetes.md#restart-kubelet)** section of the Rook deployment guide.
If you encounter this, just **restart the Kubelet process**, as described in the **[Restart Kubelet](./quickstart.md#restart-kubelet)** section of the Rook deployment guide.
### Volume Creation
The volume must first be created in the Rook cluster and then bound to a volume claim before it can be mounted to a pod.
+13
View File
@@ -0,0 +1,13 @@
---
title: Custom Resources
weight: 30
---
# Custom Resources (CRDs)
Rook allows you to create and manage your storage cluster through custom resource definitions (CRDs). Each type of resource
has its own CRD defined.
- [Cluster](cluster-crd.md): A Rook cluster provides the basis of the storage platform to serve block, object stores, and shared file systems.
- [Pool](pool-crd.md): A pool manages the backing store for a block store. Pools are also used internally by object and file stores.
- [Object Store](object-store-crd.md): An object store exposes storage with an S3-compatible interface.
- [File System](filesystem-crd.md): A file system provides shared storage for multiple Kubernetes pods.
+4 -1
View File
@@ -1,10 +1,13 @@
---
title: Contributing
weight: 50
weight: 90
---
# Contributing
Thank you for your time and effort to help us improve Rook! Here are a few steps to get started. If you have any questions,
don't hesitate to reach out to us on our [Slack](https://Rook-io.slack.com) dev channel.
## Prerequisites
1. [GO 1.8](https://golang.org/dl/) or greater installed
+3 -3
View File
@@ -1,10 +1,10 @@
---
title: File System CRD
weight: 20
title: Shared File System
weight: 38
indent: true
---
# Creating Shared File Systems
# Shared File System CRD
Rook allows creation and customization of shared file systems through the custom resource definitions (CRDs). The following settings are available
for file systems.
@@ -1,10 +1,10 @@
---
title: Shared File System
weight: 14
weight: 26
indent: true
---
# Shared File System Quickstart
# Shared File System
A shared file system can be mounted read-write from multiple pods. This may be useful for applications which can be clustered using a shared filesystem.
@@ -12,7 +12,7 @@ This example runs a shared file system for the [kube-registry](https://github.co
## Prerequisites
This guide assumes you have created a Rook cluster as explained in the main [Kubernetes guide](kubernetes.md)
This guide assumes you have created a Rook cluster as explained in the main [Kubernetes guide](quickstart.md)
## Create the File System
+40
View File
@@ -0,0 +1,40 @@
---
title: Flex Volume Configuration
weight: 12
indent: true
---
# Flex Volume Configuration
Rook uses [FlexVolume](https://github.com/kubernetes/community/blob/master/contributors/devel/flexvolume.md) to integrate with Kubernetes for performing storage operations. In some operating systems where Kubernetes is deployed, the [default Flexvolume plugin directory](https://github.com/kubernetes/community/blob/master/contributors/devel/flexvolume.md#prerequisites) (the directory where flexvolume drivers are installed) is **read-only**.
This is the case for Kubernetes deployments on CoreOS and Rancher.
In these environments, the Kubelet needs to be told to use a different flexvolume plugin directory that is accessible and writeable.
To do this, you will need to first add the `--volume-plugin-dir` flag to the Kubelet and then restart the Kubelet process.
These steps need to be carried out on **all nodes** in your cluster.
## CoreOS Container Linux
In CoreOS, our recommendation is to specify the flag as shown below:
```bash
--volume-plugin-dir=/var/lib/kubelet/volumeplugins
```
Restart Kubelet in order for this change to take effect.
## Rancher
Rancher provides an easy way to configure Kubelet. This flag can be provided to the Kubelet configuration template at deployment time or by using the `up to date` feature if Kubernetes is already deployed.
To configure Flexvolume in Rancher, specify this Kubelet flag as shown below:
```bash
--volume-plugin-dir=/var/lib/kubelet/volumeplugins
```
Restart Kubelet in order for this change to take effect.
## Tectonic
Follow [these instructions](tectonic.md) to configure Rook on Tectonic.
+5 -5
View File
@@ -1,12 +1,12 @@
---
title: Helm Chart
weight: 23
title: Operator
weight: 51
indent: true
---
# Helm Chart
# Operator Helm Chart
Installs [rook](https://github.com/rook/rook) to create,configure and manage Rook clusters atop Kubernetes.
Installs [rook](https://github.com/rook/rook) to create, configure, and manage Rook clusters on Kubernetes.
## Introduction
@@ -14,7 +14,7 @@ This chart bootstraps a [rook-operator](https://github.com/rook/rook) deployment
## Prerequisites
- Kubernetes 1.6+ with Beta APIs & ThirdPartyResources enabled
- Kubernetes 1.6+
### RBAC
+9
View File
@@ -0,0 +1,9 @@
---
title: Helm Charts
weight: 50
---
# Helm Charts
Rook has published a Helm chart for the [operator](helm-operator.md). Other Helm charts will also be developed for each of the
Pool, Object Store, and Filesystem CRDs.
+12 -43
View File
@@ -1,7 +1,6 @@
---
title: Prerequisites
weight: 11
indent: true
weight: 10
---
# Prerequisites
@@ -15,50 +14,20 @@ Kubernetes v1.6 or higher is targeted by Rook (while Rook is in alpha it will tr
Support is available for Kubernetes v1.5.2, although your mileage may vary.
You will need to use the yaml files from the [1.5 folder](/cluster/examples/kubernetes/1.5).
## Privileges
## Privileges and RBAC
Creating the Rook operator requires privileges for setting up RBAC. To launch the operator you need to have created your user certificate that is bound to ClusterRole `cluster-admin`.
One simple way to achieve it is to assign your certificate with the `system:masters` group:
```
-subj "/CN=admin/O=system:masters"
```
`system:masters` is a special group that is bound to `cluster-admin` ClusterRole, but it can't be easily revoked so be careful with taking that route in a production setting.
Binding individual certificate to ClusterRole `cluster-admin` is revocable by deleting the ClusterRoleBinding.
Rook requires privileges to manage the storage in your cluster. See the details [here](rbac.md) for setting up RBAC.
## Flexvolume Configuration
Rook uses [Flexvolume](https://github.com/kubernetes/community/blob/master/contributors/devel/flexvolume.md) to integrate with Kubernetes for performing storage operations. In some operating systems where Kubernetes is deployed, the [default Flexvolume plugin directory](https://github.com/kubernetes/community/blob/master/contributors/devel/flexvolume.md#prerequisites) (the directory where flexvolume drivers are installed) is **read-only**.
The Rook agent requires setup as a Flex volume plugin to manage the storage attachments in your cluster.
See the [Flex Volume Configuration](flexvolume.md) topic to configure your Kubernetes deployment to load the Rook volume plugin.
This is the case for Kubernetes deployments on CoreOS and Rancher.
In these environments, the Kubelet needs to be told to use a different flexvolume plugin directory that is accessible and writeable.
To do this, you will need to first add the `--volume-plugin-dir` flag to the Kubelet and then restart the Kubelet process.
These steps need to be carried out on **all nodes** in your cluster.
## Bootstrapping Kubernetes
### CoreOS Container Linux
Rook will run wherever Kubernetes is running. Here are some simple enviroments will help you get started with Rook.
In CoreOS, our recomendation is to specify the flag as shown below:
```bash
--volume-plugin-dir=/var/lib/kubelet/volumeplugins
```
Restart Kubelet in order for this change to take effect.
### Rancher
Rancher provides an easy way to configure Kubelet. This flag can be provided to the Kubelet configuration template at deployment time or by using the `up to date` feature if Kubernetes is already deployed.
To configure Flexvolume in Rancher, specify this Kubelet flag as shown below:
```bash
--volume-plugin-dir=/var/lib/kubelet/volumeplugins
```
Restart Kubelet in order for this change to take effect.
## Minikube
### Minikube
To install `minikube`, refer to this [page](https://github.com/kubernetes/minikube/releases). Once you have `minikube` installed, start a cluster by doing the following:
@@ -76,14 +45,14 @@ Kubectl is now configured to use the cluster.
After these steps, your minikube cluster is ready to install Rook on.
## Kubeadm
### Kubeadm
You can easily spin up Rook on top of a `kubeadm` cluster.
You can find the instructions on how to install kubeadm in the [Install `kubeadm`] (https://kubernetes.io/docs/setup/independent/install-kubeadm/) page.
By using `kubeadm`, you can use Rook in just a few minutes!
## New local Kubernetes cluster with Vagrant
### New local Kubernetes cluster with Vagrant
For a quick start with a new local cluster, use the Rook fork of [coreos-kubernetes](https://github.com/rook/coreos-kubernetes). This will bring up a multi-node Kubernetes cluster with `vagrant` and CoreOS virtual machines ready to use Rook immediately.
@@ -101,12 +70,12 @@ Then wait for the cluster to come up and verify that kubernetes is done initiali
kubectl cluster-info
```
Once you see a url response, your cluster is [ready for use by Rook](kubernetes.md#deploy-rook).
Once you see a url response, your cluster is [ready for use by Rook](quickstart.md#deploy-rook).
## Using Rook in Kubernetes
Now that you have a Kubernetes cluster running, you can start using Rook with [these steps](kubernetes.md#deploy-rook).
Now that you have a Kubernetes cluster running, you can start using Rook with [these steps](quickstart.md#deploy-rook).
## Using Rook on Tectonic Bare Metal
-304
View File
@@ -1,304 +0,0 @@
---
title: Kubernetes
weight: 10
---
# Rook on Kubernetes
- [Quickstart](#quickstart)
- [Design](#design)
## Quickstart
This example shows how to build a simple, multi-tier web application on Kubernetes using persistent volumes enabled by Rook.
### Minimum Version
Kubernetes **v1.6** or higher is targeted by Rook (while Rook is in alpha it will track the latest release to use the latest features).
Support is available for Kubernetes **v1.5.2**, although your mileage may vary.
You will need to use the yaml files from the [1.5 folder](/cluster/examples/kubernetes/1.5).
### Prerequisites
To make sure you have a Kubernetes cluster that is ready for `Rook`, you can [follow these quick instructions](k8s-pre-reqs.md).
If you are using `dataDirHostPath` to persist rook data on kubernetes hosts, make sure your host has at least 5GB of space available on the specified path.
### Deploy Rook
With your Kubernetes cluster running, Rook can be setup and deployed by simply creating the rook-operator deployment and creating a rook cluster. To customize the operator settings, see the [Rook Helm Chart](helm-operator.md).
```bash
cd cluster/examples/kubernetes
kubectl create -f rook-operator.yaml
# verify the rook-operator and rook-agents pods are in the `Running` state before proceeding
kubectl -n rook-system get pod
```
## RBAC for PodSecurityPolicies
If you have activated the [PodSecurityPolicy Admission Controller](https://kubernetes.io/docs/admin/admission-controllers/#podsecuritypolicy) and thus are
using [PodSecurityPolicies](https://kubernetes.io/docs/concepts/policy/pod-security-policy/), you will require additional `(Cluster)RoleBindings`
for the different `ServiceAccounts` Rook uses to start the Rook Storage Pods.
**Note**: You do not have to perform these steps if you do not have the `PodSecurityPolicy` Admission Controller activated!
##### PodSecurityPolicy
You need one `PodSecurityPolicy` that allows privileged `Pod` execution. Here is an example:
```yaml
apiVersion: extensions/v1beta1
kind: PodSecurityPolicy
metadata:
name: privileged
spec:
fsGroup:
rule: RunAsAny
privileged: true
runAsUser:
rule: RunAsAny
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
volumes:
- '*'
allowedCapabilities:
- '*'
hostPID: true
hostIPC: true
hostNetwork: false
```
**Hint**: Allowing `hostNetwork` usage is required when using `hostNetwork: true` in the Cluster `CustomResourceDefinition`!
You are then also required to allow the usage of `hostPorts` in the `PodSecurityPolicy`. The given port range is a minimal
working recommendation for rook:
```yaml
hostPorts:
# CEPH ports
- min: 6789
max: 7300
# rook-api port
- min: 8124
max: 8124
```
##### ClusterRole and ClusterRoleBinding
Next up you require a `ClusterRole` and a corresponding `ClusterRoleBinding`, which enables the Rook Agent `ServiceAccount` to run the rook-agent `Pods` on all nodes
with privileged rights. Here are the definitions:
```yaml
# privilegedPSP grants access to use the privileged PSP.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: privileged-psp-user
rules:
- apiGroups:
- extensions
resources:
- podsecuritypolicies
resourceNames:
- privileged
verbs:
- use
```
and
```yaml
# Allow the rook-agent serviceAccount to use the privileged PSP
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: rook-agent-psp
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: privileged-psp-user
subjects:
- kind: ServiceAccount
name: rook-agent
namespace: rook-system
```
Save these definitions to one or multiple yaml files and create them by executing `kubectl apply -f <nameOfYourFile>.yaml`
You will also require two more `RoleBindings` for each Rook Cluster you deploy:
Create these two `RoleBindings` in the Namespace you plan to deploy your Rook Cluster into (default is "rook" namespace):
```yaml
# Allow the default serviceAccount to use the priviliged PSP
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: rook-default-psp
namespace: rook
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: privileged-psp-user
subjects:
- kind: ServiceAccount
name: default
namespace: rook
---
# Allow the rook-ceph-osd serviceAccount to use the priviliged PSP
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: rook-ceph-osd-psp
namespace: rook
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: privileged-psp-user
subjects:
- kind: ServiceAccount
name: rook-ceph-osd
namespace: rook
```
---
#### **Restart Kubelet**
**(K8S 1.7.x and older only)**
For versions of Kubernetes prior to 1.8, the Kubelet process on all nodes will require a restart after the Rook operator and Rook agents have been deployed. As part of their initial setup, the Rook agents deploy and configure a Flexvolume plugin in order to integrate with Kubernetes' volume controller framework. In Kubernetes v1.8+, the [dynamic Flexvolume plugin discovery](https://github.com/kubernetes/community/blob/master/contributors/devel/flexvolume.md#dynamic-plugin-discovery) will find and initialize our plugin, but in older versions of Kubernetes a manual restart of the Kubelet will be required.
#### **Disable Attacher-detacher controller**
**(K8S 1.6.x only)**
For Kubernetes 1.6, it is also necessary to pass the `--enable-controller-attach-detach=false` flag to Kubelet when you restart it. This is a workaround for a [Kubernetes issue](https://github.com/kubernetes/kubernetes/issues/47109) that only affects 1.6.
---
Now that the rook-operator pod is running, we can create the Rook cluster. For the cluster to survive reboots,
make sure you set the `dataDirHostPath` property. For more settings, see the documentation on [configuring the cluster](cluster-crd.md).
Save the cluster spec as `rook-cluster.yaml`:
```yaml
apiVersion: v1
kind: Namespace
metadata:
name: rook
---
apiVersion: rook.io/v1alpha1
kind: Cluster
metadata:
name: rook
namespace: rook
spec:
dataDirHostPath: /var/lib/rook
storage:
useAllNodes: true
useAllDevices: false
storeConfig:
storeType: bluestore
databaseSizeMB: 1024
journalSizeMB: 1024
```
Create the cluster:
```bash
kubectl create -f rook-cluster.yaml
```
Use `kubectl` to list pods in the `rook` namespace. You should be able to see the following pods once they are all running:
```bash
$ kubectl -n rook get pod
NAME READY STATUS RESTARTS AGE
rook-api-1511082791-7qs0m 1/1 Running 0 5m
rook-ceph-mgr0-1279756402-wc4vt 1/1 Running 0 5m
rook-ceph-mon0-jflt5 1/1 Running 0 6m
rook-ceph-mon1-wkc8p 1/1 Running 0 6m
rook-ceph-mon2-p31dj 1/1 Running 0 6m
rook-ceph-osd-0h6nb 1/1 Running 0 5m
```
## Storage
For a walkthrough of the three types of storage exposed by Rook, see the guides for:
- **[Block](k8s-block.md)**: Create block storage to be consumed by a pod
- **[Shared File System](k8s-filesystem.md)**: Create a file system to be shared across multiple pods
- **[Object](k8s-object.md)**: Create an object store that is accessible inside or outside the Kubernetes cluster
## Tools
We have created a toolbox container that contains the full suite of Ceph clients for debugging and troubleshooting your Rook cluster. Please see the [toolbox readme](toolbox.md) for setup and usage information. Also see our [advanced configuration](advanced-configuration.md) document for helpful maintenance and tuning examples.
The toolbox also contains the `rookctl` tool as required in the [File System](k8s-filesystem.md) and [Object](k8s-object.md) walkthroughs, or a [simplified walkthrough of block, file and object storage](client.md). In the near future, `rookctl` will not be required for kubernetes scenarios.
### Monitoring
Each Rook cluster has some built in metrics collectors/exporters for monitoring with [Prometheus](https://prometheus.io/).
To learn how to set up monitoring for your Rook cluster, you can follow the steps in the [monitoring guide](./k8s-monitoring.md).
## Teardown
To clean up all the artifacts created by the demo, first cleanup the resources from the [block](k8s-block.md#teardown) and [file](k8s-filesystem.md#teardown) walkthroughs (unmount volumes, delete volume claims, etc).
Those steps have been copied below for convenience, but note that some of these may not exist if you did not complete those parts of the demo:
```console
kubectl delete -f wordpress.yaml
kubectl delete -f mysql.yaml
kubectl delete -n rook pool replicapool
kubectl delete storageclass rook-block
kubectl -n kube-system delete secret rook-admin
kubectl delete -f kube-registry.yaml
```
After those resources have been cleaned up, you can then delete your Rook cluster:
```console
kubectl delete -n rook cluster rook
```
This will begin the process of all cluster resources being cleaned up, after which you can delete the rest of the deployment with the following:
```console
kubectl delete thirdpartyresources cluster.rook.io pool.rook.io objectstore.rook.io filesystem.rook.io volumeattachment.rook.io # ignore errors if on K8s 1.7+
kubectl delete crd clusters.rook.io pools.rook.io objectstores.rook.io filesystems.rook.io volumeattachments.rook.io # ignore errors if on K8s 1.5 and 1.6
kubectl delete -n rook-system daemonset rook-agent
kubectl delete -f rook-operator.yaml
kubectl delete clusterroles rook-agent
kubectl delete clusterrolebindings rook-agent
kubectl delete namespace rook
```
If you modified the demo settings, additional cleanup is up to you for devices, host paths, etc.
## Design
Rook enables storage software systems to run on Kubernetes using Kubernetes primitives. Although Rook's reference storage system is Ceph, support for other storage systems can be added. The following image illustrates how Rook integrates with Kubernetes:
![Rook Architecture on Kubernetes](media/rook-architecture.png)
With Rook running in the Kubernetes cluster, Kubernetes applications can
mount block devices and filesystems managed by Rook, or can use the S3/Swift API for object storage. The Rook operator
automates configuration of storage components and monitors the cluster to ensure the storage remains available
and healthy. There is also a REST API service for configuring the Rook storage and a command line tool called `rookctl`.
The Rook operator is a simple container that has all that is needed to bootstrap
and monitor the storage cluster. The operator will start and monitor [ceph monitor pods](https://github.com/rook/rook/blob/master/design/mon-health.md) and a daemonset for the OSDs, which provides basic
RADOS storage as well as a deployment for a RESTful API service. When requested through the api service,
object storage (S3/Swift) is enabled by starting a deployment for RGW, while a shared file system is enabled with a deployment for MDS.
The operator will monitor the storage daemons to ensure the cluster is healthy. Ceph mons will be started or failed over when necessary, and
other adjustments are made as the cluster grows or shrinks. The operator will also watch for desired state changes
requested by the api service and apply the changes.
Rook operator also creates the Rook agents. These agents are pods deployed on every Kubernetes node. Each agent configures a Flexvolume plugin that integrates with Kubernetes' volume controller framework and also handle all storage operations required on the node, such as attaching network storage devices, mounting volumes and formating filesystem.
![Rook Components on Kubernetes](media/kubernetes.png)
The Rook daemons (Mons, OSDs, MGR, RGW, and MDS) are compiled to a single binary `rook`, and included in a minimal container.
The `rook` container includes Ceph daemons and tools to manage and store all data -- there are no changes to the data path.
Rook does not attempt to maintain full fidelity with Ceph. Many of the Ceph concepts like placement groups and crush maps
are hidden so you don't have to worry about them. Instead Rook creates a much simplified UX for admins that is in terms
of physical resources, pools, volumes, filesystems, and buckets.
Rook is implemented in golang. Ceph is implemented in C++ where the data path is highly optimized. We believe
this combination offers the best of both worlds.
For more detailed design documentation, see the [design docs](https://github.com/rook/rook/tree/master/design).
@@ -1,14 +1,13 @@
---
title: Monitoring
weight: 15
indent: true
weight: 40
---
# Monitoring
Each Rook cluster has some built in metrics collectors/exporters for monitoring with [Prometheus](https://prometheus.io/). To enable monitoring of Rook in your Kubernetes cluster, you can follow the steps below.
Note that these steps work only for Kubernetes 1.7.x or higher. For Kubernetes 1.6.x or older, refer to the Rook 0.5.0 [documentation](https://github.com/rook/rook/blob/release-0.5/Documentation/k8s-monitoring.md) and use these [manifest files](https://github.com/rook/rook/tree/release-0.5/cluster/examples/kubernetes/monitoring).
Note that these steps work only for Kubernetes 1.7.x or higher. For Kubernetes 1.6.x or older, refer to the Rook 0.5.0 [documentation](https://github.com/rook/rook/blob/release-0.5/Documentation/monitoring.md) and use these [manifest files](https://github.com/rook/rook/tree/release-0.5/cluster/examples/kubernetes/monitoring).
## Prometheus Operator
+3 -3
View File
@@ -1,10 +1,10 @@
---
title: Object Store CRD
weight: 19
title: Object Store
weight: 36
indent: true
---
# Creating Rook Object Stores
# Object Store CRD
Rook allows creation and customization of object stores through the custom resource definitions (CRDs). The following settings are available
for object stores.
@@ -1,16 +1,16 @@
---
title: Object Storage
weight: 13
weight: 24
indent: true
---
# Object Storage Quickstart
# Object Storage
Object storage exposes an S3 API to the storage cluster for applications to put and get data.
## Prerequisites
This guide assumes you have created a Rook cluster as explained in the main [Kubernetes guide](kubernetes.md)
This guide assumes you have created a Rook cluster as explained in the main [Kubernetes guide](quickstart.md)
## Object Store
@@ -60,7 +60,7 @@ kubectl -n rook get pod -l app=rook-ceph-rgw
## Create a User
Creating an object storage user requires running `rookctl` commands with the [Rook toolbox](kubernetes.md#tools) pod. This will be simplified in the future with a CRD for the object store users.
Creating an object storage user requires running `rookctl` commands with the [Rook toolbox](quickstart.md#tools) pod. This will be simplified in the future with a CRD for the object store users.
```bash
rookctl object user create my-store rook-user "A rook rgw User"
+3 -3
View File
@@ -1,10 +1,10 @@
---
title: Pool CRD
weight: 16
title: Pool
weight: 34
indent: true
---
# Creating Rook Storage Pools
# Pool CRD
Rook allows creation and customization of storage pools through the custom resource definitions (CRDs). The following settings are available
for pools.
+144
View File
@@ -0,0 +1,144 @@
---
title: Quickstart
weight: 2
---
# Quickstart Guide
This example shows how to build a simple, multi-tier web application on Kubernetes using persistent volumes enabled by Rook.
## Minimum Version
Kubernetes **v1.6** or higher is targeted by Rook (while Rook is in alpha it will track the latest release to use the latest features).
Support is available for Kubernetes **v1.5.2**, although your mileage may vary.
You will need to use the yaml files from the [1.5 folder](/cluster/examples/kubernetes/1.5).
## Prerequisites
To make sure you have a Kubernetes cluster that is ready for `Rook`, you can [follow these instructions](k8s-pre-reqs.md).
If you are using `dataDirHostPath` to persist rook data on kubernetes hosts, make sure your host has at least 5GB of space available on the specified path.
## Deploy Rook
With your Kubernetes cluster running, Rook can be setup and deployed by simply creating the rook-operator deployment and creating a rook cluster. To customize the operator settings, see the [Rook Helm Chart](helm-operator.md).
```bash
cd cluster/examples/kubernetes
kubectl create -f rook-operator.yaml
# verify the rook-operator and rook-agents pods are in the `Running` state before proceeding
kubectl -n rook-system get pod
```
---
### **Restart Kubelet**
**(K8S 1.7.x and older only)**
For versions of Kubernetes prior to 1.8, the Kubelet process on all nodes will require a restart after the Rook operator and Rook agents have been deployed. As part of their initial setup, the Rook agents deploy and configure a Flexvolume plugin in order to integrate with Kubernetes' volume controller framework. In Kubernetes v1.8+, the [dynamic Flexvolume plugin discovery](https://github.com/kubernetes/community/blob/master/contributors/devel/flexvolume.md#dynamic-plugin-discovery) will find and initialize our plugin, but in older versions of Kubernetes a manual restart of the Kubelet will be required.
### **Disable Attacher-detacher controller**
**(K8S 1.6.x only)**
For Kubernetes 1.6, it is also necessary to pass the `--enable-controller-attach-detach=false` flag to Kubelet when you restart it. This is a workaround for a [Kubernetes issue](https://github.com/kubernetes/kubernetes/issues/47109) that only affects 1.6.
---
Now that the rook-operator pod is running, we can create the Rook cluster. For the cluster to survive reboots,
make sure you set the `dataDirHostPath` property. For more settings, see the documentation on [configuring the cluster](cluster-crd.md).
Save the cluster spec as `rook-cluster.yaml`:
```yaml
apiVersion: v1
kind: Namespace
metadata:
name: rook
---
apiVersion: rook.io/v1alpha1
kind: Cluster
metadata:
name: rook
namespace: rook
spec:
dataDirHostPath: /var/lib/rook
storage:
useAllNodes: true
useAllDevices: false
storeConfig:
storeType: bluestore
databaseSizeMB: 1024
journalSizeMB: 1024
```
Create the cluster:
```bash
kubectl create -f rook-cluster.yaml
```
Use `kubectl` to list pods in the `rook` namespace. You should be able to see the following pods once they are all running:
```bash
$ kubectl -n rook get pod
NAME READY STATUS RESTARTS AGE
rook-api-1511082791-7qs0m 1/1 Running 0 5m
rook-ceph-mgr0-1279756402-wc4vt 1/1 Running 0 5m
rook-ceph-mon0-jflt5 1/1 Running 0 6m
rook-ceph-mon1-wkc8p 1/1 Running 0 6m
rook-ceph-mon2-p31dj 1/1 Running 0 6m
rook-ceph-osd-0h6nb 1/1 Running 0 5m
```
# Storage
For a walkthrough of the three types of storage exposed by Rook, see the guides for:
- **[Block](block.md)**: Create block storage to be consumed by a pod
- **[Shared File System](filesystem.md)**: Create a file system to be shared across multiple pods
- **[Object](object.md)**: Create an object store that is accessible inside or outside the Kubernetes cluster
# Tools
We have created a toolbox container that contains the full suite of Ceph clients for debugging and troubleshooting your Rook cluster. Please see the [toolbox readme](toolbox.md) for setup and usage information. Also see our [advanced configuration](advanced-configuration.md) document for helpful maintenance and tuning examples.
The toolbox also contains the `rookctl` tool as required in the [File System](filesystem.md) and [Object](object.md) walkthroughs, or a [simplified walkthrough of block, file and object storage](client.md). In the near future, `rookctl` will not be required for kubernetes scenarios.
# Monitoring
Each Rook cluster has some built in metrics collectors/exporters for monitoring with [Prometheus](https://prometheus.io/).
To learn how to set up monitoring for your Rook cluster, you can follow the steps in the [monitoring guide](./monitoring.md).
# Teardown
To clean up all the artifacts created by the demo, first cleanup the resources from the [block](block.md#teardown) and [file](filesystem.md#teardown) walkthroughs (unmount volumes, delete volume claims, etc).
Those steps have been copied below for convenience, but note that some of these may not exist if you did not complete those parts of the demo:
```console
kubectl delete -f wordpress.yaml
kubectl delete -f mysql.yaml
kubectl delete -n rook pool replicapool
kubectl delete storageclass rook-block
kubectl -n kube-system delete secret rook-admin
kubectl delete -f kube-registry.yaml
```
After those resources have been cleaned up, you can then delete your Rook cluster:
```console
kubectl delete -n rook cluster rook
```
This will begin the process of all cluster resources being cleaned up, after which you can delete the rest of the deployment with the following:
```console
kubectl delete thirdpartyresources cluster.rook.io pool.rook.io objectstore.rook.io filesystem.rook.io volumeattachment.rook.io # ignore errors if on K8s 1.7+
kubectl delete crd clusters.rook.io pools.rook.io objectstores.rook.io filesystems.rook.io volumeattachments.rook.io # ignore errors if on K8s 1.5 and 1.6
kubectl delete -n rook-system daemonset rook-agent
kubectl delete -f rook-operator.yaml
kubectl delete clusterroles rook-agent
kubectl delete clusterrolebindings rook-agent
kubectl delete namespace rook
```
IMPORTANT: The final cleanup step requires deleting files on each host in the cluster. All files under the `dataDirHostPath` property specified in the cluster CRD will need to be deleted. Otherwise, inconsistent state will remain when a new cluster is started.
If you modified the demo settings, additional cleanup is up to you for devices, host paths, etc.
+153
View File
@@ -0,0 +1,153 @@
---
title: RBAC Security
weight: 14
indent: true
---
# RBAC Security
## Cluster Role
Creating the Rook operator requires privileges for setting up RBAC. To launch the operator you need to have created your user certificate that is bound to ClusterRole `cluster-admin`.
One simple way to achieve it is to assign your certificate with the `system:masters` group:
```
-subj "/CN=admin/O=system:masters"
```
`system:masters` is a special group that is bound to `cluster-admin` ClusterRole, but it can't be easily revoked so be careful with taking that route in a production setting.
Binding individual certificate to ClusterRole `cluster-admin` is revocable by deleting the ClusterRoleBinding.
## RBAC for PodSecurityPolicies
If you have activated the [PodSecurityPolicy Admission Controller](https://kubernetes.io/docs/admin/admission-controllers/#podsecuritypolicy) and thus are
using [PodSecurityPolicies](https://kubernetes.io/docs/concepts/policy/pod-security-policy/), you will require additional `(Cluster)RoleBindings`
for the different `ServiceAccounts` Rook uses to start the Rook Storage Pods.
**Note**: You do not have to perform these steps if you do not have the `PodSecurityPolicy` Admission Controller activated!
##### PodSecurityPolicy
You need one `PodSecurityPolicy` that allows privileged `Pod` execution. Here is an example:
```yaml
apiVersion: extensions/v1beta1
kind: PodSecurityPolicy
metadata:
name: privileged
spec:
fsGroup:
rule: RunAsAny
privileged: true
runAsUser:
rule: RunAsAny
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
volumes:
- '*'
allowedCapabilities:
- '*'
hostPID: true
hostIPC: true
hostNetwork: false
```
**Hint**: Allowing `hostNetwork` usage is required when using `hostNetwork: true` in the Cluster `CustomResourceDefinition`!
You are then also required to allow the usage of `hostPorts` in the `PodSecurityPolicy`. The given port range is a minimal
working recommendation for rook:
```yaml
hostPorts:
# CEPH ports
- min: 6789
max: 7300
# rook-api port
- min: 8124
max: 8124
```
##### ClusterRole and ClusterRoleBinding
Next up you require a `ClusterRole` and a corresponding `ClusterRoleBinding`, which enables the Rook Agent `ServiceAccount` to run the rook-agent `Pods` on all nodes
with privileged rights. Here are the definitions:
```yaml
# privilegedPSP grants access to use the privileged PSP.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: privileged-psp-user
rules:
- apiGroups:
- extensions
resources:
- podsecuritypolicies
resourceNames:
- privileged
verbs:
- use
```
and
```yaml
apiVersion: v1
kind: Namespace
metadata:
name: rook-system
---
# Allow the rook-agent serviceAccount to use the privileged PSP
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: rook-agent-psp
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: privileged-psp-user
subjects:
- kind: ServiceAccount
name: rook-agent
namespace: rook-system
```
Save these definitions to one or multiple yaml files and create them by executing `kubectl apply -f <nameOfYourFile>.yaml`
You will also require two more `RoleBindings` for each Rook Cluster you deploy:
Create these two `RoleBindings` in the Namespace you plan to deploy your Rook Cluster into (default is "rook" namespace):
```yaml
apiVersion: v1
kind: Namespace
metadata:
name: rook
---
# Allow the default serviceAccount to use the priviliged PSP
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: rook-default-psp
namespace: rook
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: privileged-psp-user
subjects:
- kind: ServiceAccount
name: default
namespace: rook
---
# Allow the rook-ceph-osd serviceAccount to use the privileged PSP
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: rook-ceph-osd-psp
namespace: rook
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: privileged-psp-user
subjects:
- kind: ServiceAccount
name: rook-ceph-osd
namespace: rook
```
+11
View File
@@ -0,0 +1,11 @@
---
title: Storage
weight: 20
---
# Storage
Rook provides three types of storage to the Kubernetes cluster.
- [Block Storage](block.md): Mount storage to a single pod
- [Object Storage](object.md): Expose an S3 API to the storage cluster for applications to put and get data
- [Shared File System](filesystem.md): Mount storage to multiple pods
+9 -3
View File
@@ -1,5 +1,11 @@
# Using Rook on Tectonic Bare Metal
Here is a running guide on how to implement Rook on Tectonic. A complete guide on how to install Tectonic is out of the scope of the Rook project. But many info can be found on the [Tectonic website](https://coreos.com/tectonic/docs/latest/)
---
title: Tectonic Bare Metal
weight: 16
indent: true
---
# Tectonic Bare Metal
Here is a running guide on how to implement Rook on Tectonic. A complete guide on how to install Tectonic is out of the scope of the Rook project. More info can be found on the [Tectonic website](https://coreos.com/tectonic/docs/latest/)
## Prequisites
- An installed tectonic-installer. These steps are described on [the Tectonic website](https://coreos.com/tectonic/docs/latest/install/bare-metal/#4-tectonic-installer)
@@ -38,5 +44,5 @@ Visit the official [Tectonic manual boot](https://coreos.com/tectonic/docs/lates
**Remark:** The Tectonic installer contains the correct terraform binary out of the box. This terraform binary can be found in following directory `~/tectonic/tectonic-installer/linux`.
## Start Rook
After the Tectonic Installer ran and the Kubernetes cluster is started and ready, you can follow the [Rook installation guide](kubernetes.md).
After the Tectonic Installer ran and the Kubernetes cluster is started and ready, you can follow the [Rook installation guide](quickstart.md).
If you want to specify which disks Rook uses, follow the instructions in [creating Rook clusters](cluster-crd.md)
+3 -2
View File
@@ -1,6 +1,7 @@
---
title: Toolbox
weight: 60
weight: 72
indent: true
---
# Rook Toolbox
@@ -9,7 +10,7 @@ The toolbox is based on Ubuntu, so more tools of your choosing can be easily ins
## Running the Toolbox in Kubernetes
The rook toolbox can run as a pod in a Kubernetes cluster. After you ensure you have a running Kubernetes cluster with rook deployed (see the [Kubernetes](kubernetes.md) instructions),
The rook toolbox can run as a pod in a Kubernetes cluster. After you ensure you have a running Kubernetes cluster with rook deployed (see the [Kubernetes](quickstart.md) instructions),
launch the rook-tools pod.
Save the tools spec as `rook-tools.yaml`:
+12
View File
@@ -0,0 +1,12 @@
---
title: Tools
weight: 70
---
# Tools
Rook provides a number of tools to help you manage your cluster.
- [Toolbox](toolbox.md): A pod from which you can run all of the tools to troubleshoot the storage cluster
- [Rook Client](client.md): A command-line tool (`rookctl`) found in the toolbox
- [Advanced Configuration](advanced-configuration.md): Tips and tricks for configuring for cluster
- [Common Issues](common-problems.md): Common issues and their potential solutions
+4 -5
View File
@@ -1,10 +1,9 @@
---
title: Upgrading Rook
weight: 25
indent: true
title: Upgrades
weight: 60
---
# Upgrading the Rook Software
# Upgrades
This guide will walk you through the manual steps to upgrade the software in a Rook cluster from one version to the next.
Rook is a distributed software system and therefore there are multiple components to individually upgrade in the sequence defined in this guide.
After each component is upgraded, it is important to verify that the cluster returns to a healthy and fully functional state.
@@ -276,7 +275,7 @@ kubectl -n rook get pod -l app=rook-ceph-mgr -o jsonpath='{range .items[*]}{.met
```
### Optional Components
If you have optionally installed either [object storage](./k8s-object.md) or a [shared file system](./k8s-filesystem.md) in your Rook cluster, the sections below will provide guidance on how to update them as well.
If you have optionally installed either [object storage](./object.md) or a [shared file system](./filesystem.md) in your Rook cluster, the sections below will provide guidance on how to update them as well.
They are both managed by deployments, which we have already covered in this guide, so the instructions will be brief.
#### Object Storage (RGW)
+1 -1
View File
@@ -1 +1 @@
For documentation on running Rook in your Kubernetes cluster see the [Kubernetes Quickstart Guide](/Documentation/kubernetes.md)
For documentation on running Rook in your Kubernetes cluster see the [Kubernetes Quickstart Guide](/Documentation/quickstart.md)