diff --git a/Documentation/README.md b/Documentation/README.md index 545853fdb..94e0cb292 100644 --- a/Documentation/README.md +++ b/Documentation/README.md @@ -11,7 +11,7 @@ We plan to continue adding support for other storage systems and environments ba ## Getting Started -Starting Rook in your cluster is as simple as two `kubectl` commands. See our [Quickstart](quickstart.md) guide for the details on what you need to get going. +Starting Rook in your cluster is as simple as two `kubectl` commands. See our [Quickstart](ceph-quickstart.md) guide for the details on what you need to get going. Once you have a Rook cluster running, walk through the guides for block, object, and file to start consuming the storage in your cluster: - **[Block](block.md)**: Create block storage to be consumed by a pod diff --git a/Documentation/advanced-configuration.md b/Documentation/advanced-configuration.md index a357bff81..66bb5bed3 100644 --- a/Documentation/advanced-configuration.md +++ b/Documentation/advanced-configuration.md @@ -78,7 +78,7 @@ do done ``` -The output should look something like this: +The output should look something like this. Note that OSDs on the same node will show duplicate information. ```bash Pod: osd-m2fz2 @@ -300,7 +300,7 @@ Of course you will only have 1/3 the capacity as a tradeoff. ### Setting PG Count Be sure to read the [placement group sizing](#placement-group-sizing) section -before changing the numner of PGs. +before changing the number of PGs. ```bash # Set the number of PGs in the rbd pool to 512 diff --git a/Documentation/block.md b/Documentation/block.md index 25c5dfcad..fe61785d2 100644 --- a/Documentation/block.md +++ b/Documentation/block.md @@ -10,7 +10,7 @@ Block storage allows you to mount storage to a single pod. This example shows ho ## Prerequisites -This guide assumes you have created a Rook cluster as explained in the main [Quickstart](quickstart.md) guide. +This guide assumes you have created a Rook cluster as explained in the main [Quickstart](ceph-quickstart.md) guide. ## Provision Storage diff --git a/Documentation/ceph-cluster-crd.md b/Documentation/ceph-cluster-crd.md index 88c430eb9..7611c2866 100644 --- a/Documentation/ceph-cluster-crd.md +++ b/Documentation/ceph-cluster-crd.md @@ -18,7 +18,6 @@ Settings can be specified at the global level to apply to the cluster as a whole ### Cluster Settings - `dataDirHostPath`: The path on the host ([hostPath](https://kubernetes.io/docs/concepts/storage/volumes/#hostpath)) where config and data should be stored for each of the services. If the directory does not exist, it will be created. Because this directory persists on the host, it will remain after pods are deleted. - On **Minikube** environments, use `/data/rook`. Minikube boots into a tmpfs but it provides some [directories](https://github.com/kubernetes/minikube/blob/master/docs/persistent_volumes.md) where files can be persisted across reboots. Using one of these directories will ensure that Rook's data and configuration files are persisted and that enough storage space is available. - - If a path is not specified, an [empty dir](https://kubernetes.io/docs/concepts/storage/volumes/#emptydir) will be used and the config will be lost when the pod or host is restarted. This option is **not recommended**. - **WARNING**: For test scenarios, if you delete a cluster and start a new cluster on the same hosts, the path used by `dataDirHostPath` must be deleted. Otherwise, stale keys and other config will remain from the previous cluster and the new mons will fail to start. If this value is empty, each pod will get an ephemeral directory to store their config files that is tied to the lifetime of the pod running on that node. More details can be found in the Kubernetes [empty dir docs](https://kubernetes.io/docs/concepts/storage/volumes/#emptydir). - `dashboard`: Settings for the Ceph dashboard. To view the dashboard in your browser see the [dashboard guide](ceph-dashboard.md). @@ -44,7 +43,7 @@ This will bring up your default text editor and allow you to add and remove stor This feature is only available when `useAllNodes` has been set to `false`. ### Mon Settings -- `count`: set the number of mons to be started. The number should be odd and between `1` and `9`. If not specified will be set to `3` and `allowMultiplePerNode` is also set to `true`. +- `count`: set the number of mons to be started. The number should be odd and between `1` and `9`. If not specified the default is set to `3` and `allowMultiplePerNode` is also set to `true`. - `allowMultiplePerNode`: enable (`true`) or disable (`false`) the placement of multiple mons on one node. Default is `false`. ### Node Settings @@ -113,13 +112,10 @@ For more information on resource requests/limits see the official Kubernetes doc - `memory`: Limit for Memory (example: one gigabyte of memory `1Gi`, half a gigabyte of memory `512Mi`). ## Samples +Here are several samples for configuring Ceph clusters. Each of the samples must also include the namespace and corresponding access granted for management by the Ceph operator. See the [common cluster resources](#common-cluster-resources) below. + ### Storage configuration: All devices ```yaml -apiVersion: v1 -kind: Namespace -metadata: - name: rook-ceph ---- apiVersion: ceph.rook.io/v1beta1 kind: Cluster metadata: @@ -147,11 +143,6 @@ Individual nodes and their config can be specified so that only the named nodes Each node's 'name' field should match their 'kubernetes.io/hostname' label. ```yaml -apiVersion: v1 -kind: Namespace -metadata: - name: rook-ceph ---- apiVersion: ceph.rook.io/v1beta1 kind: Cluster metadata: @@ -189,11 +180,6 @@ This example is based up on the [Storage Configuration: Specific devices](#stora Individual nodes can override the cluster wide specified directories list. ```yaml -apiVersion: v1 -kind: Namespace -metadata: - name: rook-ceph ---- apiVersion: ceph.rook.io/v1beta1 kind: Cluster metadata: @@ -225,11 +211,6 @@ The example under 'all' would have all services scheduled on kubernetes nodes la tolerate taints with a key of 'storage-node'. ```yaml -apiVersion: v1 -kind: Namespace -metadata: - name: rook-ceph ---- apiVersion: ceph.rook.io/v1beta1 kind: Cluster metadata: @@ -269,11 +250,6 @@ You can override these requests/limits for OSDs per node when using `useAllNodes **WARNING** Before setting resource requests/limits, please take a look at the Ceph documentation for recommendations for each component: [Ceph - Hardware Recommendations](http://docs.ceph.com/docs/master/start/hardware-recommendations/). ```yaml -apiVersion: v1 -kind: Namespace -metadata: - name: rook-ceph ---- apiVersion: ceph.rook.io/v1beta1 kind: Cluster metadata: @@ -296,3 +272,59 @@ spec: cpu: "2" memory: "4096Mi" ``` + +## Common Cluster Resources +Each Ceph cluster must be created in a namespace and also give access to the Rook operator to manage the cluster in the namespace. Creating the namespace and these controls must be added to each of the examples previously shown. + +```yaml +apiVersion: v1 +kind: Namespace +metadata: + name: rook-ceph +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: rook-ceph-cluster + namespace: rook-ceph +--- +kind: Role +apiVersion: rbac.authorization.k8s.io/v1beta1 +metadata: + name: rook-ceph-cluster + namespace: rook-ceph +rules: +- apiGroups: [""] + resources: ["configmaps"] + verbs: [ "get", "list", "watch", "create", "update", "delete" ] +--- +# Allow the operator to create resources in this cluster's namespace +kind: RoleBinding +apiVersion: rbac.authorization.k8s.io/v1beta1 +metadata: + name: rook-ceph-cluster-mgmt + namespace: rook-ceph +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: rook-ceph-cluster-mgmt +subjects: +- kind: ServiceAccount + name: rook-ceph-system + namespace: rook-ceph-system +--- +# Allow the pods in this namespace to work with configmaps +kind: RoleBinding +apiVersion: rbac.authorization.k8s.io/v1beta1 +metadata: + name: rook-ceph-cluster + namespace: rook-ceph +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: rook-ceph-cluster +subjects: +- kind: ServiceAccount + name: rook-ceph-cluster + namespace: rook-ceph +``` \ No newline at end of file diff --git a/Documentation/quickstart.md b/Documentation/ceph-quickstart.md similarity index 73% rename from Documentation/quickstart.md rename to Documentation/ceph-quickstart.md index f60244458..b0d27f717 100644 --- a/Documentation/quickstart.md +++ b/Documentation/ceph-quickstart.md @@ -1,14 +1,12 @@ --- -title: Quickstart -weight: 2 +title: Ceph Storage +weight: 3 +indent: true --- -# Quickstart Guide +# Ceph Storage Quickstart -Welcome to Rook! We hope you have a great experience installing the Rook storage platform to enable highly available, durable storage -in your cluster. If you have any questions along the way, please don't hesitate to ask us in our [Slack channel](https://rook-io.slack.com). Sign up for our Slack [here](https://rook-slackin.herokuapp.com/). - -This guide will walk you through the basic setup of a Rook cluster. This will enable you to consume block, object, and file storage +This guide will walk you through the basic setup of a Ceph cluster and enable you to consume block, object, and file storage from other pods running in your cluster. ## Minimum Version @@ -40,7 +38,7 @@ The first step is to deploy the Rook system components, which include the Rook a cd cluster/examples/kubernetes/ceph kubectl create -f operator.yaml -# verify the rook-ceph-operator and rook-ceph-agent pods are in the `Running` state before proceeding +# verify the rook-ceph-operator, rook-ceph-agent, and rook-discover pods are in the `Running` state before proceeding kubectl -n rook-ceph-system get pod ``` @@ -56,7 +54,7 @@ For versions of Kubernetes prior to 1.8, the Kubelet process on all nodes will r ## Create a Rook Cluster -Now that the Rook operator and agent pods are running, we can create the Rook cluster. For the cluster to survive reboots, +Now that the Rook operator, agent, and discover pods are running, we can create the Rook cluster. For the cluster to survive reboots, make sure you set the `dataDirHostPath` property. For more settings, see the documentation on [configuring the cluster](ceph-cluster-crd.md). @@ -91,16 +89,20 @@ Create the cluster: kubectl create -f cluster.yaml ``` -Use `kubectl` to list pods in the `rook` namespace. You should be able to see the following pods once they are all running: +Use `kubectl` to list pods in the `rook` namespace. You should be able to see the following pods once they are all running. +The number of osd pods will depend on the number of nodes in the cluster and the number of devices and directories configured. ```bash $ kubectl -n rook-ceph get pod -NAME READY STATUS RESTARTS AGE -rook-ceph-mgr0-1279756402-wc4vt 1/1 Running 0 5m -rook-ceph-mon0-jflt5 1/1 Running 0 6m -rook-ceph-mon1-wkc8p 1/1 Running 0 6m -rook-ceph-mon2-p31dj 1/1 Running 0 6m -rook-ceph-osd-0h6nb 1/1 Running 0 5m +NAME READY STATUS RESTARTS AGE +rook-ceph-mgr-a-75cc4ccbf4-t8qtx 1/1 Running 0 24m +rook-ceph-mon0-72vx7 1/1 Running 0 25m +rook-ceph-mon1-rrpm6 1/1 Running 0 24m +rook-ceph-mon2-zff9r 1/1 Running 0 24m +rook-ceph-osd-id-0-5fd8cb9747-dvlsb 1/1 Running 0 23m +rook-ceph-osd-id-1-84dc695b48-r5mhf 1/1 Running 0 23m +rook-ceph-osd-id-2-558878cd84-cnp67 1/1 Running 0 23m +rook-ceph-osd-prepare-minikube-wq4f5 0/1 Completed 0 24m ``` # Storage @@ -125,4 +127,4 @@ To learn how to set up monitoring for your Rook cluster, you can follow the step # Teardown -When you are done with the test cluster, see [these instructions](teardown.md) to clean up the cluster. +When you are done with the test cluster, see [these instructions](ceph-teardown.md) to clean up the cluster. diff --git a/Documentation/teardown.md b/Documentation/ceph-teardown.md similarity index 94% rename from Documentation/teardown.md rename to Documentation/ceph-teardown.md index a65e7d0a9..fab54efa3 100644 --- a/Documentation/teardown.md +++ b/Documentation/ceph-teardown.md @@ -1,6 +1,6 @@ --- -title: Cleanup -weight: 5 +title: Ceph Cleanup +weight: 4 indent: true --- @@ -21,8 +21,8 @@ First you will need to clean up the resources created on top of the Rook cluster These commands will clean up the resources from the [block](block.md#teardown) and [file](filesystem.md#teardown) walkthroughs (unmount volumes, delete volume claims, etc). If you did not complete those parts of the walkthrough, you can skip these instructions: ```console -kubectl delete -f wordpress.yaml -kubectl delete -f mysql.yaml +kubectl delete -f ../wordpress.yaml +kubectl delete -f ../mysql.yaml kubectl delete -n rook-ceph pool replicapool kubectl delete storageclass rook-ceph-block kubectl delete -f kube-registry.yaml @@ -39,10 +39,9 @@ Verify that the cluster CRD has been deleted before continuing to the next step. kubectl -n rook-ceph get cluster.ceph.rook.io ``` -## Delete the Operator and Agent -This will begin the process of all cluster resources being cleaned up, after which you can delete the rest of the deployment with the following: +## Delete the Operator +This will begin the process of all cluster resources being cleaned up, after which you can delete the operator and related resources such as the agent and discover daemonsets with the following: ```console -kubectl delete -n rook-ceph-system daemonset rook-ceph-agent kubectl delete -f operator.yaml ``` diff --git a/Documentation/cockroachdb.md b/Documentation/cockroachdb.md index eac24583a..f1756f8f9 100644 --- a/Documentation/cockroachdb.md +++ b/Documentation/cockroachdb.md @@ -1,10 +1,10 @@ --- title: CockroachDB -weight: 27 +weight: 5 indent: true --- -# CockroachDB +# CockroachDB Quickstart CockroachDB is a cloud-native SQL database for building global, scalable cloud services that survive disasters. Rook provides an operator to deploy and manage CockroachDB clusters. diff --git a/Documentation/common-issues.md b/Documentation/common-issues.md index a01473513..a2f5482d5 100644 --- a/Documentation/common-issues.md +++ b/Documentation/common-issues.md @@ -16,6 +16,7 @@ If after trying the suggestions found on this page and the problem is not resolv - [Cluster failing to service requests](#cluster-failing-to-service-requests) - [Only a single monitor pod starts](#only-a-single-monitor-pod-starts) - [OSD pods are failing to start](#osd-pods-are-failing-to-start) +- [OSDs are not created on my devices](#osd-pods-are-not-created-on-my-devices) - [Node hangs after reboot](#node-hangs-after-reboot) - [Rook Agent modprobe exec format error](#rook-agent-modprobe-exec-format-error) - [Using multiple shared filesystem (CephFS) is attempted on a kernel version older than 4.7](#using-multiple-shared-filesystem-cephfs-is-attempted-on-a-kernel-version-older-than-47) @@ -51,7 +52,7 @@ After you verify the basic health of the running pods, next you will want to run ``` ### Tools in other pods -The Ceph tools are found in all of the Rook pods where Ceph is running, such as the operator, monitors, and OSDs. Rather than starting the toolbox pod, you can connect to the existing pods to more quickly execute the Ceph tools. For example, to connect to the operator pod: +The Ceph tools are found in all of the Rook pods where Ceph is running, such as the operator and monitors. Rather than starting the toolbox pod, you can connect to the existing pods to more quickly execute the Ceph tools. For example, to connect to the operator pod: ```bash kubectl -n rook-ceph-system exec -it $(kubectl -n rook-ceph-system get pods -l app=rook-operator -o jsonpath='{.items[0].metadata.name}') -- bash @@ -59,7 +60,7 @@ kubectl -n rook-ceph-system exec -it $(kubectl -n rook-ceph-system get pods -l a Now from inside the operator pod you can execute the Ceph tools. -It is preferred to connect to the operator pod rather than monitors and OSDs since the operator always has the latest configuration files. If you have multiple Rook clusters, it is preferred to connect to the Rook toolbox for a specific cluster. Otherwise, your ceph commands may connect to the wrong cluster. +It is preferred to connect to the operator pod rather than monitors since the operator always has the latest configuration files. If you have multiple Rook clusters, it is preferred to connect to the Rook toolbox for a specific cluster. Otherwise, your ceph commands may connect to the wrong cluster. ### Ceph Commands Here are some common commands to troubleshoot the cluster. @@ -150,7 +151,7 @@ A symptom of this can be found in the Kubelet's log/journal, with the following ```console Oct 30 22:23:03 core-02 kubelet-wrapper[31926]: E1030 22:23:03.524159 31926 desired_state_of_world_populator.go:285] Failed to add volume "mysql-persistent-storage" (specName: "pvc-9f273fbc") for pod "9f2ff89a-bdbf" to desiredStateOfWorld. err=failed to get Plugin from volumeSpec for volume "pvc-9f273fbc" err=no volume plugin matched ``` -If you encounter this, just **restart the Kubelet process**, as described in the **[Restart Kubelet](./quickstart.md#restart-kubelet)** section of the Rook deployment guide. +If you encounter this, just **restart the Kubelet process**, as described in the **[Restart Kubelet](./ceph-quickstart.md#restart-kubelet)** section of the Rook deployment guide. ### Volume Creation The volume must first be created in the Rook cluster and then bound to a volume claim before it can be mounted to a pod. @@ -365,6 +366,56 @@ This directory is the `dataDirHostPath` setting in the cluster CRD and is typica To fix the issue you will need to delete all components of Rook and then delete the contents of `/var/lib/rook` (or the directory specified by `dataDirHostPath`) on each of the hosts in the cluster. Then when the cluster CRD is applied to start a new cluster, the rook-operator should start all the pods as expected. + +# OSD pods are not created on my devices + +## Symptoms +* No OSD pods are started in the cluster +* Devices are not configured with OSDs even though specified in the Cluster CRD +* One OSD pod is started on each node instead of multiple pods for each device + +## Investigation +First, ensure that you have specified the devices correctly in the CRD. +The [Cluster CRD](ceph-cluster-crd.md#storage-selection-settings) has several ways to specify the devices that are to be consumed by the Rook storage: +- `useAllDevices: true`: Rook will consume all devices it determines to be available +- `deviceFilter`: Consume all devices that match this regular expression +- `devices`: Explicit list of device names on each node to consume + +Second, if Rook determines that a device is not available (has existing partitions or a formatted file system), Rook will skip consuming the devices. +If Rook is not starting OSDs on the devices you expect, Rook may have skipped it for this reason. To see if a device was skipped, view the OSD preparation log +on the node where the device was skipped. + +``` +# get the prepare pods in the cluster +$ kubectl -n rook-ceph get pod -l app=rook-ceph-osd-prepare +NAME READY STATUS RESTARTS AGE +rook-ceph-osd-prepare-node1-fvmrp 0/1 Completed 0 18m +rook-ceph-osd-prepare-node2-w9xv9 0/1 Completed 0 22m +rook-ceph-osd-prepare-node3-7rgnv 0/1 Completed 0 22m + +# view the logs for the node of interest +$ kubectl -n rook-ceph logs rook-ceph-osd-prepare-node1-fvmrp +``` + +Towards the begining of the log you will see messages such as the following: +``` +# message that the device sda was skipped +cephosd: skipping device sda that is in use (not by rook) + +# message that the devices sdb and sdc are being configured +cephosd: configuring osd devices: {"Entries":{"sdb":{"Data":-1,"Metadata":null},"sdc":{"Data":-1,"Metadata":null}}} +``` + +## Solution +After you have either updated the CRD with the correct settings, or you have cleaned the partitions or file system from your devices, +you can trigger the operator to analyze the devices again by restarting the operator. Each time the operator starts, it +will ensure all the desired devices are configured. + +``` +# Restart the operator to ensure devices are configured. A new pod will automatically be started when the current operator pod is deleted. +$ kubectl -n rook-ceph-system delete pod -l app=rook-ceph-operator +``` + # Node hangs after reboot ## Symptoms diff --git a/Documentation/crds.md b/Documentation/crds.md index be3496156..7f21fe35b 100644 --- a/Documentation/crds.md +++ b/Documentation/crds.md @@ -15,4 +15,7 @@ has its own CRD defined. - [File System](ceph-filesystem-crd.md): A file system provides shared storage for multiple Kubernetes pods. ## CockroachDB -- [Cluster](cockroachdb-cluster-crd.md): CockroachDB is an open-source distributed SQL database that is highly scalable across multiple global regions and also highly durable. \ No newline at end of file +- [Cluster](cockroachdb-cluster-crd.md): CockroachDB is an open-source distributed SQL database that is highly scalable across multiple global regions and also highly durable. + +## Minio +- [Object Store](minio-object-store-crd.md): Minio is a high performance distributed object storage server, designed for large-scale private cloud infrastructure. \ No newline at end of file diff --git a/Documentation/development-flow.md b/Documentation/development-flow.md index 8aed89859..ef2b81c8e 100644 --- a/Documentation/development-flow.md +++ b/Documentation/development-flow.md @@ -10,7 +10,7 @@ don't hesitate to reach out to us on our [Slack](https://Rook-io.slack.com) dev ## Prerequisites -1. [GO 1.8](https://golang.org/dl/) or greater installed +1. [GO 1.10](https://golang.org/dl/) or greater installed 2. Git client installed 3. Github account @@ -63,9 +63,9 @@ For new features of significant scope and complexity, a design document is recom For smaller, straightforward features and bug fixes, there is no need for a design document. Authoring a design document for big features has many advantages: -* helps flesh out the approach by forcing the author to think critically about the feature and can identify potential issues early on -* gets agreement amongst the community before code is written that could be wasted effort in the wrong direction -* serves as an artifact of the architecture that is easier to read for visitors to the project than just the code by itself +* Helps flesh out the approach by forcing the author to think critically about the feature and can identify potential issues early on +* Gets agreement amongst the community before code is written that could be wasted effort in the wrong direction +* Serves as an artifact of the architecture that is easier to read for visitors to the project than just the code by itself Note that writing code to prototype the feature while working on the design may be very useful to help flesh out the approach. @@ -112,13 +112,13 @@ Rebasing is a very powerful feature of Git. You need to understand how it works ## Submitting a Pull Request -Once you have implemented the feature or bug fix in your branch, you will open a PR to the upstream rook repo. Before opening the PR ensure you have added unit tests, are passing the E2E tests, cleaned your commit history, and have rebased on the latest upstream. +Once you have implemented the feature or bug fix in your branch, you will open a PR to the upstream rook repo. Before opening the PR ensure you have added unit tests, are passing the integration tests, cleaned your commit history, and have rebased on the latest upstream. In order to open a pull request (PR) it is required to be up to date with the latest changes upstream. If other commits are pushed upstream before your PR is merged, you will also need to rebase again before it will be merged. ### Regression Testing -All pull requests must pass the unit and e2e smoke tests before they can be merged. These tests automatically +All pull requests must pass the unit and integration tests before they can be merged. These tests automatically run as a part of the build process. The results of these tests along with code reviews and other criterias determine whether your request will be accepted into the `rook/rook` repo. It is prudent to run all tests locally on your development box prior to submitting a pull request to the `rook/rook` repo. @@ -137,9 +137,9 @@ go test -coverprofile=coverage.out go tool cover -html=coverage.out -o coverage.html ``` -#### Running the E2E Tests +#### Running the Integration Tests For instructions on how to execute the end to end smoke test suite, -follow the [e2e instructions](../e2e/README.md). +follow the [test instructions](https://github.com/rook/rook/blob/master/tests/README.md). ### Commit History diff --git a/Documentation/direct-tools.md b/Documentation/direct-tools.md index c9247372e..c1419a511 100644 --- a/Documentation/direct-tools.md +++ b/Documentation/direct-tools.md @@ -45,7 +45,7 @@ mount /dev/rbd0 /tmp/rook-volume Write and read a file: ```bash -echo "Hello Rook!" > /tmp/rook-volume/hello +echo "Hello Rook" > /tmp/rook-volume/hello cat /tmp/rook-volume/hello ``` @@ -88,7 +88,7 @@ ls /tmp/registry Try writing and reading a file to the shared file system. ```bash -echo "Hello Rook!" > /tmp/registry/hello +echo "Hello Rook" > /tmp/registry/hello cat /tmp/registry/hello # delete the file when you're done diff --git a/Documentation/disaster-recovery.md b/Documentation/disaster-recovery.md index ed16d4f77..c9b286a58 100644 --- a/Documentation/disaster-recovery.md +++ b/Documentation/disaster-recovery.md @@ -19,7 +19,7 @@ that it is the only mon in quorum, and then restart the good mon. ### Stop the operator First, stop the operator so it will not try to failover the mons while we are modifying the monmap ```bash -kubectl -n rook-ceph-system delete deployment rook-operator +kubectl -n rook-ceph-system delete deployment rook-ceph-operator ``` ### Inject a new monmap diff --git a/Documentation/filesystem.md b/Documentation/filesystem.md index a31f2d81d..aa6167ed3 100644 --- a/Documentation/filesystem.md +++ b/Documentation/filesystem.md @@ -12,7 +12,7 @@ This example runs a shared file system for the [kube-registry](https://github.co ### Prerequisites -This guide assumes you have created a Rook cluster as explained in the main [Kubernetes guide](quickstart.md) +This guide assumes you have created a Rook cluster as explained in the main [Kubernetes guide](ceph-quickstart.md) ### Multiple File Systems Not Supported diff --git a/Documentation/helm-operator.md b/Documentation/helm-operator.md index a4c95c4d8..bdf14ecb3 100644 --- a/Documentation/helm-operator.md +++ b/Documentation/helm-operator.md @@ -34,26 +34,25 @@ kubectl --namespace kube-system patch deploy/tiller-deploy -p '{"spec": {"templa ## Installing The Ceph Operator helm chart will install the basic components necessary to create a storage platform for your Kubernetes cluster. -After the helm chart is installed, you will need to [create a Rook cluster](quickstart.md#create-a-rook-cluster). +After the helm chart is installed, you will need to [create a Rook cluster](ceph-quickstart.md#create-a-rook-cluster). The `helm install` command deploys rook on the Kubernetes cluster in the default configuration. The [configuration](#configuration) section lists the parameters that can be configured during installation. It is recommended that the rook operator be installed into the `rook-ceph-system` namespace (you will install your clusters into separate namespaces). -Rook currently publishes builds to the `alpha` and `master` channels. In the future, `beta` and `stable` will also be available. +Rook currently publishes builds of the Ceph operator to the `beta`, `alpha`, and `master` channels. In the future `stable` will also be available. + +### Beta +The beta channel is the most recent release of Rook that is considered nearly stable for the community, starting with the v0.8 release. + +```console +helm repo add rook-beta https://charts.rook.io/beta +helm install --namespace rook-ceph-system rook-beta/rook-ceph +``` ### Alpha The alpha channel is the most recent release of Rook that is considered ready for testing by the community. ```console helm repo add rook-alpha https://charts.rook.io/alpha -``` - -For the v0.7 release (see the [v0.7 documentation](https://rook.io/docs/rook/v0.7/helm-operator.html)): -```console -helm install --namespace rook-system rook-alpha/rook -``` - -After the v0.8 release is available: -```console helm install --namespace rook-ceph-system rook-alpha/rook-ceph ``` @@ -69,7 +68,7 @@ helm install --namespace rook-ceph-system rook-master/rook-ceph --version /tmp/rookObj + echo "Hello Rook" > /tmp/rookObj s3cmd put /tmp/rookObj --no-ssl --host=${AWS_HOST} --host-bucket= s3://rookbucket ``` diff --git a/Documentation/quickstart-toc.md b/Documentation/quickstart-toc.md new file mode 100644 index 000000000..6551daa47 --- /dev/null +++ b/Documentation/quickstart-toc.md @@ -0,0 +1,17 @@ +--- +title: Quickstart +weight: 2 +--- + +# Quickstart Guides + +Welcome to Rook! We hope you have a great experience installing the Rook storage platform to enable highly available, durable storage +in your cluster. If you have any questions along the way, please don't hesitate to ask us in our [Slack channel](https://rook-io.slack.com). You can sign up for our Slack [here](https://rook-slackin.herokuapp.com/). + +Rook provides three types of storage providers to a Kubernetes cluster, each with its own operator to deploy and manage the cluster. Follow these guides to get started with each provider. + +| Storage Provider | Status | Description | +|---|---|---| +| [Ceph](ceph-quickstart.md) | Beta | Ceph is a highly scalable distributed storage solution for block storage, object storage, and shared file systems with years of production deployments. | +| [CockroachDB](cockroachdb.md) | Alpha | CockroachDB is a cloud-native SQL database for building global, scalable cloud services that survive disasters. | +| [Minio](minio-object-store.md) | Alpha | Minio is a high performance distributed object storage server, designed for large-scale private cloud infrastructure. | diff --git a/Documentation/storage.md b/Documentation/storage.md index 6fb2f52dd..2c63220d9 100644 --- a/Documentation/storage.md +++ b/Documentation/storage.md @@ -1,11 +1,11 @@ --- -title: Storage +title: Ceph Storage weight: 20 --- -# Storage +# Ceph Storage -Rook provides three types of storage to the Kubernetes cluster. +Rook provides three types of storage to the Kubernetes cluster with the Ceph operator. - [Block Storage](block.md): Mount storage to a single pod - [Object Storage](object.md): Expose an S3 API to the storage cluster for applications to put and get data that is accessible from inside or outside the Kubernetes cluster - [Shared File System](filesystem.md): Mount a file system that can be shared across multiple pods diff --git a/Documentation/tectonic.md b/Documentation/tectonic.md index 4cf0ee9dc..b0472e277 100644 --- a/Documentation/tectonic.md +++ b/Documentation/tectonic.md @@ -44,5 +44,5 @@ Visit the official [Tectonic manual boot](https://coreos.com/tectonic/docs/lates **Remark:** The Tectonic installer contains the correct terraform binary out of the box. This terraform binary can be found in following directory `~/tectonic/tectonic-installer/linux`. ## Start Rook -After the Tectonic Installer ran and the Kubernetes cluster is started and ready, you can follow the [Rook installation guide](quickstart.md). +After the Tectonic Installer ran and the Kubernetes cluster is started and ready, you can follow the [Rook installation guide](ceph-quickstart.md). If you want to specify which disks Rook uses, follow the instructions in [creating Rook clusters](ceph-cluster-crd.md) diff --git a/Documentation/toolbox.md b/Documentation/toolbox.md index 40cd754d3..c3d324ea0 100644 --- a/Documentation/toolbox.md +++ b/Documentation/toolbox.md @@ -10,7 +10,7 @@ The toolbox is based on CentOS, so more tools of your choosing can be easily ins ## Running the Toolbox in Kubernetes -The rook toolbox can run as a pod in a Kubernetes cluster. After you ensure you have a running Kubernetes cluster with rook deployed (see the [Kubernetes](quickstart.md) instructions), +The rook toolbox can run as a pod in a Kubernetes cluster. After you ensure you have a running Kubernetes cluster with rook deployed (see the [Kubernetes](ceph-quickstart.md) instructions), launch the rook-ceph-tools pod. Save the tools spec as `toolbox.yaml`: @@ -92,4 +92,6 @@ kubectl -n rook-ceph delete pod rook-ceph-tools ``` ## Troubleshooting without the Toolbox -The Ceph tools will commonly be the only tools needed to troubleshoot a cluster. In that case, you can connect to any of the rook pods and execute the ceph commands in the same way that you would in the toolbox pod. For example, you can connect to the mon, osd, or even the operator pod to execute commands such as `ceph status`. +The Ceph tools will commonly be the only tools needed to troubleshoot a cluster. In that case, you can connect to any of the rook pods and execute the ceph commands in the same way that you would in the toolbox pod such as the mon pods or the operator pod. +If connecting to the mon pods, make sure you connect to the mon most recently started. The mons keep the config updated in memory after starting and may not have the latest config on disk. +For example, after starting the cluster connect to the `mon2` pod instead of `mon0`. diff --git a/Documentation/upgrade.md b/Documentation/upgrade.md index e38a63e94..99ca4f3df 100644 --- a/Documentation/upgrade.md +++ b/Documentation/upgrade.md @@ -14,7 +14,9 @@ The goal is to provide prescriptive guidance and knowledge on how to upgrade a l We welcome feedback and opening issues! ## Supported Versions -The supported version for this upgrade guide is **from an 0.7 release to the latest builds**. Until 0.8 is released, the latest builds are labeled such as `v0.7.0-27.gbfc8ec6`. Build-to-build upgrades are not guaranteed to work. This guide is to test upgrades only between the official releases. +The supported version for this upgrade guide is **from an 0.7 release to the latest builds**. Until 0.8 is released, +the latest builds are labeled such as `v0.7.0-280.g41829b1`. Build-to-build upgrades are not guaranteed to work. +This guide is to test upgrades only between the official releases. For a guide to upgrade previous versions of Rook, please refer to the version of documentation for those releases. - [Upgrade 0.6 to 0.7](https://rook.io/docs/rook/v0.7/upgrade.html) @@ -35,7 +37,7 @@ This persists metadata on host nodes, enabling pods to be terminated during the More details about `dataDirHostPath` can be found in the [Cluster CRD readme](./ceph-cluster-crd.md#cluster-settings). * All pods consuming Rook storage should be created, running, and in a steady state. No Rook persistent volumes should be in the act of being created or deleted. -The minimal sample Cluster spec that will be used in this guide can be found below (note that the specific configuration may not be applicable to all environments): +The minimal sample v0.7 Cluster spec that will be used in this guide can be found below (note that the specific configuration may not be applicable to all environments): ``` apiVersion: v1 kind: Namespace @@ -125,7 +127,7 @@ Any pod that is using a Rook volume should also remain healthy: ## Upgrade Process The general flow of the upgrade process will be to upgrade the version of a Rook pod, verify the pod is running with the new version, then verify that the overall cluster health is still in a good state. -In this guide, we will be upgrading a live Rook cluster running `v0.7.0` to the next available version of `v0.8`. Until the `v0.8` release is completed, we will instead use the latest `v0.7` tag such as `v0.7.0-27.gbfc8ec6`. +In this guide, we will be upgrading a live Rook cluster running `v0.7.1` to the next available version of `v0.8`. Until the `v0.8` release is completed, we will instead use the latest `v0.7` tag such as `v0.7.0-280.g41829b1`. Let's get started! @@ -322,6 +324,19 @@ kubectl -n rook delete daemonset rook-ceph-osd kubectl -n rook delete replicaset rook-ceph-osd- ``` +Now that the 0.7 OSD pods have been deleted, soon you should see the new 0.8 OSD pods in the `Running` state. It may take a few minutes until Kubernetes retries starting the pods. +Once they have all been started, you will see the pods running. If they do not start in a timely manner, you can delete the pods and their K8s deployment will immediately create new pods to replace them. +``` +$ kubectl -n rook get pod -l app=rook-ceph-osd +NAME READY STATUS RESTARTS AGE +rook-ceph-osd-id-0-5675d6f5f8-r5b2g 1/1 Running 6 6m +rook-ceph-osd-id-1-69cc6bd8f6-59tcn 1/1 Running 6 6m +rook-ceph-osd-id-2-74b7cf67c5-mtl92 1/1 Running 6 6m +rook-ceph-osd-id-3-757b845567-bk259 1/1 Running 6 6m +rook-ceph-osd-id-4-6cccb5f7d8-wxl2w 1/1 Running 6 6m +rook-ceph-osd-id-5-5b8598cc9f-2pnfb 1/1 Running 6 6m +``` + ### Ceph Manager The ceph manager has been renamed in 0.8. The new manager will be started automatically by the operator. The old manager and its secret can simply be deleted. @@ -334,7 +349,7 @@ kubectl -n rook delete secret rook-ceph-mgr0 ### Legacy Custom Resource Definitions (CRDs) -During this upgrade process, the new Ceph operator automatically migrated legacy custom resources to their new `rook.io/v1alpha2` and `ceph.rook.io/v1alpha1` types. +During this upgrade process, the new Ceph operator automatically migrated legacy custom resources to their new `rook.io/v1alpha2` and `ceph.rook.io/v1beta1` types. First confirm that there are no remaining legacy CRD instances: ```bash @@ -355,6 +370,12 @@ kubectl delete crd pools.rook.io kubectl delete crd volumeattachments.rook.io ``` +After the legacy CRDs are deleted, you will see some errors in the operator log since the operator was trying to watch the legacy CRDs. +To run with a clean log, restart the operator again by deleting the pod. +```bash +kubectl -n rook-system delete pod -l app=rook-ceph-operator +``` + ### Optional Components If you have optionally installed either [object storage](./object.md) or a [shared file system](./filesystem.md) in your Rook cluster, the sections below will provide guidance on how to update them as well. They are both managed by deployments, which we have already covered in this guide, so the instructions will be brief.