diff --git a/.gitignore b/.gitignore index 1e829f088..d6ab0cf68 100644 --- a/.gitignore +++ b/.gitignore @@ -14,7 +14,7 @@ coverage.html debug.test .vagrant/ -demo/vagrant/cloud-config.yml +demo/standalone/cloud-config.yml .virtualbox/ diff --git a/Documentation/media/kubernetes.png b/Documentation/media/kubernetes.png new file mode 100644 index 000000000..2c3a16665 Binary files /dev/null and b/Documentation/media/kubernetes.png differ diff --git a/Documentation/media/cluster.png b/Documentation/media/standalone.png similarity index 100% rename from Documentation/media/cluster.png rename to Documentation/media/standalone.png diff --git a/README.md b/README.md index 50abc03af..f839d05ad 100644 --- a/README.md +++ b/README.md @@ -6,27 +6,23 @@ - [What is Rook?](#what-is-rook) - [Status](#status) -- [Kubernetes](#kubernetes) -- [Rook Standalone Service](#rook-standalone-service) -- [Block, File and Object Storage](#block-file-and-object-storage) +- [Quickstart Guides](#quickstart-guides) - [Building](#building) -- [Design](#design) - [Contributing](#contributing) - [Contact](#contact) - [Licensing](#licensing) ## What is Rook? -Rook is a distributed storage system designed for cloud native applications. It aims to be the storage solution for container-native app developers. -Just like your container apps, Rook is designed to take advantage of the resource management and orchestration of the Kubernetes platform. -This allows Rook to share cluster resources with your apps, allowing it to dynamically scale as your apps scale and save the effort to manage storage infrastructure separately. -Rook provides persistent storage for your Kubernetes apps - which is one of the largest barriers for developers to fully containerize their apps. -Rook exposes file, block, and object storage on top of shared resource pools. Rook has minimal -dependencies and can be deployed in dedicated storage clusters or converged clusters. It's -self-managing, self-protecting, self-healing, and is designed to just work without teams of -engineers managing it. It scales from a single node, to multi-PB clusters spread geographically. -It's based on the [Ceph](http://ceph.com) project with over 10 years of production deployments in some of the -largest storage clusters in the world. +Rook is a distributed storage system designed for cloud native applications. It exposes file, block, and object storage on top of shared resource pools. +Rook has minimal dependencies and can be deployed in dedicated storage clusters or converged clusters. +It's self-managing, self-protecting, self-healing, and is designed to just work without teams of engineers managing it. +It scales from a single node, to multi-PB clusters spread geographically. + +It is based on the [Ceph](http://ceph.com) project that has over 10 years of production deployments in some of the largest storage clusters in the world. + +Rook integrates deeply into popular container environments like Kubernetes and leverages facilities for lifecycle management, resource management, scale-out and upgrades. +Rook also integrates into the Kubernetes API to expose a uniform surface area for management. ## Status @@ -35,283 +31,21 @@ and other user-facing objects are subject to change. Backward-compability is not release. See our [Roadmap](https://github.com/rook/rook/wiki/Roadmap) and [Issues](https://github.com/rook/rook/issues). Please help us by [Contributing](CONTRIBUTING.md) to the project. -## Quickstart +## Quickstart Guides -Here's the quickest way to get going with Rook. +There are a few different options for running a Rook cluster for your storage needs. Kubernetes is the recommended way because of the rich orchestration and scheduling that Kubernetes provides via the Rook operator. -### Kubernetes +1. [Kubernetes](/demo/kubernetes/README.md) (recommended) +2. [Standalone](/demo/standalone/README.md) -This example shows how to build a simple, multi-tier web application on Kubernetes using persistent volumes enabled by Rook. - -#### Prerequisites - -This example requires a running Kubernetes cluster. To make sure you have a Kubernetes cluster that is ready for `rook`, you can [follow these quick instructions](demo/kubernetes/README.md). - -Note that we are striving for even more smooth integration with Kubernetes in the future such that `rook` will work out of the box with any Kubernetes cluster. - -#### Deploy Rook - -With your Kubernetes cluster running, Rook can be setup and deployed by simply deploying the [rook-operator](demo/kubernetes/rook-operator.yaml) deployment manifest. -You will find this manifest and all our example manifest files in the [demo/kubernetes](demo/kubernetes) folder. - -``` -cd demo/kubernetes -kubectl create namespace rook -kubectl create -f rook-operator.yaml -``` - -Use `kubectl` to list pods in the rook namespace. You should be able to see the following: - -``` -$ kubectl -n rook get pod -NAME READY STATUS RESTARTS AGE -mon0 1/1 Running 0 1m -mon1 1/1 Running 0 1m -mon2 1/1 Running 0 1m -osd-n1sm3 1/1 Running 0 1m -osd-pb0sh 1/1 Running 0 1m -osd-rth3q 1/1 Running 0 1m -rgw-1785797224-9xb4r 1/1 Running 0 1m -rgw-1785797224-vbg8d 1/1 Running 0 1m -rook-api-4184191414-l0wmw 1/1 Running 0 1m -rook-operator-349747813-c3dmm 1/1 Running 0 1m -``` -**NOTE:** RGW (object storage gateway) is currently deployed by default but in the future will be done only when needed (see [#413](https://github.com/rook/rook/issues/413)) - -#### Provision Storage -Before Rook can start provisioning storage, a StorageClass needs to be created. This is used to specify information needed for Kubernetes to interoperate with Rook for provisioning persistent volumes. Rook already creates a default admin and demo user, whose secrets are already specified in the sample [rook-storageclass.yaml](demo/kubernetes/rook-storageclass.yaml). - -Now we just need to specify the Ceph monitor endpoints (requires `jq`): - -``` -export MONS=$(kubectl -n rook get pod mon0 mon1 mon2 -o json|jq ".items[].status.podIP"|tr -d "\""|sed -e 's/$/:6790/'|paste -s -d, -) -sed 's#INSERT_HERE#'$MONS'#' rook-storageclass.yaml | kubectl create -f - -``` -**NOTE:** In the v0.4 release we plan to expose monitors via DNS/service names instead of IP address (see [#355](https://github.com/rook/rook/issues/355)), which will streamline the experience and remove the need for this step. - -#### Consume the storage - -Now that rook is running and integrated with Kubernetes, we can create a sample app to consume the block storage provisioned by rook. We will create the classic wordpress and mysql apps. -Both these apps will make use of block volumes provisioned by rook. - -Start mysql and wordpress: - -``` -kubectl create -f mysql.yaml -kubectl create -f wordpress.yaml -``` - -Both of these apps create a block volume and mount it to their respective pod. You can see the Kubernetes volume claims by running the following: - -``` -$ kubectl get pvc -NAME STATUS VOLUME CAPACITY ACCESSMODES AGE -mysql-pv-claim Bound pvc-95402dbc-efc0-11e6-bc9a-0cc47a3459ee 20Gi RWO 1m -wp-pv-claim Bound pvc-39e43169-efc1-11e6-bc9a-0cc47a3459ee 20Gi RWO 1m -``` - -Once the wordpress and mysql pods are in the `Running` state, get the cluster IP of the wordpress app and enter it in your brower: - -``` -$ kubectl get svc wordpress -NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE -wordpress 10.3.0.155 80:30841/TCP 2m -``` - -You should see the wordpress app running. - -**NOTE:** When running in a vagrant environment, there will be no external IP address to reach wordpress with. You will only be able to reach wordpress via the `CLUSTER-IP` from inside the Kubernetes cluster. - -#### Rook Client -You also have the option to use the `rook` client tool directly by running it in a pod that can be started in the cluster with: -``` -kubectl create -f rook-client/rook-client.yml -``` - -Starting the rook-client pod will take a bit of time to download the container, so you can check to see when it's ready with (it should be in the `Running` state): -``` -kubectl -n rook get pod rook-client -``` - -Connect to the rook-client pod and verify the `rook` client can talk to the cluster: -``` -kubectl -n rook exec rook-client -it bash -rook node ls -``` - -At this point (optional), you can follow the steps in the [Block, File and Object Storage section](#block-file-and-object-storage) to create and use those types of storage. - -## Rook Standalone Service - -Rook can also be deployed as a standalone service on any modern Linux host by running the following: - -### Linux -1. Download the latest binaries - - ```bash - $ wget https://github.com/rook/rook/releases/download/v0.2.2/rook-v0.2.2-linux-amd64.tar.gz - $ tar xvf rook-v0.2.2-linux-amd64.tar.gz - ``` - -2. Start a one node Rook cluster - - ```bash - $ ./rookd --data-dir /tmp/rook-test - ``` - -### Vagrant - -Rook is also easy to run with `vagrant` on CoreOS via `rkt`. - -``` -cd demo/vagrant -vagrant up -``` - -## Block, File and Object Storage - -### Block Storage -1. Create a new volume image (10MB) - - ```bash - $ rook block create --name test --size 10485760 - ``` - -2. Mount the block volume and format it - - ```bash - sudo rook block mount --name test --path /tmp/rook-volume - sudo chown $USER:$USER /tmp/rook-volume - ``` - -3. Write and read a file - - ```bash - echo "Hello Rook!" > /tmp/rook-volume/hello - cat /tmp/rook-volume/hello - ``` - -4. Cleanup - - ```bash - sudo rook block unmount --path /tmp/rook-volume - ``` - -## Shared File System -1. Create a shared file system - - ```bash - rook filesystem create --name testFS - ``` - -2. Verify the shared file system was created - - ```bash - rook filesystem ls - ``` - -3. Mount the shared file system from the cluster to your local machine - - ```bash - rook filesystem mount --name testFS --path /tmp/rookFS - sudo chown $USER:$USER /tmp/rookFS - ``` - -4. Write and read a file to the shared file system - - ```bash - echo "Hello Rook!" > /tmp/rookFS/hello - cat /tmp/rookFS/hello - ``` - -5. Unmount the shared file system (this does **not** delete the data from the cluster) - - ```bash - rook filesystem unmount --path /tmp/rookFS - ``` - -6. Cleanup the shared file system from the cluster (this **does** delete the data from the cluster) - - ``` - rook filesystem delete --name testFS - ``` - -### Object Storage -1. Create an object storage instance in the cluster - - ```bash - rook object create - ``` - -2. Create an object storage user - - ```bash - rook object user create rook-user "A rook rgw User" - ``` - -3. Get the connection information for accessing object storage - - ```bash - eval $(rook object connection rook-user --format env-var) - ``` - -4. Use an S3 compatible client to create a bucket in the object store - - ```bash - s3cmd mb --no-ssl --host=${AWS_ENDPOINT} --host-bucket= s3://rookbucket - ``` - -5. List all buckets in the object store - - ```bash - s3cmd ls --no-ssl --host=${AWS_ENDPOINT} --host-bucket= - ``` - -6. Upload a file to the newly created bucket - - ```bash - echo "Hello Rook!" > /tmp/rookObj - s3cmd put /tmp/rookObj --no-ssl --host=${AWS_ENDPOINT} --host-bucket= s3://rookbucket - ``` - -7. Download and verify the file from the bucket - - ```bash - s3cmd get s3://rookbucket/rookObj /tmp/rookObj-download --no-ssl --host=${AWS_ENDPOINT} --host-bucket= - cat /tmp/rookObj-download - ``` +### Using Rook +Once you have a Rook cluster running, you can use the `rook` tool to create and manage storage as shown in the following guide: +- [Using Rook Guide](/demo/client/README.md) ## Building See [Building](https://github.com/rook/rook/wiki/Building) in the wiki for more details. -## Design - -A rook cluster is made up of one or more nodes each running the Rook daemon `rookd`. Containers and Pods can -mount block devices and filesystems exposed by the cluster, or can use S3/Swift API for object storage. There is -also a REST API exposed by `rookd` as well as a command line tool called `rook`. - -![Overview](Documentation/media/cluster.png) - -The Rook daemon `rookd` is a single binary that is self-contained and has all that is needed to bootstrap, scale -and manage a storage cluster. `rookd` is typically compiled into a single static binary (just like most golang -binaries) or a dynamic binary that takes a dependency on mostly libc. It can run in minimal containers, alongside a -hypervisor, or directly on the host on most Linux distributions. - -`rookd` uses an embedded version of Ceph for storing all data -- there are no changes to the data path. An embedded version -of Ceph was created specifically for Rook scenarios and has been pushed upstream. Rook does not attempt to maintain full fidelity -with Ceph, for example, most of the Ceph concepts like OSDs, MONs, placement groups, etc. are hidden. Instead Rook creates -a much simplified UX for admins that is in terms of physical resources, pools, volumes, filesystems, and buckets. - -`rookd` embeds Etcd to store configuration and coordinate cluster-wide management operations. `rookd` will automatically -bootstrap Etcd, manage it, and scale it as the cluster grows. It's also possible to use an external Etcd instead of the embedded one -if needed. - -Rook and etcd are implemented in golang. Ceph is implemented in C++ where the data path is highly optimized. We believe -this combination offers the best of both worlds. - -See [Design](https://github.com/rook/rook/wiki/Design) wiki for more details. - ## Contributing We welcome contributions. See [Contributing](CONTRIBUTING.md) to get started. @@ -330,6 +64,6 @@ Please use the following to reach members of the community: ## Licensing -Rook and Etcd are under the Apache 2.0 license. Ceph is mostly under the LGPL 2.0 license. Some portions +Rook and Etcd are under the Apache 2.0 license. [Ceph](https://github.com/rook/ceph/blob/master/COPYING) is mostly under the LGPL 2.0 license. Some portions of the code are under different licenses. The appropriate license information can be found in the headers of the source files. diff --git a/demo/client/README.md b/demo/client/README.md new file mode 100644 index 000000000..76f662b1d --- /dev/null +++ b/demo/client/README.md @@ -0,0 +1,115 @@ +# Using Rook +The `rook` client tool can be used to manage your Rook cluster once it is running as well as manage block, file and object storage. See the sections below for details on how to configure each type of storage. + +If you don't yet have a Rook cluster running, refer to our [Quickstart Guides](../../README.md#quickstart-guides). + +## Block Storage +1. Create a new volume image (10MB) + + ```bash + $ rook block create --name test --size 10485760 + ``` + +2. Mount the block volume and format it + + ```bash + sudo rook block mount --name test --path /tmp/rook-volume + sudo chown $USER:$USER /tmp/rook-volume + ``` + +3. Write and read a file + + ```bash + echo "Hello Rook!" > /tmp/rook-volume/hello + cat /tmp/rook-volume/hello + ``` + +4. Cleanup + + ```bash + sudo rook block unmount --path /tmp/rook-volume + ``` + +## Shared File System +1. Create a shared file system + + ```bash + rook filesystem create --name testFS + ``` + +2. Verify the shared file system was created + + ```bash + rook filesystem ls + ``` + +3. Mount the shared file system from the cluster to your local machine + + ```bash + rook filesystem mount --name testFS --path /tmp/rookFS + sudo chown $USER:$USER /tmp/rookFS + ``` + +4. Write and read a file to the shared file system + + ```bash + echo "Hello Rook!" > /tmp/rookFS/hello + cat /tmp/rookFS/hello + ``` + +5. Unmount the shared file system (this does **not** delete the data from the cluster) + + ```bash + rook filesystem unmount --path /tmp/rookFS + ``` + +6. Cleanup the shared file system from the cluster (this **does** delete the data from the cluster) + + ``` + rook filesystem delete --name testFS + ``` + +## Object Storage +1. Create an object storage instance in the cluster + + ```bash + rook object create + ``` + +2. Create an object storage user + + ```bash + rook object user create rook-user "A rook rgw User" + ``` + +3. Get the connection information for accessing object storage + + ```bash + eval $(rook object connection rook-user --format env-var) + ``` + +4. Use an S3 compatible client to create a bucket in the object store + + ```bash + s3cmd mb --no-ssl --host=${AWS_ENDPOINT} --host-bucket= s3://rookbucket + ``` + +5. List all buckets in the object store + + ```bash + s3cmd ls --no-ssl --host=${AWS_ENDPOINT} --host-bucket= + ``` + +6. Upload a file to the newly created bucket + + ```bash + echo "Hello Rook!" > /tmp/rookObj + s3cmd put /tmp/rookObj --no-ssl --host=${AWS_ENDPOINT} --host-bucket= s3://rookbucket + ``` + +7. Download and verify the file from the bucket + + ```bash + s3cmd get s3://rookbucket/rookObj /tmp/rookObj-download --no-ssl --host=${AWS_ENDPOINT} --host-bucket= + cat /tmp/rookObj-download + ``` diff --git a/demo/kubernetes/README.md b/demo/kubernetes/README.md index f0a7ffa6c..7cd43e87e 100644 --- a/demo/kubernetes/README.md +++ b/demo/kubernetes/README.md @@ -1,41 +1,133 @@ -## Existing Kubernetes Cluster -If you already have a running Kubernetes cluster, you will also need to modify the kubelet service to bind mount `/sbin/modprobe` to allow access to `modprobe`. Access to modprobe is necessary for using the rbd volume plugin (). -If using RKT, you can allow modprobe by following this [doc](https://github.com/coreos/coreos-kubernetes/blob/master/Documentation/kubelet-wrapper.md#allow-pods-to-use-rbd-volumes). +# Rook on Kubernetes +- [Quickstart](#quickstart) +- [Design](#design) -Instructions also copied here for convenience: -Pods using the [rbd volume plugin](https://github.com/kubernetes/kubernetes/tree/master/examples/volumes/rbd) to consume data from ceph must ensure that the kubelet has access to modprobe. Add the following options to the `RKT_OPTS` env before launching the kubelet via kubelet-wrapper: +## Quickstart +This example shows how to build a simple, multi-tier web application on Kubernetes using persistent volumes enabled by Rook. -```ini -[Service] -Environment=KUBELET_VERSION=v1.5.3_coreos.0 -Environment="RKT_OPTS=--volume modprobe,kind=host,source=/usr/sbin/modprobe \ - --mount volume=modprobe,target=/usr/sbin/modprobe \ - --volume lib-modules,kind=host,source=/lib/modules \ - --mount volume=lib-modules,target=/lib/modules \ - --uuid-file-save=/var/run/kubelet-pod.uuid" -... -``` +### Prerequisites -Note that the kubelet also requires access to the userspace `rbd` tool that is included only in hyperkube images tagged `v1.3.6_coreos.0` or later. +This example requires a running Kubernetes cluster with access to `modprobe` by the kubelet. To make sure you have a Kubernetes cluster that is ready for `rook`, you can [follow these quick instructions](pre-reqs.md). -## New Local Kubernetes Cluster -Alternatively, for a quick start with a new local cluster, use the Rook fork of [coreos-kubernetes](https://github.com/rook/coreos-kubernetes). This will bring up a multi-node Kubernetes cluster with `vagrant` and CoreOS virtual machines ready to use `rook` immediately. -``` -git clone https://github.com/rook/coreos-kubernetes.git -cd coreos-kubernetes/multi-node/vagrant -vagrant up -export KUBECONFIG="$(pwd)/kubeconfig" -kubectl config use-context vagrant-multi -``` +Note that we are striving for even more smooth integration with Kubernetes in the future such that `rook` will work out of the box with any Kubernetes cluster. -Then wait for the cluster to come up and verify that kubernetes is done initializing (be patient, it takes a bit): +### Deploy Rook + +With your Kubernetes cluster running, Rook can be setup and deployed by simply deploying the [rook-operator](rook-operator.yaml) deployment manifest. ``` -kubectl cluster-info +cd demo/kubernetes +kubectl create namespace rook +kubectl create -f rook-operator.yaml ``` -If you see a url response, your cluster is ready to go. +Use `kubectl` to list pods in the rook namespace. You should be able to see the following: -## Using Rook in Kubernetes -Now that you have a Kubernetes cluster running, you can start using `rook` with [these steps](../../README.md#kubernetes) \ No newline at end of file +``` +$ kubectl -n rook get pod +NAME READY STATUS RESTARTS AGE +mon0 1/1 Running 0 1m +mon1 1/1 Running 0 1m +mon2 1/1 Running 0 1m +osd-n1sm3 1/1 Running 0 1m +osd-pb0sh 1/1 Running 0 1m +osd-rth3q 1/1 Running 0 1m +rgw-1785797224-9xb4r 1/1 Running 0 1m +rgw-1785797224-vbg8d 1/1 Running 0 1m +rook-api-4184191414-l0wmw 1/1 Running 0 1m +rook-operator-349747813-c3dmm 1/1 Running 0 1m +``` +**NOTE:** RGW (object storage gateway) is currently deployed by default but in the future will be done only when needed (see [#413](https://github.com/rook/rook/issues/413)) + +### Provision Storage +Before Rook can start provisioning storage, a StorageClass needs to be created. This is used to specify information needed for Kubernetes to interoperate with Rook for provisioning persistent volumes. Rook already creates a default admin and demo user, whose secrets are already specified in the sample [rook-storageclass.yaml](rook-storageclass.yaml). + +Now we just need to specify the Ceph monitor endpoints (requires `jq`): + +``` +export MONS=$(kubectl -n rook get pod mon0 mon1 mon2 -o json|jq ".items[].status.podIP"|tr -d "\""|sed -e 's/$/:6790/'|paste -s -d, -) +sed 's#INSERT_HERE#'$MONS'#' rook-storageclass.yaml | kubectl create -f - +``` +**NOTE:** In the v0.4 release we plan to expose monitors via DNS/service names instead of IP address (see [#355](https://github.com/rook/rook/issues/355)), which will streamline the experience and remove the need for this step. + +### Consume the storage + +Now that rook is running and integrated with Kubernetes, we can create a sample app to consume the block storage provisioned by rook. We will create the classic wordpress and mysql apps. +Both these apps will make use of block volumes provisioned by rook. + +Start mysql and wordpress: + +``` +kubectl create -f mysql.yaml +kubectl create -f wordpress.yaml +``` + +Both of these apps create a block volume and mount it to their respective pod. You can see the Kubernetes volume claims by running the following: + +``` +$ kubectl get pvc +NAME STATUS VOLUME CAPACITY ACCESSMODES AGE +mysql-pv-claim Bound pvc-95402dbc-efc0-11e6-bc9a-0cc47a3459ee 20Gi RWO 1m +wp-pv-claim Bound pvc-39e43169-efc1-11e6-bc9a-0cc47a3459ee 20Gi RWO 1m +``` + +Once the wordpress and mysql pods are in the `Running` state, get the cluster IP of the wordpress app and enter it in your brower: + +``` +$ kubectl get svc wordpress +NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE +wordpress 10.3.0.155 80:30841/TCP 2m +``` + +You should see the wordpress app running. + +**NOTE:** When running in a vagrant environment, there will be no external IP address to reach wordpress with. You will only be able to reach wordpress via the `CLUSTER-IP` from inside the Kubernetes cluster. + +### Rook Client +You also have the option to use the `rook` client tool directly by running it in a pod that can be started in the cluster with: +``` +kubectl create -f rook-client/rook-client.yml +``` + +Starting the rook-client pod will take a bit of time to download the container, so you can check to see when it's ready with (it should be in the `Running` state): +``` +kubectl -n rook get pod rook-client +``` + +Connect to the rook-client pod and verify the `rook` client can talk to the cluster: +``` +kubectl -n rook exec rook-client -it bash +rook node ls +``` + +At this point, you can use the `rook` tool along with some [simple steps to create and manage block, file and object storage](../client/README.md). + +## Design + +With Rook running in the Kubernetes cluster, Kubernetes applications can +mount block devices and filesystems managed by Rook, or can use the S3/Swift API for object storage. The Rook operator +automates configuration of the Ceph storage components and monitors the cluster to ensure the storage remains available +and healthy. There is also a REST API service for configuring the Rook storage and a command line tool called `rook`. + +![Rook Architecture on Kubernetes](/Documentation/media/kubernetes.png) + +The Rook operator is a simple container containing the `rook-operator` binary that has all that is needed to bootstrap +and monitor the storage cluster. The operator will start and monitor ceph monitor pods and a daemonset for the OSDs to +which provides basic RADOS storage, as well as a deployment for a RESTful API service. When requested through the api service, +object storage (S3/Swift) is enabled by starting a deployment for RGW, while a shared file system is enabled with a deployment for MDS. + +The operator will monitor the storage daemons to ensure the cluster is healthy, to start more ceph mons when necessary, to +failover mons, make adjustments as the cluster grows or shrinks, etc. The operator will also watch for desired state changes +requested by the api service and apply the changes. + +The Rook daemons (Mons, OSDs, RGW, and MDS) are compiled to a single binary `rookd`, and included in a minimal container. +`rookd` uses an embedded version of Ceph for storing all data -- there are no changes to the data path. +Rook does not attempt to maintain full fidelity with Ceph. Many of the Ceph concepts like placement groups and crush maps +are hidden so you don't have to worry about them. Instead Rook creates a much simplified UX for admins that is in terms +of physical resources, pools, volumes, filesystems, and buckets. + +Rook is implemented in golang. Ceph is implemented in C++ where the data path is highly optimized. We believe +this combination offers the best of both worlds. + +See [Design](https://github.com/rook/rook/wiki/Design) wiki for more details. \ No newline at end of file diff --git a/demo/kubernetes/pre-reqs.md b/demo/kubernetes/pre-reqs.md new file mode 100644 index 000000000..5a7abbd85 --- /dev/null +++ b/demo/kubernetes/pre-reqs.md @@ -0,0 +1,41 @@ + +## New Local Kubernetes Cluster +For a quick start with a new local cluster, use the Rook fork of [coreos-kubernetes](https://github.com/rook/coreos-kubernetes). This will bring up a multi-node Kubernetes cluster with `vagrant` and CoreOS virtual machines ready to use `rook` immediately. +``` +git clone https://github.com/rook/coreos-kubernetes.git +cd coreos-kubernetes/multi-node/vagrant +vagrant up +export KUBECONFIG="$(pwd)/kubeconfig" +kubectl config use-context vagrant-multi +``` + +Then wait for the cluster to come up and verify that kubernetes is done initializing (be patient, it takes a bit): + +``` +kubectl cluster-info +``` + +Once you see a url response, your cluster is [ready for use by Rook](README.md#deploy-rook). + +## Existing Kubernetes Cluster +Alternatively, if you already have a running Kubernetes cluster, you can deploy Rook to it with a small update to modify the kubelet service to bind mount `/sbin/modprobe`, which allows access to `modprobe`. +Access to modprobe is necessary for using the rbd volume plugin, which is being tracked in the Kubernetes code base with [#23924](https://github.com/kubernetes/kubernetes/issues/23924). + +If using RKT, you can enable `modprobe` by following this [guide](https://github.com/coreos/coreos-kubernetes/blob/master/Documentation/kubelet-wrapper.md#allow-pods-to-use-rbd-volumes). Instructions have been directly copied below for your convenience: + +Add the following options to the `RKT_OPTS` env before launching the kubelet via kubelet-wrapper: +```ini +[Service] +Environment=KUBELET_VERSION=v1.5.3_coreos.0 +Environment="RKT_OPTS=--volume modprobe,kind=host,source=/usr/sbin/modprobe \ + --mount volume=modprobe,target=/usr/sbin/modprobe \ + --volume lib-modules,kind=host,source=/lib/modules \ + --mount volume=lib-modules,target=/lib/modules \ + --uuid-file-save=/var/run/kubelet-pod.uuid" +... +``` + +Note that the kubelet also requires access to the userspace `rbd` tool that is included only in hyperkube images tagged `v1.3.6_coreos.0` or later. + +## Using Rook in Kubernetes +Now that you have a Kubernetes cluster running, you can start using `rook` with [these steps](README.md#deploy-rook). \ No newline at end of file diff --git a/demo/standalone/README.md b/demo/standalone/README.md new file mode 100644 index 000000000..d4d30bcdd --- /dev/null +++ b/demo/standalone/README.md @@ -0,0 +1,80 @@ + +# Rook Standalone + +- [Directly on Linux Host](#directly-on-linux-host) +- [Vagrant](#vagrant) +- [Design](#design) + +## Directly on Linux Host + +Rook can be deployed as a standalone service directly on any modern Linux host by running the following: + +1. Download the latest client and server binaries + +```bash +wget https://github.com/rook/rook/releases/download/v0.2.2/rook-v0.2.2-linux-amd64.tar.gz +tar xvf rook-v0.2.2-linux-amd64.tar.gz +``` + +2. Start a one node Rook cluster +```bash +./rookd --data-dir /tmp/rook-test +``` + +3. In another console (but in the same path), verify the Rook cluster is ready +```bash +./rook status +``` + +At this point, you can use the `rook` tool along with some [simple steps to create and manage block, file and object storage](../client/README.md). + + +## Vagrant + +Rook is also easy to run in virutal machines with `vagrant` using the provided [Vagrantfile](Vagrantfile) in this directory. When instructed, `vagrant` will start up CoreOS virtual machines and launch `rook` via `rkt`. Note that you can make configuration changes as desired in the provided [cloud config file](cloud-config.yml.in). + +``` +vagrant up +``` + +### Rook Client Tool +Once the Rook cluster in Vagrant is running and initialized, you can use the `rook` client tool to manage the cluster and consume the storage. First, either use a locally built `rook` tool or download the latest release from github: +``` +wget https://github.com/rook/rook/releases/download/v0.2.2/rook-v0.2.2-linux-amd64.tar.gz +tar xvf rook-v0.2.2-linux-amd64.tar.gz +``` + +Verify the cluster is up and running: +``` +export ROOK_API_SERVER_ENDPOINT="172.20.20.10:8124" +./rook status +``` + +At this point, you can use the `rook` tool along with some [simple steps to create and manage block, file and object storage](../client/README.md). + +### Clean up + +When you are all done with your Rook cluster in Vagrant, you can clean everything up by destroying the VMs with: +``` +vagrant destroy -f +rm -f .discovery-token +``` + +## Design + +Rook supports an environment where there is no orchestration platform. Rook comes with its own basic orchestration +engine based on Etcd that is activated when not running in Kubernetes. + +![Standalone Rook Architecture](/Documentation/media/standalone.png) + +The Rook daemon, `rookd`, is a single binary that is self-contained and has all that is needed to bootstrap, scale +and manage a storage cluster. `rookd` is typically compiled into a single static binary (just like most golang +binaries) or a dynamic binary that takes a dependency on mostly libc. It can run in minimal containers, alongside a +hypervisor, or directly on the host on most Linux distributions. Each machine in the cluster must run the Rook daemon. + +`rookd` embeds Etcd to store configuration and coordinate cluster-wide management operations. `rookd` will automatically +bootstrap Etcd, manage it, and scale it as the cluster grows. It's also possible to use an external Etcd instead of the embedded one +if needed. + +Standalone Rook still requires some key orchestration features such as resource management, health monitoring, failover, and upgrades. +If this scenario is interesting to you, contributions are welcome! \ No newline at end of file diff --git a/demo/vagrant/Vagrantfile b/demo/standalone/Vagrantfile similarity index 100% rename from demo/vagrant/Vagrantfile rename to demo/standalone/Vagrantfile diff --git a/demo/vagrant/cloud-config.yml.in b/demo/standalone/cloud-config.yml.in similarity index 100% rename from demo/vagrant/cloud-config.yml.in rename to demo/standalone/cloud-config.yml.in diff --git a/demo/vagrant/util.rb b/demo/standalone/util.rb similarity index 100% rename from demo/vagrant/util.rb rename to demo/standalone/util.rb