diff --git a/Documentation/k8s-block.md b/Documentation/k8s-block.md new file mode 100644 index 000000000..b7e749fb8 --- /dev/null +++ b/Documentation/k8s-block.md @@ -0,0 +1,62 @@ +# Block Storage Quickstart + +Block storage allows you to mount storage to a single pod. + +### Prerequisites + +This guide assumes you have created a Rook cluster as explained in the main [Kubernetes guide](kubernetes.md) + +### Provision Storage +Before Rook can start provisioning storage, a StorageClass and its storage pool need to be created. This is needed for Kubernetes to interoperate with Rook for provisioning persistent volumes. The rook-storageclass.yaml sample will create the storage pool automatically. For more options on pools, see the documentation on [creating storage pools](pool-tpr.md). + +Rook already creates a default admin and rbd user, whose secrets are specified in the sample [rook-storageclass.yaml](/demo/kubernetes/rook-storageclass.yaml). Now we just need to specify the Ceph monitor endpoints (requires `jq`): + +```bash +cd demo/kubernetes +export MONS=$(kubectl -n rook get pod mon0 mon1 mon2 -o json|jq ".items[].status.podIP"|tr -d "\""|sed -e 's/$/:6790/'|paste -s -d, -) +sed 's#INSERT_HERE#'$MONS'#' rook-storageclass.yaml | kubectl create -f - +``` + +**NOTE:** We are working on streamlining the experience and removing the need for this step. See [#355](https://github.com/rook/rook/issues/355). + +### Consume the storage + +We create a sample app to consume the block storage provisioned by Rook with the classic wordpress and mysql apps. +Both of these apps will make use of block volumes provisioned by Rook. + +Start mysql and wordpress from the `demo/kubernetes` folder: + +```bash +kubectl create -f mysql.yaml +kubectl create -f wordpress.yaml +``` + +Both of these apps create a block volume and mount it to their respective pod. You can see the Kubernetes volume claims by running the following: + +```bash +$ kubectl get pvc +NAME STATUS VOLUME CAPACITY ACCESSMODES AGE +mysql-pv-claim Bound pvc-95402dbc-efc0-11e6-bc9a-0cc47a3459ee 20Gi RWO 1m +wp-pv-claim Bound pvc-39e43169-efc1-11e6-bc9a-0cc47a3459ee 20Gi RWO 1m +``` + +Once the wordpress and mysql pods are in the `Running` state, get the cluster IP of the wordpress app and enter it in your brower: + +```bash +$ kubectl get svc wordpress +NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE +wordpress 10.3.0.155 80:30841/TCP 2m +``` + +You should see the wordpress app running. + +**NOTE:** When running in a vagrant environment, there will be no external IP address to reach wordpress with. You will only be able to reach wordpress via the `CLUSTER-IP` from inside the Kubernetes cluster. + +### Teardown +To clean up all the artifacts created by the block demo: +``` +kubectl delete -f wordpress.yaml +kubectl delete -f mysql.yaml +kubectl delete -n rook rookpool replicapool +kubectl delete storageclass rook-block +``` diff --git a/Documentation/k8s-filesystem.md b/Documentation/k8s-filesystem.md new file mode 100644 index 000000000..a21fe17a6 --- /dev/null +++ b/Documentation/k8s-filesystem.md @@ -0,0 +1,102 @@ +# Shared File System Quickstart + +A shared file system can be mounted read-write from multiple pods. This may be useful for applications which can be clustered using a shared filesystem. + +This example runs a shared file system for the [kube-registry](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/registry). + +### Prerequisites + +This guide assumes you have created a Rook cluster and pool as explained in the main [Kubernetes guide](kubernetes.md) + +## Rook Client +Setting up the Rook file system currently requires the Rook client. This will be simplified in the future with a TPR for the object stores. + +```bash +kubectl create -f rook-client/rook-client.yml + +# Starting the pod may take a couple minutes, so check to see when it's ready: +kubectl -n rook get pod rook-client + +# Connect to the rook-client pod +kubectl -n rook exec rook-client -it bash + +# Confirm the rook client can connect to the cluster +rook status +``` + +## Create the File System +Create the file system with the default pools. +```bash +rook filesystem create --name registryFS +``` + +### Optional: Adjust pool paramaters + +By default the pools do not have any redundancy. To create another copy of the data, let's set the replication to 2. + +First we will launch the [Rook toolbox](toolbox.md#running-the-toolbox) in order to run `ceph` commands. +```bash +# Start the Rook toolbox in order to run Ceph commands (the yml is found in the toolbox folder) +cd toolbox +kubectl create -f rook-tools.yml + +# Verify the toolbox is running +kubectl -n rook get pod rook-tools + +# Connect to the toolbox +kubectl -n rook exec -it rook-tools bash +``` + +Now we can modify the pool size +```bash +ceph osd pool set registryFS-data size 2 +ceph osd pool set registryFS-metadata size 2 +``` + +### Optional: Copy admin key to desired namespace + +If you are consuming the filesystem from a namespace other than `rook` you will need to copy the key to the desired namespace. +In this example we are copying to the `kube-system` namespace. + +```bash +kubectl get secret rook-admin -n rook -o json | jq '.metadata.namespace = "kube-system"' | kubectl apply -f - +``` + +## Deploy the Application + +The kube-registry yaml is defined [here](/demo/kubernetes/kube-registry.yaml). We will need to update the yaml with the monitor IP addresses with the following commands. +In the future this step will be improved with a Rook volume plugin. +```bash +cd demo/kubernetes +export MONS=$(kubectl -n rook get pod mon0 mon1 mon2 -o json|jq ".items[].status.podIP"|tr -d "\""|sed -e 's/$/:6790/'|paste -s -d, -) +sed "s/INSERT_MONS_HERE/$MONS/g" kube-registry.yaml | kubectl create -f - +``` + +You now have a docker registry which is HA with persistent storage. + +### Test the storage + +Verify that kube-registry is using the filesystem that was configured above. + +```bash +# Start the rook toolbox +kubectl -n rook exec rook-tools -it bash + +# Mount the same filesystem that the kube-registry is using +mkdir /tmp/registry +rook filesystem mount --name registryFS --path /tmp/registry + +# Here you should see a directory called docker created by the registry +ls /tmp/registry + +# Cleanup the filesystem mount +rook filesystem unmount --path /tmp/registry +rmdir /tmp/registry +``` + +### Teardown +To clean up all the artifacts created by the file system demo: +```bash +kubectl -n kube-system delete secret rook-admin +kubectl delete -f kube-registry.yaml +``` diff --git a/Documentation/k8s-object.md b/Documentation/k8s-object.md new file mode 100644 index 000000000..1afe21b11 --- /dev/null +++ b/Documentation/k8s-object.md @@ -0,0 +1,80 @@ +# Object Storage Quickstart + +Object storage exposes an S3 API to the storage cluster for applications to put and get data. + +### Prerequisites + +This guide assumes you have created a Rook cluster as explained in the main [Kubernetes guide](kubernetes.md) + +## Rook Client +Setting up the object storage currently requires the Rook client. This will be simplified in the future with a TPR for the object stores. + +```bash +kubectl create -f rook-client/rook-client.yml + +# Starting the pod may take a couple minutes, so check to see when it's ready: +kubectl -n rook get pod rook-client + +# Connect to the rook-client pod +kubectl -n rook exec rook-client -it bash + +# Confirm the rook client can connect to the cluster +rook status +``` + +## Create the Object Store and User +Now we will create the object store, which starts the RGW service in the cluster with the S3 API. +From within the rook client container, run the following: + +```bash +# Create an object storage instance in the cluster +rook object create + +# Create an object storage user. The first user may take a minute to create. +# If it times out, run the same command again to confirm that it finished. +rook object user create rook-user "A rook rgw User" +``` + +The object store is now available for pods to connect by using the creds of `rook-user`. + +### Environment Variables +If your s3 client uses environment variables, the client can print them for you +```bash +rook object connection rook-user --format env-var +``` + +See the [Object Storage](client.md#object-storage) documentation for more steps on consuming the object storage. + +## Access External to the Cluster + +Rook sets up the object storage so pods will have access internal to the cluster. If your applications are running outside the cluster, +you will need to setup an external service through a `NodePort`. + +First, note the service that exposes RGW internal to the cluster. We will leave this service intact and create a new service for external access. +```bash +$ kubectl -n rook get service rgw +NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE +rgw 10.3.0.248 53390/TCP 45s +``` + +Now create the external service: +```bash +cd demo/kubernetes +kubectl create -f rgw-external.yaml +``` + +See both rgw services running and notice what port the external service is running on: +```bash +$ kubectl -n rook get service rgw rgw-external +NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE +rgw 10.3.0.248 53390/TCP 1m +rgw-external 10.3.0.146 53390:30711/TCP 1m +``` + +Internally the rgw service is running on port `53390`. The external port in this case is `30711`. Now you can access the object store from anywhere! All you need is the hostname for any machine in the cluster, the external port, and the user credentials. + +If you're testing on the [coreos-kubernetes vagrant environment](k8s-pre-reqs.md#new-local-kubernetes-cluster), you can verify it is working from your host: +- If running in the single-node cluster: + - `curl 172.17.4.99:30711` +- If running in the multi-node cluster: + - `curl 172.17.4.101:30711` diff --git a/Documentation/kubernetes.md b/Documentation/kubernetes.md index 18ce0c05c..f3a3c8d8d 100644 --- a/Documentation/kubernetes.md +++ b/Documentation/kubernetes.md @@ -18,24 +18,22 @@ Note that we are striving for even more smooth integration with Kubernetes in th With your Kubernetes cluster running, Rook can be setup and deployed by simply creating the [rook-operator](/demo/kubernetes/rook-operator.yaml) deployment and creating a [rook cluster](/demo/kubernetes/rook-cluster.yaml). -``` +```bash cd demo/kubernetes kubectl create -f rook-operator.yaml -``` -This will start the rook-operator pod. Verify that it is in the `Running` state before proceeding: -``` +# This will start the rook-operator pod. Verify that it is in the `Running` state before proceeding: kubectl get pod | grep rook-operator ``` -Now that the rook-operator pod is in the `Running` state, we can create the Rook cluster. See the documentation on [configuring the cluster](cluster-tpr.md). -``` +Now that the rook-operator pod is running, we can create the Rook cluster. See the documentation on [configuring the cluster](cluster-tpr.md). +```bash kubectl create -f rook-cluster.yaml ``` -Use `kubectl` to list pods in the rook namespace. You should be able to see the following once they are all running: +Use `kubectl` to list pods in the rook namespace. You should be able to see the following pods once they are all running: -``` +```bash $ kubectl -n rook get pod NAME READY STATUS RESTARTS AGE mon0 1/1 Running 0 1m @@ -46,83 +44,44 @@ osd-6jmph 1/1 Running 0 1m rook-api-1709486253-gvdnc 1/1 Running 0 1m ``` -### Provision Storage -Before Rook can start provisioning storage, a StorageClass and its storage pool need to be created. This is needed for Kubernetes to interoperate with Rook for provisioning persistent volumes. The rook-storageclass.yaml sample will create the storage pool automatically. For more options on pools, see the documentation on [creating storage pools](pool-tpr.md). +## Storage +For a walkthrough of the three types of storage exposed by Rook, see the guides for: +- **[Block](k8s-block.md)**: Create block storage to be consumed by a pod +- **[Shared File System](k8s-filesystem.md)**: Create a file system to be shared across multiple pods +- **[Object](k8s-object.md)**: Create an object store that is accessible inside or outside the Kubernetes cluster -Rook already creates a default admin and rbd user, whose secrets are specified in the sample [rook-storageclass.yaml](/demo/kubernetes/rook-storageclass.yaml). Now we just need to specify the Ceph monitor endpoints (requires `jq`): - -``` -export MONS=$(kubectl -n rook get pod mon0 mon1 mon2 -o json|jq ".items[].status.podIP"|tr -d "\""|sed -e 's/$/:6790/'|paste -s -d, -) -sed 's#INSERT_HERE#'$MONS'#' rook-storageclass.yaml | kubectl create -f - -``` -**NOTE:** In the v0.4 release we plan to expose monitors via DNS/service names instead of IP address (see [#355](https://github.com/rook/rook/issues/355)), which will streamline the experience and remove the need for this step. - -### Consume the storage - -Now that Rook is running and integrated with Kubernetes, we can create a sample app to consume the block storage provisioned by Rook. We will create the classic wordpress and mysql apps. -Both these apps will make use of block volumes provisioned by Rook. - -Start mysql and wordpress from the `demo/kubernetes` folder: - -``` -kubectl create -f mysql.yaml -kubectl create -f wordpress.yaml -``` - -Both of these apps create a block volume and mount it to their respective pod. You can see the Kubernetes volume claims by running the following: - -``` -$ kubectl get pvc -NAME STATUS VOLUME CAPACITY ACCESSMODES AGE -mysql-pv-claim Bound pvc-95402dbc-efc0-11e6-bc9a-0cc47a3459ee 20Gi RWO 1m -wp-pv-claim Bound pvc-39e43169-efc1-11e6-bc9a-0cc47a3459ee 20Gi RWO 1m -``` - -Once the wordpress and mysql pods are in the `Running` state, get the cluster IP of the wordpress app and enter it in your brower: - -``` -$ kubectl get svc wordpress -NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE -wordpress 10.3.0.155 80:30841/TCP 2m -``` - -You should see the wordpress app running. - -**NOTE:** When running in a vagrant environment, there will be no external IP address to reach wordpress with. You will only be able to reach wordpress via the `CLUSTER-IP` from inside the Kubernetes cluster. +## Tools ### Rook Client You also have the option to use the `rook` client tool directly by running it in a pod that can be started in the cluster with: -``` +```bash kubectl create -f rook-client/rook-client.yml -``` -Starting the rook-client pod will take a bit of time to download the container, so you can check to see when it's ready with (it should be in the `Running` state): -``` +# Starting the rook-client pod will take a bit of time to download the container, so check when it's in the Running state kubectl -n rook get pod rook-client -``` -Connect to the rook-client pod and verify the `rook` client can talk to the cluster: -``` +# Connect to the rook-client pod kubectl -n rook exec rook-client -it bash + +# Verify the rook client can talk to the cluster: rook node ls ``` At this point, you can use the `rook` tool along with some [simple steps to create and manage block, file and object storage](client.md). +### Advanced Configuration and Troubleshooting +We have created a toolbox container that contains the full suite of Ceph clients for debugging and troubleshooting your Rook cluster. Please see the [toolbox readme](toolbox.md) for setup and usage information. + ### Monitoring Each Rook cluster has some built in metrics collectors/exporters for monitoring with [Prometheus](https://prometheus.io/). To learn how to set up monitoring for your Rook cluster, you can follow the steps in the [monitoring guide](./k8s-monitoring.md). -### Teardown -To clean up all the artifacts created by the demo, run the following: -``` -kubectl delete -f wordpress.yaml -kubectl delete -f mysql.yaml +## Teardown +To clean up all the artifacts created by the demo, *first cleanup the resources from the block, file, and object walkthroughs* (unmount volumes, delete volume claims, etc), then run the following: +```bash kubectl delete deployment rook-operator kubectl delete -n rook rookcluster rook -kubectl delete -n rook rookpool replicapool kubectl delete thirdpartyresources rookcluster.rook.io rookpool.rook.io -kubectl delete storageclass rook-block kubectl delete secret rook-rbd-user kubectl delete namespace rook ``` diff --git a/Documentation/toolbox.md b/Documentation/toolbox.md index bddd1da82..71115125c 100644 --- a/Documentation/toolbox.md +++ b/Documentation/toolbox.md @@ -3,45 +3,48 @@ The rook toolbox is a container with common tools used for rook debugging and te ## Installing more tools The rook toolbox is based on Ubuntu, so more tools of your choosing can be easily installed with `apt-get`. For example, to install `telnet`: -``` +```bash apt-get update apt-get install telnet ``` -## Running the toolbox -### Kubernetes +## Running the Toolbox in Kubernetes + The rook toolbox can run as a pod in a Kubernetes cluster. First, ensure you have a running Kubernetes cluster with rook deployed (see the [Kubernetes](kubernetes.md) instructions). From this directory, launch the rook-tools pod: -``` +```bash +cd toolbox kubectl create -f rook-tools.yml ``` Wait for the toolbox pod to download its container and get to the `running` state: -``` +```bash kubectl -n rook get pod rook-tools ``` Once the rook-tools pod is running, you can connect to it with: -``` +```bash kubectl -n rook exec -it rook-tools bash ``` All available tools in the toolbox are ready for your troubleshooting needs. Example: -``` +```bash rook status ceph df rados df ``` When you are completely done with the toolbox, you can clean it up by running: -``` +```bash kubectl delete -f rook-tools.yml ``` +## Running the Toolbox for Standalone + ### Container Linux by CoreOS To use the rook toolbox on CoreOS, first add the following values to the toolbox config file: -``` +```bash cat >~/.toolboxrc <