feedc0de/my-rook-config
1
0
Fork 0
forked from rook/rook
Code Pull Requests Activity

Add object and file system walkthroughs

Browse Source
This commit is contained in:
Travis Nielsen
2017-04-17 14:31:02 -07:00
parent c34d44b502
commit 6f51db6338
7 changed files with 351 additions and 78 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Documentation/k8s-block.md
+62
View File
@@ -0,0 +1,62 @@
# Block Storage Quickstart
Block storage allows you to mount storage to a single pod.
### Prerequisites
This guide assumes you have created a Rook cluster as explained in the main [Kubernetes guide](kubernetes.md)
### Provision Storage
Before Rook can start provisioning storage, a StorageClass and its storage pool need to be created. This is needed for Kubernetes to interoperate with Rook for provisioning persistent volumes. The rook-storageclass.yaml sample will create the storage pool automatically. For more options on pools, see the documentation on [creating storage pools](pool-tpr.md).
Rook already creates a default admin and rbd user, whose secrets are specified in the sample [rook-storageclass.yaml](/demo/kubernetes/rook-storageclass.yaml). Now we just need to specify the Ceph monitor endpoints (requires `jq`):
```bash
cd demo/kubernetes
export MONS=$(kubectl -n rook get pod mon0 mon1 mon2 -o json|jq ".items[].status.podIP"|tr -d "\""|sed -e 's/$/:6790/'|paste -s -d, -)
sed 's#INSERT_HERE#'$MONS'#' rook-storageclass.yaml | kubectl create -f -
```
**NOTE:** We are working on streamlining the experience and removing the need for this step. See [#355](https://github.com/rook/rook/issues/355).
### Consume the storage
We create a sample app to consume the block storage provisioned by Rook with the classic wordpress and mysql apps.
Both of these apps will make use of block volumes provisioned by Rook.
Start mysql and wordpress from the `demo/kubernetes` folder:
```bash
kubectl create -f mysql.yaml
kubectl create -f wordpress.yaml
```
Both of these apps create a block volume and mount it to their respective pod. You can see the Kubernetes volume claims by running the following:
```bash
$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESSMODES AGE
mysql-pv-claim Bound pvc-95402dbc-efc0-11e6-bc9a-0cc47a3459ee 20Gi RWO 1m
wp-pv-claim Bound pvc-39e43169-efc1-11e6-bc9a-0cc47a3459ee 20Gi RWO 1m
```
Once the wordpress and mysql pods are in the `Running` state, get the cluster IP of the wordpress app and enter it in your brower:
```bash
$ kubectl get svc wordpress
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
wordpress 10.3.0.155 <pending> 80:30841/TCP 2m
```
You should see the wordpress app running.
**NOTE:** When running in a vagrant environment, there will be no external IP address to reach wordpress with. You will only be able to reach wordpress via the `CLUSTER-IP` from inside the Kubernetes cluster.
### Teardown
To clean up all the artifacts created by the block demo:
```
kubectl delete -f wordpress.yaml
kubectl delete -f mysql.yaml
kubectl delete -n rook rookpool replicapool
kubectl delete storageclass rook-block
```
Documentation/k8s-filesystem.md
+102
View File
@@ -0,0 +1,102 @@
# Shared File System Quickstart
A shared file system can be mounted read-write from multiple pods. This may be useful for applications which can be clustered using a shared filesystem.
This example runs a shared file system for the [kube-registry](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/registry).
### Prerequisites
This guide assumes you have created a Rook cluster and pool as explained in the main [Kubernetes guide](kubernetes.md)
## Rook Client
Setting up the Rook file system currently requires the Rook client. This will be simplified in the future with a TPR for the object stores.
```bash
kubectl create -f rook-client/rook-client.yml
# Starting the pod may take a couple minutes, so check to see when it's ready:
kubectl -n rook get pod rook-client
# Connect to the rook-client pod
kubectl -n rook exec rook-client -it bash
# Confirm the rook client can connect to the cluster
rook status
```
## Create the File System
Create the file system with the default pools.
```bash
rook filesystem create --name registryFS
```
### Optional: Adjust pool paramaters
By default the pools do not have any redundancy. To create another copy of the data, let's set the replication to 2.
First we will launch the [Rook toolbox](toolbox.md#running-the-toolbox) in order to run `ceph` commands.
```bash
# Start the Rook toolbox in order to run Ceph commands (the yml is found in the toolbox folder)
cd toolbox
kubectl create -f rook-tools.yml
# Verify the toolbox is running
kubectl -n rook get pod rook-tools
# Connect to the toolbox
kubectl -n rook exec -it rook-tools bash
```
Now we can modify the pool size
```bash
ceph osd pool set registryFS-data size 2
ceph osd pool set registryFS-metadata size 2
```
### Optional: Copy admin key to desired namespace
If you are consuming the filesystem from a namespace other than `rook` you will need to copy the key to the desired namespace.
In this example we are copying to the `kube-system` namespace.
```bash
kubectl get secret rook-admin -n rook -o json | jq '.metadata.namespace = "kube-system"' | kubectl apply -f -
```
## Deploy the Application
The kube-registry yaml is defined [here](/demo/kubernetes/kube-registry.yaml). We will need to update the yaml with the monitor IP addresses with the following commands.
In the future this step will be improved with a Rook volume plugin.
```bash
cd demo/kubernetes
export MONS=$(kubectl -n rook get pod mon0 mon1 mon2 -o json|jq ".items[].status.podIP"|tr -d "\""|sed -e 's/$/:6790/'|paste -s -d, -)
sed "s/INSERT_MONS_HERE/$MONS/g" kube-registry.yaml | kubectl create -f -
```
You now have a docker registry which is HA with persistent storage.
### Test the storage
Verify that kube-registry is using the filesystem that was configured above.
```bash
# Start the rook toolbox
kubectl -n rook exec rook-tools -it bash
# Mount the same filesystem that the kube-registry is using
mkdir /tmp/registry
rook filesystem mount --name registryFS --path /tmp/registry
# Here you should see a directory called docker created by the registry
ls /tmp/registry
# Cleanup the filesystem mount
rook filesystem unmount --path /tmp/registry
rmdir /tmp/registry
```
### Teardown
To clean up all the artifacts created by the file system demo:
```bash
kubectl -n kube-system delete secret rook-admin
kubectl delete -f kube-registry.yaml
```
Documentation/k8s-object.md
+80
View File
@@ -0,0 +1,80 @@
# Object Storage Quickstart
Object storage exposes an S3 API to the storage cluster for applications to put and get data.
### Prerequisites
This guide assumes you have created a Rook cluster as explained in the main [Kubernetes guide](kubernetes.md)
## Rook Client
Setting up the object storage currently requires the Rook client. This will be simplified in the future with a TPR for the object stores.
```bash
kubectl create -f rook-client/rook-client.yml
# Starting the pod may take a couple minutes, so check to see when it's ready:
kubectl -n rook get pod rook-client
# Connect to the rook-client pod
kubectl -n rook exec rook-client -it bash
# Confirm the rook client can connect to the cluster
rook status
```
## Create the Object Store and User
Now we will create the object store, which starts the RGW service in the cluster with the S3 API.
From within the rook client container, run the following:
```bash
# Create an object storage instance in the cluster
rook object create
# Create an object storage user. The first user may take a minute to create.
# If it times out, run the same command again to confirm that it finished.
rook object user create rook-user "A rook rgw User"
```
The object store is now available for pods to connect by using the creds of `rook-user`.
### Environment Variables
If your s3 client uses environment variables, the client can print them for you
```bash
rook object connection rook-user --format env-var
```
See the [Object Storage](client.md#object-storage) documentation for more steps on consuming the object storage.
## Access External to the Cluster
Rook sets up the object storage so pods will have access internal to the cluster. If your applications are running outside the cluster,
you will need to setup an external service through a `NodePort`.
First, note the service that exposes RGW internal to the cluster. We will leave this service intact and create a new service for external access.
```bash
$ kubectl -n rook get service rgw
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
rgw 10.3.0.248 <none> 53390/TCP 45s
```
Now create the external service:
```bash
cd demo/kubernetes
kubectl create -f rgw-external.yaml
```
See both rgw services running and notice what port the external service is running on:
```bash
$ kubectl -n rook get service rgw rgw-external
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
rgw 10.3.0.248 <none> 53390/TCP 1m
rgw-external 10.3.0.146 <nodes> 53390:30711/TCP 1m
```
Internally the rgw service is running on port `53390`. The external port in this case is `30711`. Now you can access the object store from anywhere! All you need is the hostname for any machine in the cluster, the external port, and the user credentials.
If you're testing on the [coreos-kubernetes vagrant environment](k8s-pre-reqs.md#new-local-kubernetes-cluster), you can verify it is working from your host:
- If running in the single-node cluster:
- `curl 172.17.4.99:30711`
- If running in the multi-node cluster:
- `curl 172.17.4.101:30711`
Documentation/kubernetes.md
+23 -64
View File
@@ -18,24 +18,22 @@ Note that we are striving for even more smooth integration with Kubernetes in th
With your Kubernetes cluster running, Rook can be setup and deployed by simply creating the [rook-operator](/demo/kubernetes/rook-operator.yaml) deployment and creating a [rook cluster](/demo/kubernetes/rook-cluster.yaml).
```
```bash
cd demo/kubernetes
kubectl create -f rook-operator.yaml
```
This will start the rook-operator pod. Verify that it is in the `Running` state before proceeding:
```
# This will start the rook-operator pod. Verify that it is in the `Running` state before proceeding:
kubectl get pod | grep rook-operator
```
Now that the rook-operator pod is in the `Running` state, we can create the Rook cluster. See the documentation on [configuring the cluster](cluster-tpr.md).
```
Now that the rook-operator pod is running, we can create the Rook cluster. See the documentation on [configuring the cluster](cluster-tpr.md).
```bash
kubectl create -f rook-cluster.yaml
```
Use `kubectl` to list pods in the rook namespace. You should be able to see the following once they are all running:
Use `kubectl` to list pods in the rook namespace. You should be able to see the following pods once they are all running:
```
```bash
$ kubectl -n rook get pod
NAME READY STATUS RESTARTS AGE
mon0 1/1 Running 0 1m
@@ -46,83 +44,44 @@ osd-6jmph 1/1 Running 0 1m
rook-api-1709486253-gvdnc 1/1 Running 0 1m
```
### Provision Storage
Before Rook can start provisioning storage, a StorageClass and its storage pool need to be created. This is needed for Kubernetes to interoperate with Rook for provisioning persistent volumes. The rook-storageclass.yaml sample will create the storage pool automatically. For more options on pools, see the documentation on [creating storage pools](pool-tpr.md).
## Storage
For a walkthrough of the three types of storage exposed by Rook, see the guides for:
- **[Block](k8s-block.md)**: Create block storage to be consumed by a pod
- **[Shared File System](k8s-filesystem.md)**: Create a file system to be shared across multiple pods
- **[Object](k8s-object.md)**: Create an object store that is accessible inside or outside the Kubernetes cluster
Rook already creates a default admin and rbd user, whose secrets are specified in the sample [rook-storageclass.yaml](/demo/kubernetes/rook-storageclass.yaml). Now we just need to specify the Ceph monitor endpoints (requires `jq`):
```
export MONS=$(kubectl -n rook get pod mon0 mon1 mon2 -o json|jq ".items[].status.podIP"|tr -d "\""|sed -e 's/$/:6790/'|paste -s -d, -)
sed 's#INSERT_HERE#'$MONS'#' rook-storageclass.yaml | kubectl create -f -
```
**NOTE:** In the v0.4 release we plan to expose monitors via DNS/service names instead of IP address (see [#355](https://github.com/rook/rook/issues/355)), which will streamline the experience and remove the need for this step.
### Consume the storage
Now that Rook is running and integrated with Kubernetes, we can create a sample app to consume the block storage provisioned by Rook. We will create the classic wordpress and mysql apps.
Both these apps will make use of block volumes provisioned by Rook.
Start mysql and wordpress from the `demo/kubernetes` folder:
```
kubectl create -f mysql.yaml
kubectl create -f wordpress.yaml
```
Both of these apps create a block volume and mount it to their respective pod. You can see the Kubernetes volume claims by running the following:
```
$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESSMODES AGE
mysql-pv-claim Bound pvc-95402dbc-efc0-11e6-bc9a-0cc47a3459ee 20Gi RWO 1m
wp-pv-claim Bound pvc-39e43169-efc1-11e6-bc9a-0cc47a3459ee 20Gi RWO 1m
```
Once the wordpress and mysql pods are in the `Running` state, get the cluster IP of the wordpress app and enter it in your brower:
```
$ kubectl get svc wordpress
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
wordpress 10.3.0.155 <pending> 80:30841/TCP 2m
```
You should see the wordpress app running.
**NOTE:** When running in a vagrant environment, there will be no external IP address to reach wordpress with. You will only be able to reach wordpress via the `CLUSTER-IP` from inside the Kubernetes cluster.
## Tools
### Rook Client
You also have the option to use the `rook` client tool directly by running it in a pod that can be started in the cluster with:
```
```bash
kubectl create -f rook-client/rook-client.yml
```
Starting the rook-client pod will take a bit of time to download the container, so you can check to see when it's ready with (it should be in the `Running` state):
```
# Starting the rook-client pod will take a bit of time to download the container, so check when it's in the Running state
kubectl -n rook get pod rook-client
```
Connect to the rook-client pod and verify the `rook` client can talk to the cluster:
```
# Connect to the rook-client pod
kubectl -n rook exec rook-client -it bash
# Verify the rook client can talk to the cluster:
rook node ls
```
At this point, you can use the `rook` tool along with some [simple steps to create and manage block, file and object storage](client.md).
### Advanced Configuration and Troubleshooting
We have created a toolbox container that contains the full suite of Ceph clients for debugging and troubleshooting your Rook cluster. Please see the [toolbox readme](toolbox.md) for setup and usage information.
### Monitoring
Each Rook cluster has some built in metrics collectors/exporters for monitoring with [Prometheus](https://prometheus.io/).
To learn how to set up monitoring for your Rook cluster, you can follow the steps in the [monitoring guide](./k8s-monitoring.md).
### Teardown
To clean up all the artifacts created by the demo, run the following:
```
kubectl delete -f wordpress.yaml
kubectl delete -f mysql.yaml
## Teardown
To clean up all the artifacts created by the demo, *first cleanup the resources from the block, file, and object walkthroughs* (unmount volumes, delete volume claims, etc), then run the following:
```bash
kubectl delete deployment rook-operator
kubectl delete -n rook rookcluster rook
kubectl delete -n rook rookpool replicapool
kubectl delete thirdpartyresources rookcluster.rook.io rookpool.rook.io
kubectl delete storageclass rook-block
kubectl delete secret rook-rbd-user
kubectl delete namespace rook
```
Documentation/toolbox.md
+17 -14
View File
@@ -3,45 +3,48 @@ The rook toolbox is a container with common tools used for rook debugging and te
## Installing more tools
The rook toolbox is based on Ubuntu, so more tools of your choosing can be easily installed with `apt-get`. For example, to install `telnet`:
```
```bash
apt-get update
apt-get install telnet
```
## Running the toolbox
### Kubernetes
## Running the Toolbox in Kubernetes
The rook toolbox can run as a pod in a Kubernetes cluster. First, ensure you have a running Kubernetes cluster with rook deployed (see the [Kubernetes](kubernetes.md) instructions).
From this directory, launch the rook-tools pod:
```
```bash
cd toolbox
kubectl create -f rook-tools.yml
```
Wait for the toolbox pod to download its container and get to the `running` state:
```
```bash
kubectl -n rook get pod rook-tools
```
Once the rook-tools pod is running, you can connect to it with:
```
```bash
kubectl -n rook exec -it rook-tools bash
```
All available tools in the toolbox are ready for your troubleshooting needs. Example:
```
```bash
rook status
ceph df
rados df
```
When you are completely done with the toolbox, you can clean it up by running:
```
```bash
kubectl delete -f rook-tools.yml
```
## Running the Toolbox for Standalone
### Container Linux by CoreOS
To use the rook toolbox on CoreOS, first add the following values to the toolbox config file:
```
```bash
cat >~/.toolboxrc <<EOL
TOOLBOX_DOCKER_IMAGE=quay.io/rook/toolbox
TOOLBOX_DOCKER_TAG=latest
@@ -49,29 +52,29 @@ EOL
```
Then launch the toolbox as usual:
```
```bash
toolbox
```
#### Ceph Tools
To use the ceph tools from a rook host, launch the toolbox with the following options:
```
```bash
toolbox --bind=/var/lib/rook:/var/lib/rook /toolbox/entrypoint.sh
```
Then you can run `ceph` and `rados` commands like usual:
```
```bash
ceph df
rados df
```
### Other Linux Distros
The rook toolbox container can simply be run directly with `docker` on other Linux distros:
```
```bash
docker run -it quay.io/rook/toolbox
```
#### Ceph Tools
To run ceph tools such as `ceph` and `rados`, run the container with the following options:
```
```bash
docker run -it --network=host -v /var/lib/rook:/var/lib/rook quay.io/rook/toolbox
```
demo/kubernetes/kube-registry.yaml
+48
View File
@@ -0,0 +1,48 @@
apiVersion: v1
kind: ReplicationController
metadata:
name: kube-registry-v0
namespace: kube-system
labels:
k8s-app: kube-registry
version: v0
kubernetes.io/cluster-service: "true"
spec:
replicas: 3
selector:
k8s-app: kube-registry
version: v0
template:
metadata:
labels:
k8s-app: kube-registry
version: v0
kubernetes.io/cluster-service: "true"
spec:
containers:
- name: registry
image: registry:2
resources:
limits:
cpu: 100m
memory: 100Mi
env:
- name: REGISTRY_HTTP_ADDR
value: :5000
- name: REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY
value: /var/lib/registry
volumeMounts:
- name: image-store
mountPath: /var/lib/registry
ports:
- containerPort: 5000
name: registry
protocol: TCP
volumes:
- name: image-store
cephfs:
monitors:
- INSERT_MONS_HERE
user: admin
secretRef:
name: rook-admin
demo/kubernetes/rgw-external.yaml
+19
View File
@@ -0,0 +1,19 @@
apiVersion: v1
kind: Service
metadata:
name: rgw-external
namespace: rook
labels:
app: rgw
rook_cluster: rook
spec:
ports:
- name: rgw
port: 53390
protocol: TCP
targetPort: 53390
selector:
app: rgw
rook_cluster: rook
sessionAffinity: None
type: NodePort