kubernetes pod spec and documentation for running a rook-client pod

This commit is contained in:
Jared Watts committed 2016-12-15 15:45:21 -08:00
1 parent 71dec22235
commit 6bed484bad
5 files changed
+137 -46

No files matched your search

+4 -4
View File
@@ -133,26 +133,26 @@ On a modern Linux host run the following:
3. Use an S3 compatible client to create a bucket in the object store
```bash
s3cmd mb --no-ssl --host=127.0.0.1:53390 --host-bucket= s3://rookbucket
s3cmd mb --no-ssl --host=${AWS_ENDPOINT} --host-bucket= s3://rookbucket
```
4. List all buckets in the object store
```bash
s3cmd ls --no-ssl --host=127.0.0.1:53390 --host-bucket=
s3cmd ls --no-ssl --host=${AWS_ENDPOINT} --host-bucket=
```
5. Upload a file to the newly created bucket
```bash
echo "Hello Rook!" > /tmp/rookObj
s3cmd put /tmp/rookObj --no-ssl --host=127.0.0.1:53390 --host-bucket= s3://rookbucket
s3cmd put /tmp/rookObj --no-ssl --host=${AWS_ENDPOINT} --host-bucket= s3://rookbucket
```
6. Download and verify the file from the bucket
```bash
s3cmd get s3://rookbucket/rookObj /tmp/rookObj-download --no-ssl --host=127.0.0.1:53390 --host-bucket=
s3cmd get s3://rookbucket/rookObj /tmp/rookObj-download --no-ssl --host=${AWS_ENDPOINT} --host-bucket=
cat /tmp/rookObj-download
```
+83 -40
View File
@@ -4,74 +4,117 @@ A demo of persistent block storage from rook within a [Kubernetes](http://kubern
## Prerequisites
A running Kubernetes cluster is required. See the <https://github.com/rook/coreos-kubernetes/tree/rook-demo> (rook-demo branch) in the rook fork of the coreos-kubernetes repo.
The changes to coreos-kubernetes necessary to support the scenario include:
1. Devices created in the vagrant environment to provide storage in the cluster.
2. Access to modprobe, necessary for using the rbd volume plugin (<https://github.com/kubernetes/kubernetes/issues/23924>).
A running Kubernetes cluster is required. First clone <https://github.com/rook/coreos-kubernetes/tree/rook-demo> (rook-demo branch in the rook fork of the coreos-kubernetes repo):
```
git clone https://github.com/rook/coreos-kubernetes.git
cd coreos-kubernetes
git checkout rook-demo
```
The changes found in that branch that facilitate this demo scenario include:
1. Devices created in the vagrant environment to provide storage in the cluster.
2. Access to modprobe, necessary for using the rbd volume plugin (<https://github.com/kubernetes/kubernetes/issues/23924>).
### Multi-node deployment
If you are using the rook-demo branch you can get a vagrant kubernetes cluster running like this (from the root of the repo):
```
cd multi-node/vagrant
vagrant up
export KUBECONFIG="$(pwd)/kubeconfig"
kubectl config use-context vagrant-multi
```
cd multi-node/vagrant
vagrant up
export KUBECONFIG="$(pwd)/kubeconfig"
kubectl config use-context vagrant-multi
Verify that kubernetes is done initializing.
kubectl get nodes
Then wait for the cluster to come up and verify that kubernetes is done initializing (be patient, it takes a bit):
```
kubectl get nodes
```
### Single-node deployment
This walkthrough is using the multi-node deployment. To use the single-node deployment, the main change to the instructions is the ip address.
This walkthrough is using the multi-node deployment. To instead use the single-node deployment, the main change to the instructions is the IP address.
Replace `172.17.4.101` with `172.17.4.99` in these instructions and in `rook.yml`.
## Starting rook
cd <rook>/demo/kubernetes
export TOKEN=$(curl -s -w "\n" 'https://discovery.etcd.io/new?size=1')
kubectl create configmap rookd --from-literal=discovery-token=$TOKEN
kubectl create -f rook.yml
```
cd <rook>/demo/kubernetes
export TOKEN=$(curl -s -w "\n" 'https://discovery.etcd.io/new?size=1')
kubectl create configmap rookd --from-literal=discovery-token=$TOKEN
kubectl create -f rook.yml
```
This generates a discovery token so the nodes can find each other and then starts rook. `rookd` will start up on each node and form a cluster.
**Note**: In environments other than the coreos-kubernetes vagrant cluster, to have external access via the rook cli, the `rook.yml` will need to be edited and the `externalIPs` array will need to be changed to an appropriate externally routable ip to one of the cluster nodes.
## Using the storage
## Using the rook storage with a rook client pod
First let's setup access to the rook cluster via the command line client. You can download a pre-built from [github releases](https://github.com/rook/rook/releases) or [build from source](https://github.com/rook/rook/blob/master/build/README.md). (If you are not using coreos-kubernetes vagrant substitute the ip from above that you put in the `rook.yml` for `ROOK_API_SERVER_ENDPOINT`)
First let's provision some useful storage in the rook cluster by using the `rook` command line client. For convenience, there is a pod that contains the latest rook client. It can be started in the cluster with:
```
kubectl create -f rook-client/rook-client.yml
```
export ROOK_API_SERVER_ENDPOINT=172.17.4.201:8124
rook status
It may take a moment for the rook cluster to come up and `rook status` to complete successfully. Once it is successful we will want to fetch the rook secret that is used to mount rook block devices and store it in a Kubernetes secret. This only needs to be done once for a given cluster.
Starting the rook-client pod will take a bit of time to download the container, so you can check to see when it's ready with (it should be in the Running state):
```
kubectl get pod rook-client
```
SECRET=$(curl -s $ROOK_API_SERVER_ENDPOINT/client | jq -r '.secretKey')
kubectl create secret generic rookd --from-literal=key=$SECRET --type kubernetes.io/rbd
Connect to the rook-client pod and verify the `rook` client can talk to the `rookd` cluster:
```
kubectl exec rook-client -it bash
rook node ls
```
To use a block device in the kubernetes cluster you will first need to create a block image for that device.
At this point, you can follow the steps in the rook [quickstart](/README.md#block-storage) to create and use block, file and object storage.
rook block create --name demoblock --size 1073741824
## Running a pod with persistent rook storage
To use a block device in the kubernetes cluster you will first need to create a block image for that device. **In your rook-client pod** from above, run:
```
rook block create --name demoblock --size 1073741824
```
Then we must fetch the mon endpoints and substitute them in to the `mysql.yml` for the example mysql service and pipe it into `kubectl` to create the pod:
export MONS=$(curl -s $ROOK_API_SERVER_ENDPOINT/client | jq -c '.monAddresses')
sed 's#INSERT_HERE#'$MONS'#' mysql.yml | kubectl create -f -
Now, **back on your host machine**, set a variable so subsequent commands can find the rook API server:
```
export ROOK_API_SERVER_ENDPOINT=172.17.4.201:8124
```
**NOTE:** If you are not using coreos-kubernetes vagrant, you need to substitute the IP from above that you put in the `rook.yml` for `ROOK_API_SERVER_ENDPOINT`
Verify that the mysql pod has the rbd volume mounted.
Now, we will want to fetch the rook secret that is used to mount rook block devices and store it in a Kubernetes secret. This only needs to be done once for a given cluster:
```
SECRET=$(curl -s $ROOK_API_SERVER_ENDPOINT/client | jq -r '.secretKey')
kubectl create secret generic rookd --from-literal=key=$SECRET --type kubernetes.io/rbd
```
kubectl describe pod mysql
Then we must fetch the monitor endpoints and substitute them in to the `mysql.yml` file for the example mysql service and pipe it into `kubectl` to create the pod:
```
export MONS=$(curl -s $ROOK_API_SERVER_ENDPOINT/client | jq -c '.monAddresses')
sed 's#INSERT_HERE#'$MONS'#' mysql.yml | kubectl create -f -
```
Verify that the mysql pod has the rbd volume mounted (it may take a bit to first pull the mysql image):
```
kubectl describe pod mysql
```
With mysql pod running, we now have a pod using persistent storage from the `rook` cluster in the form of a block device.
## Teardown demo
```
kubectl delete pod rook-client
kubectl delete pod mysql
kubectl delete secret rookd
kubectl delete configmap rookd
kubectl delete -f rook.yml
```
kubectl delete pod mysql
kubectl delete secret rookd
kubectl delete configmap rookd
kubectl delete -f rook.yml
kubectl should show no pods running at all (be patient, they make take a bit to terminate completely). You can walk through the demo again if you desire.
## Todo
To fully clean up the environment (and **destroy all data**), run the following from the same directory you initially ran `vagrant up` in:
```
vagrant destroy -f
```
## TODO
* Kubernetes volume plugin for rook that avoids the need for the mon endpoints being hard-coded into the pod spec
* a better solution for the modprobe issue (<https://github.com/kubernetes/kubernetes/issues/23924>)
+16
View File
@@ -0,0 +1,16 @@
FROM ubuntu
MAINTAINER Jared Watts <jared.watts@quantum.com>
ENV ROOK_VERSION 0.2.1
RUN apt-get update && \
apt-get install -yq --no-install-recommends wget s3cmd kmod module-init-tools sudo
RUN wget --no-check-certificate -P /tmp https://github.com/rook/rook/releases/download/v${ROOK_VERSION}/rook-v${ROOK_VERSION}-linux-amd64.tar.gz && \
gunzip /tmp/rook-v${ROOK_VERSION}-linux-amd64.tar.gz && \
tar -xf /tmp/rook-v${ROOK_VERSION}-linux-amd64.tar -C /tmp && \
mv /tmp/rook /usr/local/bin/ && \
chmod 0755 /usr/local/bin/rook && \
rm -rf /tmp/rook*
@@ -0,0 +1,32 @@
apiVersion: v1
kind: Pod
metadata:
name: rook-client
spec:
containers:
- name: rook-client
image: quay.io/rook/k8s-demo
imagePullPolicy: IfNotPresent
command: ["sleep", "36500d"]
securityContext:
privileged: true
env:
- name: ROOK_API_SERVER_ENDPOINT
value: rookd-api:8124
volumeMounts:
- mountPath: /dev
name: dev
- mountPath: /sys
name: sys
- mountPath: /lib/modules
name: libmodules
volumes:
- name: dev
hostPath:
path: /dev
- name: sys
hostPath:
path: /sys
- name: libmodules
hostPath:
path: /lib/modules
+2 -2
View File
@@ -1,10 +1,10 @@
apiVersion: v1
kind: Service
metadata:
name: rest
name: rookd-api
labels:
app: rook
role: rest
role: rookd-api
spec:
type: NodePort
externalIPs: [ "172.17.4.201" ]