diff --git a/README.md b/README.md index 52a0aa472..019bf8957 100644 --- a/README.md +++ b/README.md @@ -133,26 +133,26 @@ On a modern Linux host run the following: 3. Use an S3 compatible client to create a bucket in the object store ```bash - s3cmd mb --no-ssl --host=127.0.0.1:53390 --host-bucket= s3://rookbucket + s3cmd mb --no-ssl --host=${AWS_ENDPOINT} --host-bucket= s3://rookbucket ``` 4. List all buckets in the object store ```bash - s3cmd ls --no-ssl --host=127.0.0.1:53390 --host-bucket= + s3cmd ls --no-ssl --host=${AWS_ENDPOINT} --host-bucket= ``` 5. Upload a file to the newly created bucket ```bash echo "Hello Rook!" > /tmp/rookObj - s3cmd put /tmp/rookObj --no-ssl --host=127.0.0.1:53390 --host-bucket= s3://rookbucket + s3cmd put /tmp/rookObj --no-ssl --host=${AWS_ENDPOINT} --host-bucket= s3://rookbucket ``` 6. Download and verify the file from the bucket ```bash - s3cmd get s3://rookbucket/rookObj /tmp/rookObj-download --no-ssl --host=127.0.0.1:53390 --host-bucket= + s3cmd get s3://rookbucket/rookObj /tmp/rookObj-download --no-ssl --host=${AWS_ENDPOINT} --host-bucket= cat /tmp/rookObj-download ``` diff --git a/demo/kubernetes/README.md b/demo/kubernetes/README.md index f57ed0703..47db04518 100644 --- a/demo/kubernetes/README.md +++ b/demo/kubernetes/README.md @@ -4,74 +4,117 @@ A demo of persistent block storage from rook within a [Kubernetes](http://kubern ## Prerequisites -A running Kubernetes cluster is required. See the (rook-demo branch) in the rook fork of the coreos-kubernetes repo. -The changes to coreos-kubernetes necessary to support the scenario include: -1. Devices created in the vagrant environment to provide storage in the cluster. -2. Access to modprobe, necessary for using the rbd volume plugin (). +A running Kubernetes cluster is required. First clone (rook-demo branch in the rook fork of the coreos-kubernetes repo): +``` +git clone https://github.com/rook/coreos-kubernetes.git +cd coreos-kubernetes +git checkout rook-demo +``` + +The changes found in that branch that facilitate this demo scenario include: +1. Devices created in the vagrant environment to provide storage in the cluster. +2. Access to modprobe, necessary for using the rbd volume plugin (). ### Multi-node deployment If you are using the rook-demo branch you can get a vagrant kubernetes cluster running like this (from the root of the repo): +``` +cd multi-node/vagrant +vagrant up +export KUBECONFIG="$(pwd)/kubeconfig" +kubectl config use-context vagrant-multi +``` - cd multi-node/vagrant - vagrant up - export KUBECONFIG="$(pwd)/kubeconfig" - kubectl config use-context vagrant-multi - -Verify that kubernetes is done initializing. - - kubectl get nodes - +Then wait for the cluster to come up and verify that kubernetes is done initializing (be patient, it takes a bit): +``` +kubectl get nodes +``` ### Single-node deployment -This walkthrough is using the multi-node deployment. To use the single-node deployment, the main change to the instructions is the ip address. +This walkthrough is using the multi-node deployment. To instead use the single-node deployment, the main change to the instructions is the IP address. Replace `172.17.4.101` with `172.17.4.99` in these instructions and in `rook.yml`. ## Starting rook - - cd /demo/kubernetes - export TOKEN=$(curl -s -w "\n" 'https://discovery.etcd.io/new?size=1') - kubectl create configmap rookd --from-literal=discovery-token=$TOKEN - kubectl create -f rook.yml +``` +cd /demo/kubernetes +export TOKEN=$(curl -s -w "\n" 'https://discovery.etcd.io/new?size=1') +kubectl create configmap rookd --from-literal=discovery-token=$TOKEN +kubectl create -f rook.yml +``` This generates a discovery token so the nodes can find each other and then starts rook. `rookd` will start up on each node and form a cluster. **Note**: In environments other than the coreos-kubernetes vagrant cluster, to have external access via the rook cli, the `rook.yml` will need to be edited and the `externalIPs` array will need to be changed to an appropriate externally routable ip to one of the cluster nodes. -## Using the storage +## Using the rook storage with a rook client pod -First let's setup access to the rook cluster via the command line client. You can download a pre-built from [github releases](https://github.com/rook/rook/releases) or [build from source](https://github.com/rook/rook/blob/master/build/README.md). (If you are not using coreos-kubernetes vagrant substitute the ip from above that you put in the `rook.yml` for `ROOK_API_SERVER_ENDPOINT`) +First let's provision some useful storage in the rook cluster by using the `rook` command line client. For convenience, there is a pod that contains the latest rook client. It can be started in the cluster with: +``` +kubectl create -f rook-client/rook-client.yml +``` - export ROOK_API_SERVER_ENDPOINT=172.17.4.201:8124 - rook status - -It may take a moment for the rook cluster to come up and `rook status` to complete successfully. Once it is successful we will want to fetch the rook secret that is used to mount rook block devices and store it in a Kubernetes secret. This only needs to be done once for a given cluster. +Starting the rook-client pod will take a bit of time to download the container, so you can check to see when it's ready with (it should be in the Running state): +``` +kubectl get pod rook-client +``` - SECRET=$(curl -s $ROOK_API_SERVER_ENDPOINT/client | jq -r '.secretKey') - kubectl create secret generic rookd --from-literal=key=$SECRET --type kubernetes.io/rbd +Connect to the rook-client pod and verify the `rook` client can talk to the `rookd` cluster: +``` +kubectl exec rook-client -it bash +rook node ls +``` -To use a block device in the kubernetes cluster you will first need to create a block image for that device. +At this point, you can follow the steps in the rook [quickstart](/README.md#block-storage) to create and use block, file and object storage. - rook block create --name demoblock --size 1073741824 +## Running a pod with persistent rook storage +To use a block device in the kubernetes cluster you will first need to create a block image for that device. **In your rook-client pod** from above, run: +``` +rook block create --name demoblock --size 1073741824 +``` -Then we must fetch the mon endpoints and substitute them in to the `mysql.yml` for the example mysql service and pipe it into `kubectl` to create the pod: - - export MONS=$(curl -s $ROOK_API_SERVER_ENDPOINT/client | jq -c '.monAddresses') - sed 's#INSERT_HERE#'$MONS'#' mysql.yml | kubectl create -f - +Now, **back on your host machine**, set a variable so subsequent commands can find the rook API server: +``` +export ROOK_API_SERVER_ENDPOINT=172.17.4.201:8124 +``` +**NOTE:** If you are not using coreos-kubernetes vagrant, you need to substitute the IP from above that you put in the `rook.yml` for `ROOK_API_SERVER_ENDPOINT` -Verify that the mysql pod has the rbd volume mounted. +Now, we will want to fetch the rook secret that is used to mount rook block devices and store it in a Kubernetes secret. This only needs to be done once for a given cluster: +``` +SECRET=$(curl -s $ROOK_API_SERVER_ENDPOINT/client | jq -r '.secretKey') +kubectl create secret generic rookd --from-literal=key=$SECRET --type kubernetes.io/rbd +``` - kubectl describe pod mysql +Then we must fetch the monitor endpoints and substitute them in to the `mysql.yml` file for the example mysql service and pipe it into `kubectl` to create the pod: +``` +export MONS=$(curl -s $ROOK_API_SERVER_ENDPOINT/client | jq -c '.monAddresses') +sed 's#INSERT_HERE#'$MONS'#' mysql.yml | kubectl create -f - +``` + +Verify that the mysql pod has the rbd volume mounted (it may take a bit to first pull the mysql image): +``` +kubectl describe pod mysql +``` + +With mysql pod running, we now have a pod using persistent storage from the `rook` cluster in the form of a block device. ## Teardown demo +``` +kubectl delete pod rook-client +kubectl delete pod mysql +kubectl delete secret rookd +kubectl delete configmap rookd +kubectl delete -f rook.yml +``` - kubectl delete pod mysql - kubectl delete secret rookd - kubectl delete configmap rookd - kubectl delete -f rook.yml +kubectl should show no pods running at all (be patient, they make take a bit to terminate completely). You can walk through the demo again if you desire. -## Todo +To fully clean up the environment (and **destroy all data**), run the following from the same directory you initially ran `vagrant up` in: +``` +vagrant destroy -f +``` + +## TODO * Kubernetes volume plugin for rook that avoids the need for the mon endpoints being hard-coded into the pod spec * a better solution for the modprobe issue () diff --git a/demo/kubernetes/rook-client/Dockerfile b/demo/kubernetes/rook-client/Dockerfile new file mode 100644 index 000000000..3f6e69b93 --- /dev/null +++ b/demo/kubernetes/rook-client/Dockerfile @@ -0,0 +1,16 @@ +FROM ubuntu +MAINTAINER Jared Watts + +ENV ROOK_VERSION 0.2.1 + +RUN apt-get update && \ + apt-get install -yq --no-install-recommends wget s3cmd kmod module-init-tools sudo + +RUN wget --no-check-certificate -P /tmp https://github.com/rook/rook/releases/download/v${ROOK_VERSION}/rook-v${ROOK_VERSION}-linux-amd64.tar.gz && \ + gunzip /tmp/rook-v${ROOK_VERSION}-linux-amd64.tar.gz && \ + tar -xf /tmp/rook-v${ROOK_VERSION}-linux-amd64.tar -C /tmp && \ + mv /tmp/rook /usr/local/bin/ && \ + chmod 0755 /usr/local/bin/rook && \ + rm -rf /tmp/rook* + + diff --git a/demo/kubernetes/rook-client/rook-client.yml b/demo/kubernetes/rook-client/rook-client.yml new file mode 100644 index 000000000..ce560cfee --- /dev/null +++ b/demo/kubernetes/rook-client/rook-client.yml @@ -0,0 +1,32 @@ +apiVersion: v1 +kind: Pod +metadata: + name: rook-client +spec: + containers: + - name: rook-client + image: quay.io/rook/k8s-demo + imagePullPolicy: IfNotPresent + command: ["sleep", "36500d"] + securityContext: + privileged: true + env: + - name: ROOK_API_SERVER_ENDPOINT + value: rookd-api:8124 + volumeMounts: + - mountPath: /dev + name: dev + - mountPath: /sys + name: sys + - mountPath: /lib/modules + name: libmodules + volumes: + - name: dev + hostPath: + path: /dev + - name: sys + hostPath: + path: /sys + - name: libmodules + hostPath: + path: /lib/modules \ No newline at end of file diff --git a/demo/kubernetes/rook.yml b/demo/kubernetes/rook.yml index de25f1281..064ccd513 100644 --- a/demo/kubernetes/rook.yml +++ b/demo/kubernetes/rook.yml @@ -1,10 +1,10 @@ apiVersion: v1 kind: Service metadata: - name: rest + name: rookd-api labels: app: rook - role: rest + role: rookd-api spec: type: NodePort externalIPs: [ "172.17.4.201" ]