Sébastien Han
3c6fa79518
This commit adds instructions to develop on a multi-node deployment, which is somehow similar to a production setup. It also proposes a development workflow approach to write/build/test/ship code to Rook. Signed-off-by: Sébastien Han <seb@redhat.com>
143 lines
7.0 KiB
Markdown
143 lines
7.0 KiB
Markdown
---
|
|
title: Multi-Node Test Environment
|
|
weight: 91
|
|
indent: true
|
|
---
|
|
|
|
# Multi-Node Test Environment
|
|
|
|
## Setup expectation
|
|
|
|
There are a bunch of pre-requisites to be able to deploy the following environment. Such as:
|
|
|
|
* A Linux workstation (CentOS or Fedora)
|
|
* KVM/QEMU installation
|
|
|
|
For other Linux distribution, there is no guarantee the following will work.
|
|
However adapting commands (apt/yum/dnf) could just work.
|
|
|
|
## Prerequisites installation
|
|
|
|
On your host machine, execute `tests/scripts/multi-node/rpm-system-prerequisites.sh`.
|
|
|
|
## Deploy Kubernetes with Kubespray
|
|
|
|
Clone it:
|
|
|
|
```bash
|
|
git clone https://github.com/kubernetes-incubator/kubespray
|
|
cd kubespray
|
|
```
|
|
|
|
In order to successfully deploy Kubernetes with Kubespray, you must have this code: https://github.com/kubernetes-incubator/kubespray/pull/2153 and https://github.com/kubernetes-incubator/kubespray/pull/2271.
|
|
|
|
Edit `inventory/group_vars/k8s-cluster.yml` with:
|
|
|
|
```bash
|
|
docker_options: "--insecure-registry=172.17.8.1:5000 --insecure-registry={{ kube_service_addresses }} --graph={{ docker_daemon_graph }} {{ docker_log_opts }}"
|
|
```
|
|
|
|
FYI: `172.17.8.1` is the libvirt bridge IP, so it's reachable from all your virtual machines.
|
|
This means a registry running on the host machine is reachable from the virtual machines running the Kubernetes cluster.
|
|
|
|
Create Vagrant's variable directory:
|
|
|
|
```bash
|
|
mkdir vagrant/
|
|
```
|
|
|
|
Put `tests/scripts/multi-node/config.rb` in `vagrant/`. You can adapt it at will.
|
|
Feel free to adapt `num_instances`.
|
|
|
|
Deploy!
|
|
|
|
```bash
|
|
vagrant up --no-provision ; vagrant provision
|
|
```
|
|
|
|
Go grab a coffee:
|
|
|
|
```
|
|
PLAY RECAP *********************************************************************
|
|
k8s-01 : ok=351 changed=111 unreachable=0 failed=0
|
|
k8s-02 : ok=230 changed=65 unreachable=0 failed=0
|
|
k8s-03 : ok=230 changed=65 unreachable=0 failed=0
|
|
k8s-04 : ok=229 changed=65 unreachable=0 failed=0
|
|
k8s-05 : ok=229 changed=65 unreachable=0 failed=0
|
|
k8s-06 : ok=229 changed=65 unreachable=0 failed=0
|
|
k8s-07 : ok=229 changed=65 unreachable=0 failed=0
|
|
k8s-08 : ok=229 changed=65 unreachable=0 failed=0
|
|
k8s-09 : ok=229 changed=65 unreachable=0 failed=0
|
|
|
|
Friday 12 January 2018 10:25:45 +0100 (0:00:00.017) 0:17:24.413 ********
|
|
===============================================================================
|
|
download : container_download | Download containers if pull is required or told to always pull (all nodes) - 192.44s
|
|
kubernetes/preinstall : Update package management cache (YUM) --------- 178.26s
|
|
download : container_download | Download containers if pull is required or told to always pull (all nodes) - 102.24s
|
|
docker : ensure docker packages are installed -------------------------- 57.20s
|
|
download : container_download | Download containers if pull is required or told to always pull (all nodes) -- 52.33s
|
|
kubernetes/preinstall : Install packages requirements ------------------ 25.18s
|
|
download : container_download | Download containers if pull is required or told to always pull (all nodes) -- 23.74s
|
|
download : container_download | Download containers if pull is required or told to always pull (all nodes) -- 18.90s
|
|
download : container_download | Download containers if pull is required or told to always pull (all nodes) -- 15.39s
|
|
kubernetes/master : Master | wait for the apiserver to be running ------ 12.44s
|
|
download : container_download | Download containers if pull is required or told to always pull (all nodes) -- 11.83s
|
|
download : container_download | Download containers if pull is required or told to always pull (all nodes) -- 11.66s
|
|
kubernetes/node : install | Copy kubelet from hyperkube container ------ 11.44s
|
|
download : container_download | Download containers if pull is required or told to always pull (all nodes) -- 11.41s
|
|
download : container_download | Download containers if pull is required or told to always pull (all nodes) -- 11.00s
|
|
docker : Docker | pause while Docker restarts -------------------------- 10.22s
|
|
kubernetes/secrets : Check certs | check if a cert already exists on node --- 6.05s
|
|
kubernetes-apps/network_plugin/flannel : Flannel | Wait for flannel subnet.env file presence --- 5.33s
|
|
kubernetes/master : Master | wait for kube-scheduler -------------------- 5.30s
|
|
kubernetes/master : Copy kubectl from hyperkube container --------------- 4.77s
|
|
[leseb@tarox kubespray]$
|
|
[leseb@tarox kubespray]$
|
|
[leseb@tarox kubespray]$ vagrant ssh k8s-01
|
|
Last login: Fri Jan 12 09:22:18 2018 from 192.168.121.1
|
|
|
|
[vagrant@k8s-01 ~]$ kubectl get nodes
|
|
NAME STATUS ROLES AGE VERSION
|
|
k8s-01 Ready master,node 2m v1.9.0+coreos.0
|
|
k8s-02 Ready node 2m v1.9.0+coreos.0
|
|
k8s-03 Ready node 2m v1.9.0+coreos.0
|
|
k8s-04 Ready node 2m v1.9.0+coreos.0
|
|
k8s-05 Ready node 2m v1.9.0+coreos.0
|
|
k8s-06 Ready node 2m v1.9.0+coreos.0
|
|
k8s-07 Ready node 2m v1.9.0+coreos.0
|
|
k8s-08 Ready node 2m v1.9.0+coreos.0
|
|
k8s-09 Ready node 2m v1.9.0+coreos.0
|
|
```
|
|
|
|
## Development workflow on the host
|
|
|
|
Everything should happen on the host, your development environment will reside on the host machine NOT inside the virtual machines running the Kubernetes cluster.
|
|
|
|
Now, please refer to [https://rook.io/docs/rook/master/development-flow.html](https://rook.io/docs/rook/master/development-flow.html) to setup your development environment (go, git etc).
|
|
|
|
At this stage, Rook should be cloned on your host.
|
|
|
|
From your Rook repository (should be $GOPATH/src/github.com/rook) location execute `bash tests/scripts/multi-node/build-rook.sh`.
|
|
During its execution, `build-rook.sh` will purge all running Rook pods from the cluster, so that your latest container image can be deployed.
|
|
Furthermore, **all Ceph data and config will be purged** as well.
|
|
Ensure that you are done with all existing state on your test cluster before executing `build-rook.sh` as it will clear everything.
|
|
|
|
Each time you build and deploy with `build-rook.sh`, the virtual machines (k8s-0X) will pull the new container image and run your new Rook code.
|
|
You can run `bash tests/scripts/multi-node/build-rook.sh` as many times as you want to rebuild your new rook image and redeploy a cluster that is running your new code.
|
|
|
|
From here, resume your dev, change your code and test it by running `bash tests/scripts/multi-node/build-rook.sh`.
|
|
|
|
|
|
## Teardown
|
|
|
|
Typically, to flush your environment you will run the following from within kubespray's git repository.
|
|
This action will be performed on the host:
|
|
|
|
```bash
|
|
[user@host-machine kubespray]$ vagrant destroy -f
|
|
```
|
|
|
|
Also, if you were using `kubectl` on that host machine, you can resurrect your old configuration by renaming `$HOME/.kube/config.before.rook.$TIMESTAMP` with `$HOME/.kube/config`.
|
|
|
|
If you were not using `kubectl`, feel free to simply remove `$HOME/.kube/config.rook`.
|