my-rook-config/design/local-node-agent.md

Rook Local Node Agent

Overview

In a distributed storage system, there are operations that must be performed on a specific node in the cluster where storage is to be consumed from. For example, a database pod that requires persistent storage needs to attach and mount a volume backed by the storage cluster on the same node that the pod is scheduled to run on. In this document, we propose a design for a Rook "agent" that will be deployed in the cluster to run on nodes that have a need to perform operations to consume storage from the cluster.

Background: Flexvolume Issues

In Kubernetes 1.8, the Kubernetes Storage SIG recommends storage providers to implement out-of-tree plugins to provide persistent storage. With the full implementation of Container Storage Interface (CSI) several months away from completion, the recommended approach in the meantime is to implement a Flexvolume. Deployment of a Flexvolume will be improved in 1.8 to make it less of a manual process (e.g., dynamic discovery), as described in the following pull request: https://github.com/kubernetes/community/pull/833

However, there are still some limitations to the capabilities of a Flexvolume implementation. For example, a Flexvolume plugin does not execute in a context that has cluster credentials, so it cannot communicate with the Kubernetes API to perform such operations as creating Custom Resource Definitions (CRDs). This document will describe how a Rook local node agent would work with the Flexvolume architecture to provide storage in a Kubernetes cluster, in addition to other responsibilities that must be performed on specific nodes in the cluster.

Detailed Design

Responsibilities

The Rook agent will have multiple responsibilities beyond just performing storage operations for the Rook Flexvolume driver. One can think of the agent as a mini operator that functions at the node level. The initial proposed responsibilities of the agent are:

1. Deploy the Rook Flexvolume driver to the volume-plugin-dir directory on every node
2. Perform storage operations on behalf of the Flexvolume driver, such as attaching, detaching, mounting and unmounting Rook cluster storage
3. Cluster clean up operations, such as forcefully unmapping RBD devices when the Rook cluster is being deleted while there are still pods consuming those volumes
4. Proxy I/O traffic from kernel modules to user space (e.g. Network Block Device (NBD) kernel module to librbd in userspace)

Deployment

The Rook operator will deploy the Rook agent to all nodes in the cluster via a Daemonset in the same namespace in which the operator is running. It is a permanent (long running) daemon that has a lifecycle tied to the node that it is scheduled on. The agent deployment will happen when the operator is first created, in the same flow where the operator is declaring CRDs for clusters, pools, etc. This means that the Rook agents are not associated with a specific Rook cluster and that they will be able to handle operations for any Rook cluster instance. The Rook operator CRD will be updated to allow selectors to control the set of nodes that the agent is scheduled on, but in most cases it is desirable for it to be running on all nodes.

Each agent pod will be running the same rook/rook container image in use today, but with a new agent command (similar to the existing mon or osd commands). This image will have all the tools necessary to provision and manage storage (e.g., rbd, iSCSI tools and the Flexvolume driver). When the agent starts up, it will immediately copy the Rook Flexvolume driver from its image to the volume-plugin-dir on its host node. In Kubernetes 1.8+, the dynamic Flexvolume plugin discovery will find and initialize our driver, but in older versions of Kubernetes a manual restart of the Kubelet will be required.

After the driver has been deployed, the Rook agent will create a Unix domain socket that will serve as the communications channel between the agent and the Flexvolume driver. The driver will initiate a handshake to the agent over the socket during its Init function when it is executed after being discovered by the Controller manager. The agent will wait for the driver to complete the handshake before moving on and being ready to accept Flexvolume storage requests.

Note that the Rook operator will continue to run its existing dynamic volume provisioner to provision and delete persistent volumes as needed.

Flexvolume

The Rook Flexvolume driver will be very lightweight, simply implementing Mount() and Unmount() from the required interface and then offloading the storage provider work to the Rook agent over the Unix domain socket. Note this means that the attach/detach controller and its centralized attaching/detaching will not be used. Instead, attaching will be performed as part of Mount() and detaching as part of Unmount(), as described in the next section. This is important because it means that the Rook Flexvolume driver and Agent do not need to be installed on the Kubernetes master node (control plane), which is not possible in some environments such as Google Container Engine (GKE). This makes this proposed design more portable.

Control Flow Overview

Below is a simplified description of the control flow for providing block storage ready to be consumed by a pod:

1. Rook operator is running and a Rook agent pod is running on every node in the cluster. The Rook Flexvolume driver has been deployed to the volume-plugin-dir on each node.
2. A user creates a Persistent Volume Claim (PVC) specifying a storageclass that uses the rook.io/block provisioner
3. Provision() is called on the operator's provisioner to create a block image in the cluster. At this point, the Provision phase is complete and the PVC/PV are considered Bound together.
4. Once a pod has been created that consumes the PVC, Mount() is called by the Kubelet on the Rook Flexvolume on the node that will be consuming the storage, which calls into its local Rook agent via the Unix domain socket.
   • Mount() is a blocking call by the Kubelet and it will wait while the entire mapping/mounting is performed by the driver and agent synchronously.
5. The agent then creates a volume attach CRD that represents the attachment of the cluster volume to the node.
6. Next, the agent performs the mapping of the volume to a local device on the node and updates the status of the CRD and its device path field (e.g., /dev/rbd0).
7. Control is returned to the driver, and if the mapping was successful then the driver will proceed with mounting the new local device to the requested path on the host. If necessary, the driver will also format the volume according to the filesystem type expressed on the storageclass for the volume.
8. The driver then returns from Mount() with a successful result to the Kubelet.

Usage of CRDs

The control flow described above explained how the agent uses the Kubernetes API to create a volume attach CRD that represents the attachment of a cluster volume to a specific cluster node. This usage of the Kubernetes API is a reason why the Flexvolume driver, which does not run in a context that has any cluster credentials, is insufficient for this design, and thus why a Rook agent pod is desirable. Let's enumerate some of the benefits of using a volume attach CRD:

1. The admin gets a native kubectl experience for viewing and getting information on attached volumes in their cluster.
2. The CRD helps provide fencing for the volume in a generalized way, it is not specific to the underlying storage Rook is creating and managing. The existence of the CRD provides a means of bookkeeping to signal that the volume is locked and in use.
3. In the event that a node dies that had a volume attached, a CRD allows centralized detachment of the volume by the Rook operator. This will be explained in more detail in the fencing section.

Improvements Over Rook's Current Support for Persistent Volumes

Rook currently has a hybrid approach of providing persistent storage in a Kubernetes cluster. While the Rook operator implements a dynamic volume provisioner, the attach and mounting is offloaded to the existing RBD plugin. This requires that the Ceph tools are installed alongside the Kubelet on every node in the cluster, which causes friction in the experience for users of Rook. Furthermore, if users want to consume a Rook volume outside of the default namespace, then they must manually copy secrets to their namespace of choice.

Both of these issues are handled by this proposed design for the Rook agent, greatly streamlining the Rook storage experience in Kubernetes and removing the common causes for errors and failure. This is a big win for our users.

This proposed design also allows Rook to normalize on a single path for providing persistent volumes across all Kubernetes versions that it supports (1.5+). Since this design is fully out-of-tree for Kubernetes, it is not tied to Kubernetes release timelines. Updates, fixes and new features can be released through the normal Rook release process and timelines.

The experience is also normalized across other distributed storage platforms that Rook will support in the future, such as GlusterFS. Instead of the user having to create a storageclass specific for the underlying distributed storage (e.g., RBD if Rook is using Ceph), this will abstract that decision away, allowing the user to simply request storage. This was already true of the hybrid approach in Rook v0.5, but it's worth noting here still.

Fencing

Ensuring exclusive single client access (ReadWriteOnce) to the persistent volume can be achieved by the Rook agent, independently from the underlying storage protocol that is being used (RBD, NBD, iSCSI, etc.). Remember that during the attach operation, the Rook agent will create a volume attach CRD. When the agent performs the attach on its local node, it can update the CRD with information stating that the volume is now locked. Thus, the CRD itself is providing the accounting for the volume's lock in a generalized way, independent of the of the underlying storage protocol the agent chose to use for the volume. Of course, this lock can be augmented at the lower layer of the specific storage protocol, e.g., rbd lock add for RBD.

Similarly, during the detach operation, the CRD will be deleted to signify that the volume has been unlocked. If the node where the volume has attached has died, then we need a centralized way to perform the unlocking of the volume. Therefore, the Rook operator will be watching for Kubernetes Node events that will signal when a node has died and been removed from the cluster. The operator will act on this node deletion event by executing the storage protocol specific command (e.g., rbd lock remove) and the deleting the volume attach CRD, signaling that the volume is now unlocked.

Security

The only interface for communicating with and invoking operations on the Rook agent is the Unix domain socket. This socket will have read/write only accessible by root and it is only accessible on the local node (not remotely accessible).

Cluster Cleanup

Rook makes storage as a service a deeply integrated part of the Kubernetes cluster as opposed to an external entity. This integration makes more attention to lifecycle management of the storage components necessary.

Hung RBD Kernel Module

If the monitor pods of a Rook cluster are no longer accessible while block storage is mapped to a node, the kernel RBD module will be hung and require a power cycle of the machine.

The Rook agent can help mitigate this scenario, by watching for the cluster CRD delete event. When a Rook cluster is being deleted, there may still be consumers of the storage in the cluster in the form of pods with PVCs. When a Rook agent receives a cluster CRD delete event, they will respond by checking for any Rook storage on the local node they are running on and then forcefully remove them.

To forcefully remove the storage from local pods, the agent will perform the following sequence of steps for each Rook PVC:

$ kubectl delete pvc <pvc name>
$ sudo rbd unmap -o force /dev/rbdX
# wait for it to time out or send SIGINT
$ kubectl delete pv <pv name>
$ sudo umount <mount point>

As mentioned above, each agent will be watching for events on cluster CRDs. Even in a large scale cluster, this is appropriate to have many watchers on the cluster CRDs for a couple reasons:

1. Cluster CRD events are relatively rare since they are tied to cluster lifecycles
2. Each agent truly does need to be informed about cluster CRD events since they all may need to act on them

Open Questions

1. Reliability: Does the centralized Detach() fallback by the operator work reliably when the node consuming the volume has died? We will vet this further while we are testing the implementation under real world scenarios.
2. Security: Are there further security considerations that need to be made?
3. Portability: Since this design does not need to run anything on the master node (control plane), it is fairly portable across Kubernetes deployments and environments. However, some environments, such as Google Container Engine, do not have suitable kernel drivers for I/O traffic. Those environments need to be updated with a lowest common denominator kernel driver, such as NBD.