forked from rook/rook
Tobias Florek
db814dcd4b
It has only been used by the flex driver which is long gone. This removes the nodes/proxy RBAC grant from the helm chart, the example YAMl and the security design docs. Signed-off-by: Tobias Florek <me@ibotty.net>
356 lines
10 KiB
Markdown
356 lines
10 KiB
Markdown
# Security Model
|
|
|
|
The Rook operator currently uses a highly privileged service account with permissions to create namespaces, roles, role bindings, etc. Our approach would not pass a security audit and this design explores an improvement to this. Furthermore given our use of multiple service accounts and namespace, setting policies and quotas is harder than it needs to be.
|
|
|
|
## Goals
|
|
* Reduce the number of service accounts and privileges used by Rook
|
|
* Reduce the number of namespaces that are used by Rook
|
|
* Only use services accounts and namespaces used by the cluster admin -- this enables them to set security policies and quotas that rook adheres to
|
|
* Continue to support a least privileged model
|
|
|
|
## What we do today
|
|
|
|
Today the cluster admin creates the rook system namespace, rook-operator service account and RBAC rules as follows:
|
|
|
|
```yaml
|
|
apiVersion: v1
|
|
kind: Namespace
|
|
metadata:
|
|
name: rook-system
|
|
---
|
|
apiVersion: v1
|
|
kind: ServiceAccount
|
|
metadata:
|
|
name: rook-operator
|
|
namespace: rook-system
|
|
---
|
|
kind: ClusterRole
|
|
apiVersion: rbac.authorization.k8s.io/v1
|
|
metadata:
|
|
name: rook-operator
|
|
rules:
|
|
- apiGroups: [""]
|
|
resources: ["namespaces", "serviceaccounts", "secrets", "pods", "services", "nodes", "configmaps", "events", "persistentvolumes", "persistentvolumeclaims"]
|
|
verbs: [ "get", "list", "watch", "patch", "create", "update", "delete" ]
|
|
- apiGroups: ["extensions"]
|
|
resources: ["thirdpartyresources", "deployments", "daemonsets", "replicasets"]
|
|
verbs: [ "get", "list", "watch", "create", "delete" ]
|
|
- apiGroups: ["apiextensions.k8s.io"]
|
|
resources: ["customresourcedefinitions"]
|
|
verbs: [ "get", "list", "watch", "create", "delete" ]
|
|
- apiGroups: ["rbac.authorization.k8s.io"]
|
|
resources: ["clusterroles", "clusterrolebindings", "roles", "rolebindings"]
|
|
verbs: [ "get", "list", "watch", "create", "update", "delete" ]
|
|
- apiGroups: ["storage.k8s.io"]
|
|
resources: ["storageclasses"]
|
|
verbs: [ "get", "list", "watch", "delete" ]
|
|
- apiGroups: ["rook.io"]
|
|
resources: ["*"]
|
|
verbs: [ "*" ]
|
|
---
|
|
kind: ClusterRoleBinding
|
|
apiVersion: rbac.authorization.k8s.io/v1
|
|
metadata:
|
|
name: rook-operator
|
|
roleRef:
|
|
apiGroup: rbac.authorization.k8s.io
|
|
kind: ClusterRole
|
|
name: rook-operator
|
|
subjects:
|
|
- kind: ServiceAccount
|
|
name: rook-operator
|
|
namespace: rook-system
|
|
```
|
|
|
|
`rook-operator` is a highly privileged service account with cluster wide scope. It likely has more privileges than is currently needed, for example, the operator does not create namespaces today. Note the name `rook-system` and `rook-operator` are not important and can be set to anything.
|
|
|
|
Once the rook operator is up and running it will automatically create the service account for the rook agent and the following RBAC rules:
|
|
|
|
```yaml
|
|
apiVersion: v1
|
|
kind: ServiceAccount
|
|
metadata:
|
|
name: rook-ceph-agent
|
|
namespace: rook-system
|
|
---
|
|
kind: ClusterRole
|
|
apiVersion: rbac.authorization.k8s.io/v1
|
|
metadata:
|
|
name: rook-ceph-agent
|
|
rules:
|
|
- apiGroups: [""]
|
|
resources: ["pods", "secrets", "configmaps", "persistentvolumes", "nodes"]
|
|
verbs: [ "get", "list" ]
|
|
- apiGroups: ["storage.k8s.io"]
|
|
resources: ["storageclasses"]
|
|
verbs: [ "get" ]
|
|
- apiGroups: ["rook.io"]
|
|
resources: ["volumeattachment"]
|
|
verbs: [ "get", "list", "watch", "create", "update" ]
|
|
---
|
|
kind: ClusterRoleBinding
|
|
apiVersion: rbac.authorization.k8s.io/v1
|
|
metadata:
|
|
name: rook-ceph-agent
|
|
roleRef:
|
|
apiGroup: rbac.authorization.k8s.io
|
|
kind: ClusterRole
|
|
name: rook-ceph-agent
|
|
subjects:
|
|
- kind: ServiceAccount
|
|
name: rook-ceph-agent
|
|
namespace: rook-system
|
|
```
|
|
|
|
When the cluster admin create a new Rook cluster they do so by adding a namespace and the rook cluster spec:
|
|
|
|
```yaml
|
|
apiVersion: v1
|
|
kind: Namespace
|
|
metadata:
|
|
name: mycluster
|
|
---
|
|
apiVersion: rook.io/v1alpha1
|
|
kind: Cluster
|
|
metadata:
|
|
name: myrookcluster
|
|
namespace: mycluster
|
|
...
|
|
```
|
|
|
|
At this point the rook operator will notice that a new rook cluster CRD showed up and proceeds to create a service account for the `rook-api` and `rook-ceph-osd`. It will also use the `default` service account in the `mycluster` namespace for some pods.
|
|
|
|
The `rook-api` service account and RBAC rules are as follows:
|
|
|
|
```yaml
|
|
apiVersion: v1
|
|
kind: ServiceAccount
|
|
metadata:
|
|
name: rook-api
|
|
namespace: mycluster
|
|
---
|
|
kind: Role
|
|
apiVersion: rbac.authorization.k8s.io/v1
|
|
metadata:
|
|
name: rook-api
|
|
namespace: mycluster
|
|
rules:
|
|
- apiGroups: [""]
|
|
resources: ["namespaces", "secrets", "pods", "services", "nodes", "configmaps", "events"]
|
|
verbs: [ "get", "list", "watch", "create", "update" ]
|
|
- apiGroups: ["extensions"]
|
|
resources: ["thirdpartyresources", "deployments", "daemonsets", "replicasets"]
|
|
verbs: [ "get", "list", "create", "update" ]
|
|
- apiGroups: ["storage.k8s.io"]
|
|
resources: ["storageclasses"]
|
|
verbs: [ "get", "list" ]
|
|
- apiGroups: ["apiextensions.k8s.io"]
|
|
resources: ["customresourcedefinitions"]
|
|
verbs: [ "get", "list", "create" ]
|
|
---
|
|
kind: RoleBinding
|
|
apiVersion: rbac.authorization.k8s.io/v1
|
|
metadata:
|
|
name: rook-api
|
|
namespace: mycluster
|
|
roleRef:
|
|
apiGroup: rbac.authorization.k8s.io
|
|
kind: Role
|
|
name: rook-api
|
|
subjects:
|
|
- kind: ServiceAccount
|
|
name: rook-api
|
|
namespace: mycluster
|
|
```
|
|
|
|
The `rook-ceph-osd` service account and RBAC rules are as follows:
|
|
|
|
```yaml
|
|
apiVersion: v1
|
|
kind: ServiceAccount
|
|
metadata:
|
|
name: rook-ceph-osd
|
|
namespace: mycluster
|
|
---
|
|
kind: Role
|
|
apiVersion: rbac.authorization.k8s.io/v1
|
|
metadata:
|
|
name: rook-ceph-osd
|
|
namespace: mycluster
|
|
rules:
|
|
- apiGroups: [""]
|
|
resources: ["configmaps"]
|
|
verbs: [ "get", "list", "watch", "create", "update", "delete" ]
|
|
---
|
|
kind: RoleBinding
|
|
apiVersion: rbac.authorization.k8s.io/v1
|
|
metadata:
|
|
name: rook-ceph-osd
|
|
namespace: mycluster
|
|
roleRef:
|
|
apiGroup: rbac.authorization.k8s.io
|
|
kind: Role
|
|
name: rook-ceph-osd
|
|
subjects:
|
|
- kind: ServiceAccount
|
|
name: rook-ceph-osd
|
|
namespace: mycluster
|
|
```
|
|
|
|
## Proposed Changes
|
|
|
|
Just as we do today the cluster admin is responsible for creating the `rook-system` namespace. I propose we have a single service account in this namespace and call it `rook-system` by default. The names used are inconsequential and can be set to something different by the cluster admin.
|
|
|
|
```yaml
|
|
apiVersion: v1
|
|
kind: Namespace
|
|
metadata:
|
|
name: rook-system
|
|
---
|
|
apiVersion: v1
|
|
kind: ServiceAccount
|
|
metadata:
|
|
name: rook-system
|
|
namespace: rook-system
|
|
```
|
|
|
|
The `rook-system` service account is responsible for launching all pods, services, daemonsets, etc. for Rook and should have enough privilege to do and nothing more. I've not audited all the RBAC rules but a good tool to do is [here](https://github.com/liggitt/audit2rbac). For example:
|
|
|
|
```yaml
|
|
kind: ClusterRole
|
|
apiVersion: rbac.authorization.k8s.io/v1
|
|
metadata:
|
|
name: rook-system
|
|
rules:
|
|
- apiGroups: [""]
|
|
resources: ["pods", "services", "configmaps"]
|
|
verbs: [ "get", "list", "watch", "patch", "create", "update, "delete" ]
|
|
- apiGroups: ["extensions"]
|
|
resources: ["deployments", "daemonsets", "replicasets"]
|
|
verbs: [ "get", "list", "watch", "patch", "create", "update, "delete" ]
|
|
- apiGroups: ["apiextensions.k8s.io"]
|
|
resources: ["customresourcedefinitions"]
|
|
verbs: [ "get", "list", "watch", "patch", "create", "update, "delete" ]
|
|
- apiGroups: ["rook.io"]
|
|
resources: ["*"]
|
|
verbs: [ "*" ]
|
|
---
|
|
kind: RoleBinding
|
|
apiVersion: rbac.authorization.k8s.io/v1
|
|
metadata:
|
|
name: rook-system
|
|
namespace: rook-system
|
|
roleRef:
|
|
apiGroup: rbac.authorization.k8s.io
|
|
kind: ClusterRole
|
|
name: rook-system
|
|
subjects:
|
|
- kind: ServiceAccount
|
|
name: rook-system
|
|
namespace: rook-system
|
|
```
|
|
|
|
Notably absent here are privileges to set other RBAC rules and create read cluster-wide secrets and other resources. Because the admin created the `rook-system` namespace and service account they are free to set policies on them using namespace quotas.
|
|
|
|
Also note that while we use a `ClusterRole` for rook-system we only use a `RoleBinding` to grant it access to the `rook-system` namespace. It does not have cluster-wide privileges.
|
|
|
|
When creating a Rook cluster the cluster admin will continue to define the namespace and cluster CRD as follows:
|
|
|
|
```yaml
|
|
apiVersion: v1
|
|
kind: Namespace
|
|
metadata:
|
|
name: mycluster
|
|
---
|
|
apiVersion: rook.io/v1alpha1
|
|
kind: Cluster
|
|
metadata:
|
|
name: myrookcluster
|
|
namespace: mycluster
|
|
...
|
|
```
|
|
|
|
In addition we will require that the cluster-admin define a service account and role binding as follows:
|
|
|
|
```yaml
|
|
apiVersion: v1
|
|
kind: ServiceAccount
|
|
metadata:
|
|
name: rook-cluster
|
|
namespace: mycluster
|
|
---
|
|
kind: Role
|
|
apiVersion: rbac.authorization.k8s.io/v1
|
|
metadata:
|
|
name: rook-cluster
|
|
namespace: mycluster
|
|
rules:
|
|
- apiGroups: [""]
|
|
resources: ["configmaps"]
|
|
verbs: [ "get", "list", "watch", "create", "update", "delete" ]
|
|
---
|
|
kind: RoleBinding
|
|
apiVersion: rbac.authorization.k8s.io/v1
|
|
metadata:
|
|
name: rook-cluster
|
|
namespace: mycluster
|
|
roleRef:
|
|
apiGroup: rbac.authorization.k8s.io
|
|
kind: ClusterRole
|
|
name: rook-system
|
|
subjects:
|
|
- kind: ServiceAccount
|
|
name: rook-system
|
|
namespace: rook-system
|
|
---
|
|
kind: RoleBinding
|
|
apiVersion: rbac.authorization.k8s.io/v1
|
|
metadata:
|
|
name: rook-system
|
|
namespace: mycluster
|
|
roleRef:
|
|
apiGroup: rbac.authorization.k8s.io
|
|
kind: Role
|
|
name: rook-cluster
|
|
namespace: rook-cluster
|
|
subjects:
|
|
- kind: ServiceAccount
|
|
name: rook-cluster
|
|
namespace: mycluster
|
|
```
|
|
|
|
This will grant the `rook-system` service account access to the new namespace and also setup a least privileged service account `rook-cluster` to be used for pods in this namespace that need K8S api access.
|
|
|
|
With this approach `rook-system` will only have access to namespaces nominated by the cluster admin. Also we will no longer create any service accounts or namespaces enabling admins to set stable policies and quotas.
|
|
|
|
Also all rook pods except the rook operator pod should run using `rook-cluster` service account in the namespace they're in.
|
|
|
|
### Supporting common namespaces
|
|
|
|
Finally, we should support running multiple rook clusters in the same namespaces. While namespaces are a great organizational unit for pods etc. they are also a unit of policy and quotas. While we can force the cluster admin to go to an approach where they need to manage multiple namespaces, we would be better off if we give the option to cluster admin decide how they use namespace.
|
|
|
|
For example, it should be possible to run rook-operator, rook-agent, and multiple independent rook clusters in a single namespace. This is going to require setting a prefix for pod names and other resources that could collide.
|
|
|
|
The following should be possible:
|
|
|
|
```yaml
|
|
apiVersion: v1
|
|
kind: Namespace
|
|
metadata:
|
|
name: myrook
|
|
---
|
|
apiVersion: rook.io/v1alpha1
|
|
kind: Cluster
|
|
metadata:
|
|
name: red
|
|
namespace: mycluster
|
|
...
|
|
---
|
|
apiVersion: rook.io/v1alpha1
|
|
kind: Cluster
|
|
metadata:
|
|
name: blue
|
|
namespace: mycluster
|
|
...
|
|
```
|