forked from rook/rook
Blaine Gardner
0c721e05d5
Add the ability to specify node profiles in the multus validation test. This addresses a few points of early feedback on the validation tool. Statements below critique the tool's behavior before this patch. 1. The tool assumes all daemons are on public and cluster network, which means users who have a significantly smaller cluster net (a design choice) cannot run a single test to determine if Rook is likely to install correctly. 2. The tool does not have placement options to select only a subset of Kubernetes nodes to run validation on. 3. Users of multus seem to have a dedicated pool of storage nodes more often than the average Rook install. This makes sense for security- and perforance-minded users. The tool cannot run a single test to verify storage-only and general-workload nodes at one time. These points are addressed by allowing users to specify configurations for different "NodeTypes." Each NodeType config has options for selecting the number of OSDs as well as the number of other (non-OSD) Ceph daemons. This limits the unnecessary exhaustion of cluster network addresses from critique 1. Each NodeType config has its own placement (critique 2). Users can define as many NodeTypes as needed to test the network for their planned CephCluster. Specifically, this allows the tool to test storage-only nodes and generalized-workload nodes at the same time. An arbitrary number of NodeTypes are allowed to support even more highly specialized cluster setups, such as multiple tiers of storage nodes where some storage-only nodes may run more OSDs than others. Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
173 lines
7.8 KiB
YAML
173 lines
7.8 KiB
YAML
####################################################################################################
|
|
# This manifest contains a Kubernetes Job and supporting definitions for running Rook's Multus
|
|
# validation tool. The Multus validation tool's help text is copied below. Modify the Job
|
|
# definition in this manifest based on the needs of your cluster.
|
|
####################################################################################################
|
|
# THE BELOW HELP TEXT IS AUTO-GENERATED BY `make docs`
|
|
#
|
|
# Run a validation test that determines whether the current Multus and system
|
|
# configurations will support Rook with Multus.
|
|
#
|
|
# This should be run BEFORE Rook is installed.
|
|
#
|
|
# This is a fairly long-running test. It starts up a web server and many
|
|
# clients to verify that Multus network communication works properly.
|
|
#
|
|
# It does *not* perform any load testing. Networks that cannot support high
|
|
# volumes of Ceph traffic may still encounter runtime issues. This may be
|
|
# particularly noticeable with high I/O load or during OSD rebalancing
|
|
# (see: https://docs.ceph.com/en/latest/architecture/#rebalancing).
|
|
# For example, during Rook or Ceph cluster upgrade.
|
|
#
|
|
# Override the kube config file location by setting the KUBECONFIG environment variable.
|
|
#
|
|
# Usage:
|
|
# rook multus validation run [--public-network=<nad-name>] [--cluster-network=<nad-name>] [flags]
|
|
#
|
|
# Flags:
|
|
# --cluster-network string The name of the Network Attachment Definition (NAD) that will be used for Ceph's cluster network. This should be a namespaced name in the form <namespace>/<name> if the NAD is defined in a different namespace from the cluster namespace.
|
|
# -c, --config string The validation test config file to use. This cannot be used with other flags.
|
|
# --daemons-per-node int The number of validation test daemons to run per node. It is recommended to set this to the maximum number of Ceph daemons that can run on any node in the worst case of node failure(s). The default value is set to the worst-case value for a Rook Ceph cluster with 3 portable OSDs, 3 portable monitors, and where all optional child resources have been created with 1 daemon such that they all might run on a single node in a failure scenario. If you aren't sure what to choose for this value, add 1 for each additional OSD beyond 3. (default 19)
|
|
# -h, --help help for run
|
|
# -n, --namespace string The namespace for validation test resources. It is recommended to set this to the namespace in which Rook's Ceph cluster will be installed. (default "rook-ceph")
|
|
# --nginx-image string The Nginx image used for the validation server and clients. (default "nginxinc/nginx-unprivileged:stable-alpine")
|
|
# --public-network string The name of the Network Attachment Definition (NAD) that will be used for Ceph's public network. This should be a namespaced name in the form <namespace>/<name> if the NAD is defined in a different namespace from the cluster namespace.
|
|
# --timeout-minutes timeoutMinutes The time to wait for resources to change to the expected state. For example, for the test web server to start, for test clients to become ready, or for test resources to be deleted. At longest, this may need to reflect the time it takes for client pods to to pull images, get address assignments, and then for each client to determine that its network connection is stable. Minimum: 1 minute. Recommended: 2 minutes or more. (default 3m0s)
|
|
#
|
|
# Global Flags:
|
|
# --log-level string logging level for logging/tracing output (valid values: ERROR,WARNING,INFO,DEBUG) (default "INFO")
|
|
# THE ABOVE HELP TEXT IS AUTO-GENERATED BY `make docs`
|
|
####################################################################################################
|
|
---
|
|
# Service account for job that validates multus configuration
|
|
apiVersion: v1
|
|
kind: ServiceAccount
|
|
metadata:
|
|
name: rook-ceph-multus-validation
|
|
namespace: rook-ceph # namespace:cluster
|
|
# imagePullSecrets:
|
|
# - name: my-registry-secret
|
|
---
|
|
# Aspects of multus validation job that require access to the operator/cluster namespace
|
|
kind: Role
|
|
apiVersion: rbac.authorization.k8s.io/v1
|
|
metadata:
|
|
name: rook-ceph-multus-validation
|
|
namespace: rook-ceph # namespace:cluster
|
|
rules:
|
|
- apiGroups: [""]
|
|
resources: ["configmaps", "configmaps/finalizers", "pods"]
|
|
verbs: ["get", "list", "create", "update", "delete"]
|
|
- apiGroups: ["apps"]
|
|
resources: ["daemonsets"]
|
|
verbs: ["list", "create", "delete", "deletecollection"]
|
|
- apiGroups: ["k8s.cni.cncf.io"]
|
|
resources: ["network-attachment-definitions"]
|
|
verbs: ["get"]
|
|
- apiGroups: ["batch"]
|
|
resources: ["jobs"]
|
|
verbs: ["get", "list", "delete"]
|
|
---
|
|
# Allow the multus validation job to run in this namespace
|
|
apiVersion: rbac.authorization.k8s.io/v1
|
|
kind: RoleBinding
|
|
metadata:
|
|
name: rook-ceph-multus-validation
|
|
namespace: rook-ceph # namespace:cluster
|
|
roleRef:
|
|
apiGroup: rbac.authorization.k8s.io
|
|
kind: Role
|
|
name: rook-ceph-multus-validation
|
|
subjects:
|
|
- kind: ServiceAccount
|
|
name: rook-ceph-multus-validation
|
|
namespace: rook-ceph # namespace:cluster
|
|
---
|
|
# A job that runs the multus validation tool
|
|
apiVersion: batch/v1
|
|
kind: Job
|
|
metadata:
|
|
name: rook-ceph-multus-validation
|
|
namespace: rook-ceph # namespace:cluster
|
|
labels:
|
|
app: rook-ceph-multus-validation
|
|
spec:
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app: rook-ceph-multus-validation
|
|
spec:
|
|
serviceAccountName: rook-ceph-multus-validation
|
|
containers:
|
|
- name: multus-validation
|
|
image: rook/ceph:master
|
|
command: ["rook"]
|
|
args:
|
|
- "multus"
|
|
- "validation"
|
|
- "run"
|
|
# - "--public-network=<NAD-NAME>" # uncomment and replace NAD name if using public network
|
|
# - "--cluster-network=<NAD-NAME>" # uncomment and replace NAD name if using cluster network
|
|
# - "--nginx-image=<IMAGE>" # uncomment and replace IMAGE with the nginx image you want use for the validation server and clients
|
|
# - "--daemons-per-node=<COUNT>" # uncomment and replace COUNT with the maximum number of daemons that should be running on each node during validation
|
|
env:
|
|
- name: POD_NAMESPACE
|
|
valueFrom:
|
|
fieldRef:
|
|
fieldPath: metadata.namespace
|
|
- name: ROOK_LOG_LEVEL
|
|
value: DEBUG
|
|
restartPolicy: Never
|
|
---
|
|
# This Pod Security Policy (PSP) allows the job to run in Kubernetes environments using PSPs
|
|
# apiVersion: rbac.authorization.k8s.io/v1
|
|
# kind: RoleBinding
|
|
# metadata:
|
|
# name: rook-ceph-multus-validation-psp
|
|
# namespace: rook-ceph # namespace:cluster
|
|
# roleRef:
|
|
# apiGroup: rbac.authorization.k8s.io
|
|
# kind: ClusterRole
|
|
# name: psp:rook
|
|
# subjects:
|
|
# - kind: ServiceAccount
|
|
# name: rook-ceph-multus-validation
|
|
# namespace: rook-ceph # namespace:cluster
|
|
# ---
|
|
# SecurityContextConstraints(SCC) for the Rook and Ceph daemons
|
|
# kind: SecurityContextConstraints
|
|
# apiVersion: security.openshift.io/v1
|
|
# metadata:
|
|
# name: rook-ceph-multus-validation
|
|
# allowPrivilegedContainer: true
|
|
# allowHostDirVolumePlugin: true
|
|
# allowHostPID: false
|
|
# # set to true if running rook with host networking enabled
|
|
# allowHostNetwork: true
|
|
# # set to true if running rook with the provider as host
|
|
# allowHostPorts: true
|
|
# priority:
|
|
# allowedCapabilities: ["MKNOD"]
|
|
# allowHostIPC: true
|
|
# readOnlyRootFilesystem: false
|
|
# # drop all default privileges
|
|
# requiredDropCapabilities: ["All"]
|
|
# defaultAddCapabilities: []
|
|
# runAsUser:
|
|
# type: RunAsAny
|
|
# seLinuxContext:
|
|
# type: RunAsAny
|
|
# fsGroup:
|
|
# type: RunAsAny
|
|
# supplementalGroups:
|
|
# type: RunAsAny
|
|
# seccompProfiles:
|
|
# - "*"
|
|
# volumes:
|
|
# - configMap
|
|
# - emptyDir
|
|
# - projected
|
|
# users:
|
|
# - system:serviceaccount:rook-ceph:rook-ceph-multus-validation # serviceaccount:namespace:cluster
|
|
---
|