Blaine Gardner
52f5119cef
Add the ability to specify node profiles in the multus validation test.
This addresses a few points of early feedback on the validation tool.
Statements below critique the tool's behavior before this patch.
1. The tool assumes all daemons are on public and cluster network, which
means users who have a significantly smaller cluster net (a
design choice) cannot run a single test to determine if Rook is
likely to install correctly.
2. The tool does not have placement options to select only a subset of
Kubernetes nodes to run validation on.
3. Users of multus seem to have a dedicated pool of storage nodes more
often than the average Rook install. This makes sense for security-
and perforance-minded users. The tool cannot run a single test to
verify storage-only and general-workload nodes at one time.
These points are addressed by allowing users to specify configurations
for different "NodeTypes."
Each NodeType config has options for selecting the number of OSDs as
well as the number of other (non-OSD) Ceph daemons. This limits the
unnecessary exhaustion of cluster network addresses from critique 1.
Each NodeType config has its own placement (critique 2).
Users can define as many NodeTypes as needed to test the network for
their planned CephCluster. Specifically, this allows the tool to test
storage-only nodes and generalized-workload nodes at the same time. An
arbitrary number of NodeTypes are allowed to support even more highly
specialized cluster setups, such as multiple tiers of storage nodes
where some storage-only nodes may run more OSDs than others.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
(cherry picked from commit 0c721e05d5)
70 lines
4.0 KiB
YAML
70 lines
4.0 KiB
YAML
# The intended namespace where the validation test will be run. This is should be where the
|
|
# Rook-Ceph cluster will be installed.
|
|
namespace: "{{ .Namespace }}"
|
|
|
|
# These fields should be set to the name of the Network Attachment Definition (NAD) which will be
|
|
# used for the Ceph cluster's public or cluster network, respectively. This should be a namespaced
|
|
# name in the form <namespace>/<name> if the NAD is defined in a different namespace from the
|
|
# CephCluster namespace. One or both of the fields may be set.
|
|
publicNetwork: "{{ .PublicNetwork }}"
|
|
clusterNetwork: "{{ .ClusterNetwork }}"
|
|
|
|
# This is the time to wait for resources to change to the expected state. For example, for the test
|
|
# web server to start, for test clients to become ready, or for test resources to be deleted. At
|
|
# longest, this may need to reflect the time it takes for client pods to get address assignments and
|
|
# then for each client to determine that its network connection is stable.
|
|
# This should be at least 1 minute. 2 minutes or more is recommended.
|
|
resourceTimeout: "{{ .ResourceTimeout }}"
|
|
|
|
# The Nginx image which will be used for the web server and clients.
|
|
nginxImage: "{{ .NginxImage }}"
|
|
|
|
# Specify validation test config for groups of nodes. A Ceph cluster may span more than one type of
|
|
# node, but usually not more than 3. Node types defined here should not overlap with each other.
|
|
# Common examples of different node types are below:
|
|
# - General, converged nodes that are used for both Ceph daemons and user applications
|
|
# - Dedicated storage nodes for hosting Ceph storage daemons
|
|
# - Non-storage nodes running workloads other than Ceph (running only Ceph CSI plugin daemons)
|
|
# - Dedicated nodes for hosting Ceph arbiter mons
|
|
# - Dedicated nodes for hosting only Ceph core (RADOS) components: mons, mgrs, OSDs
|
|
# - Dedicated nodes for hosting non-core storage components: MDS, RGW, CSI provisioner, etc.
|
|
nodeTypes:
|
|
{{- range $type, $config := .NodeTypes }}
|
|
{{ $type }}: # Unique name for this node type (lower-case alphanumeric characters and dashes only)
|
|
# This should reflect the maximum number of OSDs that may run on any given node of this type
|
|
# during a worst-case failure scenario. This may be a static number expected by the
|
|
# StorageClassDeviceSets planned for the CephCluster. For portable OSDs, this may be larger to
|
|
# account for OSDs being rescheduled during node failures.
|
|
osdsPerNode: {{ $config.OSDsPerNode }}
|
|
|
|
# This should reflect the maximum number of Ceph and CSI daemons that may run on any given node
|
|
# during a worst-case failure scenario. CSI requires 2 Multus addresses per plugin daemon.
|
|
#
|
|
# In the converged case, a Rook cluster with a single instance of all resource types might run
|
|
# these daemons on a single node during worst-case failure:
|
|
# 1 mon, 1 mgr, 1 MDS, 1 NFS server, 1 RGW, 1 rbdmirror, 1 cephfsmirror, plus
|
|
# 1 CSI provisioner and 2 CSI plugins for all 3 CSI types (RBD, CephFS, NFS) -- 16 daemons
|
|
#
|
|
# A node that is running only client workloads would run only the CSI plugin daemons per node:
|
|
# 2 CSI plugins for all 3 CSI type (RBD, CephFS, NFS) -- 6 daemons
|
|
otherDaemonsPerNode: {{ $config.OtherDaemonsPerNode }}
|
|
|
|
# Placement options for informing the validation tool how to identify this type of node.
|
|
# Affinities are not supported because their behavior is too relaxed for ensuring a specific
|
|
# number of daemons per node.
|
|
placement:
|
|
# Kubernetes Pod spec node selector for assigning validation Pods to this node type.
|
|
# See https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector
|
|
nodeSelector:
|
|
{{- range $k, $v := $config.Placement.NodeSelector }}
|
|
{{ $k }}: {{ $v }}
|
|
{{- end }}
|
|
|
|
# Kubernetes tolerations for assigning validation Pods to this node type.
|
|
# See https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/
|
|
tolerations:
|
|
{{- range $idx, $toleration := $config.Placement.Tolerations }}
|
|
- {{ $toleration.ToJSON }}
|
|
{{- end }}
|
|
{{ end }}
|