Merge pull request #12866 from rook/mergify/bp/release-1.12/pr-12778
multus: allow using NADs without inspectable CIDRs (backport #12778)
This commit is contained in:
36 files changed
+2571
-900
No files matched your search
@@ -187,7 +187,12 @@ If not specified, the default SDN will be used.
|
||||
Configure the network that will be enabled for the cluster and services.
|
||||
|
||||
* `provider`: Specifies the network provider that will be used to connect the network interface. You can choose between `host`, and `multus`.
|
||||
* `selectors`: List the network selector(s) that will be used associated by a key.
|
||||
* `selectors`: Used for `multus` provider only. Select NetworkAttachmentDefinitions to use for Ceph networks.
|
||||
* `public`: Select the NetworkAttachmentDefinition to use for the public network.
|
||||
* `cluster`: Select the NetworkAttachmentDefinition to use for the cluster network.
|
||||
* `addressRanges`: Used for `host` or `multus` providers only. Allows overriding the address ranges (CIDRs) that Ceph will listen on.
|
||||
* `public`: A list of individual network ranges in CIDR format to use for Ceph's public network.
|
||||
* `cluster`: A list of individual network ranges in CIDR format to use for Ceph's cluster network.
|
||||
* `ipFamily`: Specifies the network stack Ceph daemons should listen on.
|
||||
* `dualStack`: Specifies that Ceph daemon should listen on both IPv4 and IPv6 network stacks.
|
||||
* `connections`: Settings for network connections using Ceph's msgr2 protocol
|
||||
@@ -211,10 +216,52 @@ Configure the network that will be enabled for the cluster and services.
|
||||
Changing networking configuration after a Ceph cluster has been deployed is NOT
|
||||
supported and will result in a non-functioning cluster.
|
||||
|
||||
#### Ceph public and cluster networks
|
||||
|
||||
Ceph daemons can operate on up to two distinct networks: public, and cluster.
|
||||
|
||||
Ceph daemons always use the public network, which is the Kubernetes pod network by default. The
|
||||
public network is used for client communications with the Ceph cluster (reads/writes).
|
||||
|
||||
If specified, the cluster network is used to isolate internal Ceph replication traffic. This includes
|
||||
additional copies of data replicated between OSDs during client reads/writes. This also includes OSD
|
||||
data recovery (re-replication) when OSDs or nodes go offline. If the cluster network is unspecified,
|
||||
the public network is used for this traffic instead.
|
||||
|
||||
Some Rook network providers allow manually specifying the public and network interfaces that Ceph
|
||||
will use for data traffic. Use `addressRanges` to specify this. For example:
|
||||
|
||||
```yaml
|
||||
network:
|
||||
provider: host
|
||||
addressRanges:
|
||||
public:
|
||||
- "192.168.100.0/24"
|
||||
- "192.168.101.0/24"
|
||||
cluster:
|
||||
- "192.168.200.0/24"
|
||||
```
|
||||
|
||||
This spec translates directly to Ceph's `public_network` and `host_network` configurations.
|
||||
Refer to [Ceph networking documentation](https://docs.ceph.com/docs/master/rados/configuration/network-config-ref/)
|
||||
for more details.
|
||||
|
||||
The default, unspecified network provider cannot make use of these configurations.
|
||||
|
||||
Ceph public and cluster network configurations are allowed to change, but this should be done with
|
||||
great care. When updating underlying networks or Ceph network settings, Rook assumes that the
|
||||
current network configuration used by Ceph daemons will continue to operate as intended. Network
|
||||
changes are not applied to Ceph daemon pods (like OSDs and MDSes) until the pod is restarted. When
|
||||
making network changes, ensure that restarted pods will not lose connectivity to existing pods, and
|
||||
vice versa.
|
||||
|
||||
#### Host Networking
|
||||
|
||||
To use host networking, set `provider: host`.
|
||||
|
||||
To instruct Ceph to operate on specific host interfaces or networks, use `addressRanges` to select
|
||||
the network CIDRs Ceph will bind to on the host.
|
||||
|
||||
If the host networking setting is changed in a cluster where mons are already running, the existing mons will
|
||||
remain running with the same network settings with which they were created. To complete the conversion
|
||||
to or from host networking after you update this setting, you will need to
|
||||
@@ -223,93 +270,73 @@ in order to have mons on the desired network configuration.
|
||||
|
||||
#### Multus
|
||||
|
||||
Rook supports addition of public and cluster network for ceph using Multus
|
||||
Rook supports using Multus NetworkAttachmentDefinitions for Ceph public and cluster networks.
|
||||
|
||||
The selector keys are required to be `public` and `cluster` where each represent:
|
||||
Refer to [Multus documentation](https://github.com/k8snetworkplumbingwg/multus-cni/blob/master/docs/how-to-use.md)
|
||||
for details about how to set up and select Multus networks.
|
||||
|
||||
* `public`: client communications with the cluster (reads/writes)
|
||||
* `cluster`: internal Ceph replication network
|
||||
Rook will attempt to auto-discover the network CIDRs for selected public and/or cluster networks.
|
||||
This process is not guaranteed to succeed. Furthermore, this process will get a new network lease
|
||||
for each CephCluster reconcile. Specify `addressRanges` manually if the auto-detection process
|
||||
fails or if the selected network configuration cannot automatically recycle released network leases.
|
||||
|
||||
If you want to learn more, please read:
|
||||
Only OSD pods will have both public and cluster networks attached (if specified). The rest of the
|
||||
Ceph component pods and CSI pods will only have the public network attached. The Rook operator will
|
||||
not have any networks attached; it proxies Ceph commands via a sidecar container in the mgr pod.
|
||||
|
||||
* [Ceph Networking reference](https://docs.ceph.com/docs/master/rados/configuration/network-config-ref/).
|
||||
* [Multus documentation](https://intel.github.io/multus-cni/doc/how-to-use.html)
|
||||
|
||||
Based on the configuration, the operator will do the following:
|
||||
|
||||
1. If only the `public` selector is specified, all communication will happen on that network
|
||||
|
||||
```yaml
|
||||
network:
|
||||
provider: multus
|
||||
selectors:
|
||||
public: rook-ceph/rook-public-nw
|
||||
```
|
||||
|
||||
2. If only the `cluster` selector is specified, the internal cluster traffic\* will happen on that network. All other traffic to mons, OSDs, and other daemons will be on the default network.
|
||||
|
||||
```yaml
|
||||
network:
|
||||
provider: multus
|
||||
selectors:
|
||||
cluster: rook-ceph/rook-cluster-nw
|
||||
```
|
||||
|
||||
3. If both `public` and `cluster` selectors are specified the first one will run all the communication network and the second the internal cluster network\*
|
||||
|
||||
```yaml
|
||||
network:
|
||||
provider: multus
|
||||
selectors:
|
||||
public: rook-ceph/rook-public-nw
|
||||
cluster: rook-ceph/rook-cluster-nw
|
||||
```
|
||||
|
||||
\* Internal cluster traffic includes OSD heartbeats, data replication, and data recovery
|
||||
|
||||
Only OSD pods will have both Public and Cluster networks attached. The rest of the Ceph component pods and CSI pods will only have the Public network attached.
|
||||
Rook Ceph operator will not have any networks attached as it proxies the required commands via a sidecar container in the mgr pod.
|
||||
|
||||
In order to work, each selector value must match a `NetworkAttachmentDefinition` object name in Multus.
|
||||
|
||||
For `multus` network provider, an already working cluster with Multus networking is required. Network attachment definition that later will be attached to the cluster needs to be created before the Cluster CRD.
|
||||
The Network attachment definitions should be using whereabouts cni.
|
||||
If Rook cannot find the provided Network attachment definition it will fail running the Ceph OSD pods.
|
||||
You can add the Multus network attachment selection annotation selecting the created network attachment definition on `selectors`.
|
||||
|
||||
A valid NetworkAttachmentDefinition will look like following:
|
||||
A NetworkAttachmentDefinition must exist before it can be used by Multus for a Ceph network. A
|
||||
recommended definition will look like the following:
|
||||
|
||||
```yaml
|
||||
apiVersion: "k8s.cni.cncf.io/v1"
|
||||
kind: NetworkAttachmentDefinition
|
||||
metadata:
|
||||
name: rook-public-nw
|
||||
name: ceph-multus-net
|
||||
namespace: rook-ceph
|
||||
spec:
|
||||
config: '{
|
||||
"cniVersion": "0.3.0",
|
||||
"name": "public-nad",
|
||||
"type": "macvlan",
|
||||
"master": "ens5",
|
||||
"master": "eth0",
|
||||
"mode": "bridge",
|
||||
"ipam": {
|
||||
"type": "whereabouts",
|
||||
"range": "192.168.1.0/24"
|
||||
"range": "192.168.200.0/24"
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
* Ensure that `master` matches the network interface of the host that you want to use.
|
||||
* IPAM type `whereabouts` is required because it makes sure that all the pods get a unique IP address from the multus network.
|
||||
* The NetworkAttachmentDefinition should be referenced along with the namespace in which it is present like `public: <namespace>/<name of NAD>`.
|
||||
e.g., the network attachment definition are in `default` namespace:
|
||||
* Ensure that `master` matches the network interface on hosts that you want to use.
|
||||
It must be the same across all hosts.
|
||||
* CNI type `macvlan` is highly recommended.
|
||||
It has less CPU and memory overhead compared to traditional Linux `bridge` configurations.
|
||||
* IPAM type `whereabouts` is recommended because it ensures each pod gets an IP address unique
|
||||
within the Kubernetes cluster. No DHCP server is required. If a DHCP server is present on the
|
||||
network, ensure the IP range does not overlap with the DHCP server's range.
|
||||
|
||||
```yaml
|
||||
public: default/rook-public-nw
|
||||
cluster: default/rook-cluster-nw
|
||||
```
|
||||
NetworkAttachmentDefinitions are selected for the desired Ceph network using `selectors`. Selector
|
||||
values should include the namespace in which the NAD is present. `public` and `cluster` may be
|
||||
selected independently. If `public` is left unspecified, Rook will configure Ceph to use the
|
||||
Kubernetes pod network for Ceph client traffic.
|
||||
|
||||
* This format is required in order to use the NetworkAttachmentDefinition across namespaces.
|
||||
* In Openshift, to use a NetworkAttachmentDefinition (NAD) across namespaces, the NAD must be deployed in the `default` namespace. The NAD is then referenced with the namespace: `default/rook-public-nw`
|
||||
Consider the example below which selects a hypothetical Kubernetes-wide Multus network in the
|
||||
default namespace for Ceph's public network and selects a Ceph-specific network in the `rook-ceph`
|
||||
namespace for Ceph's cluster network. The commented-out portion shows an example of how address
|
||||
ranges could be manually specified for the networks if needed.
|
||||
|
||||
```yaml
|
||||
network:
|
||||
provider: multus
|
||||
selectors:
|
||||
public: default/kube-multus-net
|
||||
cluster: rook-ceph/ceph-multus-net
|
||||
# addressRanges:
|
||||
# public:
|
||||
# - "192.168.100.0/24"
|
||||
# - "192.168.101.0/24"
|
||||
# cluster:
|
||||
# - "192.168.200.0/24"
|
||||
```
|
||||
|
||||
##### Validating Multus configuration
|
||||
|
||||
|
||||
@@ -2505,6 +2505,51 @@ string
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<h3 id="ceph.rook.io/v1.AddressRangesSpec">AddressRangesSpec
|
||||
</h3>
|
||||
<p>
|
||||
(<em>Appears on:</em><a href="#ceph.rook.io/v1.NetworkSpec">NetworkSpec</a>)
|
||||
</p>
|
||||
<div>
|
||||
</div>
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Field</th>
|
||||
<th>Description</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td>
|
||||
<code>public</code><br/>
|
||||
<em>
|
||||
<a href="#ceph.rook.io/v1.CIDRList">
|
||||
CIDRList
|
||||
</a>
|
||||
</em>
|
||||
</td>
|
||||
<td>
|
||||
<em>(Optional)</em>
|
||||
<p>Public defines a list of CIDRs to use for Ceph public network communication.</p>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>
|
||||
<code>cluster</code><br/>
|
||||
<em>
|
||||
<a href="#ceph.rook.io/v1.CIDRList">
|
||||
CIDRList
|
||||
</a>
|
||||
</em>
|
||||
</td>
|
||||
<td>
|
||||
<em>(Optional)</em>
|
||||
<p>Cluster defines a list of CIDRs to use for Ceph cluster network communication.</p>
|
||||
</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<h3 id="ceph.rook.io/v1.Annotations">Annotations
|
||||
(<code>map[string]string</code> alias)</h3>
|
||||
<p>
|
||||
@@ -2716,6 +2761,13 @@ int64
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<h3 id="ceph.rook.io/v1.CIDR">CIDR
|
||||
(<code>string</code> alias)</h3>
|
||||
<div>
|
||||
<p>An IPv4 or IPv6 network CIDR.</p>
|
||||
<p>This naive kubebuilder regex provides immediate feedback for some typos and for a common problem
|
||||
case where the range spec is forgotten (e.g., /24). Rook does in-depth validation in code.</p>
|
||||
</div>
|
||||
<h3 id="ceph.rook.io/v1.COSIDeploymentStrategy">COSIDeploymentStrategy
|
||||
(<code>string</code> alias)</h3>
|
||||
<p>
|
||||
@@ -3508,6 +3560,25 @@ string
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<h3 id="ceph.rook.io/v1.CephNetworkType">CephNetworkType
|
||||
(<code>string</code> alias)</h3>
|
||||
<div>
|
||||
<p>CephNetworkType should be “public” or “cluster”.
|
||||
Allow any string so that over-specified legacy clusters do not break on CRD update.</p>
|
||||
</div>
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Value</th>
|
||||
<th>Description</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody><tr><td><p>"cluster"</p></td>
|
||||
<td></td>
|
||||
</tr><tr><td><p>"public"</p></td>
|
||||
<td></td>
|
||||
</tr></tbody>
|
||||
</table>
|
||||
<h3 id="ceph.rook.io/v1.CephStatus">CephStatus
|
||||
</h3>
|
||||
<p>
|
||||
@@ -7979,6 +8050,29 @@ PoolSpec
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<h3 id="ceph.rook.io/v1.NetworkProviderType">NetworkProviderType
|
||||
(<code>string</code> alias)</h3>
|
||||
<p>
|
||||
(<em>Appears on:</em><a href="#ceph.rook.io/v1.NetworkSpec">NetworkSpec</a>)
|
||||
</p>
|
||||
<div>
|
||||
<p>NetworkProviderType defines valid network providers for Rook.</p>
|
||||
</div>
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Value</th>
|
||||
<th>Description</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody><tr><td><p>""</p></td>
|
||||
<td></td>
|
||||
</tr><tr><td><p>"host"</p></td>
|
||||
<td></td>
|
||||
</tr><tr><td><p>"multus"</p></td>
|
||||
<td></td>
|
||||
</tr></tbody>
|
||||
</table>
|
||||
<h3 id="ceph.rook.io/v1.NetworkSpec">NetworkSpec
|
||||
</h3>
|
||||
<p>
|
||||
@@ -7999,7 +8093,9 @@ PoolSpec
|
||||
<td>
|
||||
<code>provider</code><br/>
|
||||
<em>
|
||||
string
|
||||
<a href="#ceph.rook.io/v1.NetworkProviderType">
|
||||
NetworkProviderType
|
||||
</a>
|
||||
</em>
|
||||
</td>
|
||||
<td>
|
||||
@@ -8011,14 +8107,45 @@ string
|
||||
<td>
|
||||
<code>selectors</code><br/>
|
||||
<em>
|
||||
map[string]string
|
||||
map[github.com/rook/rook/pkg/apis/ceph.rook.io/v1.CephNetworkType]string
|
||||
</em>
|
||||
</td>
|
||||
<td>
|
||||
<em>(Optional)</em>
|
||||
<p>Selectors string values describe what networks will be used to connect the cluster.
|
||||
Meanwhile the keys describe each network respective responsibilities or any metadata
|
||||
storage provider decide.</p>
|
||||
<p>Selectors define NetworkAttachmentDefinitions to be used for Ceph public and/or cluster
|
||||
networks when the “multus” network provider is used. This config section is not used for
|
||||
other network providers.</p>
|
||||
<p>Valid keys are “public” and “cluster”. Refer to Ceph networking documentation for more:
|
||||
<a href="https://docs.ceph.com/en/reef/rados/configuration/network-config-ref/">https://docs.ceph.com/en/reef/rados/configuration/network-config-ref/</a></p>
|
||||
<p>Refer to Multus network annotation documentation for help selecting values:
|
||||
<a href="https://github.com/k8snetworkplumbingwg/multus-cni/blob/master/docs/how-to-use.md#run-pod-with-network-annotation">https://github.com/k8snetworkplumbingwg/multus-cni/blob/master/docs/how-to-use.md#run-pod-with-network-annotation</a></p>
|
||||
<p>Rook will make a best-effort attempt to automatically detect CIDR address ranges for given
|
||||
network attachment definitions. Rook’s methods are robust but may be imprecise for
|
||||
sufficiently complicated networks. Rook’s auto-detection process obtains a new IP address
|
||||
lease for each CephCluster reconcile. If Rook fails to detect, incorrectly detects, only
|
||||
partially detects, or if underlying networks do not support reusing old IP addresses, it is
|
||||
best to use the ‘addressRanges’ config section to specify CIDR ranges for the Ceph cluster.</p>
|
||||
<p>As a contrived example, one can use a theoretical Kubernetes-wide network for Ceph client
|
||||
traffic and a theoretical Rook-only network for Ceph replication traffic as shown:
|
||||
selectors:
|
||||
public: “default/cluster-fast-net”
|
||||
cluster: “rook-ceph/ceph-backend-net”</p>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>
|
||||
<code>addressRanges</code><br/>
|
||||
<em>
|
||||
<a href="#ceph.rook.io/v1.AddressRangesSpec">
|
||||
AddressRangesSpec
|
||||
</a>
|
||||
</em>
|
||||
</td>
|
||||
<td>
|
||||
<em>(Optional)</em>
|
||||
<p>AddressRanges specify a list of CIDRs that Rook will apply to Ceph’s ‘public_network’ and/or
|
||||
‘cluster_network’ configurations. This config section may be used for the “host” or “multus”
|
||||
network providers.</p>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
|
||||
@@ -363,12 +363,17 @@ ceph osd primary-affinity osd.0 0
|
||||
|
||||
## OSD Dedicated Network
|
||||
|
||||
!!! tip
|
||||
This documentation is left for historical purposes. It is still valid, but Rook offers native
|
||||
support for this feature via the
|
||||
[CephCluster network configuration](../../CRDs/Cluster/ceph-cluster-crd.md#ceph-public-and-cluster-networks).
|
||||
|
||||
It is possible to configure ceph to leverage a dedicated network for the OSDs to
|
||||
communicate across. A useful overview is the [CEPH Networks](http://docs.ceph.com/docs/master/rados/configuration/network-config-ref/#ceph-networks)
|
||||
communicate across. A useful overview is the [Ceph Networks](http://docs.ceph.com/docs/master/rados/configuration/network-config-ref/#ceph-networks)
|
||||
section of the Ceph documentation. If you declare a cluster network, OSDs will
|
||||
route heartbeat, object replication and recovery traffic over the cluster
|
||||
route heartbeat, object replication, and recovery traffic over the cluster
|
||||
network. This may improve performance compared to using a single network,
|
||||
especially when slower network technologies are used, with the tradeoff of
|
||||
especially when slower network technologies are used. The tradeoff is
|
||||
additional expense and subtle failure modes.
|
||||
|
||||
Two changes are necessary to the configuration to enable this capability:
|
||||
@@ -404,7 +409,7 @@ apiVersion: v1
|
||||
data:
|
||||
config: |
|
||||
[global]
|
||||
public network = 10.0.7.0/24
|
||||
public network = 10.0.7.0/24
|
||||
cluster network = 10.0.10.0/24
|
||||
public addr = ""
|
||||
cluster addr = ""
|
||||
|
||||
@@ -2125,6 +2125,25 @@ spec:
|
||||
description: Network related configuration
|
||||
nullable: true
|
||||
properties:
|
||||
addressRanges:
|
||||
description: AddressRanges specify a list of CIDRs that Rook will apply to Ceph's 'public_network' and/or 'cluster_network' configurations. This config section may be used for the "host" or "multus" network providers.
|
||||
nullable: true
|
||||
properties:
|
||||
cluster:
|
||||
description: Cluster defines a list of CIDRs to use for Ceph cluster network communication.
|
||||
items:
|
||||
description: "An IPv4 or IPv6 network CIDR. \n This naive kubebuilder regex provides immediate feedback for some typos and for a common problem case where the range spec is forgotten (e.g., /24). Rook does in-depth validation in code."
|
||||
pattern: ^[0-9a-fA-F:.]{2,}\/[0-9]{1,3}$
|
||||
type: string
|
||||
type: array
|
||||
public:
|
||||
description: Public defines a list of CIDRs to use for Ceph public network communication.
|
||||
items:
|
||||
description: "An IPv4 or IPv6 network CIDR. \n This naive kubebuilder regex provides immediate feedback for some typos and for a common problem case where the range spec is forgotten (e.g., /24). Rook does in-depth validation in code."
|
||||
pattern: ^[0-9a-fA-F:.]{2,}\/[0-9]{1,3}$
|
||||
type: string
|
||||
type: array
|
||||
type: object
|
||||
connections:
|
||||
description: Settings for network connections such as compression and encryption across the wire.
|
||||
nullable: true
|
||||
@@ -2174,12 +2193,16 @@ spec:
|
||||
type: object
|
||||
provider:
|
||||
description: Provider is what provides network connectivity to the cluster e.g. "host" or "multus"
|
||||
enum:
|
||||
- ""
|
||||
- host
|
||||
- multus
|
||||
nullable: true
|
||||
type: string
|
||||
selectors:
|
||||
additionalProperties:
|
||||
type: string
|
||||
description: Selectors string values describe what networks will be used to connect the cluster. Meanwhile the keys describe each network respective responsibilities or any metadata storage provider decide.
|
||||
description: "Selectors define NetworkAttachmentDefinitions to be used for Ceph public and/or cluster networks when the \"multus\" network provider is used. This config section is not used for other network providers. \n Valid keys are \"public\" and \"cluster\". Refer to Ceph networking documentation for more: https://docs.ceph.com/en/reef/rados/configuration/network-config-ref/ \n Refer to Multus network annotation documentation for help selecting values: https://github.com/k8snetworkplumbingwg/multus-cni/blob/master/docs/how-to-use.md#run-pod-with-network-annotation \n Rook will make a best-effort attempt to automatically detect CIDR address ranges for given network attachment definitions. Rook's methods are robust but may be imprecise for sufficiently complicated networks. Rook's auto-detection process obtains a new IP address lease for each CephCluster reconcile. If Rook fails to detect, incorrectly detects, only partially detects, or if underlying networks do not support reusing old IP addresses, it is best to use the 'addressRanges' config section to specify CIDR ranges for the Ceph cluster. \n As a contrived example, one can use a theoretical Kubernetes-wide network for Ceph client traffic and a theoretical Rook-only network for Ceph replication traffic as shown: selectors: public: \"default/cluster-fast-net\" cluster: \"rook-ceph/ceph-backend-net\""
|
||||
nullable: true
|
||||
type: object
|
||||
type: object
|
||||
|
||||
@@ -2123,6 +2123,25 @@ spec:
|
||||
description: Network related configuration
|
||||
nullable: true
|
||||
properties:
|
||||
addressRanges:
|
||||
description: AddressRanges specify a list of CIDRs that Rook will apply to Ceph's 'public_network' and/or 'cluster_network' configurations. This config section may be used for the "host" or "multus" network providers.
|
||||
nullable: true
|
||||
properties:
|
||||
cluster:
|
||||
description: Cluster defines a list of CIDRs to use for Ceph cluster network communication.
|
||||
items:
|
||||
description: "An IPv4 or IPv6 network CIDR. \n This naive kubebuilder regex provides immediate feedback for some typos and for a common problem case where the range spec is forgotten (e.g., /24). Rook does in-depth validation in code."
|
||||
pattern: ^[0-9a-fA-F:.]{2,}\/[0-9]{1,3}$
|
||||
type: string
|
||||
type: array
|
||||
public:
|
||||
description: Public defines a list of CIDRs to use for Ceph public network communication.
|
||||
items:
|
||||
description: "An IPv4 or IPv6 network CIDR. \n This naive kubebuilder regex provides immediate feedback for some typos and for a common problem case where the range spec is forgotten (e.g., /24). Rook does in-depth validation in code."
|
||||
pattern: ^[0-9a-fA-F:.]{2,}\/[0-9]{1,3}$
|
||||
type: string
|
||||
type: array
|
||||
type: object
|
||||
connections:
|
||||
description: Settings for network connections such as compression and encryption across the wire.
|
||||
nullable: true
|
||||
@@ -2172,12 +2191,16 @@ spec:
|
||||
type: object
|
||||
provider:
|
||||
description: Provider is what provides network connectivity to the cluster e.g. "host" or "multus"
|
||||
enum:
|
||||
- ""
|
||||
- host
|
||||
- multus
|
||||
nullable: true
|
||||
type: string
|
||||
selectors:
|
||||
additionalProperties:
|
||||
type: string
|
||||
description: Selectors string values describe what networks will be used to connect the cluster. Meanwhile the keys describe each network respective responsibilities or any metadata storage provider decide.
|
||||
description: "Selectors define NetworkAttachmentDefinitions to be used for Ceph public and/or cluster networks when the \"multus\" network provider is used. This config section is not used for other network providers. \n Valid keys are \"public\" and \"cluster\". Refer to Ceph networking documentation for more: https://docs.ceph.com/en/reef/rados/configuration/network-config-ref/ \n Refer to Multus network annotation documentation for help selecting values: https://github.com/k8snetworkplumbingwg/multus-cni/blob/master/docs/how-to-use.md#run-pod-with-network-annotation \n Rook will make a best-effort attempt to automatically detect CIDR address ranges for given network attachment definitions. Rook's methods are robust but may be imprecise for sufficiently complicated networks. Rook's auto-detection process obtains a new IP address lease for each CephCluster reconcile. If Rook fails to detect, incorrectly detects, only partially detects, or if underlying networks do not support reusing old IP addresses, it is best to use the 'addressRanges' config section to specify CIDR ranges for the Ceph cluster. \n As a contrived example, one can use a theoretical Kubernetes-wide network for Ceph client traffic and a theoretical Rook-only network for Ceph replication traffic as shown: selectors: public: \"default/cluster-fast-net\" cluster: \"rook-ceph/ceph-backend-net\""
|
||||
nullable: true
|
||||
type: object
|
||||
type: object
|
||||
|
||||
@@ -117,6 +117,7 @@ require (
|
||||
github.com/prometheus/procfs v0.11.1 // indirect
|
||||
github.com/ryanuber/go-glob v1.0.0 // indirect
|
||||
github.com/sirupsen/logrus v1.9.3 // indirect
|
||||
github.com/stretchr/objx v0.5.0 // indirect
|
||||
github.com/xlab/treeprint v1.2.0 // indirect
|
||||
go.starlark.net v0.0.0-20230814145427-12f4cb8177e4 // indirect
|
||||
go.uber.org/multierr v1.11.0 // indirect
|
||||
|
||||
@@ -67,6 +67,11 @@ func (c *CephCluster) ValidateCreate() (admission.Warnings, error) {
|
||||
return nil, errors.New("invalid create : external mode enabled cannot have mon,dashboard,monitoring,network,disruptionManagement,storage fields in CR")
|
||||
}
|
||||
}
|
||||
|
||||
if err := ValidateNetworkSpec(c.GetNamespace(), c.Spec.Network); err != nil {
|
||||
return nil, errors.Errorf("invalid create: %s", err)
|
||||
}
|
||||
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
@@ -85,11 +90,8 @@ func validateUpdatedCephCluster(updatedCephCluster *CephCluster, found *CephClus
|
||||
return nil, errors.Errorf("invalid update: DataDirHostPath change from %q to %q is not allowed", found.Spec.DataDirHostPath, updatedCephCluster.Spec.DataDirHostPath)
|
||||
}
|
||||
|
||||
// Allow an attempt to enable or disable host networking, but not other provider changes
|
||||
oldProvider := updatedCephCluster.Spec.Network.Provider
|
||||
newProvider := found.Spec.Network.Provider
|
||||
if oldProvider != newProvider && oldProvider != "host" && newProvider != "host" {
|
||||
return nil, errors.Errorf("invalid update: Provider change from %q to %q is not allowed", found.Spec.Network.Provider, updatedCephCluster.Spec.Network.Provider)
|
||||
if err := ValidateNetworkSpecUpdate(found.GetNamespace(), found.Spec.Network, updatedCephCluster.Spec.Network); err != nil {
|
||||
return nil, errors.Errorf("invalid update: %s", err)
|
||||
}
|
||||
|
||||
if len(updatedCephCluster.Spec.Storage.StorageClassDeviceSets) == len(found.Spec.Storage.StorageClassDeviceSets) {
|
||||
|
||||
@@ -16,13 +16,160 @@ limitations under the License.
|
||||
|
||||
package v1
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net"
|
||||
"strings"
|
||||
|
||||
nadv1 "github.com/k8snetworkplumbingwg/network-attachment-definition-client/pkg/apis/k8s.cni.cncf.io/v1"
|
||||
nadutils "github.com/k8snetworkplumbingwg/network-attachment-definition-client/pkg/utils"
|
||||
"github.com/pkg/errors"
|
||||
)
|
||||
|
||||
// IsMultus get whether to use multus network provider
|
||||
func (n *NetworkSpec) IsMultus() bool {
|
||||
return n.Provider == "multus"
|
||||
return n.Provider == NetworkProviderMultus
|
||||
}
|
||||
|
||||
// IsHost get whether to use host network provider. This method also preserve
|
||||
// compatibility with the old HostNetwork field.
|
||||
func (n *NetworkSpec) IsHost() bool {
|
||||
return (n.HostNetwork && n.Provider == "") || n.Provider == "host"
|
||||
return (n.HostNetwork && n.Provider == NetworkProviderDefault) || n.Provider == NetworkProviderHost
|
||||
}
|
||||
|
||||
func ValidateNetworkSpec(clusterNamespace string, spec NetworkSpec) error {
|
||||
if spec.IsMultus() {
|
||||
if len(spec.Selectors) == 0 {
|
||||
return errors.Errorf("at least one network selector must be specified when using the %q network provider", NetworkProviderMultus)
|
||||
}
|
||||
|
||||
if _, err := spec.GetNetworkSelection(clusterNamespace, CephNetworkPublic); err != nil {
|
||||
return errors.Wrap(err, "ceph public network selector provided for multus is invalid")
|
||||
}
|
||||
if _, err := spec.GetNetworkSelection(clusterNamespace, CephNetworkCluster); err != nil {
|
||||
return errors.Wrap(err, "ceph cluster network selector provided for multus is invalid")
|
||||
}
|
||||
}
|
||||
|
||||
if !spec.AddressRanges.IsEmpty() {
|
||||
if !spec.IsMultus() && !spec.IsHost() {
|
||||
// TODO: be sure to update docs that AddressRanges can be specified for host networking as
|
||||
// well as multus so that the override configmap doesn't need to be set
|
||||
return errors.Errorf("network ranges can only be specified for %q and %q network providers", NetworkProviderHost, NetworkProviderMultus)
|
||||
}
|
||||
if spec.IsMultus() {
|
||||
if len(spec.AddressRanges.Public) > 0 && !spec.NetworkHasSelection(CephNetworkPublic) {
|
||||
return errors.Errorf("public address range can only be specified for multus if there is a public network selection")
|
||||
}
|
||||
if len(spec.AddressRanges.Cluster) > 0 && !spec.NetworkHasSelection(CephNetworkCluster) {
|
||||
return errors.Errorf("cluster address range can only be specified for multus if there is a cluster network selection")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if err := spec.AddressRanges.Validate(); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func ValidateNetworkSpecUpdate(clusterNamespace string, oldSpec, newSpec NetworkSpec) error {
|
||||
// Allow an attempt to enable or disable host networking, but not other provider changes
|
||||
oldProvider := oldSpec.Provider
|
||||
newProvider := newSpec.Provider
|
||||
if oldProvider != newProvider && oldProvider != "host" && newProvider != "host" {
|
||||
return errors.Errorf("invalid update: network provider change from %q to %q is not allowed", oldProvider, newProvider)
|
||||
}
|
||||
|
||||
return ValidateNetworkSpec(clusterNamespace, newSpec)
|
||||
}
|
||||
|
||||
// NetworkHasSelection returns true if the given Ceph network has a selection.
|
||||
func (n *NetworkSpec) NetworkHasSelection(network CephNetworkType) bool {
|
||||
s, ok := n.Selectors[network]
|
||||
if !ok || s == "" {
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// GetNetworkSelection gets the network selection for a given Ceph network, or nil if the network
|
||||
// doesn't have a selection.
|
||||
func (n *NetworkSpec) GetNetworkSelection(clusterNamespace string, network CephNetworkType) (*nadv1.NetworkSelectionElement, error) {
|
||||
if !n.NetworkHasSelection(network) {
|
||||
return nil, nil // no selection for network
|
||||
}
|
||||
s := n.Selectors[network]
|
||||
// From documentation of the "k8s.v1.cni.cncf.io/network-status" annotation, valid JSON inputs
|
||||
// must be in list form, surrounded with brackets. The NAD utility library will only parse
|
||||
// list-format JSON input. However, old versions of Rook code allowed non-list JSON objects.
|
||||
// In order to support legacy users, make an attempt to turn single-JSON-object inputs into
|
||||
// len(1) lists so that they parse correctly by the util library. Do not advertise this
|
||||
// "feature" in documentation since it is not technically the correct format.
|
||||
if strings.HasPrefix(s, "{") && strings.HasSuffix(s, "}") {
|
||||
s = "[" + s + "]"
|
||||
}
|
||||
selection, err := nadutils.ParseNetworkAnnotation(s, clusterNamespace)
|
||||
if err != nil {
|
||||
return nil, errors.Wrapf(err, "failed to parse %q network selector %q", network, s)
|
||||
}
|
||||
if len(selection) != 1 {
|
||||
return nil, errors.Errorf("%q network selector %q has multiple (%d) selections, which is not supported", network, s, len(selection))
|
||||
}
|
||||
return selection[0], nil
|
||||
}
|
||||
|
||||
// NetworkSelectionsToAnnotationValue converts NetworkAttachmentDefinition network selection
|
||||
// elements to an annotation value for the "k8s.v1.cni.cncf.io/networks" annotation key.
|
||||
func NetworkSelectionsToAnnotationValue(selections ...*nadv1.NetworkSelectionElement) (string, error) {
|
||||
reduced := []*nadv1.NetworkSelectionElement{}
|
||||
for _, s := range selections {
|
||||
if s != nil {
|
||||
reduced = append(reduced, s)
|
||||
}
|
||||
}
|
||||
if len(reduced) == 0 {
|
||||
return "", nil
|
||||
}
|
||||
b, err := json.Marshal(reduced)
|
||||
if err != nil {
|
||||
return "", errors.Wrap(err, "failed to convert network selections to annotation value")
|
||||
}
|
||||
return string(b), nil
|
||||
}
|
||||
|
||||
func (n *AddressRangesSpec) IsEmpty() bool {
|
||||
return n == nil || len(n.Public) == 0 && len(n.Cluster) == 0
|
||||
}
|
||||
|
||||
func (n *AddressRangesSpec) Validate() error {
|
||||
if n.IsEmpty() {
|
||||
return nil
|
||||
}
|
||||
|
||||
allRanges := append(n.Public, n.Cluster...)
|
||||
invalid := []string{}
|
||||
for _, cidr := range allRanges {
|
||||
_, _, err := net.ParseCIDR(string(cidr))
|
||||
if err != nil {
|
||||
// returned err is "invalid CIDR: <addr>" & not more useful than invalid list below
|
||||
invalid = append(invalid, string(cidr))
|
||||
}
|
||||
}
|
||||
if len(invalid) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
return fmt.Errorf("%d network ranges are invalid: %v", len(invalid), invalid)
|
||||
}
|
||||
|
||||
// String turns a CIDR list into a comma-delimited string of CIDRs
|
||||
func (l *CIDRList) String() string {
|
||||
sl := []string{}
|
||||
for _, c := range *l {
|
||||
sl = append(sl, string(c))
|
||||
}
|
||||
return strings.Join(sl, ", ")
|
||||
}
|
||||
@@ -18,8 +18,10 @@ package v1
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"testing"
|
||||
|
||||
nadv1 "github.com/k8snetworkplumbingwg/network-attachment-definition-client/pkg/apis/k8s.cni.cncf.io/v1"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"k8s.io/apimachinery/pkg/util/yaml"
|
||||
)
|
||||
@@ -63,7 +65,7 @@ selectors:
|
||||
|
||||
expected := NetworkSpec{
|
||||
Provider: "host",
|
||||
Selectors: map[string]string{
|
||||
Selectors: map[CephNetworkType]string{
|
||||
"server": "enp2s0f0",
|
||||
"broker": "enp2s0f0",
|
||||
},
|
||||
@@ -71,3 +73,292 @@ selectors:
|
||||
|
||||
assert.Equal(t, expected, net)
|
||||
}
|
||||
|
||||
func TestAddressRangesSpec_IsEmpty(t *testing.T) {
|
||||
var specNil *AddressRangesSpec
|
||||
assert.True(t, specNil.IsEmpty())
|
||||
|
||||
empty := &AddressRangesSpec{}
|
||||
assert.True(t, empty.IsEmpty())
|
||||
|
||||
someCIDR := CIDR("1.1.1.1/16")
|
||||
nonEmptyTests := []AddressRangesSpec{
|
||||
{Public: []CIDR{someCIDR}},
|
||||
{Public: []CIDR{someCIDR, someCIDR}},
|
||||
{Cluster: []CIDR{someCIDR}},
|
||||
{Cluster: []CIDR{someCIDR, someCIDR}},
|
||||
{Public: []CIDR{someCIDR}, Cluster: []CIDR{someCIDR}},
|
||||
{Public: []CIDR{someCIDR, someCIDR}, Cluster: []CIDR{someCIDR, someCIDR}},
|
||||
}
|
||||
for _, spec := range nonEmptyTests {
|
||||
assert.False(t, spec.IsEmpty())
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
func TestAddressRangesSpec_Validate(t *testing.T) {
|
||||
// only test a small subset of CIDRs since Rook should definitely use the Go stdlib underneath
|
||||
v1 := CIDR("123.123.123.123/24")
|
||||
v2 := CIDR("1.0.0.1/24")
|
||||
v3 := CIDR("2000::/64")
|
||||
v4 := CIDR("2000:2000:2000:2000:2000:2000:2000:2000/64")
|
||||
v5 := CIDR("2000::128.128.128.128/96") // ipv4 expressed as subnet of ipv6 is valid
|
||||
|
||||
// invalid CIDRs
|
||||
i1 := CIDR("123.123.123/24")
|
||||
i2 := CIDR("123.123.123.123/33")
|
||||
i4 := CIDR("2000/64")
|
||||
i3 := CIDR("2000:/64")
|
||||
i5 := CIDR("2000::128.128.128.128/129")
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
spec AddressRangesSpec
|
||||
numErrs int
|
||||
}{
|
||||
{"empty", AddressRangesSpec{}, 0},
|
||||
{"all valid", AddressRangesSpec{
|
||||
Public: []CIDR{v1},
|
||||
Cluster: []CIDR{v2, v3, v4, v5},
|
||||
}, 0},
|
||||
{"all invalid", AddressRangesSpec{
|
||||
Public: []CIDR{i1},
|
||||
Cluster: []CIDR{i2, i3, i4, i5},
|
||||
}, 5},
|
||||
{"public only, valid", AddressRangesSpec{Public: []CIDR{v1}}, 0},
|
||||
{"public only, invalid", AddressRangesSpec{Public: []CIDR{i1}}, 1},
|
||||
{"cluster only, valid", AddressRangesSpec{Cluster: []CIDR{v2}}, 0},
|
||||
{"cluster only, invalid", AddressRangesSpec{Cluster: []CIDR{i2}}, 1},
|
||||
{"public valid, cluster valid", AddressRangesSpec{
|
||||
Public: []CIDR{v1},
|
||||
Cluster: []CIDR{v2},
|
||||
}, 0},
|
||||
{"public valid, cluster invalid", AddressRangesSpec{
|
||||
Public: []CIDR{v2},
|
||||
Cluster: []CIDR{i2},
|
||||
}, 1},
|
||||
{"public invalid, cluster valid", AddressRangesSpec{
|
||||
Public: []CIDR{i3},
|
||||
Cluster: []CIDR{v2},
|
||||
}, 1},
|
||||
{"public invalid, cluster invalid", AddressRangesSpec{
|
||||
Public: []CIDR{i3},
|
||||
Cluster: []CIDR{i4},
|
||||
}, 2},
|
||||
{"both, valid and invalid", AddressRangesSpec{
|
||||
Public: []CIDR{v1, i2},
|
||||
Cluster: []CIDR{v3, i4},
|
||||
}, 2},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
err := tt.spec.Validate()
|
||||
if tt.numErrs > 0 {
|
||||
assert.Error(t, err)
|
||||
t.Log(err)
|
||||
assert.ErrorContains(t, err, fmt.Sprintf("%d network ranges are invalid", tt.numErrs))
|
||||
} else {
|
||||
assert.NoError(t, err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// these two functions are should almost always used together and can be unit tested together more
|
||||
// easily than apart
|
||||
func TestNetworkSpec_GetNetworkSelection_NetworkSelectionsToAnnotationValue(t *testing.T) {
|
||||
// inputs are the same definition expressed in json format or non-json format
|
||||
input1 := func(json bool) string {
|
||||
if json {
|
||||
return `[{"name": "macvlan", "interface": "net1"}]`
|
||||
}
|
||||
return "macvlan@net1"
|
||||
}
|
||||
input2 := func(json bool) string {
|
||||
if json {
|
||||
return `[{"name": "macvlan", "interface": "net2"}]`
|
||||
}
|
||||
return "macvlan@net2"
|
||||
}
|
||||
|
||||
// allow running the test suite with json-format or non-json-format inputs
|
||||
testGetNetworkAnnotationValue := func(t *testing.T, json bool) {
|
||||
t.Helper()
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
specSelectors map[CephNetworkType]string
|
||||
cephNets []CephNetworkType
|
||||
want string
|
||||
wantErr bool
|
||||
}{
|
||||
{
|
||||
name: "public want public",
|
||||
specSelectors: map[CephNetworkType]string{
|
||||
"public": input1(json),
|
||||
},
|
||||
cephNets: []CephNetworkType{CephNetworkPublic},
|
||||
want: `[{"name":"macvlan","namespace":"ns","interface":"net1"}]`,
|
||||
wantErr: false,
|
||||
},
|
||||
{
|
||||
name: "cluster want cluster",
|
||||
specSelectors: map[CephNetworkType]string{
|
||||
"cluster": input1(json),
|
||||
},
|
||||
cephNets: []CephNetworkType{CephNetworkCluster},
|
||||
want: `[{"name":"macvlan","namespace":"ns","interface":"net1"}]`,
|
||||
wantErr: false,
|
||||
},
|
||||
{
|
||||
name: "public want cluster",
|
||||
specSelectors: map[CephNetworkType]string{
|
||||
"public": input1(json),
|
||||
},
|
||||
cephNets: []CephNetworkType{CephNetworkCluster},
|
||||
want: ``,
|
||||
wantErr: false,
|
||||
},
|
||||
{
|
||||
name: "cluster want public",
|
||||
specSelectors: map[CephNetworkType]string{
|
||||
"cluster": input1(json),
|
||||
},
|
||||
cephNets: []CephNetworkType{CephNetworkPublic},
|
||||
want: ``,
|
||||
wantErr: false,
|
||||
},
|
||||
{
|
||||
name: "nothing want public",
|
||||
specSelectors: map[CephNetworkType]string{},
|
||||
cephNets: []CephNetworkType{CephNetworkPublic},
|
||||
want: ``,
|
||||
wantErr: false,
|
||||
},
|
||||
{
|
||||
name: "nothing want cluster",
|
||||
specSelectors: map[CephNetworkType]string{},
|
||||
cephNets: []CephNetworkType{CephNetworkCluster},
|
||||
want: ``,
|
||||
wantErr: false,
|
||||
},
|
||||
{
|
||||
name: "unknown want public",
|
||||
specSelectors: map[CephNetworkType]string{
|
||||
"uncleKnown": input1(json),
|
||||
},
|
||||
cephNets: []CephNetworkType{CephNetworkPublic},
|
||||
want: ``,
|
||||
wantErr: false,
|
||||
},
|
||||
{
|
||||
name: "unknown want cluster",
|
||||
specSelectors: map[CephNetworkType]string{
|
||||
"uncleKnown": input1(json),
|
||||
},
|
||||
cephNets: []CephNetworkType{CephNetworkCluster},
|
||||
want: ``,
|
||||
wantErr: false,
|
||||
},
|
||||
{
|
||||
name: "public want public and cluster",
|
||||
specSelectors: map[CephNetworkType]string{
|
||||
"public": input1(json),
|
||||
},
|
||||
cephNets: []CephNetworkType{CephNetworkPublic, CephNetworkCluster},
|
||||
want: `[{"name":"macvlan","namespace":"ns","interface":"net1"}]`,
|
||||
wantErr: false,
|
||||
},
|
||||
{
|
||||
name: "cluster want public and cluster",
|
||||
specSelectors: map[CephNetworkType]string{
|
||||
"cluster": input1(json),
|
||||
},
|
||||
cephNets: []CephNetworkType{CephNetworkPublic, CephNetworkCluster},
|
||||
want: `[{"name":"macvlan","namespace":"ns","interface":"net1"}]`,
|
||||
wantErr: false,
|
||||
},
|
||||
{
|
||||
name: "public and cluster want public and cluster",
|
||||
specSelectors: map[CephNetworkType]string{
|
||||
"public": input1(json),
|
||||
"cluster": input2(json),
|
||||
},
|
||||
cephNets: []CephNetworkType{CephNetworkPublic, CephNetworkCluster},
|
||||
want: `[{"name":"macvlan","namespace":"ns","interface":"net1"},{"name":"macvlan","namespace":"ns","interface":"net2"}]`,
|
||||
wantErr: false,
|
||||
},
|
||||
{
|
||||
name: "support mixed json-non-json spec",
|
||||
specSelectors: map[CephNetworkType]string{
|
||||
"public": input1(json),
|
||||
"cluster": input2(!json), // invert json-ness of this one
|
||||
},
|
||||
cephNets: []CephNetworkType{CephNetworkPublic, CephNetworkCluster},
|
||||
want: `[{"name":"macvlan","namespace":"ns","interface":"net1"},{"name":"macvlan","namespace":"ns","interface":"net2"}]`,
|
||||
wantErr: false,
|
||||
},
|
||||
{
|
||||
name: "public and cluster want nothing",
|
||||
specSelectors: map[CephNetworkType]string{
|
||||
"public": input1(json),
|
||||
"cluster": input2(json),
|
||||
},
|
||||
cephNets: []CephNetworkType{},
|
||||
want: ``,
|
||||
wantErr: false,
|
||||
},
|
||||
{
|
||||
name: "legacy single json object support",
|
||||
specSelectors: map[CephNetworkType]string{
|
||||
"public": `{"name": "legacyJsonObject"}`,
|
||||
},
|
||||
cephNets: []CephNetworkType{CephNetworkPublic, CephNetworkCluster},
|
||||
want: `[{"name":"legacyJsonObject","namespace":"ns"}]`,
|
||||
wantErr: false,
|
||||
},
|
||||
{
|
||||
name: "invalid network selections",
|
||||
specSelectors: map[CephNetworkType]string{
|
||||
"public": `[{"name": "jsonWithNoClosingBracket"}`,
|
||||
"cluster": "multus%net",
|
||||
},
|
||||
cephNets: []CephNetworkType{CephNetworkPublic, CephNetworkCluster},
|
||||
want: ``,
|
||||
wantErr: true,
|
||||
},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
n := &NetworkSpec{
|
||||
Selectors: tt.specSelectors,
|
||||
}
|
||||
|
||||
selections := []*nadv1.NetworkSelectionElement{}
|
||||
errs := []error{}
|
||||
for _, net := range tt.cephNets {
|
||||
s, err := n.GetNetworkSelection("ns", net)
|
||||
if err != nil {
|
||||
errs = append(errs, err)
|
||||
}
|
||||
selections = append(selections, s)
|
||||
}
|
||||
got, err := NetworkSelectionsToAnnotationValue(selections...)
|
||||
if err != nil {
|
||||
errs = append(errs, err)
|
||||
}
|
||||
|
||||
assert.Equal(t, tt.wantErr, len(errs) > 0, "wantErr %v but got errs %v", tt.wantErr, errs)
|
||||
assert.Equal(t, tt.want, got)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// Actual subtests
|
||||
t.Run("non-JSON input", func(t *testing.T) {
|
||||
testGetNetworkAnnotationValue(t, false)
|
||||
})
|
||||
t.Run("JSON input", func(t *testing.T) {
|
||||
testGetNetworkAnnotationValue(t, true)
|
||||
})
|
||||
}
|
||||
@@ -2267,14 +2267,41 @@ type NetworkSpec struct {
|
||||
// Provider is what provides network connectivity to the cluster e.g. "host" or "multus"
|
||||
// +nullable
|
||||
// +optional
|
||||
Provider string `json:"provider,omitempty"`
|
||||
Provider NetworkProviderType `json:"provider,omitempty"`
|
||||
|
||||
// Selectors string values describe what networks will be used to connect the cluster.
|
||||
// Meanwhile the keys describe each network respective responsibilities or any metadata
|
||||
// storage provider decide.
|
||||
// Selectors define NetworkAttachmentDefinitions to be used for Ceph public and/or cluster
|
||||
// networks when the "multus" network provider is used. This config section is not used for
|
||||
// other network providers.
|
||||
//
|
||||
// Valid keys are "public" and "cluster". Refer to Ceph networking documentation for more:
|
||||
// https://docs.ceph.com/en/reef/rados/configuration/network-config-ref/
|
||||
//
|
||||
// Refer to Multus network annotation documentation for help selecting values:
|
||||
// https://github.com/k8snetworkplumbingwg/multus-cni/blob/master/docs/how-to-use.md#run-pod-with-network-annotation
|
||||
//
|
||||
// Rook will make a best-effort attempt to automatically detect CIDR address ranges for given
|
||||
// network attachment definitions. Rook's methods are robust but may be imprecise for
|
||||
// sufficiently complicated networks. Rook's auto-detection process obtains a new IP address
|
||||
// lease for each CephCluster reconcile. If Rook fails to detect, incorrectly detects, only
|
||||
// partially detects, or if underlying networks do not support reusing old IP addresses, it is
|
||||
// best to use the 'addressRanges' config section to specify CIDR ranges for the Ceph cluster.
|
||||
//
|
||||
// As a contrived example, one can use a theoretical Kubernetes-wide network for Ceph client
|
||||
// traffic and a theoretical Rook-only network for Ceph replication traffic as shown:
|
||||
// selectors:
|
||||
// public: "default/cluster-fast-net"
|
||||
// cluster: "rook-ceph/ceph-backend-net"
|
||||
//
|
||||
// +nullable
|
||||
// +optional
|
||||
Selectors map[string]string `json:"selectors,omitempty"`
|
||||
Selectors map[CephNetworkType]string `json:"selectors,omitempty"`
|
||||
|
||||
// AddressRanges specify a list of CIDRs that Rook will apply to Ceph's 'public_network' and/or
|
||||
// 'cluster_network' configurations. This config section may be used for the "host" or "multus"
|
||||
// network providers.
|
||||
// +nullable
|
||||
// +optional
|
||||
AddressRanges *AddressRangesSpec `json:"addressRanges,omitempty"`
|
||||
|
||||
// Settings for network connections such as compression and encryption across the
|
||||
// wire.
|
||||
@@ -2301,6 +2328,45 @@ type NetworkSpec struct {
|
||||
MultiClusterService MultiClusterServiceSpec `json:"multiClusterService,omitempty"`
|
||||
}
|
||||
|
||||
// NetworkProviderType defines valid network providers for Rook.
|
||||
// +kubebuilder:validation:Enum="";host;multus
|
||||
type NetworkProviderType string
|
||||
|
||||
const (
|
||||
NetworkProviderDefault = NetworkProviderType("")
|
||||
NetworkProviderHost = NetworkProviderType("host")
|
||||
NetworkProviderMultus = NetworkProviderType("multus")
|
||||
)
|
||||
|
||||
// CephNetworkType should be "public" or "cluster".
|
||||
// Allow any string so that over-specified legacy clusters do not break on CRD update.
|
||||
type CephNetworkType string
|
||||
|
||||
const (
|
||||
CephNetworkPublic = CephNetworkType("public")
|
||||
CephNetworkCluster = CephNetworkType("cluster")
|
||||
)
|
||||
|
||||
type AddressRangesSpec struct {
|
||||
// Public defines a list of CIDRs to use for Ceph public network communication.
|
||||
// +optional
|
||||
Public CIDRList `json:"public"`
|
||||
|
||||
// Cluster defines a list of CIDRs to use for Ceph cluster network communication.
|
||||
// +optional
|
||||
Cluster CIDRList `json:"cluster"`
|
||||
}
|
||||
|
||||
// An IPv4 or IPv6 network CIDR.
|
||||
//
|
||||
// This naive kubebuilder regex provides immediate feedback for some typos and for a common problem
|
||||
// case where the range spec is forgotten (e.g., /24). Rook does in-depth validation in code.
|
||||
// +kubebuilder:validation:Pattern=`^[0-9a-fA-F:.]{2,}\/[0-9]{1,3}$`
|
||||
type CIDR string
|
||||
|
||||
// A list of CIDRs.
|
||||
type CIDRList []CIDR
|
||||
|
||||
type MultiClusterServiceSpec struct {
|
||||
// Enable multiClusterService to export the mon and OSD services to peer cluster.
|
||||
// Ensure that peer clusters are connected using an MCS API compatible application,
|
||||
|
||||
@@ -43,6 +43,32 @@ func (in *AMQPEndpointSpec) DeepCopy() *AMQPEndpointSpec {
|
||||
return out
|
||||
}
|
||||
|
||||
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
|
||||
func (in *AddressRangesSpec) DeepCopyInto(out *AddressRangesSpec) {
|
||||
*out = *in
|
||||
if in.Public != nil {
|
||||
in, out := &in.Public, &out.Public
|
||||
*out = make(CIDRList, len(*in))
|
||||
copy(*out, *in)
|
||||
}
|
||||
if in.Cluster != nil {
|
||||
in, out := &in.Cluster, &out.Cluster
|
||||
*out = make(CIDRList, len(*in))
|
||||
copy(*out, *in)
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new AddressRangesSpec.
|
||||
func (in *AddressRangesSpec) DeepCopy() *AddressRangesSpec {
|
||||
if in == nil {
|
||||
return nil
|
||||
}
|
||||
out := new(AddressRangesSpec)
|
||||
in.DeepCopyInto(out)
|
||||
return out
|
||||
}
|
||||
|
||||
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
|
||||
func (in Annotations) DeepCopyInto(out *Annotations) {
|
||||
{
|
||||
@@ -161,6 +187,26 @@ func (in *BucketTopicStatus) DeepCopy() *BucketTopicStatus {
|
||||
return out
|
||||
}
|
||||
|
||||
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
|
||||
func (in CIDRList) DeepCopyInto(out *CIDRList) {
|
||||
{
|
||||
in := &in
|
||||
*out = make(CIDRList, len(*in))
|
||||
copy(*out, *in)
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new CIDRList.
|
||||
func (in CIDRList) DeepCopy() CIDRList {
|
||||
if in == nil {
|
||||
return nil
|
||||
}
|
||||
out := new(CIDRList)
|
||||
in.DeepCopyInto(out)
|
||||
return *out
|
||||
}
|
||||
|
||||
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
|
||||
func (in *Capacity) DeepCopyInto(out *Capacity) {
|
||||
*out = *in
|
||||
@@ -2909,6 +2955,13 @@ func (in *Module) DeepCopy() *Module {
|
||||
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
|
||||
func (in *MonSpec) DeepCopyInto(out *MonSpec) {
|
||||
*out = *in
|
||||
if in.Zones != nil {
|
||||
in, out := &in.Zones, &out.Zones
|
||||
*out = make([]MonZoneSpec, len(*in))
|
||||
for i := range *in {
|
||||
(*in)[i].DeepCopyInto(&(*out)[i])
|
||||
}
|
||||
}
|
||||
if in.StretchCluster != nil {
|
||||
in, out := &in.StretchCluster, &out.StretchCluster
|
||||
*out = new(StretchClusterSpec)
|
||||
@@ -2932,6 +2985,27 @@ func (in *MonSpec) DeepCopy() *MonSpec {
|
||||
return out
|
||||
}
|
||||
|
||||
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
|
||||
func (in *MonZoneSpec) DeepCopyInto(out *MonZoneSpec) {
|
||||
*out = *in
|
||||
if in.VolumeClaimTemplate != nil {
|
||||
in, out := &in.VolumeClaimTemplate, &out.VolumeClaimTemplate
|
||||
*out = new(corev1.PersistentVolumeClaim)
|
||||
(*in).DeepCopyInto(*out)
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new MonZoneSpec.
|
||||
func (in *MonZoneSpec) DeepCopy() *MonZoneSpec {
|
||||
if in == nil {
|
||||
return nil
|
||||
}
|
||||
out := new(MonZoneSpec)
|
||||
in.DeepCopyInto(out)
|
||||
return out
|
||||
}
|
||||
|
||||
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
|
||||
func (in *MonitoringSpec) DeepCopyInto(out *MonitoringSpec) {
|
||||
*out = *in
|
||||
@@ -3064,11 +3138,16 @@ func (in *NetworkSpec) DeepCopyInto(out *NetworkSpec) {
|
||||
*out = *in
|
||||
if in.Selectors != nil {
|
||||
in, out := &in.Selectors, &out.Selectors
|
||||
*out = make(map[string]string, len(*in))
|
||||
*out = make(map[CephNetworkType]string, len(*in))
|
||||
for key, val := range *in {
|
||||
(*out)[key] = val
|
||||
}
|
||||
}
|
||||
if in.AddressRanges != nil {
|
||||
in, out := &in.AddressRanges, &out.AddressRanges
|
||||
*out = new(AddressRangesSpec)
|
||||
(*in).DeepCopyInto(*out)
|
||||
}
|
||||
if in.Connections != nil {
|
||||
in, out := &in.Connections, &out.Connections
|
||||
*out = new(ConnectionsSpec)
|
||||
@@ -4294,27 +4373,6 @@ func (in *StretchClusterSpec) DeepCopy() *StretchClusterSpec {
|
||||
return out
|
||||
}
|
||||
|
||||
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
|
||||
func (in *MonZoneSpec) DeepCopyInto(out *MonZoneSpec) {
|
||||
*out = *in
|
||||
if in.VolumeClaimTemplate != nil {
|
||||
in, out := &in.VolumeClaimTemplate, &out.VolumeClaimTemplate
|
||||
*out = new(corev1.PersistentVolumeClaim)
|
||||
(*in).DeepCopyInto(*out)
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new MonZoneSpec.
|
||||
func (in *MonZoneSpec) DeepCopy() *MonZoneSpec {
|
||||
if in == nil {
|
||||
return nil
|
||||
}
|
||||
out := new(MonZoneSpec)
|
||||
in.DeepCopyInto(out)
|
||||
return out
|
||||
}
|
||||
|
||||
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
|
||||
func (in *TopicEndpointSpec) DeepCopyInto(out *TopicEndpointSpec) {
|
||||
*out = *in
|
||||
|
||||
@@ -44,7 +44,6 @@ import (
|
||||
"github.com/rook/rook/pkg/operator/k8sutil"
|
||||
rookversion "github.com/rook/rook/pkg/version"
|
||||
v1 "k8s.io/api/core/v1"
|
||||
kerrors "k8s.io/apimachinery/pkg/api/errors"
|
||||
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
|
||||
"k8s.io/apimachinery/pkg/types"
|
||||
)
|
||||
@@ -290,34 +289,9 @@ func preClusterStartValidation(cluster *cluster) error {
|
||||
if err := validateStretchCluster(cluster); err != nil {
|
||||
return err
|
||||
}
|
||||
if cluster.Spec.Network.IsMultus() {
|
||||
_, isPublic := cluster.Spec.Network.Selectors[config.PublicNetworkSelectorKeyName]
|
||||
_, isCluster := cluster.Spec.Network.Selectors[config.ClusterNetworkSelectorKeyName]
|
||||
if !isPublic && !isCluster {
|
||||
return errors.New("both network selector values for public and cluster selector cannot be empty for multus provider")
|
||||
}
|
||||
|
||||
for _, selector := range config.NetworkSelectors {
|
||||
// If one selector is empty, we continue
|
||||
// This means a single interface is used both public and cluster network
|
||||
if _, ok := cluster.Spec.Network.Selectors[selector]; !ok {
|
||||
continue
|
||||
}
|
||||
|
||||
multusNamespace, nad := config.GetMultusNamespace(cluster.Spec.Network.Selectors[selector])
|
||||
if multusNamespace == "" {
|
||||
multusNamespace = cluster.Namespace
|
||||
}
|
||||
|
||||
// Get network attachment definition
|
||||
_, err := cluster.context.NetworkClient.NetworkAttachmentDefinitions(multusNamespace).Get(cluster.ClusterInfo.Context, nad, metav1.GetOptions{})
|
||||
if err != nil {
|
||||
if kerrors.IsNotFound(err) {
|
||||
return errors.Wrapf(err, "specified network attachment definition for selector %q does not exist", selector)
|
||||
}
|
||||
return errors.Wrapf(err, "failed to fetch network attachment definition for selector %q", selector)
|
||||
}
|
||||
}
|
||||
if err := cephv1.ValidateNetworkSpec(cluster.Namespace, cluster.Spec.Network); err != nil {
|
||||
return errors.Wrapf(err, "failed to validate network spec for cluster in namespace %q", cluster.Namespace)
|
||||
}
|
||||
|
||||
// Validate on-PVC cluster encryption KMS settings
|
||||
@@ -597,7 +571,7 @@ func (c *cluster) reportTelemetry() {
|
||||
telemetry.ReportKeyValue(c.context, c.ClusterInfo, telemetry.DeviceSetNonPortableKey, strconv.Itoa(nonportableDeviceSets))
|
||||
|
||||
// Set the telemetry for network settings
|
||||
telemetry.ReportKeyValue(c.context, c.ClusterInfo, telemetry.NetworkProviderKey, c.Spec.Network.Provider)
|
||||
telemetry.ReportKeyValue(c.context, c.ClusterInfo, telemetry.NetworkProviderKey, string(c.Spec.Network.Provider))
|
||||
|
||||
// Set the telemetry for external cluster settings
|
||||
telemetry.ReportKeyValue(c.context, c.ClusterInfo, telemetry.ExternalModeEnabledKey, strconv.FormatBool(c.Spec.External.Enable))
|
||||
|
||||
@@ -97,7 +97,7 @@ func (c *Cluster) makeDeployment(mgrConfig *mgrConfig) (*apps.Deployment, error)
|
||||
if c.spec.Network.IsHost() {
|
||||
podSpec.Spec.DNSPolicy = v1.DNSClusterFirstWithHostNet
|
||||
} else if c.spec.Network.IsMultus() {
|
||||
if err := k8sutil.ApplyMultus(c.spec.Network, &podSpec.ObjectMeta); err != nil {
|
||||
if err := k8sutil.ApplyMultus(c.clusterInfo.Namespace, &c.spec.Network, &podSpec.ObjectMeta); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
podSpec.Spec.Containers = append(podSpec.Spec.Containers, c.makeCmdProxySidecarContainer(mgrConfig))
|
||||
|
||||
@@ -105,7 +105,7 @@ type Cluster struct {
|
||||
spec cephv1.ClusterSpec
|
||||
Namespace string
|
||||
Keyring string
|
||||
rookVersion string
|
||||
rookImage string
|
||||
orchestrationMutex sync.Mutex
|
||||
Port int32
|
||||
maxMonID int
|
||||
@@ -170,7 +170,7 @@ func (c *Cluster) MaxMonID() int {
|
||||
}
|
||||
|
||||
// Start begins the process of running a cluster of Ceph mons.
|
||||
func (c *Cluster) Start(clusterInfo *cephclient.ClusterInfo, rookVersion string, cephVersion cephver.CephVersion, spec cephv1.ClusterSpec) (*cephclient.ClusterInfo, error) {
|
||||
func (c *Cluster) Start(clusterInfo *cephclient.ClusterInfo, rookImage string, cephVersion cephver.CephVersion, spec cephv1.ClusterSpec) (*cephclient.ClusterInfo, error) {
|
||||
// Only one goroutine can orchestrate the mons at a time
|
||||
c.acquireOrchestrationLock()
|
||||
defer c.releaseOrchestrationLock()
|
||||
@@ -180,7 +180,7 @@ func (c *Cluster) Start(clusterInfo *cephclient.ClusterInfo, rookVersion string,
|
||||
if c.ClusterInfo.Context == nil {
|
||||
panic("nil context")
|
||||
}
|
||||
c.rookVersion = rookVersion
|
||||
c.rookImage = rookImage
|
||||
c.spec = spec
|
||||
|
||||
// fail if we were instructed to deploy more than one mon on the same machine with host networking
|
||||
@@ -287,6 +287,11 @@ func (c *Cluster) startMons(targetCount int) error {
|
||||
}
|
||||
}
|
||||
|
||||
// apply network settings after mons have been created b/c they are set in the mon k-v store
|
||||
if err := controller.ApplyCephNetworkSettings(c.ClusterInfo.Context, c.rookImage, c.context, &c.spec, c.ClusterInfo); err != nil {
|
||||
return errors.Wrap(err, "failed to apply ceph network settings")
|
||||
}
|
||||
|
||||
if c.spec.IsStretchCluster() {
|
||||
if err := c.configureStretchCluster(mons); err != nil {
|
||||
return errors.Wrap(err, "failed to configure stretch mons")
|
||||
@@ -658,7 +663,7 @@ func scheduleMonitor(c *Cluster, mon *monConfig) (*apps.Deployment, error) {
|
||||
// avoid issues with the real deployment, the canary should be careful not
|
||||
// to modify the storage by instead running an innocuous command.
|
||||
d.Spec.Template.Spec.InitContainers = []v1.Container{}
|
||||
d.Spec.Template.Spec.Containers[0].Image = c.rookVersion
|
||||
d.Spec.Template.Spec.Containers[0].Image = c.rookImage
|
||||
d.Spec.Template.Spec.Containers[0].Command = []string{"sleep"} // sleep responds to signals so we don't need to wrap it
|
||||
d.Spec.Template.Spec.Containers[0].Args = []string{"3600"}
|
||||
// remove the startup and liveness probes on the canary pod
|
||||
|
||||
@@ -104,7 +104,7 @@ func newCluster(context *clusterd.Context, namespace string, allowMultiplePerNod
|
||||
ClusterInfo: nil,
|
||||
context: context,
|
||||
Namespace: namespace,
|
||||
rookVersion: "myversion",
|
||||
rookImage: "myversion",
|
||||
spec: cephv1.ClusterSpec{
|
||||
Mon: cephv1.MonSpec{
|
||||
Count: 3,
|
||||
@@ -123,11 +123,11 @@ func newCluster(context *clusterd.Context, namespace string, allowMultiplePerNod
|
||||
}
|
||||
|
||||
// setCommonMonProperties is a convenience helper for setting common test properties
|
||||
func setCommonMonProperties(c *Cluster, currentMons int, mon cephv1.MonSpec, rookVersion string) {
|
||||
func setCommonMonProperties(c *Cluster, currentMons int, mon cephv1.MonSpec, rookImage string) {
|
||||
c.ClusterInfo = clienttest.CreateTestClusterInfo(currentMons)
|
||||
c.spec.Mon.Count = mon.Count
|
||||
c.spec.Mon.AllowMultiplePerNode = mon.AllowMultiplePerNode
|
||||
c.rookVersion = rookVersion
|
||||
c.rookImage = rookImage
|
||||
}
|
||||
|
||||
func TestResourceName(t *testing.T) {
|
||||
@@ -186,7 +186,7 @@ func TestStartMonPods(t *testing.T) {
|
||||
}
|
||||
|
||||
// start a basic cluster
|
||||
_, err = c.Start(c.ClusterInfo, c.rookVersion, cephver.Quincy, c.spec)
|
||||
_, err = c.Start(c.ClusterInfo, c.rookImage, cephver.Quincy, c.spec)
|
||||
assert.NoError(t, err)
|
||||
|
||||
// test annotations
|
||||
@@ -197,7 +197,7 @@ func TestStartMonPods(t *testing.T) {
|
||||
validateStart(t, c)
|
||||
|
||||
// starting again should be a no-op
|
||||
_, err = c.Start(c.ClusterInfo, c.rookVersion, cephver.Quincy, c.spec)
|
||||
_, err = c.Start(c.ClusterInfo, c.rookImage, cephver.Quincy, c.spec)
|
||||
assert.NoError(t, err)
|
||||
|
||||
validateStart(t, c)
|
||||
@@ -211,7 +211,7 @@ func TestOperatorRestart(t *testing.T) {
|
||||
c.ClusterInfo = clienttest.CreateTestClusterInfo(1)
|
||||
|
||||
// start a basic cluster
|
||||
info, err := c.Start(c.ClusterInfo, c.rookVersion, cephver.Quincy, c.spec)
|
||||
info, err := c.Start(c.ClusterInfo, c.rookImage, cephver.Quincy, c.spec)
|
||||
assert.NoError(t, err)
|
||||
assert.NoError(t, info.IsInitialized())
|
||||
|
||||
@@ -221,7 +221,7 @@ func TestOperatorRestart(t *testing.T) {
|
||||
c.ClusterInfo = clienttest.CreateTestClusterInfo(1)
|
||||
|
||||
// starting again should be a no-op, but will not result in an error
|
||||
info, err = c.Start(c.ClusterInfo, c.rookVersion, cephver.Quincy, c.spec)
|
||||
info, err = c.Start(c.ClusterInfo, c.rookImage, cephver.Quincy, c.spec)
|
||||
assert.NoError(t, err)
|
||||
assert.NoError(t, info.IsInitialized())
|
||||
|
||||
@@ -239,7 +239,7 @@ func TestOperatorRestartHostNetwork(t *testing.T) {
|
||||
c.ClusterInfo = clienttest.CreateTestClusterInfo(1)
|
||||
|
||||
// start a basic cluster
|
||||
info, err := c.Start(c.ClusterInfo, c.rookVersion, cephver.Quincy, c.spec)
|
||||
info, err := c.Start(c.ClusterInfo, c.rookImage, cephver.Quincy, c.spec)
|
||||
assert.NoError(t, err)
|
||||
assert.NoError(t, info.IsInitialized())
|
||||
|
||||
@@ -251,7 +251,7 @@ func TestOperatorRestartHostNetwork(t *testing.T) {
|
||||
c.ClusterInfo = clienttest.CreateTestClusterInfo(1)
|
||||
|
||||
// starting again should be a no-op, but still results in an error
|
||||
info, err = c.Start(c.ClusterInfo, c.rookVersion, cephver.Quincy, c.spec)
|
||||
info, err = c.Start(c.ClusterInfo, c.rookImage, cephver.Quincy, c.spec)
|
||||
assert.NoError(t, err)
|
||||
assert.NoError(t, info.IsInitialized(), info)
|
||||
|
||||
|
||||
@@ -86,7 +86,7 @@ func TestHostNetworkSameNode(t *testing.T) {
|
||||
c.ClusterInfo = clienttest.CreateTestClusterInfo(1)
|
||||
|
||||
// start a basic cluster
|
||||
_, err = c.Start(c.ClusterInfo, c.rookVersion, cephver.Quincy, c.spec)
|
||||
_, err = c.Start(c.ClusterInfo, c.rookImage, cephver.Quincy, c.spec)
|
||||
assert.Error(t, err)
|
||||
}
|
||||
|
||||
@@ -104,7 +104,7 @@ func TestPodMemory(t *testing.T) {
|
||||
c := newCluster(context, namespace, true, r)
|
||||
c.ClusterInfo = clienttest.CreateTestClusterInfo(1)
|
||||
// start a basic cluster
|
||||
_, err = c.Start(c.ClusterInfo, c.rookVersion, cephver.Quincy, c.spec)
|
||||
_, err = c.Start(c.ClusterInfo, c.rookImage, cephver.Quincy, c.spec)
|
||||
assert.NoError(t, err)
|
||||
|
||||
// Test REQUEST == LIMIT
|
||||
@@ -120,7 +120,7 @@ func TestPodMemory(t *testing.T) {
|
||||
c = newCluster(context, namespace, true, r)
|
||||
c.ClusterInfo = clienttest.CreateTestClusterInfo(1)
|
||||
// start a basic cluster
|
||||
_, err = c.Start(c.ClusterInfo, c.rookVersion, cephver.Quincy, c.spec)
|
||||
_, err = c.Start(c.ClusterInfo, c.rookImage, cephver.Quincy, c.spec)
|
||||
assert.NoError(t, err)
|
||||
|
||||
// Test LIMIT != REQUEST but obviously LIMIT > REQUEST
|
||||
@@ -136,7 +136,7 @@ func TestPodMemory(t *testing.T) {
|
||||
c = newCluster(context, namespace, true, r)
|
||||
c.ClusterInfo = clienttest.CreateTestClusterInfo(1)
|
||||
// start a basic cluster
|
||||
_, err = c.Start(c.ClusterInfo, c.rookVersion, cephver.Quincy, c.spec)
|
||||
_, err = c.Start(c.ClusterInfo, c.rookImage, cephver.Quincy, c.spec)
|
||||
assert.NoError(t, err)
|
||||
|
||||
// Test valid case where pod resource is set appropriately
|
||||
@@ -152,7 +152,7 @@ func TestPodMemory(t *testing.T) {
|
||||
c = newCluster(context, namespace, true, r)
|
||||
c.ClusterInfo = clienttest.CreateTestClusterInfo(1)
|
||||
// start a basic cluster
|
||||
_, err = c.Start(c.ClusterInfo, c.rookVersion, cephver.Quincy, c.spec)
|
||||
_, err = c.Start(c.ClusterInfo, c.rookImage, cephver.Quincy, c.spec)
|
||||
assert.NoError(t, err)
|
||||
|
||||
// Test no resources were specified on the pod
|
||||
@@ -160,7 +160,7 @@ func TestPodMemory(t *testing.T) {
|
||||
c = newCluster(context, namespace, true, r)
|
||||
c.ClusterInfo = clienttest.CreateTestClusterInfo(1)
|
||||
// start a basic cluster
|
||||
_, err = c.Start(c.ClusterInfo, c.rookVersion, cephver.Quincy, c.spec)
|
||||
_, err = c.Start(c.ClusterInfo, c.rookImage, cephver.Quincy, c.spec)
|
||||
assert.NoError(t, err)
|
||||
|
||||
}
|
||||
|
||||
@@ -217,7 +217,13 @@ func (c *Cluster) makeMonPod(monConfig *monConfig, canary bool) (*corev1.Pod, er
|
||||
if monConfig.UseHostNetwork {
|
||||
pod.Spec.DNSPolicy = corev1.DNSClusterFirstWithHostNet
|
||||
} else if c.spec.Network.IsMultus() {
|
||||
if err := k8sutil.ApplyMultus(c.spec.Network, &pod.ObjectMeta); err != nil {
|
||||
cluster := cephv1.CephCluster{
|
||||
ObjectMeta: metav1.ObjectMeta{
|
||||
Namespace: c.Namespace,
|
||||
},
|
||||
Spec: c.spec,
|
||||
}
|
||||
if err := k8sutil.ApplyMultus(cluster.GetNamespace(), &cluster.Spec.Network, &pod.ObjectMeta); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
|
||||
@@ -166,7 +166,7 @@ func (c *Cluster) getKeyRotationPodTemplateSpec(osdProps osdProperties, osd OSDI
|
||||
if c.spec.Network.IsHost() {
|
||||
podTemplateSpec.Spec.DNSPolicy = v1.DNSClusterFirstWithHostNet
|
||||
} else if c.spec.Network.IsMultus() {
|
||||
if err := k8sutil.ApplyMultus(c.spec.Network, &podTemplateSpec.ObjectMeta); err != nil {
|
||||
if err := k8sutil.ApplyMultus(c.clusterInfo.Namespace, &c.spec.Network, &podTemplateSpec.ObjectMeta); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
|
||||
@@ -638,7 +638,7 @@ func (c *Cluster) makeDeployment(osdProps osdProperties, osd OSDInfo, provisionC
|
||||
if c.spec.Network.IsHost() {
|
||||
podTemplateSpec.Spec.DNSPolicy = v1.DNSClusterFirstWithHostNet
|
||||
} else if c.spec.Network.IsMultus() {
|
||||
if err := k8sutil.ApplyMultus(c.spec.Network, &podTemplateSpec.ObjectMeta); err != nil {
|
||||
if err := k8sutil.ApplyMultus(c.clusterInfo.Namespace, &c.spec.Network, &podTemplateSpec.ObjectMeta); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
|
||||
@@ -66,7 +66,7 @@ func (r *ReconcileCephRBDMirror) makeDeployment(daemonConfig *daemonConfig, rbdM
|
||||
if r.cephClusterSpec.Network.IsHost() {
|
||||
podSpec.Spec.DNSPolicy = v1.DNSClusterFirstWithHostNet
|
||||
} else if r.cephClusterSpec.Network.IsMultus() {
|
||||
if err := k8sutil.ApplyMultus(r.cephClusterSpec.Network, &podSpec.ObjectMeta); err != nil {
|
||||
if err := k8sutil.ApplyMultus(r.clusterInfo.Namespace, &r.cephClusterSpec.Network, &podSpec.ObjectMeta); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
|
||||
@@ -137,20 +137,6 @@ func SetOrRemoveDefaultConfigs(
|
||||
}
|
||||
}
|
||||
|
||||
// Apply Multus if needed
|
||||
if clusterSpec.Network.IsMultus() {
|
||||
logger.Info("configuring ceph network(s) with multus")
|
||||
cephNetworks, err := generateNetworkSettings(clusterInfo.Context, context, clusterInfo.Namespace, clusterSpec.Network.Selectors)
|
||||
if err != nil {
|
||||
return errors.Wrap(err, "failed to generate network settings")
|
||||
}
|
||||
|
||||
// Apply ceph network settings to the mon config store
|
||||
if err := monStore.SetAll("global", cephNetworks); err != nil {
|
||||
return errors.Wrap(err, "failed to network config overrides")
|
||||
}
|
||||
}
|
||||
|
||||
// This section will remove any previously configured option(s) from the mon centralized store
|
||||
// This is useful for scenarios where options are not needed anymore and we just want to reset to internal's default
|
||||
// On upgrade, the flag will be removed
|
||||
|
||||
@@ -56,8 +56,22 @@ type Option struct {
|
||||
Value string
|
||||
}
|
||||
|
||||
// SetIfChanged sets a config in the centralized mon configuration database if the config has
|
||||
// changed value.
|
||||
// https://docs.ceph.com/docs/master/rados/configuration/ceph-conf/#monitor-configuration-database
|
||||
//
|
||||
// There is a bug through at least Ceph v18 where `ceph config get global <option>` does not work.
|
||||
// As a workaround it is possible to use `ceph config get client <option>` as long as the config
|
||||
// option won't be overridden by clients. SetIfChanged uses this workaround assuming it is valid.
|
||||
// Any new uses of this function should take extreme care when using `who="global"` to check that
|
||||
// the workaround is valid for usage with the given option.
|
||||
// Options validated for workaround by Ceph devs: public_network, cluster_network
|
||||
func (m *MonStore) SetIfChanged(who, option, value string) (bool, error) {
|
||||
currentVal, err := m.Get(who, option)
|
||||
getWho := who
|
||||
if who == "global" {
|
||||
getWho = "client"
|
||||
}
|
||||
currentVal, err := m.Get(getWho, option)
|
||||
if err != nil {
|
||||
return false, errors.Wrapf(err, "failed to get value %q", option)
|
||||
}
|
||||
|
||||
@@ -1,124 +0,0 @@
|
||||
/*
|
||||
Copyright 2020 The Rook Authors. All rights reserved.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
*/
|
||||
|
||||
// Package config provides default configurations which Rook will set in Ceph clusters.
|
||||
package config
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"github.com/pkg/errors"
|
||||
"github.com/rook/rook/pkg/clusterd"
|
||||
"github.com/rook/rook/pkg/operator/k8sutil"
|
||||
kerrors "k8s.io/apimachinery/pkg/api/errors"
|
||||
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
|
||||
)
|
||||
|
||||
const (
|
||||
// PublicNetworkSelectorKeyName is the network selector key for the ceph public network
|
||||
PublicNetworkSelectorKeyName = "public"
|
||||
// ClusterNetworkSelectorKeyName is the network selector key for the ceph cluster network
|
||||
ClusterNetworkSelectorKeyName = "cluster"
|
||||
// WhereaboutsIPAMType is Whereabouts IPAM type
|
||||
WhereaboutsIPAMType = "whereabouts"
|
||||
hostLocalIPAMType = "host-local"
|
||||
staticIPAMType = "static"
|
||||
)
|
||||
|
||||
var (
|
||||
// NetworkSelectors is a slice of ceph network selector key name
|
||||
NetworkSelectors = []string{PublicNetworkSelectorKeyName, ClusterNetworkSelectorKeyName}
|
||||
)
|
||||
|
||||
func generateNetworkSettings(ctx context.Context, clusterdContext *clusterd.Context, namespace string, networkSelectors map[string]string) (map[string]string, error) {
|
||||
cephNetworks := map[string]string{}
|
||||
|
||||
for _, selectorKey := range NetworkSelectors {
|
||||
// skip if selector is not specified
|
||||
if _, ok := networkSelectors[selectorKey]; !ok {
|
||||
continue
|
||||
}
|
||||
|
||||
multusNamespace, nad := GetMultusNamespace(networkSelectors[selectorKey])
|
||||
if multusNamespace == "" {
|
||||
multusNamespace = namespace
|
||||
}
|
||||
// Get network attachment definition
|
||||
netDefinition, err := clusterdContext.NetworkClient.NetworkAttachmentDefinitions(multusNamespace).Get(ctx, nad, metav1.GetOptions{})
|
||||
if err != nil {
|
||||
if kerrors.IsNotFound(err) {
|
||||
return cephNetworks, errors.Wrapf(err, "specified network attachment definition %q in namespace %q for selector %q does not exist", nad, namespace, selectorKey)
|
||||
}
|
||||
return cephNetworks, errors.Wrapf(err, "failed to fetch network attachment definition for selector %q", selectorKey)
|
||||
}
|
||||
|
||||
// Get network attachment definition configuration
|
||||
netConfig, err := k8sutil.GetNetworkAttachmentConfig(*netDefinition)
|
||||
if err != nil {
|
||||
return cephNetworks, errors.Wrapf(err, "failed to get network attachment definition configuration for selector %q", selectorKey)
|
||||
}
|
||||
|
||||
networkRange := getNetworkRange(netConfig)
|
||||
if networkRange != "" {
|
||||
key := fmt.Sprintf("%s_network", selectorKey)
|
||||
cephNetworks[key] = networkRange
|
||||
} else {
|
||||
return cephNetworks, errors.Errorf("empty subnet from network attachment definition %q", networkSelectors[selectorKey])
|
||||
}
|
||||
}
|
||||
|
||||
return cephNetworks, nil
|
||||
}
|
||||
|
||||
func GetMultusNamespace(nad string) (string, string) {
|
||||
tmp := strings.Split(nad, "/")
|
||||
if len(tmp) == 2 {
|
||||
return tmp[0], tmp[1]
|
||||
}
|
||||
return "", nad
|
||||
}
|
||||
|
||||
func getNetworkRange(netConfig k8sutil.NetworkAttachmentConfig) string {
|
||||
var subnets []string
|
||||
|
||||
switch netConfig.Ipam.Type {
|
||||
case hostLocalIPAMType:
|
||||
if netConfig.Ipam.Subnet != "" {
|
||||
return netConfig.Ipam.Subnet
|
||||
}
|
||||
for _, netRanges := range netConfig.Ipam.Ranges {
|
||||
for _, netRange := range netRanges {
|
||||
subnets = append(subnets, netRange.Subnet)
|
||||
}
|
||||
}
|
||||
return strings.Join(subnets, ",")
|
||||
|
||||
case staticIPAMType:
|
||||
for _, subnet := range netConfig.Ipam.Addresses {
|
||||
subnets = append(subnets, subnet.Address)
|
||||
}
|
||||
|
||||
return strings.Join(subnets, ",")
|
||||
|
||||
case WhereaboutsIPAMType:
|
||||
return netConfig.Ipam.Range
|
||||
|
||||
default:
|
||||
return ""
|
||||
}
|
||||
}
|
||||
@@ -1,327 +0,0 @@
|
||||
/*
|
||||
Copyright 2020 The Rook Authors. All rights reserved.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
*/
|
||||
|
||||
package config
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
networkv1 "github.com/k8snetworkplumbingwg/network-attachment-definition-client/pkg/apis/k8s.cni.cncf.io/v1"
|
||||
fakenetclient "github.com/k8snetworkplumbingwg/network-attachment-definition-client/pkg/client/clientset/versioned/fake"
|
||||
"github.com/rook/rook/pkg/clusterd"
|
||||
"github.com/rook/rook/pkg/operator/k8sutil"
|
||||
testop "github.com/rook/rook/pkg/operator/test"
|
||||
"github.com/stretchr/testify/assert"
|
||||
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
|
||||
)
|
||||
|
||||
func TestGenerateNetworkSettings(t *testing.T) {
|
||||
t.Run("no network definition exists", func(*testing.T) {
|
||||
ctx := context.TODO()
|
||||
ns := "rook-ceph"
|
||||
clientset := testop.New(t, 1)
|
||||
clusterdContext := &clusterd.Context{
|
||||
Clientset: clientset,
|
||||
NetworkClient: fakenetclient.NewSimpleClientset().K8sCniCncfIoV1(),
|
||||
}
|
||||
netSelector := map[string]string{"public": "public-network-attach-def"}
|
||||
_, err := generateNetworkSettings(ctx, clusterdContext, ns, netSelector)
|
||||
assert.Error(t, err)
|
||||
|
||||
})
|
||||
|
||||
t.Run("only cluster network", func(*testing.T) {
|
||||
netSelector := map[string]string{"cluster": "cluster-network-attach-def"}
|
||||
networks := []networkv1.NetworkAttachmentDefinition{getClusterNetwork()}
|
||||
ctxt := context.TODO()
|
||||
ns := "rook-ceph"
|
||||
clientset := testop.New(t, 1)
|
||||
clusterdContext := &clusterd.Context{
|
||||
Clientset: clientset,
|
||||
NetworkClient: fakenetclient.NewSimpleClientset().K8sCniCncfIoV1(),
|
||||
}
|
||||
|
||||
for i := range networks {
|
||||
_, err := clusterdContext.NetworkClient.NetworkAttachmentDefinitions(ns).Create(ctxt, &networks[i], metav1.CreateOptions{})
|
||||
assert.NoError(t, err)
|
||||
}
|
||||
cephNetwork, err := generateNetworkSettings(context.TODO(), clusterdContext, ns, netSelector)
|
||||
assert.NoError(t, err)
|
||||
assert.Equal(t, "172.18.0.0/16", cephNetwork["cluster_network"])
|
||||
assert.Equal(t, 1, len(cephNetwork))
|
||||
})
|
||||
|
||||
t.Run("only public network", func(*testing.T) {
|
||||
ctx := context.TODO()
|
||||
ns := "rook-ceph"
|
||||
clientset := testop.New(t, 1)
|
||||
clusterdContext := &clusterd.Context{
|
||||
Clientset: clientset,
|
||||
NetworkClient: fakenetclient.NewSimpleClientset().K8sCniCncfIoV1(),
|
||||
}
|
||||
netSelector := map[string]string{"public": "public-network-attach-def"}
|
||||
networks := []networkv1.NetworkAttachmentDefinition{getPublicNetwork()}
|
||||
for i := range networks {
|
||||
_, err := clusterdContext.NetworkClient.NetworkAttachmentDefinitions(ns).Create(ctx, &networks[i], metav1.CreateOptions{})
|
||||
assert.NoError(t, err)
|
||||
}
|
||||
cephNetwork, err := generateNetworkSettings(ctx, clusterdContext, ns, netSelector)
|
||||
assert.NoError(t, err)
|
||||
assert.Equal(t, "192.168.0.0/24", cephNetwork["public_network"])
|
||||
assert.Equal(t, 1, len(cephNetwork))
|
||||
})
|
||||
|
||||
t.Run("public and cluster network", func(*testing.T) {
|
||||
ctx := context.TODO()
|
||||
ns := "rook-ceph"
|
||||
clientset := testop.New(t, 1)
|
||||
clusterdContext := &clusterd.Context{
|
||||
Clientset: clientset,
|
||||
NetworkClient: fakenetclient.NewSimpleClientset().K8sCniCncfIoV1(),
|
||||
}
|
||||
netSelector := map[string]string{
|
||||
"public": "public-network-attach-def",
|
||||
"cluster": "cluster-network-attach-def",
|
||||
}
|
||||
networks := []networkv1.NetworkAttachmentDefinition{getPublicNetwork(), getClusterNetwork()}
|
||||
for i := range networks {
|
||||
_, err := clusterdContext.NetworkClient.NetworkAttachmentDefinitions(ns).Create(ctx, &networks[i], metav1.CreateOptions{})
|
||||
assert.NoError(t, err)
|
||||
}
|
||||
cephNetwork, err := generateNetworkSettings(ctx, clusterdContext, ns, netSelector)
|
||||
assert.NoError(t, err)
|
||||
assert.Equal(t, "172.18.0.0/16", cephNetwork["cluster_network"])
|
||||
assert.Equal(t, "192.168.0.0/24", cephNetwork["public_network"])
|
||||
assert.Equal(t, 2, len(cephNetwork))
|
||||
})
|
||||
}
|
||||
|
||||
func getPublicNetwork() networkv1.NetworkAttachmentDefinition {
|
||||
return networkv1.NetworkAttachmentDefinition{
|
||||
ObjectMeta: metav1.ObjectMeta{
|
||||
Name: "public-network-attach-def",
|
||||
},
|
||||
Spec: networkv1.NetworkAttachmentDefinitionSpec{
|
||||
Config: `{
|
||||
"cniVersion": "0.3.0",
|
||||
"type": "macvlan",
|
||||
"master": "eth2",
|
||||
"mode": "bridge",
|
||||
"ipam": {
|
||||
"type": "host-local",
|
||||
"subnet": "192.168.0.0/24",
|
||||
"gateway": "172.18.8.1"
|
||||
}
|
||||
}`,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func getClusterNetwork() networkv1.NetworkAttachmentDefinition {
|
||||
return networkv1.NetworkAttachmentDefinition{
|
||||
ObjectMeta: metav1.ObjectMeta{
|
||||
Name: "cluster-network-attach-def",
|
||||
},
|
||||
Spec: networkv1.NetworkAttachmentDefinitionSpec{
|
||||
Config: `{
|
||||
"cniVersion": "0.3.0",
|
||||
"type": "macvlan",
|
||||
"master": "eth2",
|
||||
"mode": "bridge",
|
||||
"ipam": {
|
||||
"type": "host-local",
|
||||
"subnet": "172.18.0.0/16",
|
||||
"gateway": "172.18.0.1"
|
||||
}
|
||||
}`,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func TestGetNetworkRange(t *testing.T) {
|
||||
t.Run("simple host-local IPAM test", func(t *testing.T) {
|
||||
ns := "rook-ceph"
|
||||
nad := &networkv1.NetworkAttachmentDefinition{
|
||||
ObjectMeta: metav1.ObjectMeta{
|
||||
Name: "public-network-attach-def",
|
||||
Namespace: ns,
|
||||
},
|
||||
Spec: networkv1.NetworkAttachmentDefinitionSpec{
|
||||
Config: `{
|
||||
"cniVersion": "0.3.0",
|
||||
"type": "macvlan",
|
||||
"master": "eth2",
|
||||
"mode": "bridge",
|
||||
"ipam": {
|
||||
"type": "host-local",
|
||||
"subnet": "",
|
||||
"gateway": "172.18.8.1"
|
||||
}
|
||||
}`,
|
||||
},
|
||||
}
|
||||
|
||||
netConfig, err := k8sutil.GetNetworkAttachmentConfig(*nad)
|
||||
assert.NoError(t, err)
|
||||
|
||||
//
|
||||
// TEST 1: subnet/range is empty
|
||||
//
|
||||
networkRange := getNetworkRange(netConfig)
|
||||
assert.Empty(t, networkRange)
|
||||
|
||||
//
|
||||
// TEST 2: subnet is not empty
|
||||
//
|
||||
netConfig.Ipam.Type = "host-local"
|
||||
netConfig.Ipam.Subnet = "192.168.0.0/24"
|
||||
networkRange = getNetworkRange(netConfig)
|
||||
assert.Equal(t, "192.168.0.0/24", networkRange)
|
||||
})
|
||||
|
||||
t.Run("advanced host-local IPAM test", func(t *testing.T) {
|
||||
ns := "rook-ceph"
|
||||
nad := &networkv1.NetworkAttachmentDefinition{
|
||||
ObjectMeta: metav1.ObjectMeta{
|
||||
Name: "public-network-attach-def",
|
||||
Namespace: ns,
|
||||
},
|
||||
Spec: networkv1.NetworkAttachmentDefinitionSpec{
|
||||
Config: `{
|
||||
"ipam": {
|
||||
"type": "host-local",
|
||||
"ranges": [
|
||||
[
|
||||
{
|
||||
"subnet": "10.10.0.0/16",
|
||||
"rangeStart": "10.10.1.20",
|
||||
"rangeEnd": "10.10.3.50",
|
||||
"gateway": "10.10.0.254"
|
||||
},
|
||||
{
|
||||
"subnet": "172.16.5.0/24"
|
||||
}
|
||||
],
|
||||
[
|
||||
{
|
||||
"subnet": "3ffe:ffff:0:01ff::/64",
|
||||
"rangeStart": "3ffe:ffff:0:01ff::0010",
|
||||
"rangeEnd": "3ffe:ffff:0:01ff::0020"
|
||||
}
|
||||
]
|
||||
],
|
||||
"routes": [
|
||||
{ "dst": "0.0.0.0/0" },
|
||||
{ "dst": "192.168.0.0/16", "gw": "10.10.5.1" },
|
||||
{ "dst": "3ffe:ffff:0:01ff::1/64" }
|
||||
],
|
||||
"dataDir": "/run/my-orchestrator/container-ipam-state"
|
||||
}
|
||||
}`,
|
||||
},
|
||||
}
|
||||
|
||||
netConfig, err := k8sutil.GetNetworkAttachmentConfig(*nad)
|
||||
assert.NoError(t, err)
|
||||
networkRange := getNetworkRange(netConfig)
|
||||
assert.Equal(t, "10.10.0.0/16,172.16.5.0/24,3ffe:ffff:0:01ff::/64", networkRange)
|
||||
})
|
||||
|
||||
t.Run("advanced static IPAM test", func(t *testing.T) {
|
||||
ns := "rook-ceph"
|
||||
nad := &networkv1.NetworkAttachmentDefinition{
|
||||
ObjectMeta: metav1.ObjectMeta{
|
||||
Name: "public-network-attach-def",
|
||||
Namespace: ns,
|
||||
},
|
||||
Spec: networkv1.NetworkAttachmentDefinitionSpec{
|
||||
Config: `{
|
||||
"ipam": {
|
||||
"type": "static",
|
||||
"addresses": [
|
||||
{
|
||||
"address": "10.10.0.1/24",
|
||||
"gateway": "10.10.0.254"
|
||||
},
|
||||
{
|
||||
"address": "3ffe:ffff:0:01ff::1/64",
|
||||
"gateway": "3ffe:ffff:0::1"
|
||||
}
|
||||
],
|
||||
"routes": [
|
||||
{ "dst": "0.0.0.0/0" },
|
||||
{ "dst": "192.168.0.0/16", "gw": "10.10.5.1" },
|
||||
{ "dst": "3ffe:ffff:0:01ff::1/64" }
|
||||
],
|
||||
"dns": {
|
||||
"nameservers" : ["8.8.8.8"],
|
||||
"domain": "example.com",
|
||||
"search": [ "example.com" ]
|
||||
}
|
||||
}
|
||||
}`,
|
||||
},
|
||||
}
|
||||
netConfig, err := k8sutil.GetNetworkAttachmentConfig(*nad)
|
||||
assert.NoError(t, err)
|
||||
networkRange := getNetworkRange(netConfig)
|
||||
assert.Equal(t, "10.10.0.1/24,3ffe:ffff:0:01ff::1/64", networkRange)
|
||||
})
|
||||
|
||||
t.Run("advanced whereabouts IPAM test", func(t *testing.T) {
|
||||
ns := "rook-ceph"
|
||||
nad := &networkv1.NetworkAttachmentDefinition{
|
||||
ObjectMeta: metav1.ObjectMeta{
|
||||
Name: "public-network-attach-def",
|
||||
Namespace: ns,
|
||||
},
|
||||
Spec: networkv1.NetworkAttachmentDefinitionSpec{
|
||||
Config: `{
|
||||
"cniVersion": "0.3.0",
|
||||
"name": "whereaboutsexample",
|
||||
"type": "macvlan",
|
||||
"master": "eth0",
|
||||
"mode": "bridge",
|
||||
"ipam": {
|
||||
"type": "whereabouts",
|
||||
"range": "192.168.2.225/28",
|
||||
"exclude": [
|
||||
"192.168.2.229/30",
|
||||
"192.168.2.236/32"
|
||||
]
|
||||
}
|
||||
}`,
|
||||
},
|
||||
}
|
||||
netConfig, err := k8sutil.GetNetworkAttachmentConfig(*nad)
|
||||
assert.NoError(t, err)
|
||||
networkRange := getNetworkRange(netConfig)
|
||||
assert.Equal(t, "192.168.2.225/28", networkRange)
|
||||
})
|
||||
}
|
||||
|
||||
func TestGetMultusNamespace(t *testing.T) {
|
||||
// TEST 1: When namespace is specified with the NAD
|
||||
namespace, nad := GetMultusNamespace("multus-ns/public-nad")
|
||||
assert.Equal(t, "multus-ns", namespace)
|
||||
assert.Equal(t, "public-nad", nad)
|
||||
|
||||
// TEST 2: When only NAD is specified
|
||||
namespace, nad = GetMultusNamespace("public-nad")
|
||||
assert.Empty(t, namespace)
|
||||
assert.Equal(t, "public-nad", nad)
|
||||
}
|
||||
@@ -54,10 +54,9 @@ func TestCreateClusterSecrets(t *testing.T) {
|
||||
Clientset: clientset,
|
||||
Executor: executor,
|
||||
}
|
||||
provider := "Multus"
|
||||
cephClusterSpec := &cephv1.ClusterSpec{
|
||||
Network: cephv1.NetworkSpec{
|
||||
Provider: provider},
|
||||
Provider: cephv1.NetworkProviderMultus},
|
||||
}
|
||||
namespace := "ns"
|
||||
ownerInfo := cephclient.NewMinimumOwnerInfoWithOwnerRef()
|
||||
|
||||
@@ -0,0 +1,426 @@
|
||||
/*
|
||||
Copyright 2023 The Rook Authors. All rights reserved.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
*/
|
||||
|
||||
package controller
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"net"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
nadv1 "github.com/k8snetworkplumbingwg/network-attachment-definition-client/pkg/apis/k8s.cni.cncf.io/v1"
|
||||
"github.com/pkg/errors"
|
||||
cephv1 "github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
|
||||
"github.com/rook/rook/pkg/clusterd"
|
||||
cephclient "github.com/rook/rook/pkg/daemon/ceph/client"
|
||||
"github.com/rook/rook/pkg/operator/ceph/config"
|
||||
"github.com/rook/rook/pkg/operator/k8sutil"
|
||||
"github.com/rook/rook/pkg/operator/k8sutil/cmdreporter"
|
||||
corev1 "k8s.io/api/core/v1"
|
||||
v1 "k8s.io/api/core/v1"
|
||||
)
|
||||
|
||||
const detectNetworkCIDRTimeout = 15 * time.Minute
|
||||
|
||||
var (
|
||||
newCmdReporter = cmdreporter.New
|
||||
discoverCephAddressRangesFunc = discoverCephAddressRanges
|
||||
)
|
||||
|
||||
func ApplyCephNetworkSettings(
|
||||
ctx context.Context,
|
||||
rookImage string,
|
||||
clusterdContext *clusterd.Context,
|
||||
clusterSpec *cephv1.ClusterSpec,
|
||||
clusterInfo *cephclient.ClusterInfo,
|
||||
) error {
|
||||
netSpec := clusterSpec.Network
|
||||
|
||||
if !netSpec.IsHost() && !netSpec.IsMultus() {
|
||||
// do not apply specs when using k8s pod network, and for safety, only apply net specs for
|
||||
// nets where it is definitely safe to do so (e.g., multus, hostnet)
|
||||
logger.Infof("not applying network settings for cluster %q ceph networks", clusterInfo.Namespace)
|
||||
return nil
|
||||
}
|
||||
|
||||
var publicCIDRs cephv1.CIDRList = []cephv1.CIDR{}
|
||||
var clusterCIDRs cephv1.CIDRList = []cephv1.CIDR{}
|
||||
discoverPublic := true
|
||||
discoverCluster := true
|
||||
|
||||
if !netSpec.AddressRanges.IsEmpty() {
|
||||
// if user specifies CIDRs in the network config, that is the definitive source of truth
|
||||
// do not auto-detect CIDRs if given by user
|
||||
if len(netSpec.AddressRanges.Public) > 0 {
|
||||
logger.Infof("using user-provided network CIDR(s) for cluster %q public network", clusterInfo.Namespace)
|
||||
discoverPublic = false
|
||||
publicCIDRs = netSpec.AddressRanges.Public
|
||||
}
|
||||
if len(netSpec.AddressRanges.Cluster) > 0 {
|
||||
logger.Infof("using user-provided network CIDR(s) for cluster %q cluster network", clusterInfo.Namespace)
|
||||
discoverCluster = false
|
||||
clusterCIDRs = netSpec.AddressRanges.Cluster
|
||||
}
|
||||
}
|
||||
|
||||
if netSpec.IsMultus() && (discoverPublic || discoverCluster) {
|
||||
pub, clus, err := discoverCephAddressRangesFunc(ctx, rookImage, clusterdContext, clusterSpec, clusterInfo, discoverPublic, discoverCluster)
|
||||
if err != nil {
|
||||
return errors.Wrap(err, "failed to discover network CIDRs for multus, "+
|
||||
"please correct any possible errors in the CephCluster spec.network.selectors, or "+
|
||||
"use CephCluster spec.network.addressRanges to manually specify which network ranges to use for public/cluster networks")
|
||||
}
|
||||
if discoverPublic {
|
||||
publicCIDRs = pub
|
||||
}
|
||||
if discoverCluster {
|
||||
clusterCIDRs = clus
|
||||
}
|
||||
}
|
||||
|
||||
if err := setNetworkCIDRs(clusterdContext, clusterInfo, cephv1.CephNetworkPublic, publicCIDRs); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := setNetworkCIDRs(clusterdContext, clusterInfo, cephv1.CephNetworkCluster, clusterCIDRs); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
type monStoreInterface interface {
|
||||
SetIfChanged(who string, option string, value string) (bool, error)
|
||||
}
|
||||
|
||||
var getMonStoreFunc = func(context *clusterd.Context, clusterInfo *cephclient.ClusterInfo) monStoreInterface {
|
||||
return config.GetMonStore(context, clusterInfo)
|
||||
}
|
||||
|
||||
func setNetworkCIDRs(clusterdCtx *clusterd.Context, clusterInfo *cephclient.ClusterInfo, cephNet cephv1.CephNetworkType, cidrs cephv1.CIDRList) error {
|
||||
ns := clusterInfo.Namespace
|
||||
settingKey := fmt.Sprintf("%s_network", string(cephNet))
|
||||
settingVal := cidrs.String()
|
||||
|
||||
logger.Infof("ensuring cluster %q %q network is configured to use CIDR(s) %q", ns, cephNet, settingVal)
|
||||
if len(cidrs) > 0 {
|
||||
var s monStoreInterface = getMonStoreFunc(clusterdCtx, clusterInfo)
|
||||
changed, err := s.SetIfChanged("global", settingKey, settingVal)
|
||||
if err != nil {
|
||||
return errors.Wrapf(err, "failed to set CIDR(s) %q on cluster %q %q network", settingVal, cephNet, ns)
|
||||
}
|
||||
if changed {
|
||||
logger.Infof("modified cluster %q %q network config to use CIDR(s) %q", ns, cephNet, settingVal)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// simple wrapper to discover public and cluster nets at the same time in parallel
|
||||
func discoverCephAddressRanges(
|
||||
ctx context.Context,
|
||||
rookImage string,
|
||||
clusterdContext *clusterd.Context,
|
||||
clusterSpec *cephv1.ClusterSpec,
|
||||
clusterInfo *cephclient.ClusterInfo,
|
||||
discoverPublic, discoverCluster bool,
|
||||
) (
|
||||
publicRanges []cephv1.CIDR,
|
||||
clusterRanges []cephv1.CIDR,
|
||||
err error,
|
||||
) {
|
||||
type rangeResult struct {
|
||||
ranges []string
|
||||
err error
|
||||
}
|
||||
publicChan := make(chan rangeResult, 1)
|
||||
defer close(publicChan)
|
||||
clusterChan := make(chan rangeResult, 1)
|
||||
defer close(clusterChan)
|
||||
|
||||
ctx, cancel := context.WithDeadline(ctx, time.Now().Add(detectNetworkCIDRTimeout))
|
||||
defer cancel()
|
||||
|
||||
if discoverPublic && clusterSpec.Network.NetworkHasSelection(cephv1.CephNetworkPublic) {
|
||||
go func() {
|
||||
ranges, err := discoverAddressRanges(ctx, rookImage, clusterdContext, clusterSpec, clusterInfo, cephv1.CephNetworkPublic)
|
||||
publicChan <- rangeResult{ranges, err}
|
||||
}()
|
||||
} else {
|
||||
publicChan <- rangeResult{[]string{}, nil}
|
||||
}
|
||||
|
||||
if discoverCluster && clusterSpec.Network.NetworkHasSelection(cephv1.CephNetworkCluster) {
|
||||
go func() {
|
||||
ranges, err := discoverAddressRanges(ctx, rookImage, clusterdContext, clusterSpec, clusterInfo, cephv1.CephNetworkCluster)
|
||||
clusterChan <- rangeResult{ranges, err}
|
||||
}()
|
||||
} else {
|
||||
clusterChan <- rangeResult{[]string{}, nil}
|
||||
}
|
||||
|
||||
publicResult := <-publicChan
|
||||
clusterResult := <-clusterChan
|
||||
|
||||
if publicResult.err != nil {
|
||||
return publicRanges, clusterRanges, publicResult.err
|
||||
}
|
||||
if clusterResult.err != nil {
|
||||
return publicRanges, clusterRanges, clusterResult.err
|
||||
}
|
||||
|
||||
return toCephCIDRs(publicResult.ranges), toCephCIDRs(clusterResult.ranges), nil
|
||||
}
|
||||
|
||||
func toCephCIDRs(rawCIDRs []string) []cephv1.CIDR {
|
||||
cidrs := make([]cephv1.CIDR, 0, len(rawCIDRs))
|
||||
for _, c := range rawCIDRs {
|
||||
cidrs = append(cidrs, cephv1.CIDR(c))
|
||||
}
|
||||
return cidrs
|
||||
}
|
||||
|
||||
func discoverAddressRanges(
|
||||
ctx context.Context,
|
||||
rookImage string,
|
||||
clusterdContext *clusterd.Context,
|
||||
clusterSpec *cephv1.ClusterSpec,
|
||||
clusterInfo *cephclient.ClusterInfo,
|
||||
cephNetwork cephv1.CephNetworkType,
|
||||
) (
|
||||
ranges []string,
|
||||
err error,
|
||||
) {
|
||||
ranges = []string{}
|
||||
clusterNamespace := clusterInfo.Namespace
|
||||
|
||||
logger.Infof("discovering ceph %q network CIDR(s) for cluster %q", cephNetwork, clusterNamespace)
|
||||
|
||||
netSelection, err := clusterInfo.NetworkSpec.GetNetworkSelection(clusterNamespace, cephNetwork)
|
||||
if err != nil {
|
||||
return ranges, errors.Wrapf(err, "failed to get %q network selection", cephNetwork)
|
||||
}
|
||||
if netSelection == nil {
|
||||
// no selection for this network
|
||||
return ranges, nil
|
||||
}
|
||||
|
||||
// Modify the user-provided network selection to set the interface name in the pod be the Ceph
|
||||
// network (e.g., public, cluster). This is important for accurately identifying the CIDR range
|
||||
// provided by the network selection in the canary. It is safe to make this change to the
|
||||
// user-defined spec in the canary for this limited purpose, but not safe during Ceph runtime.
|
||||
netSelection.InterfaceRequest = string(cephNetwork)
|
||||
|
||||
netSelectionValue, err := cephv1.NetworkSelectionsToAnnotationValue(netSelection)
|
||||
if err != nil {
|
||||
return ranges, errors.Wrapf(err, "failed to get %q network annotation value for cluster in namespace %q", cephNetwork, clusterNamespace)
|
||||
}
|
||||
|
||||
// This job is complex to overcome 2 specific limitations:
|
||||
// - CNI/Multus' `k8s.v1.cni.cncf.io/network-status` annotation only reports IP addrs, not CIDRs
|
||||
// - `ip addr show` in the pod shows CIDRs but also contains extra IPv6 SLAAC addrs in addition
|
||||
// to the addr(s) attached by CNI/Multus
|
||||
// Rook must cross-reference both pieces of info to accurately read the CIDR(s) for the net.
|
||||
// Both pieces of info must come from the same Pod+Container in order to be cross-ref'd.
|
||||
// Use downward API to allow the cmd reporter job to report both pieces of info at the same time
|
||||
networkCanary, err := newCmdReporter(
|
||||
clusterdContext.Clientset,
|
||||
clusterInfo.OwnerInfo,
|
||||
"rook-ceph-network-canary",
|
||||
"rook-ceph-network-"+string(cephNetwork)+"-canary",
|
||||
clusterNamespace,
|
||||
[]string{"bash", "-c", `set -e
|
||||
cat /var/lib/rook/multus/network-status
|
||||
echo "" # newline
|
||||
echo "` + separator() + `"
|
||||
ip --json address show dev ` + string(cephNetwork) + `
|
||||
`},
|
||||
[]string{},
|
||||
rookImage,
|
||||
clusterSpec.CephVersion.Image,
|
||||
clusterSpec.CephVersion.ImagePullPolicy,
|
||||
)
|
||||
if err != nil {
|
||||
return ranges, errors.Wrapf(err, "failed to set up ceph %q network canary", cephNetwork)
|
||||
}
|
||||
|
||||
job := networkCanary.Job()
|
||||
job.Spec.Template.Spec.ServiceAccountName = "rook-ceph-cmd-reporter"
|
||||
|
||||
// put command reporter job on the multus network
|
||||
job.Spec.Template.Annotations = map[string]string{
|
||||
nadv1.NetworkAttachmentAnnot: netSelectionValue,
|
||||
}
|
||||
|
||||
// use osd placement for net canaries b/c osd pods are present on both public and cluster nets
|
||||
clusterSpec.Placement[cephv1.KeyOSD].ApplyToPodSpec(&job.Spec.Template.Spec)
|
||||
|
||||
// set up net status vol from downward api, plus init container to wait for net status to be available
|
||||
netStatusVol, netStatusMount := networkStatusVolumeAndMount()
|
||||
job.Spec.Template.Spec.Volumes = append(job.Spec.Template.Spec.Volumes, netStatusVol)
|
||||
job.Spec.Template.Spec.Containers[0].VolumeMounts = append(job.Spec.Template.Spec.Containers[0].VolumeMounts, netStatusMount)
|
||||
job.Spec.Template.Spec.InitContainers = append(job.Spec.Template.Spec.InitContainers, containerWaitForNetworkStatus(clusterSpec))
|
||||
|
||||
stdout, stderr, retcode, err := networkCanary.Run(ctx, detectNetworkCIDRTimeout)
|
||||
if err != nil {
|
||||
return ranges, errors.Wrapf(err, "failed to complete ceph %q network canary job", cephNetwork)
|
||||
}
|
||||
if retcode != 0 {
|
||||
return ranges, errors.Errorf("ceph %q network canary ceph CSI version job returned failure code %d: stdout: %q: stderr: %q", cephNetwork, retcode, stdout, stderr)
|
||||
}
|
||||
|
||||
splitOutput := strings.Split(stdout, separator())
|
||||
if len(splitOutput) != 2 {
|
||||
// don't log the raw outputs in case there could be sensitive info
|
||||
return ranges, errors.Errorf("ceph %q network canary job did not return two separate outputs (instead %d)", cephNetwork, len(splitOutput))
|
||||
}
|
||||
|
||||
rawNetStatus := strings.TrimSpace(splitOutput[0])
|
||||
if rawNetStatus == "" {
|
||||
return ranges, errors.Errorf("ceph %q network canary job did not return network status", cephNetwork)
|
||||
}
|
||||
|
||||
rawIpAddrs := strings.TrimSpace(splitOutput[1])
|
||||
if rawIpAddrs == "" {
|
||||
return ranges, errors.Errorf("ceph %q network canary job did not return ip address info", cephNetwork)
|
||||
}
|
||||
|
||||
netStatuses, err := k8sutil.ParseNetworkStatusAnnotation(rawNetStatus)
|
||||
if err != nil {
|
||||
return ranges, errors.Wrapf(err, "failed to parse ceph %q network canary status annotation", cephNetwork)
|
||||
}
|
||||
netStatus, ok := k8sutil.FindNetworkStatusByInterface(netStatuses, string(cephNetwork))
|
||||
if !ok {
|
||||
return ranges, errors.Errorf("failed to find ceph %q network status in canary status annotation", cephNetwork)
|
||||
}
|
||||
|
||||
ipAddrs, err := k8sutil.ParseLinuxIpAddrOutput(rawIpAddrs)
|
||||
if err != nil {
|
||||
return ranges, errors.Wrapf(err, "failed to get IP address info for ceph %q network in canary pod", cephNetwork)
|
||||
}
|
||||
if len(ipAddrs) != 1 {
|
||||
return []string{}, errors.Errorf("should have only one (found %d) 'ip address' result from network %q canary pod", len(ipAddrs), cephNetwork)
|
||||
}
|
||||
ifaceInfo := ipAddrs[0]
|
||||
|
||||
ranges, err = crossReferenceNetworkStatusAndIpResult(netStatus, ifaceInfo)
|
||||
if err != nil {
|
||||
return []string{}, errors.Wrapf(err, "failed to cross reference network %q canary pod status annotation and internal ip results", cephNetwork)
|
||||
}
|
||||
|
||||
return ranges, nil
|
||||
}
|
||||
|
||||
func separator() string {
|
||||
return "===== " + nadv1.NetworkStatusAnnot + " above ===== ip address below ====="
|
||||
}
|
||||
|
||||
// use downward api to make the network status available at /tmp/network-status
|
||||
// NOTE: env var might be preferable, but in testing, putting annotation into env var via downward
|
||||
// api had a race condition where the value was almost always empty when the container first ran,
|
||||
// even after waiting for the annotation in an init container; so use volume instead
|
||||
func networkStatusVolumeAndMount() (corev1.Volume, corev1.VolumeMount) {
|
||||
vol := corev1.Volume{
|
||||
Name: "network-status",
|
||||
VolumeSource: corev1.VolumeSource{
|
||||
DownwardAPI: &corev1.DownwardAPIVolumeSource{
|
||||
Items: []v1.DownwardAPIVolumeFile{{
|
||||
Path: "network-status",
|
||||
FieldRef: &v1.ObjectFieldSelector{
|
||||
FieldPath: `metadata.annotations['` + nadv1.NetworkStatusAnnot + `']`,
|
||||
}}}}}}
|
||||
mnt := corev1.VolumeMount{
|
||||
Name: "network-status",
|
||||
MountPath: "/var/lib/rook/multus",
|
||||
}
|
||||
return vol, mnt
|
||||
}
|
||||
|
||||
// cmd reporter expects its command (or script) to run correctly the first time, but the job needs
|
||||
// to wait until the network status annotation is present to output info before running so that it
|
||||
// reports all the info. separate step of waiting for network status annotation into an init
|
||||
// container so that it can be more easily debugged (output not present in cmd reporter's result)
|
||||
func containerWaitForNetworkStatus(clusterSpec *cephv1.ClusterSpec) corev1.Container {
|
||||
_, mount := networkStatusVolumeAndMount()
|
||||
return corev1.Container{
|
||||
Name: "wait-for-network-status-annotation",
|
||||
Command: []string{
|
||||
"bash",
|
||||
"-c",
|
||||
`set -e
|
||||
while [ ! -s /var/lib/rook/multus/network-status ]; do
|
||||
echo "waiting for network status"
|
||||
sleep 2
|
||||
done
|
||||
cat /var/lib/rook/multus/network-status
|
||||
echo "" # newline
|
||||
`,
|
||||
},
|
||||
Image: clusterSpec.CephVersion.Image,
|
||||
ImagePullPolicy: clusterSpec.CephVersion.ImagePullPolicy,
|
||||
VolumeMounts: []corev1.VolumeMount{
|
||||
mount,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// for every IP reported in the network status annotation, find the network info reported by
|
||||
// 'ip --json address show' inside the pod, and for the result into a list of reduced net CIDRs
|
||||
func crossReferenceNetworkStatusAndIpResult(
|
||||
netStatus nadv1.NetworkStatus,
|
||||
ifaceInfo k8sutil.LinuxIpAddrResult,
|
||||
) ([]string, error) {
|
||||
if netStatus.Interface != ifaceInfo.InterfaceName {
|
||||
// should never happen, but good to do this final check
|
||||
return []string{}, errors.Errorf("network status and internal ip interfaces do not match: %q != %q", netStatus.Interface, ifaceInfo.InterfaceName)
|
||||
}
|
||||
cidrs := []string{}
|
||||
for _, ip := range netStatus.IPs {
|
||||
c, err := cidrForIp(ifaceInfo.AddrInfo, ip)
|
||||
if err != nil {
|
||||
return []string{}, errors.Wrapf(err, "failed getting CIDR for IP %q", ip)
|
||||
}
|
||||
cidrs = append(cidrs, c)
|
||||
}
|
||||
return cidrs, nil
|
||||
}
|
||||
|
||||
func findAddrInfoForIp(infos []k8sutil.LinuxIpAddrInfo, ip string) (k8sutil.LinuxIpAddrInfo, bool) {
|
||||
for _, info := range infos {
|
||||
if info.Local == ip {
|
||||
return info, true
|
||||
}
|
||||
}
|
||||
return k8sutil.LinuxIpAddrInfo{}, false
|
||||
}
|
||||
|
||||
func cidrForIp(infos []k8sutil.LinuxIpAddrInfo, ip string) (string, error) {
|
||||
info, ok := findAddrInfoForIp(infos, ip)
|
||||
if !ok {
|
||||
return "", errors.Errorf("no info for ip %q", ip)
|
||||
}
|
||||
|
||||
// <ip>/<range> works but isn't reduced. the CIDR must be reduced so that it doesn't change
|
||||
// every tme the operator reconciles and the net canary gets a new IP
|
||||
naiveCIDR := fmt.Sprintf("%s/%d", info.Local, info.PrefixLen)
|
||||
|
||||
_, reduced, err := net.ParseCIDR(naiveCIDR)
|
||||
if err != nil {
|
||||
return "", errors.Wrapf(err, "failed to convert %q into a reduced CIDR", naiveCIDR)
|
||||
}
|
||||
return reduced.String(), nil
|
||||
}
|
||||
@@ -0,0 +1,830 @@
|
||||
/*
|
||||
Copyright 2023 The Rook Authors. All rights reserved.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
*/
|
||||
|
||||
package controller
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/pkg/errors"
|
||||
cephv1 "github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
|
||||
"github.com/rook/rook/pkg/clusterd"
|
||||
"github.com/rook/rook/pkg/daemon/ceph/client"
|
||||
"github.com/rook/rook/pkg/operator/k8sutil"
|
||||
"github.com/rook/rook/pkg/operator/k8sutil/cmdreporter"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/mock"
|
||||
batch "k8s.io/api/batch/v1"
|
||||
v1 "k8s.io/api/core/v1"
|
||||
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
|
||||
"k8s.io/client-go/kubernetes"
|
||||
)
|
||||
|
||||
type mockCmdReporter struct {
|
||||
mock.Mock
|
||||
|
||||
job *batch.Job
|
||||
}
|
||||
|
||||
func (m *mockCmdReporter) Job() *batch.Job {
|
||||
m.Called()
|
||||
return m.job
|
||||
}
|
||||
|
||||
func (m *mockCmdReporter) Run(ctx context.Context, timeout time.Duration) (stdout, stderr string, retcode int, retErr error) {
|
||||
args := m.Called(ctx, timeout)
|
||||
return args.String(0), args.String(1), args.Int(2), args.Error(3)
|
||||
}
|
||||
|
||||
// returns a newCmdReporter function that returns the given mockCmdReporter and error
|
||||
func mockNewCmdReporter(m *mockCmdReporter, err error) func(clientset kubernetes.Interface, ownerInfo *k8sutil.OwnerInfo, appName string, jobName string, jobNamespace string, cmd []string, args []string, rookImage string, runImage string, imagePullPolicy v1.PullPolicy) (cmdreporter.CmdReporterInterface, error) {
|
||||
return func(clientset kubernetes.Interface, ownerInfo *k8sutil.OwnerInfo, appName, jobName, jobNamespace string, cmd, args []string, rookImage, runImage string, imagePullPolicy v1.PullPolicy) (cmdreporter.CmdReporterInterface, error) {
|
||||
job, err := cmdreporter.MockCmdReporterJob(clientset, ownerInfo, appName, jobName, jobNamespace, cmd, args, rookImage, runImage, imagePullPolicy)
|
||||
if err != nil {
|
||||
// okay to panic here because this is a unit test setup failure, not part of code testing
|
||||
panic(fmt.Sprintf("error setting up mock CmdReporter job: %v", err))
|
||||
}
|
||||
m.job = job
|
||||
return m, err
|
||||
}
|
||||
}
|
||||
|
||||
func newTestConfigsWithNetworkSpec(n cephv1.NetworkSpec) (*clusterd.Context, *cephv1.ClusterSpec, *client.ClusterInfo) {
|
||||
clusterdCtx := &clusterd.Context{}
|
||||
clusterSpec := &cephv1.ClusterSpec{
|
||||
Network: n,
|
||||
}
|
||||
clusterInfo := &client.ClusterInfo{
|
||||
Namespace: "ns",
|
||||
NetworkSpec: clusterSpec.Network,
|
||||
OwnerInfo: k8sutil.NewOwnerInfoWithOwnerRef(&metav1.OwnerReference{}, "ns"),
|
||||
}
|
||||
return clusterdCtx, clusterSpec, clusterInfo
|
||||
}
|
||||
|
||||
func Test_discoverAddressRanges(t *testing.T) {
|
||||
oldNewCmdReporter := newCmdReporter
|
||||
defer func() { newCmdReporter = oldNewCmdReporter }()
|
||||
|
||||
t.Run("public network selected", func(t *testing.T) {
|
||||
clusterdCtx, clusterSpec, clusterInfo := newTestConfigsWithNetworkSpec(
|
||||
cephv1.NetworkSpec{
|
||||
Provider: cephv1.NetworkProviderMultus,
|
||||
Selectors: map[cephv1.CephNetworkType]string{
|
||||
cephv1.CephNetworkPublic: "macvlan-public",
|
||||
},
|
||||
AddressRanges: nil, // should auto-detect
|
||||
})
|
||||
|
||||
t.Run("discover public cidrs", func(t *testing.T) {
|
||||
cmdReporter := new(mockCmdReporter)
|
||||
cmdReporter.On("Job")
|
||||
goodOutput := netStatus("public", "2000::1") +
|
||||
"\n" + separator() + "\n" +
|
||||
ipAddrOutput("public", "2000::1/112")
|
||||
cmdReporter.On("Run", mock.Anything, mock.Anything).Return(goodOutput, "", 0, nil)
|
||||
newCmdReporter = mockNewCmdReporter(cmdReporter, nil)
|
||||
|
||||
ranges, err := discoverAddressRanges(context.Background(), "rook/ceph:master", clusterdCtx, clusterSpec, clusterInfo, cephv1.CephNetworkPublic)
|
||||
assert.NoError(t, err)
|
||||
assert.Equal(t, []string{"2000::/112"}, ranges)
|
||||
})
|
||||
t.Run("discover cluster cidrs", func(t *testing.T) {
|
||||
cmdReporter := new(mockCmdReporter)
|
||||
// no cmd reporter methods should be called b/c cluster net is not selected
|
||||
newCmdReporter = mockNewCmdReporter(cmdReporter, nil)
|
||||
|
||||
ranges, err := discoverAddressRanges(context.Background(), "rook/ceph:master", clusterdCtx, clusterSpec, clusterInfo, cephv1.CephNetworkCluster)
|
||||
assert.NoError(t, err)
|
||||
assert.Equal(t, []string{}, ranges)
|
||||
})
|
||||
t.Run("public ip mismatch", func(t *testing.T) {
|
||||
cmdReporter := new(mockCmdReporter)
|
||||
cmdReporter.On("Job")
|
||||
goodOutput := netStatus("public", "192.168.1.1") +
|
||||
"\n" + separator() + "\n" +
|
||||
ipAddrOutput("public", "192.168.1.2/24") // IP is one more than above
|
||||
cmdReporter.On("Run", mock.Anything, mock.Anything).Return(goodOutput, "", 0, nil)
|
||||
newCmdReporter = mockNewCmdReporter(cmdReporter, nil)
|
||||
|
||||
ranges, err := discoverAddressRanges(context.Background(), "rook/ceph:master", clusterdCtx, clusterSpec, clusterInfo, cephv1.CephNetworkPublic)
|
||||
assert.Error(t, err)
|
||||
assert.Equal(t, []string{}, ranges)
|
||||
})
|
||||
t.Run("public ip iface mismatch", func(t *testing.T) {
|
||||
cmdReporter := new(mockCmdReporter)
|
||||
cmdReporter.On("Job")
|
||||
goodOutput := netStatus("public", "192.168.1.1") +
|
||||
"\n" + separator() + "\n" +
|
||||
ipAddrOutput("cluster", "192.168.1.1/24") // cluster != public above
|
||||
cmdReporter.On("Run", mock.Anything, mock.Anything).Return(goodOutput, "", 0, nil)
|
||||
newCmdReporter = mockNewCmdReporter(cmdReporter, nil)
|
||||
|
||||
ranges, err := discoverAddressRanges(context.Background(), "rook/ceph:master", clusterdCtx, clusterSpec, clusterInfo, cephv1.CephNetworkPublic)
|
||||
assert.Error(t, err)
|
||||
assert.Equal(t, []string{}, ranges)
|
||||
})
|
||||
t.Run("iface wrong", func(t *testing.T) {
|
||||
cmdReporter := new(mockCmdReporter)
|
||||
cmdReporter.On("Job")
|
||||
goodOutput := netStatus("cluster", "192.168.1.1") +
|
||||
"\n" + separator() + "\n" +
|
||||
ipAddrOutput("cluster", "192.168.1.1/24") // cluster != public above
|
||||
cmdReporter.On("Run", mock.Anything, mock.Anything).Return(goodOutput, "", 0, nil)
|
||||
newCmdReporter = mockNewCmdReporter(cmdReporter, nil)
|
||||
|
||||
ranges, err := discoverAddressRanges(context.Background(), "rook/ceph:master", clusterdCtx, clusterSpec, clusterInfo, cephv1.CephNetworkPublic)
|
||||
assert.Error(t, err)
|
||||
assert.Equal(t, []string{}, ranges)
|
||||
})
|
||||
t.Run("no net status", func(t *testing.T) {
|
||||
cmdReporter := new(mockCmdReporter)
|
||||
cmdReporter.On("Job")
|
||||
goodOutput := "" + // no net status
|
||||
"\n" + separator() + "\n" +
|
||||
ipAddrOutput("public", "192.168.1.1/24")
|
||||
cmdReporter.On("Run", mock.Anything, mock.Anything).Return(goodOutput, "", 0, nil)
|
||||
newCmdReporter = mockNewCmdReporter(cmdReporter, nil)
|
||||
|
||||
ranges, err := discoverAddressRanges(context.Background(), "rook/ceph:master", clusterdCtx, clusterSpec, clusterInfo, cephv1.CephNetworkPublic)
|
||||
assert.Error(t, err)
|
||||
assert.Equal(t, []string{}, ranges)
|
||||
})
|
||||
t.Run("no ip addr output", func(t *testing.T) {
|
||||
cmdReporter := new(mockCmdReporter)
|
||||
cmdReporter.On("Job")
|
||||
goodOutput := netStatus("public", "2000::1") +
|
||||
"\n" + separator() + "\n" +
|
||||
"" // no ip addr output
|
||||
cmdReporter.On("Run", mock.Anything, mock.Anything).Return(goodOutput, "", 0, nil)
|
||||
newCmdReporter = mockNewCmdReporter(cmdReporter, nil)
|
||||
|
||||
ranges, err := discoverAddressRanges(context.Background(), "rook/ceph:master", clusterdCtx, clusterSpec, clusterInfo, cephv1.CephNetworkPublic)
|
||||
assert.Error(t, err)
|
||||
assert.Equal(t, []string{}, ranges)
|
||||
})
|
||||
t.Run("cmdreporter run failure", func(t *testing.T) {
|
||||
cmdReporter := new(mockCmdReporter)
|
||||
cmdReporter.On("Job")
|
||||
cmdReporter.On("Run", mock.Anything, mock.Anything).Return("", "", 0, errors.New("fake err"))
|
||||
newCmdReporter = mockNewCmdReporter(cmdReporter, nil)
|
||||
|
||||
ranges, err := discoverAddressRanges(context.Background(), "rook/ceph:master", clusterdCtx, clusterSpec, clusterInfo, cephv1.CephNetworkPublic)
|
||||
assert.Error(t, err)
|
||||
assert.Equal(t, []string{}, ranges)
|
||||
})
|
||||
t.Run("cmdreporter nonzero exit", func(t *testing.T) {
|
||||
cmdReporter := new(mockCmdReporter)
|
||||
cmdReporter.On("Job")
|
||||
cmdReporter.On("Run", mock.Anything, mock.Anything).Return("some garbage", "", 1, nil)
|
||||
newCmdReporter = mockNewCmdReporter(cmdReporter, nil)
|
||||
|
||||
ranges, err := discoverAddressRanges(context.Background(), "rook/ceph:master", clusterdCtx, clusterSpec, clusterInfo, cephv1.CephNetworkPublic)
|
||||
assert.Error(t, err)
|
||||
assert.Equal(t, []string{}, ranges)
|
||||
})
|
||||
})
|
||||
|
||||
t.Run("cluster network selected", func(t *testing.T) {
|
||||
clusterdCtx, clusterSpec, clusterInfo := newTestConfigsWithNetworkSpec(
|
||||
cephv1.NetworkSpec{
|
||||
Provider: cephv1.NetworkProviderMultus,
|
||||
Selectors: map[cephv1.CephNetworkType]string{
|
||||
cephv1.CephNetworkCluster: "macvlan-cluster",
|
||||
},
|
||||
AddressRanges: nil, // should auto-detect
|
||||
})
|
||||
|
||||
t.Run("discover cluster cidrs", func(t *testing.T) {
|
||||
cmdReporter := new(mockCmdReporter)
|
||||
cmdReporter.On("Job")
|
||||
goodOutput := netStatus("cluster", "10.144.1.5") +
|
||||
"\n" + separator() + "\n" +
|
||||
ipAddrOutput("cluster", "10.144.1.5/16")
|
||||
cmdReporter.On("Run", mock.Anything, mock.Anything).Return(goodOutput, "", 0, nil)
|
||||
newCmdReporter = mockNewCmdReporter(cmdReporter, nil)
|
||||
|
||||
ranges, err := discoverAddressRanges(context.Background(), "rook/ceph:master", clusterdCtx, clusterSpec, clusterInfo, cephv1.CephNetworkCluster)
|
||||
assert.NoError(t, err)
|
||||
assert.Equal(t, []string{"10.144.0.0/16"}, ranges)
|
||||
})
|
||||
t.Run("discover public cidrs", func(t *testing.T) {
|
||||
cmdReporter := new(mockCmdReporter)
|
||||
// no cmd reporter methods should be called b/c public net is not selected
|
||||
newCmdReporter = mockNewCmdReporter(cmdReporter, nil)
|
||||
|
||||
ranges, err := discoverAddressRanges(context.Background(), "rook/ceph:master", clusterdCtx, clusterSpec, clusterInfo, cephv1.CephNetworkPublic)
|
||||
assert.NoError(t, err)
|
||||
assert.Equal(t, []string{}, ranges)
|
||||
})
|
||||
})
|
||||
|
||||
t.Run("public and cluster networks selected", func(t *testing.T) {
|
||||
clusterdCtx, clusterSpec, clusterInfo := newTestConfigsWithNetworkSpec(
|
||||
cephv1.NetworkSpec{
|
||||
Provider: cephv1.NetworkProviderMultus,
|
||||
Selectors: map[cephv1.CephNetworkType]string{
|
||||
cephv1.CephNetworkPublic: "macvlan-public",
|
||||
cephv1.CephNetworkCluster: "macvlan-cluster",
|
||||
},
|
||||
AddressRanges: nil, // should auto-detect
|
||||
})
|
||||
|
||||
t.Run("discover public cidrs", func(t *testing.T) {
|
||||
cmdReporter := new(mockCmdReporter)
|
||||
cmdReporter.On("Job")
|
||||
goodOutput := netStatus("public", "10.144.1.5") +
|
||||
"\n" + separator() + "\n" +
|
||||
ipAddrOutput("public", "10.144.1.5/16")
|
||||
cmdReporter.On("Run", mock.Anything, mock.Anything).Return(goodOutput, "", 0, nil)
|
||||
newCmdReporter = mockNewCmdReporter(cmdReporter, nil)
|
||||
|
||||
ranges, err := discoverAddressRanges(context.Background(), "rook/ceph:master", clusterdCtx, clusterSpec, clusterInfo, cephv1.CephNetworkPublic)
|
||||
assert.NoError(t, err)
|
||||
assert.Equal(t, []string{"10.144.0.0/16"}, ranges)
|
||||
})
|
||||
t.Run("discover cluster cidrs", func(t *testing.T) {
|
||||
cmdReporter := new(mockCmdReporter)
|
||||
cmdReporter.On("Job")
|
||||
goodOutput := netStatus("cluster", "1abc::1") +
|
||||
"\n" + separator() + "\n" +
|
||||
ipAddrOutput("cluster", "1abc::1/112")
|
||||
cmdReporter.On("Run", mock.Anything, mock.Anything).Return(goodOutput, "", 0, nil)
|
||||
newCmdReporter = mockNewCmdReporter(cmdReporter, nil)
|
||||
|
||||
ranges, err := discoverAddressRanges(context.Background(), "rook/ceph:master", clusterdCtx, clusterSpec, clusterInfo, cephv1.CephNetworkCluster)
|
||||
assert.NoError(t, err)
|
||||
assert.Equal(t, []string{"1abc::/112"}, ranges)
|
||||
})
|
||||
t.Run("public multiple ips", func(t *testing.T) {
|
||||
cmdReporter := new(mockCmdReporter)
|
||||
cmdReporter.On("Job")
|
||||
goodOutput := netStatus("public", "10.144.1.5", "fe05::6") +
|
||||
"\n" + separator() + "\n" +
|
||||
ipAddrOutput("public", "10.144.1.5/16", "fe05::6/96")
|
||||
cmdReporter.On("Run", mock.Anything, mock.Anything).Return(goodOutput, "", 0, nil)
|
||||
newCmdReporter = mockNewCmdReporter(cmdReporter, nil)
|
||||
|
||||
ranges, err := discoverAddressRanges(context.Background(), "rook/ceph:master", clusterdCtx, clusterSpec, clusterInfo, cephv1.CephNetworkPublic)
|
||||
assert.NoError(t, err)
|
||||
assert.Equal(t, []string{"10.144.0.0/16", "fe05::/96"}, ranges)
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
func mockDiscoverAddressRangesFunc(publicReturn []string, clusterReturn []string, panicIfDiscoverPublic, panicIfDiscoverCluster bool, shouldErr error) func(ctx context.Context, rookImage string, clusterdContext *clusterd.Context, clusterSpec *cephv1.ClusterSpec, clusterInfo *client.ClusterInfo, discoverPublic bool, discoverCluster bool) (publicRanges []cephv1.CIDR, clusterRanges []cephv1.CIDR, err error) {
|
||||
return func(ctx context.Context, rookImage string, clusterdContext *clusterd.Context, clusterSpec *cephv1.ClusterSpec, clusterInfo *client.ClusterInfo, discoverPublic bool, discoverCluster bool) (publicRanges []cephv1.CIDR, clusterRanges []cephv1.CIDR, err error) {
|
||||
// panic is fine for unit testing
|
||||
if panicIfDiscoverPublic && discoverPublic {
|
||||
panic("should not discover public net")
|
||||
}
|
||||
if panicIfDiscoverCluster && discoverCluster {
|
||||
panic("should not discover cluster net")
|
||||
}
|
||||
return toCephCIDRs(publicReturn), toCephCIDRs(clusterReturn), shouldErr
|
||||
}
|
||||
}
|
||||
|
||||
type mockMonStore struct {
|
||||
mock.Mock
|
||||
}
|
||||
|
||||
func (m *mockMonStore) SetIfChanged(who string, option string, value string) (bool, error) {
|
||||
args := m.Called(who, option, value)
|
||||
return args.Bool(0), args.Error(1)
|
||||
}
|
||||
|
||||
func mockGetMonStoreFunc(m *mockMonStore) func(context *clusterd.Context, clusterInfo *client.ClusterInfo) monStoreInterface {
|
||||
return func(context *clusterd.Context, clusterInfo *client.ClusterInfo) monStoreInterface {
|
||||
return m
|
||||
}
|
||||
}
|
||||
|
||||
func TestApplyCephNetworkSettings(t *testing.T) {
|
||||
oldDiscoverCephAddressRangesFunc := discoverCephAddressRangesFunc
|
||||
defer func() { discoverCephAddressRangesFunc = oldDiscoverCephAddressRangesFunc }()
|
||||
|
||||
oldGetMonStoreFunc := getMonStoreFunc
|
||||
defer func() { getMonStoreFunc = oldGetMonStoreFunc }()
|
||||
|
||||
t.Run("no net defined", func(t *testing.T) {
|
||||
clusterdCtx, clusterSpec, clusterInfo := newTestConfigsWithNetworkSpec(cephv1.NetworkSpec{})
|
||||
|
||||
discoverCephAddressRangesFunc = mockDiscoverAddressRangesFunc([]string{}, []string{}, false, false, errors.New("should not be called"))
|
||||
|
||||
monStore := new(mockMonStore)
|
||||
// implicit: mon store should not have SetIfChanged called on it
|
||||
getMonStoreFunc = mockGetMonStoreFunc(monStore)
|
||||
|
||||
err := ApplyCephNetworkSettings(context.Background(), "rook/ceph:master", clusterdCtx, clusterSpec, clusterInfo)
|
||||
assert.NoError(t, err)
|
||||
})
|
||||
t.Run("multus", func(t *testing.T) {
|
||||
t.Run("public selected", func(t *testing.T) {
|
||||
clusterdCtx, clusterSpec, clusterInfo := newTestConfigsWithNetworkSpec(
|
||||
cephv1.NetworkSpec{
|
||||
Provider: cephv1.NetworkProviderMultus,
|
||||
Selectors: map[cephv1.CephNetworkType]string{
|
||||
cephv1.CephNetworkPublic: "macvlan-public",
|
||||
},
|
||||
AddressRanges: nil, // should auto-detect
|
||||
})
|
||||
|
||||
t.Run("single public cidr", func(t *testing.T) {
|
||||
discoverCephAddressRangesFunc = mockDiscoverAddressRangesFunc([]string{"192.168.0.0/16"}, []string{}, false, false, nil)
|
||||
|
||||
monStore := new(mockMonStore)
|
||||
monStore.On("SetIfChanged", "global", "public_network", "192.168.0.0/16").Return(true, nil)
|
||||
getMonStoreFunc = mockGetMonStoreFunc(monStore)
|
||||
|
||||
err := ApplyCephNetworkSettings(context.Background(), "rook/ceph:master", clusterdCtx, clusterSpec, clusterInfo)
|
||||
assert.NoError(t, err)
|
||||
})
|
||||
t.Run("multiple public cidrs", func(t *testing.T) {
|
||||
discoverCephAddressRangesFunc = mockDiscoverAddressRangesFunc([]string{"192.168.0.0/16", "feab::/112"}, []string{}, false, false, nil)
|
||||
|
||||
monStore := new(mockMonStore)
|
||||
monStore.On("SetIfChanged", "global", "public_network", "192.168.0.0/16, feab::/112").Return(true, nil)
|
||||
getMonStoreFunc = mockGetMonStoreFunc(monStore)
|
||||
|
||||
err := ApplyCephNetworkSettings(context.Background(), "rook/ceph:master", clusterdCtx, clusterSpec, clusterInfo)
|
||||
assert.NoError(t, err)
|
||||
})
|
||||
})
|
||||
t.Run("cluster selected", func(t *testing.T) {
|
||||
clusterdCtx, clusterSpec, clusterInfo := newTestConfigsWithNetworkSpec(
|
||||
cephv1.NetworkSpec{
|
||||
Provider: cephv1.NetworkProviderMultus,
|
||||
Selectors: map[cephv1.CephNetworkType]string{
|
||||
cephv1.CephNetworkCluster: "macvlan-cluster",
|
||||
},
|
||||
AddressRanges: nil, // should auto-detect
|
||||
})
|
||||
|
||||
t.Run("single cluster cidr", func(t *testing.T) {
|
||||
discoverCephAddressRangesFunc = mockDiscoverAddressRangesFunc([]string{}, []string{"192.168.0.0/16"}, false, false, nil)
|
||||
|
||||
monStore := new(mockMonStore)
|
||||
monStore.On("SetIfChanged", "global", "cluster_network", "192.168.0.0/16").Return(true, nil)
|
||||
getMonStoreFunc = mockGetMonStoreFunc(monStore)
|
||||
|
||||
err := ApplyCephNetworkSettings(context.Background(), "rook/ceph:master", clusterdCtx, clusterSpec, clusterInfo)
|
||||
assert.NoError(t, err)
|
||||
})
|
||||
t.Run("multiple cluster cidrs", func(t *testing.T) {
|
||||
discoverCephAddressRangesFunc = mockDiscoverAddressRangesFunc([]string{}, []string{"192.168.0.0/16", "feab::/112"}, false, false, nil)
|
||||
|
||||
monStore := new(mockMonStore)
|
||||
monStore.On("SetIfChanged", "global", "cluster_network", "192.168.0.0/16, feab::/112").Return(true, nil)
|
||||
getMonStoreFunc = mockGetMonStoreFunc(monStore)
|
||||
|
||||
err := ApplyCephNetworkSettings(context.Background(), "rook/ceph:master", clusterdCtx, clusterSpec, clusterInfo)
|
||||
assert.NoError(t, err)
|
||||
})
|
||||
})
|
||||
t.Run("public and cluster selected", func(t *testing.T) {
|
||||
clusterdCtx, clusterSpec, clusterInfo := newTestConfigsWithNetworkSpec(
|
||||
cephv1.NetworkSpec{
|
||||
Provider: cephv1.NetworkProviderMultus,
|
||||
Selectors: map[cephv1.CephNetworkType]string{
|
||||
cephv1.CephNetworkPublic: "macvlan-public",
|
||||
cephv1.CephNetworkCluster: "macvlan-cluster",
|
||||
},
|
||||
AddressRanges: nil, // should auto-detect
|
||||
})
|
||||
|
||||
t.Run("single cidr", func(t *testing.T) {
|
||||
discoverCephAddressRangesFunc = mockDiscoverAddressRangesFunc([]string{"192.168.0.0/16"}, []string{"192.168.10.0/16"}, false, false, nil)
|
||||
|
||||
monStore := new(mockMonStore)
|
||||
monStore.On("SetIfChanged", "global", "public_network", "192.168.0.0/16").Return(true, nil)
|
||||
monStore.On("SetIfChanged", "global", "cluster_network", "192.168.10.0/16").Return(true, nil)
|
||||
getMonStoreFunc = mockGetMonStoreFunc(monStore)
|
||||
|
||||
err := ApplyCephNetworkSettings(context.Background(), "rook/ceph:master", clusterdCtx, clusterSpec, clusterInfo)
|
||||
assert.NoError(t, err)
|
||||
})
|
||||
t.Run("multiple cidrs", func(t *testing.T) {
|
||||
discoverCephAddressRangesFunc = mockDiscoverAddressRangesFunc([]string{"192.168.0.0/16", "feab::/112"}, []string{"192.168.10.0/16", "feab:10::/112"}, false, false, nil)
|
||||
|
||||
monStore := new(mockMonStore)
|
||||
monStore.On("SetIfChanged", "global", "public_network", "192.168.0.0/16, feab::/112").Return(true, nil)
|
||||
monStore.On("SetIfChanged", "global", "cluster_network", "192.168.10.0/16, feab:10::/112").Return(true, nil)
|
||||
getMonStoreFunc = mockGetMonStoreFunc(monStore)
|
||||
|
||||
err := ApplyCephNetworkSettings(context.Background(), "rook/ceph:master", clusterdCtx, clusterSpec, clusterInfo)
|
||||
assert.NoError(t, err)
|
||||
})
|
||||
t.Run("single and multiple cidrs", func(t *testing.T) {
|
||||
discoverCephAddressRangesFunc = mockDiscoverAddressRangesFunc([]string{"192.168.0.0/16"}, []string{"192.168.10.0/16", "feab:10::/112"}, false, false, nil)
|
||||
|
||||
monStore := new(mockMonStore)
|
||||
monStore.On("SetIfChanged", "global", "public_network", "192.168.0.0/16").Return(true, nil)
|
||||
monStore.On("SetIfChanged", "global", "cluster_network", "192.168.10.0/16, feab:10::/112").Return(true, nil)
|
||||
getMonStoreFunc = mockGetMonStoreFunc(monStore)
|
||||
|
||||
err := ApplyCephNetworkSettings(context.Background(), "rook/ceph:master", clusterdCtx, clusterSpec, clusterInfo)
|
||||
assert.NoError(t, err)
|
||||
})
|
||||
t.Run("multiple and single cidrs", func(t *testing.T) {
|
||||
discoverCephAddressRangesFunc = mockDiscoverAddressRangesFunc([]string{"192.168.0.0/16", "feab::/112"}, []string{"192.168.10.0/16"}, false, false, nil)
|
||||
|
||||
monStore := new(mockMonStore)
|
||||
monStore.On("SetIfChanged", "global", "public_network", "192.168.0.0/16, feab::/112").Return(true, nil)
|
||||
monStore.On("SetIfChanged", "global", "cluster_network", "192.168.10.0/16").Return(true, nil)
|
||||
getMonStoreFunc = mockGetMonStoreFunc(monStore)
|
||||
|
||||
err := ApplyCephNetworkSettings(context.Background(), "rook/ceph:master", clusterdCtx, clusterSpec, clusterInfo)
|
||||
assert.NoError(t, err)
|
||||
})
|
||||
t.Run("discover err", func(t *testing.T) {
|
||||
discoverCephAddressRangesFunc = mockDiscoverAddressRangesFunc([]string{}, []string{}, false, false, errors.New("fake err"))
|
||||
|
||||
monStore := new(mockMonStore)
|
||||
getMonStoreFunc = mockGetMonStoreFunc(monStore)
|
||||
|
||||
err := ApplyCephNetworkSettings(context.Background(), "rook/ceph:master", clusterdCtx, clusterSpec, clusterInfo)
|
||||
assert.ErrorContains(t, err, "fake err")
|
||||
})
|
||||
t.Run("public apply err", func(t *testing.T) {
|
||||
discoverCephAddressRangesFunc = mockDiscoverAddressRangesFunc([]string{"192.168.0.0/16"}, []string{"192.168.10.0/16"}, false, false, nil)
|
||||
|
||||
monStore := new(mockMonStore)
|
||||
monStore.On("SetIfChanged", "global", "public_network", "192.168.0.0/16").Return(true, errors.New("fake err"))
|
||||
monStore.On("SetIfChanged", "global", "cluster_network", "192.168.10.0/16").Return(true, nil)
|
||||
getMonStoreFunc = mockGetMonStoreFunc(monStore)
|
||||
|
||||
err := ApplyCephNetworkSettings(context.Background(), "rook/ceph:master", clusterdCtx, clusterSpec, clusterInfo)
|
||||
assert.ErrorContains(t, err, "fake err")
|
||||
})
|
||||
t.Run("cluster apply err", func(t *testing.T) {
|
||||
discoverCephAddressRangesFunc = mockDiscoverAddressRangesFunc([]string{"192.168.0.0/16"}, []string{"192.168.10.0/16"}, false, false, nil)
|
||||
|
||||
monStore := new(mockMonStore)
|
||||
monStore.On("SetIfChanged", "global", "public_network", "192.168.0.0/16").Return(true, nil)
|
||||
monStore.On("SetIfChanged", "global", "cluster_network", "192.168.10.0/16").Return(true, errors.New("fake err"))
|
||||
getMonStoreFunc = mockGetMonStoreFunc(monStore)
|
||||
|
||||
err := ApplyCephNetworkSettings(context.Background(), "rook/ceph:master", clusterdCtx, clusterSpec, clusterInfo)
|
||||
assert.ErrorContains(t, err, "fake err")
|
||||
})
|
||||
})
|
||||
t.Run("public selected with address ranges", func(t *testing.T) {
|
||||
clusterdCtx, clusterSpec, clusterInfo := newTestConfigsWithNetworkSpec(
|
||||
cephv1.NetworkSpec{
|
||||
Provider: cephv1.NetworkProviderMultus,
|
||||
Selectors: map[cephv1.CephNetworkType]string{
|
||||
cephv1.CephNetworkPublic: "macvlan-public",
|
||||
},
|
||||
AddressRanges: &cephv1.AddressRangesSpec{
|
||||
Public: []cephv1.CIDR{
|
||||
"192.168.0.0/24",
|
||||
},
|
||||
},
|
||||
})
|
||||
|
||||
discoverCephAddressRangesFunc = mockDiscoverAddressRangesFunc([]string{}, []string{}, true, false, nil)
|
||||
|
||||
monStore := new(mockMonStore)
|
||||
monStore.On("SetIfChanged", "global", "public_network", "192.168.0.0/24").Return(true, nil)
|
||||
getMonStoreFunc = mockGetMonStoreFunc(monStore)
|
||||
|
||||
err := ApplyCephNetworkSettings(context.Background(), "rook/ceph:master", clusterdCtx, clusterSpec, clusterInfo)
|
||||
assert.NoError(t, err)
|
||||
})
|
||||
t.Run("cluster selected with address ranges", func(t *testing.T) {
|
||||
clusterdCtx, clusterSpec, clusterInfo := newTestConfigsWithNetworkSpec(
|
||||
cephv1.NetworkSpec{
|
||||
Provider: cephv1.NetworkProviderMultus,
|
||||
Selectors: map[cephv1.CephNetworkType]string{
|
||||
cephv1.CephNetworkCluster: "macvlan-cluster",
|
||||
},
|
||||
AddressRanges: &cephv1.AddressRangesSpec{
|
||||
Cluster: []cephv1.CIDR{
|
||||
"192.168.0.0/24",
|
||||
},
|
||||
},
|
||||
})
|
||||
|
||||
discoverCephAddressRangesFunc = mockDiscoverAddressRangesFunc([]string{}, []string{}, false, true, nil)
|
||||
|
||||
monStore := new(mockMonStore)
|
||||
monStore.On("SetIfChanged", "global", "cluster_network", "192.168.0.0/24").Return(true, nil)
|
||||
getMonStoreFunc = mockGetMonStoreFunc(monStore)
|
||||
|
||||
err := ApplyCephNetworkSettings(context.Background(), "rook/ceph:master", clusterdCtx, clusterSpec, clusterInfo)
|
||||
assert.NoError(t, err)
|
||||
})
|
||||
t.Run("public and cluster selected with address ranges", func(t *testing.T) {
|
||||
clusterdCtx, clusterSpec, clusterInfo := newTestConfigsWithNetworkSpec(
|
||||
cephv1.NetworkSpec{
|
||||
Provider: cephv1.NetworkProviderMultus,
|
||||
Selectors: map[cephv1.CephNetworkType]string{
|
||||
cephv1.CephNetworkPublic: "macvlan-public",
|
||||
cephv1.CephNetworkCluster: "macvlan-cluster",
|
||||
},
|
||||
AddressRanges: &cephv1.AddressRangesSpec{
|
||||
Public: []cephv1.CIDR{
|
||||
"192.168.0.0/24",
|
||||
},
|
||||
Cluster: []cephv1.CIDR{
|
||||
"192.168.50.0/24",
|
||||
},
|
||||
},
|
||||
})
|
||||
|
||||
discoverCephAddressRangesFunc = mockDiscoverAddressRangesFunc([]string{}, []string{}, true, true, errors.New("should not be called"))
|
||||
|
||||
monStore := new(mockMonStore)
|
||||
monStore.On("SetIfChanged", "global", "public_network", "192.168.0.0/24").Return(true, nil)
|
||||
monStore.On("SetIfChanged", "global", "cluster_network", "192.168.50.0/24").Return(true, nil)
|
||||
getMonStoreFunc = mockGetMonStoreFunc(monStore)
|
||||
|
||||
err := ApplyCephNetworkSettings(context.Background(), "rook/ceph:master", clusterdCtx, clusterSpec, clusterInfo)
|
||||
assert.NoError(t, err)
|
||||
})
|
||||
t.Run("public and cluster selected with multi-cidr address ranges", func(t *testing.T) {
|
||||
clusterdCtx, clusterSpec, clusterInfo := newTestConfigsWithNetworkSpec(
|
||||
cephv1.NetworkSpec{
|
||||
Provider: cephv1.NetworkProviderMultus,
|
||||
Selectors: map[cephv1.CephNetworkType]string{
|
||||
cephv1.CephNetworkPublic: "macvlan-public",
|
||||
cephv1.CephNetworkCluster: "macvlan-cluster",
|
||||
},
|
||||
AddressRanges: &cephv1.AddressRangesSpec{
|
||||
Public: []cephv1.CIDR{
|
||||
"192.168.0.0/24",
|
||||
"fabc::/112",
|
||||
},
|
||||
Cluster: []cephv1.CIDR{
|
||||
"192.168.50.0/24",
|
||||
"fabc:50::/112",
|
||||
},
|
||||
},
|
||||
})
|
||||
|
||||
discoverCephAddressRangesFunc = mockDiscoverAddressRangesFunc([]string{}, []string{}, true, true, errors.New("should not be called"))
|
||||
|
||||
monStore := new(mockMonStore)
|
||||
monStore.On("SetIfChanged", "global", "public_network", "192.168.0.0/24, fabc::/112").Return(true, nil)
|
||||
monStore.On("SetIfChanged", "global", "cluster_network", "192.168.50.0/24, fabc:50::/112").Return(true, nil)
|
||||
getMonStoreFunc = mockGetMonStoreFunc(monStore)
|
||||
|
||||
err := ApplyCephNetworkSettings(context.Background(), "rook/ceph:master", clusterdCtx, clusterSpec, clusterInfo)
|
||||
assert.NoError(t, err)
|
||||
})
|
||||
t.Run("public and cluster selected, only cluster addr ranges specified", func(t *testing.T) {
|
||||
clusterdCtx, clusterSpec, clusterInfo := newTestConfigsWithNetworkSpec(
|
||||
cephv1.NetworkSpec{
|
||||
Provider: cephv1.NetworkProviderMultus,
|
||||
Selectors: map[cephv1.CephNetworkType]string{
|
||||
cephv1.CephNetworkPublic: "macvlan-public",
|
||||
cephv1.CephNetworkCluster: "macvlan-cluster",
|
||||
},
|
||||
AddressRanges: &cephv1.AddressRangesSpec{
|
||||
Cluster: []cephv1.CIDR{
|
||||
"192.168.50.0/24",
|
||||
},
|
||||
},
|
||||
})
|
||||
|
||||
clusterAddrsShouldNotBeDiscovered := true
|
||||
discoverCephAddressRangesFunc = mockDiscoverAddressRangesFunc([]string{"192.168.0.0/24"}, []string{}, false, clusterAddrsShouldNotBeDiscovered, nil)
|
||||
|
||||
monStore := new(mockMonStore)
|
||||
monStore.On("SetIfChanged", "global", "public_network", "192.168.0.0/24").Return(true, nil)
|
||||
monStore.On("SetIfChanged", "global", "cluster_network", "192.168.50.0/24").Return(true, nil)
|
||||
getMonStoreFunc = mockGetMonStoreFunc(monStore)
|
||||
|
||||
err := ApplyCephNetworkSettings(context.Background(), "rook/ceph:master", clusterdCtx, clusterSpec, clusterInfo)
|
||||
assert.NoError(t, err)
|
||||
})
|
||||
t.Run("public and cluster selected, only cluster public ranges specified", func(t *testing.T) {
|
||||
clusterdCtx, clusterSpec, clusterInfo := newTestConfigsWithNetworkSpec(
|
||||
cephv1.NetworkSpec{
|
||||
Provider: cephv1.NetworkProviderMultus,
|
||||
Selectors: map[cephv1.CephNetworkType]string{
|
||||
cephv1.CephNetworkPublic: "macvlan-public",
|
||||
cephv1.CephNetworkCluster: "macvlan-cluster",
|
||||
},
|
||||
AddressRanges: &cephv1.AddressRangesSpec{
|
||||
Public: []cephv1.CIDR{
|
||||
"192.168.0.0/24",
|
||||
},
|
||||
},
|
||||
})
|
||||
|
||||
publicAddrsShouldNotBeDiscovered := true
|
||||
discoverCephAddressRangesFunc = mockDiscoverAddressRangesFunc([]string{}, []string{"192.168.50.0/24"}, publicAddrsShouldNotBeDiscovered, false, nil)
|
||||
|
||||
monStore := new(mockMonStore)
|
||||
monStore.On("SetIfChanged", "global", "public_network", "192.168.0.0/24").Return(true, nil)
|
||||
monStore.On("SetIfChanged", "global", "cluster_network", "192.168.50.0/24").Return(true, nil)
|
||||
getMonStoreFunc = mockGetMonStoreFunc(monStore)
|
||||
|
||||
err := ApplyCephNetworkSettings(context.Background(), "rook/ceph:master", clusterdCtx, clusterSpec, clusterInfo)
|
||||
assert.NoError(t, err)
|
||||
})
|
||||
})
|
||||
t.Run("hostnet", func(t *testing.T) {
|
||||
t.Run("no ranges specified", func(t *testing.T) {
|
||||
clusterdCtx, clusterSpec, clusterInfo := newTestConfigsWithNetworkSpec(
|
||||
cephv1.NetworkSpec{
|
||||
Provider: cephv1.NetworkProviderHost,
|
||||
Selectors: map[cephv1.CephNetworkType]string{},
|
||||
AddressRanges: nil,
|
||||
})
|
||||
|
||||
discoverCephAddressRangesFunc = mockDiscoverAddressRangesFunc([]string{}, []string{}, false, false, errors.New("should not be called"))
|
||||
|
||||
monStore := new(mockMonStore)
|
||||
getMonStoreFunc = mockGetMonStoreFunc(monStore)
|
||||
|
||||
err := ApplyCephNetworkSettings(context.Background(), "rook/ceph:master", clusterdCtx, clusterSpec, clusterInfo)
|
||||
assert.NoError(t, err)
|
||||
})
|
||||
t.Run("public ranges specified", func(t *testing.T) {
|
||||
clusterdCtx, clusterSpec, clusterInfo := newTestConfigsWithNetworkSpec(
|
||||
cephv1.NetworkSpec{
|
||||
Provider: cephv1.NetworkProviderHost,
|
||||
Selectors: map[cephv1.CephNetworkType]string{},
|
||||
AddressRanges: &cephv1.AddressRangesSpec{
|
||||
Public: []cephv1.CIDR{
|
||||
"192.168.0.0/24",
|
||||
"feab::/112",
|
||||
},
|
||||
},
|
||||
})
|
||||
|
||||
discoverCephAddressRangesFunc = mockDiscoverAddressRangesFunc([]string{}, []string{}, false, false, errors.New("should not be called"))
|
||||
|
||||
monStore := new(mockMonStore)
|
||||
monStore.On("SetIfChanged", "global", "public_network", "192.168.0.0/24, feab::/112").Return(true, nil)
|
||||
getMonStoreFunc = mockGetMonStoreFunc(monStore)
|
||||
|
||||
err := ApplyCephNetworkSettings(context.Background(), "rook/ceph:master", clusterdCtx, clusterSpec, clusterInfo)
|
||||
assert.NoError(t, err)
|
||||
})
|
||||
t.Run("cluster ranges specified", func(t *testing.T) {
|
||||
clusterdCtx, clusterSpec, clusterInfo := newTestConfigsWithNetworkSpec(
|
||||
cephv1.NetworkSpec{
|
||||
Provider: cephv1.NetworkProviderHost,
|
||||
Selectors: map[cephv1.CephNetworkType]string{},
|
||||
AddressRanges: &cephv1.AddressRangesSpec{
|
||||
Cluster: []cephv1.CIDR{
|
||||
"192.168.0.0/24",
|
||||
"feab::/112",
|
||||
},
|
||||
},
|
||||
})
|
||||
|
||||
discoverCephAddressRangesFunc = mockDiscoverAddressRangesFunc([]string{}, []string{}, false, false, errors.New("should not be called"))
|
||||
|
||||
monStore := new(mockMonStore)
|
||||
monStore.On("SetIfChanged", "global", "cluster_network", "192.168.0.0/24, feab::/112").Return(true, nil)
|
||||
getMonStoreFunc = mockGetMonStoreFunc(monStore)
|
||||
|
||||
err := ApplyCephNetworkSettings(context.Background(), "rook/ceph:master", clusterdCtx, clusterSpec, clusterInfo)
|
||||
assert.NoError(t, err)
|
||||
})
|
||||
t.Run("public and cluster ranges specified", func(t *testing.T) {
|
||||
clusterdCtx, clusterSpec, clusterInfo := newTestConfigsWithNetworkSpec(
|
||||
cephv1.NetworkSpec{
|
||||
Provider: cephv1.NetworkProviderHost,
|
||||
Selectors: map[cephv1.CephNetworkType]string{},
|
||||
AddressRanges: &cephv1.AddressRangesSpec{
|
||||
Public: []cephv1.CIDR{
|
||||
"192.168.0.0/24",
|
||||
"feab::/112",
|
||||
},
|
||||
Cluster: []cephv1.CIDR{
|
||||
"192.168.100.0/24",
|
||||
"feab:100::/112",
|
||||
},
|
||||
},
|
||||
})
|
||||
|
||||
discoverCephAddressRangesFunc = mockDiscoverAddressRangesFunc([]string{}, []string{}, false, false, errors.New("should not be called"))
|
||||
|
||||
monStore := new(mockMonStore)
|
||||
monStore.On("SetIfChanged", "global", "public_network", "192.168.0.0/24, feab::/112").Return(true, nil)
|
||||
monStore.On("SetIfChanged", "global", "cluster_network", "192.168.100.0/24, feab:100::/112").Return(true, nil)
|
||||
getMonStoreFunc = mockGetMonStoreFunc(monStore)
|
||||
|
||||
err := ApplyCephNetworkSettings(context.Background(), "rook/ceph:master", clusterdCtx, clusterSpec, clusterInfo)
|
||||
assert.NoError(t, err)
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
// generate a net status for interface (expected to be "public" or "cluster") with some number of IPs
|
||||
func netStatus(iface string, ips ...string) string {
|
||||
ipsInJson := ""
|
||||
for _, ip := range ips {
|
||||
if ipsInJson != "" {
|
||||
ipsInJson += ","
|
||||
}
|
||||
ipsInJson = ipsInJson + "\n" + ` "` + ip + `"`
|
||||
}
|
||||
out := `[{
|
||||
"name": "bridge",
|
||||
"interface": "eth0",
|
||||
"ips": [
|
||||
"10.244.1.152"
|
||||
],
|
||||
"mac": "1a:15:5d:61:18:50",
|
||||
"default": true,
|
||||
"dns": {},
|
||||
"gateway": [
|
||||
"10.244.0.1"
|
||||
]
|
||||
},{
|
||||
"name": "dont/care",
|
||||
"interface": "` + iface + `",
|
||||
"ips": [` + ipsInJson + `
|
||||
],
|
||||
"mac": "8e:ec:4a:e3:1b:96",
|
||||
"dns": {}
|
||||
}]`
|
||||
return out
|
||||
}
|
||||
|
||||
// generate 'ip --json address show dev <iface>' output with some number of <ip>/<prefix>-es
|
||||
// also contains extra IPv6 SLAAC addrs that are often present in containers
|
||||
func ipAddrOutput(iface string, ipsWithPrefixLens ...string) string {
|
||||
ipsInJson := ""
|
||||
for _, ipWithPrefix := range ipsWithPrefixLens {
|
||||
s := strings.Split(ipWithPrefix, "/")
|
||||
ip := s[0]
|
||||
prefix := s[1]
|
||||
// specifically omit "family": "inet" from the output so it's certain that field isn't used
|
||||
// when parsing IPv4 vs IPv6 addrs
|
||||
ipsInJson = ipsInJson + `
|
||||
{
|
||||
"local": "` + ip + `",
|
||||
"prefixlen": ` + prefix + `,
|
||||
"scope": "global",
|
||||
"valid_life_time": 4294967295,
|
||||
"preferred_life_time": 4294967295
|
||||
},`
|
||||
}
|
||||
out := `[
|
||||
{
|
||||
"ifindex": 3,
|
||||
"link_index": 2,
|
||||
"ifname": "` + iface + `",
|
||||
"flags": [
|
||||
"BROADCAST",
|
||||
"MULTICAST",
|
||||
"UP",
|
||||
"LOWER_UP"
|
||||
],
|
||||
"mtu": 1500,
|
||||
"qdisc": "noqueue",
|
||||
"operstate": "UP",
|
||||
"group": "default",
|
||||
"link_type": "ether",
|
||||
"address": "8e:ec:4a:e3:1b:96",
|
||||
"broadcast": "ff:ff:ff:ff:ff:ff",
|
||||
"link_netnsid": 0,
|
||||
"addr_info": [
|
||||
{
|
||||
"family": "inet6",
|
||||
"local": "fd4e:7658:764f:15c4:8cec:4aff:fee3:1b96",
|
||||
"prefixlen": 64,
|
||||
"scope": "global",
|
||||
"dynamic": true,
|
||||
"mngtmpaddr": true,
|
||||
"valid_life_time": 2591998,
|
||||
"preferred_life_time": 604798
|
||||
},` + ipsInJson + `
|
||||
{
|
||||
"family": "inet6",
|
||||
"local": "fe80::8cec:4aff:fee3:1b96",
|
||||
"prefixlen": 64,
|
||||
"scope": "link",
|
||||
"valid_life_time": 4294967295,
|
||||
"preferred_life_time": 4294967295
|
||||
}
|
||||
]
|
||||
}
|
||||
]`
|
||||
return out
|
||||
}
|
||||
@@ -177,8 +177,8 @@ func TestCephCSIController(t *testing.T) {
|
||||
},
|
||||
Spec: cephv1.ClusterSpec{
|
||||
Network: cephv1.NetworkSpec{
|
||||
Provider: "multus",
|
||||
Selectors: map[string]string{"public": "public-net", "cluster": "cluster-net"},
|
||||
Provider: cephv1.NetworkProviderMultus,
|
||||
Selectors: map[cephv1.CephNetworkType]string{"public": "public-net", "cluster": "cluster-net"},
|
||||
},
|
||||
},
|
||||
Status: cephv1.ClusterStatus{
|
||||
@@ -239,6 +239,6 @@ func TestCephCSIController(t *testing.T) {
|
||||
assert.Equal(t, "csi-cephfsplugin-holder-rook-ceph", ds.Items[1].Name)
|
||||
assert.Equal(t, "csi-rbdplugin", ds.Items[2].Name)
|
||||
assert.Equal(t, "csi-rbdplugin-holder-rook-ceph", ds.Items[3].Name)
|
||||
assert.Equal(t, "public-net", ds.Items[1].Spec.Template.Annotations["k8s.v1.cni.cncf.io/networks"], ds.Items[1].Spec.Template.Annotations)
|
||||
assert.Equal(t, `[{"name":"public-net","namespace":"rook-ceph"}]`, ds.Items[1].Spec.Template.Annotations["k8s.v1.cni.cncf.io/networks"], ds.Items[1].Spec.Template.Annotations)
|
||||
})
|
||||
}
|
||||
@@ -716,7 +716,7 @@ func (r *ReconcileCSI) applyCephClusterNetworkConfig(ctx context.Context, object
|
||||
}
|
||||
for _, cephCluster := range cephClusters.Items {
|
||||
if cephCluster.Spec.Network.IsMultus() {
|
||||
err = k8sutil.ApplyMultus(cephCluster.Spec.Network, objectMeta)
|
||||
err = k8sutil.ApplyMultus(cephCluster.GetNamespace(), &cephCluster.Spec.Network, objectMeta)
|
||||
if err != nil {
|
||||
return errors.Wrapf(err, "failed to apply multus configuration to CephCluster %q", cephCluster.Name)
|
||||
}
|
||||
@@ -851,7 +851,7 @@ func (r *ReconcileCSI) configureHolder(driver driverDetails, c ClusterDetail, tp
|
||||
// If multus is enabled, add the multus plugin label
|
||||
if c.cluster.Spec.Network.IsMultus() {
|
||||
// Apply Multus annotations to daemonset spec
|
||||
err = k8sutil.ApplyMultus(c.cluster.Spec.Network, &cephPluginHolder.Spec.Template.ObjectMeta)
|
||||
err = k8sutil.ApplyMultus(c.cluster.GetNamespace(), &c.cluster.Spec.Network, &cephPluginHolder.Spec.Template.ObjectMeta)
|
||||
if err != nil {
|
||||
return errors.Wrapf(err, "failed to apply multus configuration for holder %q in cluster %q", cephPluginHolder.Name, c.cluster.Namespace)
|
||||
}
|
||||
|
||||
@@ -102,7 +102,7 @@ func (c *Cluster) makeDeployment(mdsConfig *mdsConfig, namespace string) (*apps.
|
||||
if c.clusterSpec.Network.IsHost() {
|
||||
d.Spec.Template.Spec.DNSPolicy = v1.DNSClusterFirstWithHostNet
|
||||
} else if c.clusterSpec.Network.IsMultus() {
|
||||
if err := k8sutil.ApplyMultus(c.clusterSpec.Network, &podSpec.ObjectMeta); err != nil {
|
||||
if err := k8sutil.ApplyMultus(c.clusterInfo.Namespace, &c.clusterSpec.Network, &podSpec.ObjectMeta); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
|
||||
@@ -64,7 +64,7 @@ func (r *ReconcileFilesystemMirror) makeDeployment(daemonConfig *daemonConfig, f
|
||||
if r.cephClusterSpec.Network.IsHost() {
|
||||
podSpec.Spec.DNSPolicy = v1.DNSClusterFirstWithHostNet
|
||||
} else if r.cephClusterSpec.Network.IsMultus() {
|
||||
if err := k8sutil.ApplyMultus(r.cephClusterSpec.Network, &podSpec.ObjectMeta); err != nil {
|
||||
if err := k8sutil.ApplyMultus(r.clusterInfo.Namespace, &r.cephClusterSpec.Network, &podSpec.ObjectMeta); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
|
||||
@@ -177,7 +177,7 @@ func (r *ReconcileCephNFS) makeDeployment(nfs *cephv1.CephNFS, cfg daemonConfig)
|
||||
if hostNetwork {
|
||||
podSpec.DNSPolicy = v1.DNSClusterFirstWithHostNet
|
||||
} else if r.cephClusterSpec.Network.IsMultus() {
|
||||
if err := k8sutil.ApplyMultus(r.cephClusterSpec.Network, &podTemplateSpec.ObjectMeta); err != nil {
|
||||
if err := k8sutil.ApplyMultus(r.clusterInfo.Namespace, &r.cephClusterSpec.Network, &podTemplateSpec.ObjectMeta); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
|
||||
@@ -248,7 +248,7 @@ func (c *clusterConfig) makeRGWPodSpec(rgwConfig *rgwConfig) (v1.PodTemplateSpec
|
||||
if hostNetwork {
|
||||
podTemplateSpec.Spec.DNSPolicy = v1.DNSClusterFirstWithHostNet
|
||||
} else if c.clusterSpec.Network.IsMultus() {
|
||||
if err := k8sutil.ApplyMultus(c.clusterSpec.Network, &podTemplateSpec.ObjectMeta); err != nil {
|
||||
if err := k8sutil.ApplyMultus(c.clusterInfo.Namespace, &c.clusterSpec.Network, &podTemplateSpec.ObjectMeta); err != nil {
|
||||
return podTemplateSpec, err
|
||||
}
|
||||
}
|
||||
|
||||
@@ -26,7 +26,6 @@ import (
|
||||
"github.com/coreos/pkg/capnslog"
|
||||
"github.com/pkg/errors"
|
||||
"github.com/rook/rook/pkg/daemon/util"
|
||||
|
||||
"github.com/rook/rook/pkg/operator/k8sutil"
|
||||
batch "k8s.io/api/batch/v1"
|
||||
v1 "k8s.io/api/core/v1"
|
||||
@@ -54,6 +53,11 @@ var (
|
||||
logger = capnslog.NewPackageLogger("github.com/rook/rook", "CmdReporter")
|
||||
)
|
||||
|
||||
type CmdReporterInterface interface {
|
||||
Job() *batch.Job
|
||||
Run(ctx context.Context, timeout time.Duration) (stdout, stderr string, retcode int, retErr error)
|
||||
}
|
||||
|
||||
// CmdReporter is a wrapper for Rook's cmd-reporter commandline utility allowing operators to use
|
||||
// the utility without fully specifying the job, pod, and container templates manually.
|
||||
type CmdReporter struct {
|
||||
@@ -96,7 +100,7 @@ func New(
|
||||
cmd, args []string,
|
||||
rookImage, runImage string,
|
||||
imagePullPolicy v1.PullPolicy,
|
||||
) (*CmdReporter, error) {
|
||||
) (CmdReporterInterface, error) {
|
||||
cfg := &cmdReporterCfg{
|
||||
clientset: clientset,
|
||||
ownerInfo: ownerInfo,
|
||||
@@ -397,3 +401,32 @@ func copyBinariesVolAndMount() (v1.Volume, v1.VolumeMount) {
|
||||
}
|
||||
|
||||
func newInt32(i int32) *int32 { return &i }
|
||||
|
||||
// MockCmdReporterJob creates a job using the package's internal creation mechanism without
|
||||
// validating any inputs. Use only for unit testing.
|
||||
func MockCmdReporterJob(
|
||||
clientset kubernetes.Interface,
|
||||
ownerInfo *k8sutil.OwnerInfo,
|
||||
appName string,
|
||||
jobName string,
|
||||
jobNamespace string,
|
||||
cmd []string,
|
||||
args []string,
|
||||
rookImage string,
|
||||
runImage string,
|
||||
imagePullPolicy v1.PullPolicy,
|
||||
) (*batch.Job, error) {
|
||||
cfg := &cmdReporterCfg{
|
||||
clientset: clientset,
|
||||
ownerInfo: ownerInfo,
|
||||
appName: appName,
|
||||
jobName: jobName,
|
||||
jobNamespace: jobNamespace,
|
||||
cmd: cmd,
|
||||
args: args,
|
||||
rookImage: rookImage,
|
||||
runImage: runImage,
|
||||
imagePullPolicy: imagePullPolicy,
|
||||
}
|
||||
return cfg.initJobSpec()
|
||||
}
|
||||
@@ -18,124 +18,95 @@ package k8sutil
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
netapi "github.com/k8snetworkplumbingwg/network-attachment-definition-client/pkg/apis/k8s.cni.cncf.io/v1"
|
||||
nadv1 "github.com/k8snetworkplumbingwg/network-attachment-definition-client/pkg/apis/k8s.cni.cncf.io/v1"
|
||||
"github.com/pkg/errors"
|
||||
cephv1 "github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
|
||||
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
|
||||
)
|
||||
|
||||
const (
|
||||
// publicNetworkSelectorKeyName is the network selector key for the ceph public network
|
||||
publicNetworkSelectorKeyName = "public"
|
||||
// clusterNetworkSelectorKeyName is the network selector key for the ceph cluster network
|
||||
clusterNetworkSelectorKeyName = "cluster"
|
||||
)
|
||||
|
||||
// NetworkAttachmentConfig represents the configuration of the NetworkAttachmentDefinitions object
|
||||
type NetworkAttachmentConfig struct {
|
||||
CniVersion string `json:"cniVersion,omitempty"`
|
||||
Type string `json:"type,omitempty"`
|
||||
Master string `json:"master,omitempty"`
|
||||
Mode string `json:"mode,omitempty"`
|
||||
Ipam struct {
|
||||
Type string `json:"type,omitempty"`
|
||||
Subnet string `json:"subnet,omitempty"`
|
||||
Addresses []struct {
|
||||
Address string `json:"address,omitempty"`
|
||||
Gateway string `json:"gateway,omitempty"`
|
||||
} `json:"addresses,omitempty"`
|
||||
Ranges [][]struct {
|
||||
Subnet string `json:"subnet,omitempty"`
|
||||
RangeStart string `json:"rangeStart,omitempty"`
|
||||
RangeEnd string `json:"rangeEnd,omitempty"`
|
||||
Gateway string `json:"gateway,omitempty"`
|
||||
} `json:"ranges,omitempty"`
|
||||
Range string `json:"range,omitempty"`
|
||||
RangeStart string `json:"rangeStart,omitempty"`
|
||||
RangeEnd string `json:"rangeEnd,omitempty"`
|
||||
Routes []struct {
|
||||
Dst string `json:"dst,omitempty"`
|
||||
} `json:"routes,omitempty"`
|
||||
Gateway string `json:"gateway,omitempty"`
|
||||
} `json:"ipam,omitempty"`
|
||||
}
|
||||
|
||||
// ApplyMultus apply multus selector to Pods
|
||||
// Multus supports short and json syntax, use only one kind at a time.
|
||||
func ApplyMultus(net cephv1.NetworkSpec, objectMeta *metav1.ObjectMeta) error {
|
||||
v := make([]string, 0, 2)
|
||||
shortSyntax := false
|
||||
jsonSyntax := false
|
||||
|
||||
for k, ns := range net.Selectors {
|
||||
var multusMap map[string]string
|
||||
err := json.Unmarshal([]byte(ns), &multusMap)
|
||||
|
||||
if err == nil {
|
||||
jsonSyntax = true
|
||||
} else {
|
||||
shortSyntax = true
|
||||
}
|
||||
|
||||
app, ok := objectMeta.Labels["app"]
|
||||
if !ok {
|
||||
app = "" // unknown app
|
||||
}
|
||||
isClusterNetApp := false
|
||||
for _, clusterNetworkApp := range getClusterNetworkApps() {
|
||||
if app == clusterNetworkApp {
|
||||
isClusterNetApp = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if isClusterNetApp {
|
||||
// append all networks to apps that are cluster network apps
|
||||
v = append(v, string(ns))
|
||||
} else {
|
||||
// only append public networks to apps that are not cluster network apps
|
||||
if k == publicNetworkSelectorKeyName {
|
||||
v = append(v, string(ns))
|
||||
}
|
||||
}
|
||||
func ApplyMultus(clusterNamespace string, netSpec *cephv1.NetworkSpec, objectMeta *metav1.ObjectMeta) error {
|
||||
app, ok := objectMeta.Labels["app"]
|
||||
if !ok {
|
||||
app = "" // unknown app
|
||||
}
|
||||
|
||||
if shortSyntax && jsonSyntax {
|
||||
return fmt.Errorf("ApplyMultus: Can't mix short and JSON form")
|
||||
netSelections := []*nadv1.NetworkSelectionElement{}
|
||||
|
||||
// all apps get the public network
|
||||
pub, err := netSpec.GetNetworkSelection(clusterNamespace, cephv1.CephNetworkPublic)
|
||||
if err != nil {
|
||||
return errors.Wrapf(err, "failed to get %q network selection for app %q", cephv1.CephNetworkPublic, app)
|
||||
}
|
||||
netSelections = append(netSelections, pub)
|
||||
|
||||
if isClusterNetApp(app) {
|
||||
cluster, err := netSpec.GetNetworkSelection(clusterNamespace, cephv1.CephNetworkCluster)
|
||||
if err != nil {
|
||||
return errors.Wrapf(err, "failed to get %q network selection for app %q", cephv1.CephNetworkCluster, app)
|
||||
}
|
||||
netSelections = append(netSelections, cluster)
|
||||
}
|
||||
|
||||
// Sort network strings so that pods/deployments won't need updated in a loop if nothing changes
|
||||
sort.Strings(v)
|
||||
|
||||
networks := strings.Join(v, ", ")
|
||||
if jsonSyntax {
|
||||
networks = "[" + networks + "]"
|
||||
annotationValue, err := cephv1.NetworkSelectionsToAnnotationValue(netSelections...)
|
||||
if err != nil {
|
||||
return errors.Wrapf(err, "failed to generate annotation value to apply nets %v to app %q", netSelections, app)
|
||||
}
|
||||
|
||||
t := cephv1.Annotations{
|
||||
"k8s.v1.cni.cncf.io/networks": networks,
|
||||
"k8s.v1.cni.cncf.io/networks": annotationValue,
|
||||
}
|
||||
t.ApplyToObjectMeta(objectMeta)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// getClusterNetworkApps returns the list of ceph apps that utilize cluster network
|
||||
func getClusterNetworkApps() []string {
|
||||
return []string{"rook-ceph-osd"}
|
||||
func isClusterNetApp(app string) bool {
|
||||
return app == "rook-ceph-osd"
|
||||
}
|
||||
|
||||
// GetNetworkAttachmentConfig returns the NetworkAttachmentDefinitions configuration
|
||||
func GetNetworkAttachmentConfig(n netapi.NetworkAttachmentDefinition) (NetworkAttachmentConfig, error) {
|
||||
netConfigJSON := n.Spec.Config
|
||||
var netConfig NetworkAttachmentConfig
|
||||
|
||||
err := json.Unmarshal([]byte(netConfigJSON), &netConfig)
|
||||
if err != nil {
|
||||
return netConfig, fmt.Errorf("failed to unmarshal netconfig json %q. %v", netConfigJSON, err)
|
||||
// ParseNetworkStatusAnnotation takes the annotation value from k8s.v1.cni.cncf.io/network-status
|
||||
// and returns the network status struct.
|
||||
func ParseNetworkStatusAnnotation(annotationValue string) ([]nadv1.NetworkStatus, error) {
|
||||
netStatuses := []nadv1.NetworkStatus{}
|
||||
if err := json.Unmarshal([]byte(annotationValue), &netStatuses); err != nil {
|
||||
return nil, errors.Wrapf(err, "failed to parse network status annotation %q", annotationValue)
|
||||
}
|
||||
return netStatuses, nil
|
||||
}
|
||||
|
||||
return netConfig, nil
|
||||
func FindNetworkStatusByInterface(statuses []nadv1.NetworkStatus, ifaceName string) (nadv1.NetworkStatus, bool) {
|
||||
for _, s := range statuses {
|
||||
if s.Interface == ifaceName {
|
||||
return s, true
|
||||
}
|
||||
}
|
||||
return nadv1.NetworkStatus{}, false
|
||||
}
|
||||
|
||||
// LinuxIpAddrResult provides a pared down Go struct for codifying json-formatted output from
|
||||
// `ip --json address show`. Each result contains addresses associated with a single interface.
|
||||
type LinuxIpAddrResult struct {
|
||||
InterfaceName string `json:"ifname"`
|
||||
AddrInfo []LinuxIpAddrInfo `json:"addr_info"`
|
||||
}
|
||||
|
||||
type LinuxIpAddrInfo struct {
|
||||
Local string `json:"local"`
|
||||
PrefixLen int `json:"prefixlen"`
|
||||
}
|
||||
|
||||
// ParseLinuxIpAddrOutput parses raw json-encoded `ip --json address show` output
|
||||
func ParseLinuxIpAddrOutput(rawOutput string) ([]LinuxIpAddrResult, error) {
|
||||
results := []LinuxIpAddrResult{}
|
||||
if strings.TrimSpace(rawOutput) == "" {
|
||||
return results, errors.Errorf("cannot parse empty 'ip --json address' output")
|
||||
}
|
||||
if err := json.Unmarshal([]byte(rawOutput), &results); err != nil {
|
||||
return []LinuxIpAddrResult{}, errors.Wrap(err, "failed to parse 'ip --json address' output")
|
||||
}
|
||||
return results, nil
|
||||
}
|
||||
@@ -17,184 +17,292 @@ limitations under the License.
|
||||
package k8sutil
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
netapi "github.com/k8snetworkplumbingwg/network-attachment-definition-client/pkg/apis/k8s.cni.cncf.io/v1"
|
||||
cephv1 "github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
|
||||
"github.com/stretchr/testify/assert"
|
||||
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
|
||||
)
|
||||
|
||||
func TestApplyMultus(t *testing.T) {
|
||||
t.Run("short format", func(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
netSelectors map[string]string
|
||||
labels map[string]string
|
||||
want string
|
||||
}{
|
||||
{
|
||||
name: "no applicable networks for non-osd pod",
|
||||
netSelectors: map[string]string{
|
||||
"unknown": "macvlan@net1",
|
||||
},
|
||||
want: "",
|
||||
json1 := `[{"name": "macvlan", "interface": "net1"}]`
|
||||
json2 := `[{"name": "macvlan", "interface": "net2"}]`
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
netSelectors map[cephv1.CephNetworkType]string
|
||||
labels map[string]string
|
||||
want string
|
||||
}{
|
||||
{
|
||||
name: "non-osd pods don't get unknown nets",
|
||||
netSelectors: map[cephv1.CephNetworkType]string{
|
||||
"aruba": "macvlan@net1",
|
||||
"bahama": "macvlan@net2",
|
||||
},
|
||||
{
|
||||
name: "for a non-osd pod",
|
||||
netSelectors: map[string]string{
|
||||
publicNetworkSelectorKeyName: "macvlan@net1",
|
||||
clusterNetworkSelectorKeyName: "macvlan@net2",
|
||||
},
|
||||
want: "macvlan@net1",
|
||||
want: ``,
|
||||
},
|
||||
{
|
||||
name: "osd pods don't get unknown nets",
|
||||
netSelectors: map[cephv1.CephNetworkType]string{
|
||||
"key-largo": "macvlan@net1",
|
||||
"montego": "macvlan@net2",
|
||||
},
|
||||
{
|
||||
name: "for an osd pod",
|
||||
netSelectors: map[string]string{
|
||||
publicNetworkSelectorKeyName: "macvlan@net1",
|
||||
clusterNetworkSelectorKeyName: "macvlan@net2",
|
||||
},
|
||||
labels: map[string]string{"app": "rook-ceph-osd"},
|
||||
want: "macvlan@net1, macvlan@net2",
|
||||
labels: map[string]string{"app": "rook-ceph-osd"},
|
||||
want: ``,
|
||||
},
|
||||
|
||||
{
|
||||
name: "non-osd pods get only public networks",
|
||||
netSelectors: map[cephv1.CephNetworkType]string{
|
||||
"public": "macvlan@net1",
|
||||
"cluster": "macvlan@net2",
|
||||
},
|
||||
{
|
||||
name: "for an osd pod (reverse ordering)",
|
||||
netSelectors: map[string]string{
|
||||
publicNetworkSelectorKeyName: "macvlan@net2",
|
||||
clusterNetworkSelectorKeyName: "macvlan@net1",
|
||||
},
|
||||
labels: map[string]string{"app": "rook-ceph-osd"},
|
||||
want: "macvlan@net1, macvlan@net2", // should not change the order of output
|
||||
want: `[{"name":"macvlan","namespace":"ns","interface":"net1"}]`,
|
||||
},
|
||||
{
|
||||
name: "osd pods get both networks",
|
||||
netSelectors: map[cephv1.CephNetworkType]string{
|
||||
"public": "macvlan@net1",
|
||||
"cluster": "macvlan@net2",
|
||||
},
|
||||
}
|
||||
for _, test := range tests {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
net := cephv1.NetworkSpec{
|
||||
Provider: "multus",
|
||||
Selectors: test.netSelectors,
|
||||
}
|
||||
objMeta := metav1.ObjectMeta{}
|
||||
objMeta.Labels = test.labels
|
||||
err := ApplyMultus(net, &objMeta)
|
||||
assert.NoError(t, err)
|
||||
assert.Equal(t, test.want, objMeta.Annotations["k8s.v1.cni.cncf.io/networks"])
|
||||
})
|
||||
}
|
||||
})
|
||||
labels: map[string]string{"app": "rook-ceph-osd"},
|
||||
want: `[{"name":"macvlan","namespace":"ns","interface":"net1"},{"name":"macvlan","namespace":"ns","interface":"net2"}]`,
|
||||
},
|
||||
|
||||
t.Run("JSON format", func(t *testing.T) {
|
||||
json1 := `{"name": "macvlan", "interface": "net1"}`
|
||||
json2 := `{"name": "macvlan", "interface": "net2"}`
|
||||
|
||||
t.Run("no applicable networks for non-osd pod", func(t *testing.T) {
|
||||
net := cephv1.NetworkSpec{
|
||||
Provider: "multus",
|
||||
Selectors: map[string]string{
|
||||
"server": json1,
|
||||
"broker": json2,
|
||||
},
|
||||
}
|
||||
objMeta := metav1.ObjectMeta{}
|
||||
err := ApplyMultus(net, &objMeta)
|
||||
assert.NoError(t, err)
|
||||
// non-osd pods should not get any network annotations here
|
||||
assert.Equal(t, "[]", objMeta.Annotations["k8s.v1.cni.cncf.io/networks"])
|
||||
})
|
||||
|
||||
t.Run("for a non-osd pod", func(t *testing.T) {
|
||||
net := cephv1.NetworkSpec{
|
||||
Provider: "multus",
|
||||
Selectors: map[string]string{
|
||||
"public": json1,
|
||||
"cluster": json2,
|
||||
},
|
||||
}
|
||||
objMeta := metav1.ObjectMeta{}
|
||||
err := ApplyMultus(net, &objMeta)
|
||||
assert.NoError(t, err)
|
||||
// non-osd pods should only get public networks
|
||||
assert.Equal(t, "["+json1+"]", objMeta.Annotations["k8s.v1.cni.cncf.io/networks"])
|
||||
})
|
||||
|
||||
t.Run("for an osd pod", func(t *testing.T) {
|
||||
net := cephv1.NetworkSpec{
|
||||
Provider: "multus",
|
||||
Selectors: map[string]string{
|
||||
"server": json1,
|
||||
"broker": json2,
|
||||
},
|
||||
}
|
||||
objMeta := metav1.ObjectMeta{
|
||||
Labels: map[string]string{
|
||||
"app": "rook-ceph-osd",
|
||||
},
|
||||
}
|
||||
err := ApplyMultus(net, &objMeta)
|
||||
assert.NoError(t, err)
|
||||
assert.Equal(t, "["+json1+", "+json2+"]", objMeta.Annotations["k8s.v1.cni.cncf.io/networks"])
|
||||
})
|
||||
|
||||
t.Run("for an osd pod (reverse ordering)", func(t *testing.T) {
|
||||
net := cephv1.NetworkSpec{
|
||||
Provider: "multus",
|
||||
Selectors: map[string]string{
|
||||
"server": json2,
|
||||
"broker": json1,
|
||||
},
|
||||
}
|
||||
objMeta := metav1.ObjectMeta{
|
||||
Labels: map[string]string{
|
||||
"app": "rook-ceph-osd",
|
||||
},
|
||||
}
|
||||
err := ApplyMultus(net, &objMeta)
|
||||
assert.NoError(t, err)
|
||||
// should not change the order of output
|
||||
assert.Equal(t, "["+json1+", "+json2+"]", objMeta.Annotations["k8s.v1.cni.cncf.io/networks"])
|
||||
})
|
||||
})
|
||||
|
||||
t.Run("mixed format (error)", func(t *testing.T) {
|
||||
net := cephv1.NetworkSpec{
|
||||
Provider: "multus",
|
||||
Selectors: map[string]string{
|
||||
"server": `{"name": "macvlan", "interface": "net1"}`,
|
||||
"broker": `macvlan@net2`,
|
||||
{
|
||||
name: "osd pod network ordering is not reversed",
|
||||
netSelectors: map[cephv1.CephNetworkType]string{
|
||||
"cluster": "macvlan@net2",
|
||||
"public": "macvlan@net1",
|
||||
},
|
||||
}
|
||||
labels: map[string]string{"app": "rook-ceph-osd"},
|
||||
want: `[{"name":"macvlan","namespace":"ns","interface":"net1"},{"name":"macvlan","namespace":"ns","interface":"net2"}]`, // should not change the order of output
|
||||
},
|
||||
|
||||
objMeta := metav1.ObjectMeta{}
|
||||
err := ApplyMultus(net, &objMeta)
|
||||
{
|
||||
name: "non-osd pods take json input",
|
||||
netSelectors: map[cephv1.CephNetworkType]string{
|
||||
"public": json1,
|
||||
"cluster": json2,
|
||||
},
|
||||
want: `[{"name":"macvlan","namespace":"ns","interface":"net1"}]`,
|
||||
},
|
||||
{
|
||||
name: "osd pods take json input",
|
||||
netSelectors: map[cephv1.CephNetworkType]string{
|
||||
"public": json1,
|
||||
"cluster": json2,
|
||||
},
|
||||
labels: map[string]string{"app": "rook-ceph-osd"},
|
||||
want: `[{"name":"macvlan","namespace":"ns","interface":"net1"},{"name":"macvlan","namespace":"ns","interface":"net2"}]`,
|
||||
},
|
||||
{
|
||||
name: "osd pod network ordering is not reversed with json input",
|
||||
netSelectors: map[cephv1.CephNetworkType]string{
|
||||
"cluster": json2,
|
||||
"public": json1,
|
||||
},
|
||||
labels: map[string]string{"app": "rook-ceph-osd"},
|
||||
want: `[{"name":"macvlan","namespace":"ns","interface":"net1"},{"name":"macvlan","namespace":"ns","interface":"net2"}]`,
|
||||
},
|
||||
|
||||
assert.Error(t, err)
|
||||
})
|
||||
}
|
||||
|
||||
func TestGetNetworkAttachmentConfig(t *testing.T) {
|
||||
dummyNetAttachDef := netapi.NetworkAttachmentDefinition{
|
||||
Spec: netapi.NetworkAttachmentDefinitionSpec{
|
||||
Config: `{
|
||||
"cniVersion": "0.3.0",
|
||||
"type": "macvlan",
|
||||
"master": "eth2",
|
||||
"mode": "bridge",
|
||||
"ipam": {
|
||||
"type": "host-local",
|
||||
"subnet": "172.18.8.0/24",
|
||||
"rangeStart": "172.18.8.200",
|
||||
"rangeEnd": "172.18.8.216",
|
||||
"routes": [
|
||||
{
|
||||
"dst": "0.0.0.0/0"
|
||||
}
|
||||
],
|
||||
"gateway": "172.18.8.1"
|
||||
}
|
||||
}`,
|
||||
{
|
||||
name: "mixed json-non-json format is allowed",
|
||||
netSelectors: map[cephv1.CephNetworkType]string{
|
||||
"public": "macvlan@net1",
|
||||
"cluster": json2,
|
||||
},
|
||||
labels: map[string]string{"app": "rook-ceph-osd"},
|
||||
want: `[{"name":"macvlan","namespace":"ns","interface":"net1"},{"name":"macvlan","namespace":"ns","interface":"net2"}]`,
|
||||
},
|
||||
}
|
||||
for _, test := range tests {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
netSpec := cephv1.NetworkSpec{
|
||||
Provider: "multus",
|
||||
Selectors: test.netSelectors,
|
||||
}
|
||||
objMeta := metav1.ObjectMeta{}
|
||||
objMeta.Labels = test.labels
|
||||
err := ApplyMultus("ns", &netSpec, &objMeta)
|
||||
assert.NoError(t, err)
|
||||
assert.Equal(t, test.want, objMeta.Annotations["k8s.v1.cni.cncf.io/networks"])
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
config, err := GetNetworkAttachmentConfig(dummyNetAttachDef)
|
||||
assert.NoError(t, err)
|
||||
assert.Equal(t, "172.18.8.0/24", config.Ipam.Subnet)
|
||||
func TestParseLinuxIpAddrOutput(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
ipAddrRawOutput string
|
||||
want []LinuxIpAddrResult
|
||||
wantErr bool
|
||||
}{
|
||||
{
|
||||
name: "empty string",
|
||||
ipAddrRawOutput: "",
|
||||
want: []LinuxIpAddrResult{},
|
||||
wantErr: true,
|
||||
}, {
|
||||
name: "full output",
|
||||
ipAddrRawOutput: ipAddrOutputMixedIPv4v6,
|
||||
want: []LinuxIpAddrResult{
|
||||
{
|
||||
InterfaceName: "public",
|
||||
AddrInfo: []LinuxIpAddrInfo{
|
||||
{
|
||||
Local: "fd4e:7658:764f:15c4:8cec:4aff:fee3:1b96",
|
||||
PrefixLen: 64,
|
||||
}, {
|
||||
Local: "2000::6",
|
||||
PrefixLen: 112,
|
||||
}, {
|
||||
Local: "fe80::8cec:4aff:fee3:1b96",
|
||||
PrefixLen: 64,
|
||||
},
|
||||
},
|
||||
}, {
|
||||
InterfaceName: "net1",
|
||||
AddrInfo: []LinuxIpAddrInfo{
|
||||
{
|
||||
Local: "192.168.20.8",
|
||||
PrefixLen: 24,
|
||||
},
|
||||
{
|
||||
Local: "fd4e:7658:764f:15c4:9878:30ff:fec3:e504",
|
||||
PrefixLen: 64,
|
||||
},
|
||||
{
|
||||
Local: "fe80::9878:30ff:fec3:e504",
|
||||
PrefixLen: 64,
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
wantErr: false,
|
||||
},
|
||||
{
|
||||
name: "full output syntax error",
|
||||
ipAddrRawOutput: removeLastChar(ipAddrOutputMixedIPv4v6),
|
||||
want: []LinuxIpAddrResult{},
|
||||
wantErr: true,
|
||||
},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
got, err := ParseLinuxIpAddrOutput(tt.ipAddrRawOutput)
|
||||
if (err != nil) != tt.wantErr {
|
||||
t.Errorf("RuntimeLinuxIpAddrParser.Parse() error = %v, wantErr %v", err, tt.wantErr)
|
||||
return
|
||||
}
|
||||
assert.Equal(t, tt.want, got)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
var ipAddrOutputMixedIPv4v6 = `[
|
||||
{
|
||||
"ifindex": 3,
|
||||
"link_index": 2,
|
||||
"ifname": "public",
|
||||
"flags": [
|
||||
"BROADCAST",
|
||||
"MULTICAST",
|
||||
"UP",
|
||||
"LOWER_UP"
|
||||
],
|
||||
"mtu": 1500,
|
||||
"qdisc": "noqueue",
|
||||
"operstate": "UP",
|
||||
"group": "default",
|
||||
"link_type": "ether",
|
||||
"address": "8e:ec:4a:e3:1b:96",
|
||||
"broadcast": "ff:ff:ff:ff:ff:ff",
|
||||
"link_netnsid": 0,
|
||||
"addr_info": [
|
||||
{
|
||||
"family": "inet6",
|
||||
"local": "fd4e:7658:764f:15c4:8cec:4aff:fee3:1b96",
|
||||
"prefixlen": 64,
|
||||
"scope": "global",
|
||||
"dynamic": true,
|
||||
"mngtmpaddr": true,
|
||||
"valid_life_time": 2591998,
|
||||
"preferred_life_time": 604798
|
||||
},
|
||||
{
|
||||
"family": "inet6",
|
||||
"local": "2000::6",
|
||||
"prefixlen": 112,
|
||||
"scope": "global",
|
||||
"valid_life_time": 4294967295,
|
||||
"preferred_life_time": 4294967295
|
||||
},
|
||||
{
|
||||
"family": "inet6",
|
||||
"local": "fe80::8cec:4aff:fee3:1b96",
|
||||
"prefixlen": 64,
|
||||
"scope": "link",
|
||||
"valid_life_time": 4294967295,
|
||||
"preferred_life_time": 4294967295
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"ifindex": 3,
|
||||
"link_index": 2,
|
||||
"ifname": "net1",
|
||||
"flags": [
|
||||
"BROADCAST",
|
||||
"MULTICAST",
|
||||
"UP",
|
||||
"LOWER_UP"
|
||||
],
|
||||
"mtu": 1500,
|
||||
"qdisc": "noqueue",
|
||||
"operstate": "UP",
|
||||
"group": "default",
|
||||
"link_type": "ether",
|
||||
"address": "9a:78:30:c3:e5:04",
|
||||
"broadcast": "ff:ff:ff:ff:ff:ff",
|
||||
"link_netnsid": 0,
|
||||
"addr_info": [
|
||||
{
|
||||
"family": "inet",
|
||||
"local": "192.168.20.8",
|
||||
"prefixlen": 24,
|
||||
"broadcast": "192.168.20.255",
|
||||
"scope": "global",
|
||||
"label": "net1",
|
||||
"valid_life_time": 4294967295,
|
||||
"preferred_life_time": 4294967295
|
||||
},
|
||||
{
|
||||
"family": "inet6",
|
||||
"local": "fd4e:7658:764f:15c4:9878:30ff:fec3:e504",
|
||||
"prefixlen": 64,
|
||||
"scope": "global",
|
||||
"dynamic": true,
|
||||
"mngtmpaddr": true,
|
||||
"valid_life_time": 2591910,
|
||||
"preferred_life_time": 604710
|
||||
},
|
||||
{
|
||||
"family": "inet6",
|
||||
"local": "fe80::9878:30ff:fec3:e504",
|
||||
"prefixlen": 64,
|
||||
"scope": "link",
|
||||
"valid_life_time": 4294967295,
|
||||
"preferred_life_time": 4294967295
|
||||
}
|
||||
]
|
||||
}
|
||||
]`
|
||||
|
||||
func removeLastChar(in string) string {
|
||||
chars := []byte(strings.TrimSpace(in))
|
||||
len := len(chars)
|
||||
return string(chars[0 : len-1])
|
||||
}
|
||||
Reference in new issue
Block a user