csi: remove automated node fencing code

since node fencing is disabled in rook for sometime
and this feature is implemented in csi. So, this
commits remove obsolete code related to node loss

Signed-off-by: subhamkrai <srai@redhat.com>
(cherry picked from commit 20f350a71a)

# Conflicts:
#	pkg/operator/ceph/cluster/watcher_test.go
This commit is contained in:
subhamkrai authored and Mergify committed 2026-02-05 14:47:26 +00:00
1 parent 6ff6716ab5
commit 617a347d97
7 files changed
+5 -765

No files matched your search

-2
View File
@@ -567,8 +567,6 @@ data:
ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS: "15"
# Enable the csi addons sidecar.
CSI_ENABLE_CSIADDONS: "false"
# Enable watch for faster recovery from rbd rwo node loss
ROOK_WATCH_FOR_NODE_FAILURE: "true"
# ROOK_CSIADDONS_IMAGE: "quay.io/csiaddons/k8s-sidecar:v0.14.0"
# The GCSI RPC timeout value (in seconds). It should be >= 120. If this variable is not set or is an invalid value, it's default to 150.
CSI_GRPC_TIMEOUT_SECONDS: "150"
-2
View File
@@ -509,8 +509,6 @@ data:
ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS: "15"
# Enable the csi addons sidecar.
CSI_ENABLE_CSIADDONS: "false"
# Enable watch for faster recovery from rbd rwo node loss
ROOK_WATCH_FOR_NODE_FAILURE: "true"
# ROOK_CSIADDONS_IMAGE: "quay.io/csiaddons/k8s-sidecar:v0.14.0"
# The CSI GRPC timeout value (in seconds). It should be >= 120. If this variable is not set or is an invalid value, it's default to 150.
CSI_GRPC_TIMEOUT_SECONDS: "150"
-150
View File
@@ -1,150 +0,0 @@
# Recover RBD/CephFS RWO PVC in case of Node Loss
## Goal
Faster RBD/CephFS RWO recovery in case of node loss.
## Problem
For RBD RWO recovery:
When a node is lost where a pod is running with the RBD RWO volume is mounted, the volume cannot automatically be mounted on another node. If two clients are write to the same volume it could cause corruption. The node must be guaranteed to be down before the volume can be mounted on another node.
For CephFS recovery:
With the current design the node recovery will be faster for CephFS.
## Current Solution
For RBD RWO recovery:
We have a manual solution to the problem which involves forceful deletion of a pod so that forced detachment and attachment work is possible. The problem with the current solution is that even after the forced pod deletion it takes around 11 minutes for the volume to mount on the new node. Also there are still chances of data corruption if the old pod on the lost node comes back online, causing multiple writers and lead to data corruption if the [documentation](https://rook.github.io/docs/rook/v1.11/Troubleshooting/ceph-csi-common-issues/#node-loss) is not followed to manually block nodes.
For CephFS recovery:
Currently, CephFS recovery is slower in case of node loss.
## Proposed Solution
> Note: This solution requires minimum kubernetes version 1.26.0
The kubernetes feature [Non-Graceful Node Shutdown](https://kubernetes.io/blog/2022/12/16/kubernetes-1-26-non-graceful-node-shutdown-beta/) is available starting in Kubernetes 1.26 to help improve the volume recovery during node loss. When a node is lost, the admin is required to add the taint `out-of-service` manually to the node. After the node is tainted, Kubernetes will:
- Remove the volume attachment from the lost node
- Delete the old pod on the lost node
- Create a new pod on the new node
- Allow the volume to be attached to the new node
Once this taint is applied manually, Rook will create a [NetworkFence CR](https://github.com/csi-addons/kubernetes-csi-addons/blob/main/docs/networkfence.md). The [csi-addons operator](https://github.com/csi-addons/kubernetes-csi-addons) will then blocklist the node to prevent any ceph rbd/CephFS client on the lost node from writing any more data.
After the new pod is running on the new node and the old node which was lost comes back, Rook will delete the [NetworkFence CR](https://github.com/csi-addons/kubernetes-csi-addons/blob/main/docs/networkfence.md).
example of taint to be applied to lost node:
```console
kubectl taint nodes <node-name> node.kubernetes.io/out-of-service=nodeshutdown:NoExecute
# or
kubectl taint nodes <node-name> node.kubernetes.io/out-of-service=nodeshutdown:NoSchedule
```
> Note: This will be enabled by default in Rook if the NetworkFence CR is found, in the case for some reason user wants to disable this feature in Rook can edit the `rook-ceph-operator-config` configmap and update the `ROOK_WATCH_FOR_NODE_FAILURE: "false"`.
## How to get which IP to blocklist
There are multiple networking options available for example, Host Networking, Pod networking, Multus etc. This make it difficult to know which NodeIP address to blocklist.
For this we'll follow the following approach which will work for all networking options, except when connected to an external Ceph cluster.
1. Get the `volumesInUse` from the node which has the taint `out-of-service`.
2. List all the pv and compare the pv `spec.volumeHandle` with the node `volumesInUse` field `volumeHandle`
1. Example:
Below is sample Node volumeInUse field
```
volumesInUse:
- kubernetes.io/csi/rook-ceph.rbd.csi.ceph.com^0001-0009-Rook-ceph-0000000000000002-24862838-240d-4215-9183-abfc0e9e4002
# Note: The volumeInUse naming convention are `kubernetes.io/csi/ + CSI driver name + ^ + volumeHandle`
```
and the following is pv `volumeInHandle`
```
volumeHandle: 0001-0009-rook-ceph-0000000000000002-24862838-240d-4215-9183-abfc0e9e4002
```
3. For Ceph volumes on that node:
1. If RBD PVC makes use of the rbd status API
1. example:
```console
$ rbd status <poolname>/<image_name>
Watchers:
watcher=172.21.12.201:0/4225036114 client.17881 cookie=18446462598732840961
```
2. If CephFS PVC uses below CLI to clients connect to subvolume
1. example:
```console
$ ceph tell mds.* client ls
...
...
...
"addr": {
"type": "v1",
"addr": "192.168.39.214:0",
"nonce": 1301050887
}
...
...
...
```
4. Get IPs from step 3 (in above example `172.21.12.201`)
5. blocklist the IP where the volumes are mounted.
Suggested change
Example of a NetworkFence CR that the Rook operator would create when a `node.kubernetes.io/out-of-service` taint is added on the node:
```yaml
apiVersion: csiaddons.openshift.io/v1alpha1
kind: NetworkFence
metadata:
name: <name> # We will keep the name the same as the node name
namespace: <ceph-cluster-namespace>
spec:
driver: <driver-name> # extract the driver name from the PV object
fenceState: <fence-state> # For us it will be `Fenced`
cidrs:
- 172.21.12.201
secret:
name: <csi-rbd-provisioner-secret-name/csi-cephfs-provisioner-secret-name> # from pv object
namespace: <ceph-cluster-namespace>
parameters:
clusterID: <clusterID> # from pv.spec.csi.volumeAttributes
```
## Bringing a node back online
Once the node is back online, the admin removes the taint.
1. Remove the taint
```console
kubectl taint nodes <node-name> node.kubernetes.io/out-of-service=nodeshutdown:NoExecute-
# or
kubectl taint nodes <node-name> node.kubernetes.io/out-of-service=nodeshutdown:NoSchedule-
```
1. Rook will detect the taint is removed from the node, and immediately unfence the node by deleting the corresponding networkFence CR.
## Automation to taint a node
Rook will not automate tainting the node when they go offline. This is a decision the admin needs to make. But Rook will consider creating a sample script to watch for unavailable nodes and automatically taint the node based on how long node is offline. The admin can choose to enable the automated taints by running this example script.
## Open Design Questions
1. How to handle MultiCluster scenarios where two different node on different clusters have the same overlapping IP's.
-33
View File
@@ -43,7 +43,6 @@ import (
corev1 "k8s.io/api/core/v1"
kerrors "k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/labels"
apituntime "k8s.io/apimachinery/pkg/runtime"
"k8s.io/apimachinery/pkg/types"
"k8s.io/client-go/tools/record"
@@ -501,38 +500,6 @@ func (c *ClusterController) requestClusterDelete(clusterObj *cephv1.CephCluster)
return reconcile.Result{}, nil // do not requeue the delete
}
}
_, err := c.context.ApiExtensionsClient.ApiextensionsV1().CustomResourceDefinitions().Get(c.OpManagerCtx, "networkfences.csiaddons.openshift.io", metav1.GetOptions{})
if err == nil {
log.NamespacedInfo(clusterObj.Namespace, logger, "removing networkFence if matching cephCluster UID found")
networkFenceList := &addonsv1alpha1.NetworkFenceList{}
labelSelector := labels.SelectorFromSet(map[string]string{
networkFenceLabel: string(clusterObj.GetUID()),
})
opts := []client.DeleteAllOfOption{
client.MatchingLabels{
networkFenceLabel: string(clusterObj.GetUID()),
},
client.GracePeriodSeconds(0),
}
err = c.client.DeleteAllOf(c.OpManagerCtx, &addonsv1alpha1.NetworkFence{}, opts...)
if err != nil && !kerrors.IsNotFound(err) {
return reconcile.Result{}, errors.Wrapf(err, "failed to delete networkFence with label %s", networkFenceLabel)
}
err = c.client.List(c.OpManagerCtx, networkFenceList, &client.MatchingLabelsSelector{Selector: labelSelector})
if err != nil && !kerrors.IsNotFound(err) {
return reconcile.Result{}, errors.Wrap(err, "failed to list networkFence")
}
if len(networkFenceList.Items) > 0 {
for i := range networkFenceList.Items {
err = opcontroller.RemoveFinalizerWithName(c.OpManagerCtx, c.client, &networkFenceList.Items[i], "csiaddons.openshift.io/network-fence")
if err != nil {
return reconcile.Result{}, errors.Wrap(err, "failed to remove finalizer")
}
}
}
}
log.NamespacedInfo(clusterObj.Namespace, logger, "cleaning up CephCluster %q", nsName)
if existingCluster != nil {
+1 -496
View File
@@ -19,14 +19,8 @@ package cluster
import (
"context"
"encoding/json"
stderrors "errors"
"fmt"
"strings"
"time"
addonsv1alpha1 "github.com/csi-addons/kubernetes-csi-addons/api/csiaddons/v1alpha1"
pkgerror "github.com/pkg/errors"
cephv1 "github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
"github.com/rook/rook/pkg/clusterd"
cephclient "github.com/rook/rook/pkg/daemon/ceph/client"
@@ -35,12 +29,8 @@ import (
"github.com/rook/rook/pkg/operator/k8sutil"
corev1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apimachinery/pkg/types"
"k8s.io/apimachinery/pkg/util/sets"
"k8s.io/apimachinery/pkg/util/wait"
"sigs.k8s.io/controller-runtime/pkg/client"
)
@@ -52,17 +42,7 @@ type clientCluster struct {
context *clusterd.Context
}
var (
nodesCheckedForReconcile = sets.New[string]()
networkFenceLabel = "cephClusterUID"
errActiveClientNotFound = stderrors.New("active client not found")
)
// drivers that supports fencing, used in naming networkFence object
const (
rbdDriver = "rbd"
cephfsDriver = "cephfs"
)
var nodesCheckedForReconcile = sets.New[string]()
func newClientCluster(client client.Client, namespace string, context *clusterd.Context) *clientCluster {
return &clientCluster{
@@ -90,11 +70,6 @@ func (c *clientCluster) onK8sNode(ctx context.Context, object runtime.Object, op
// Get CephCluster
cluster := c.getCephCluster()
// Continue reconcile in case of failure too since we don't want to block other node reconcile
// if err := c.handleNodeFailure(ctx, cluster, node, opNamespace); err != nil {
// logger.Errorf("failed to handle node failure. %v", err)
// }
// skip reconcile if node is already checked in a previous reconcile
if nodesCheckedForReconcile.Has(node.Name) {
return false
@@ -158,476 +133,6 @@ func (c *clientCluster) onK8sNode(ctx context.Context, object runtime.Object, op
return false
}
func (c *clientCluster) handleNodeFailure(ctx context.Context, cluster *cephv1.CephCluster, node *corev1.Node, opNamespace string) error {
watchForNodeLoss := k8sutil.GetOperatorSetting("ROOK_WATCH_FOR_NODE_FAILURE", "true")
if strings.ToLower(watchForNodeLoss) != "true" {
logger.Debugf("not watching for node failures since `ROOK_WATCH_FOR_NODE_FAILURE` is set to %q", watchForNodeLoss)
return nil
}
disabledCSI := k8sutil.GetOperatorSetting("ROOK_CSI_DISABLE_DRIVER", "false")
if strings.ToLower(disabledCSI) != "false" {
logger.Debugf("not watching for node failures since `ROOK_CSI_DISABLE_DRIVER` is set to %q, skip creating networkFence", disabledCSI)
return nil
}
_, err := c.context.ApiExtensionsClient.ApiextensionsV1().CustomResourceDefinitions().Get(ctx, "networkfences.csiaddons.openshift.io", metav1.GetOptions{})
if err != nil {
if errors.IsNotFound(err) {
logger.Debug("networkfences.csiaddons.openshift.io CRD not found, skip creating networkFence")
return nil
}
return pkgerror.Wrapf(err, "failed to get networkfences.csiaddons.openshift.io CRD, skip creating networkFence")
}
nodeHasOutOfServiceTaint := false
for _, taint := range node.Spec.Taints {
if taint.Key == corev1.TaintNodeOutOfService {
nodeHasOutOfServiceTaint = true
logger.Infof("Found taint: Key=%v, Value=%v on node %s\n", taint.Key, taint.Value, node.Name)
break
}
}
if nodeHasOutOfServiceTaint {
err := c.fenceNode(ctx, node, cluster, opNamespace)
if err != nil {
return pkgerror.Wrapf(err, "failed to create network fence for node %q.", node.Name)
}
return nil
}
err = c.unfenceAndDeleteNetworkFence(ctx, *node, cluster, rbdDriver)
if err != nil {
return pkgerror.Wrapf(err, "failed to delete rbd network fence for node %q.", node.Name)
}
err = c.unfenceAndDeleteNetworkFence(ctx, *node, cluster, cephfsDriver)
if err != nil {
return pkgerror.Wrapf(err, "failed to delete cephFS network fence for node %q.", node.Name)
}
return nil
}
func (c *clientCluster) fenceNode(ctx context.Context, node *corev1.Node, cluster *cephv1.CephCluster, opNamespace string) error {
volumesInuse := node.Status.VolumesInUse
if len(volumesInuse) == 0 {
logger.Debugf("no volumes in use for node %q", node.Name)
return nil
}
logger.Debugf("volumesInuse %s", volumesInuse)
rbdVolumesInUse, cephFSVolumeInUse := getCephVolumesInUse(cluster, volumesInuse, opNamespace)
if len(rbdVolumesInUse) == 0 && len(cephFSVolumeInUse) == 0 {
logger.Debugf("no rbd or cephfs subvolumes in use for out of service node %q", node.Name)
return nil
}
listPVs, err := c.context.Clientset.CoreV1().PersistentVolumes().List(ctx, metav1.ListOptions{})
if err != nil {
return pkgerror.Wrapf(err, "failed to list PV")
}
if len(rbdVolumesInUse) != 0 {
rbdPVList := listRBDPV(listPVs, cluster, rbdVolumesInUse, opNamespace)
if len(rbdPVList) == 0 {
logger.Debug("No rbd PVs found on the node")
} else {
logger.Infof("node %q require fencing, found rbd volumes in use", node.Name)
clusterInfo, _, _, err := opcontroller.LoadClusterInfo(c.context, ctx, cluster.Namespace, &cluster.Spec)
if err != nil {
return pkgerror.Wrapf(err, "Failed to load cluster info.")
}
for i := range rbdPVList {
err = c.fenceRbdImage(ctx, node, cluster, clusterInfo, rbdPVList[i])
// We only need to create the network fence for any one of rbd pv.
if err == nil {
break
}
// continue to fence next rbd volume if active client not found
if stderrors.Is(err, errActiveClientNotFound) {
continue
}
if i == len(rbdPVList)-1 {
return pkgerror.Wrapf(err, "failed to fence rbd volumes")
}
logger.Errorf("failed to fence rbd volumes %q, trying next rbd volume", rbdPVList[i].Name)
}
}
}
if len(cephFSVolumeInUse) != 0 {
cephFSVolumeInUseMap := make(map[string]struct{})
for _, vol := range cephFSVolumeInUse {
cephFSVolumeInUseMap[vol] = struct{}{}
}
cephFSPVList := listRWOCephFSPV(listPVs, cluster, cephFSVolumeInUseMap, opNamespace)
if len(cephFSPVList) == 0 {
logger.Debug("No cephfs PVs found on the node %s", node.Name)
return nil
}
logger.Infof("node %q require fencing, found cephfs subvolumes in use", node.Name)
clusterInfo, _, _, err := opcontroller.LoadClusterInfo(c.context, ctx, cluster.Namespace, &cluster.Spec)
if err != nil {
return pkgerror.Wrap(err, "Failed to load cluster info.")
}
for i := range cephFSPVList {
err = c.fenceCephFSSubvolume(ctx, node, cluster, clusterInfo, cephFSPVList[i])
// We only need to create the network fence for any one of cephFS pv.
if err == nil {
break
}
// continue to fence next cephfs subvolume if active client not found
if stderrors.Is(err, errActiveClientNotFound) {
continue
}
if i == len(cephFSPVList)-1 {
return pkgerror.Wrap(err, "failed to fence cephfs subvolumes")
}
logger.Errorf("failed to fence cephfs subvolume %q, trying next cephfs subvolume", cephFSPVList[i].Name)
}
}
return nil
}
func getCephVolumesInUse(cluster *cephv1.CephCluster, volumesInUse []corev1.UniqueVolumeName, opNamespace string) ([]string, []string) {
var rbdVolumesInUse, cephFSVolumeInUse []string
for _, volume := range volumesInUse {
splitVolumeInUseBased := trimeVolumeInUse(volume)
logger.Infof("volumeInUse after split based on '^' %v", splitVolumeInUseBased)
if len(splitVolumeInUseBased) == 2 && splitVolumeInUseBased[0] == fmt.Sprintf("%s.rbd.csi.ceph.com", opNamespace) {
rbdVolumesInUse = append(rbdVolumesInUse, splitVolumeInUseBased[1])
}
if len(splitVolumeInUseBased) == 2 && splitVolumeInUseBased[0] == fmt.Sprintf("%s.cephfs.csi.ceph.com", opNamespace) {
cephFSVolumeInUse = append(cephFSVolumeInUse, splitVolumeInUseBased[1])
}
}
return rbdVolumesInUse, cephFSVolumeInUse
}
func trimeVolumeInUse(volume corev1.UniqueVolumeName) []string {
volumesInuseRemoveK8sPrefix := strings.TrimPrefix(string(volume), "kubernetes.io/csi/")
splitVolumeInUseBased := strings.Split(volumesInuseRemoveK8sPrefix, "^")
return splitVolumeInUseBased
}
func listRBDPV(listPVs *corev1.PersistentVolumeList, cluster *cephv1.CephCluster, rbdVolumesInUse []string, opNamespace string) []corev1.PersistentVolume {
var listRbdPV []corev1.PersistentVolume
for _, pv := range listPVs.Items {
// Skip if pv is not provisioned by CSI
if pv.Spec.CSI == nil {
logger.Debugf("pv %q is not provisioned by CSI", pv.Name)
continue
}
if pv.Spec.CSI.Driver == fmt.Sprintf("%s.rbd.csi.ceph.com", opNamespace) {
// Ignore PVs that support multinode access (RWX, ROX), since they can be mounted on multiple nodes.
if pvSupportsMultiNodeAccess(pv.Spec.AccessModes) {
continue
}
if pv.Spec.CSI.VolumeAttributes["staticVolume"] == "true" || pv.Spec.CSI.VolumeAttributes["pool"] == "" || pv.Spec.CSI.VolumeAttributes["imageName"] == "" {
logger.Debugf("skipping, static pv %q", pv.Name)
continue
}
for _, rbdVolume := range rbdVolumesInUse {
if pv.Spec.CSI.VolumeHandle == rbdVolume {
listRbdPV = append(listRbdPV, pv)
}
}
}
}
return listRbdPV
}
func listRWOCephFSPV(listPVs *corev1.PersistentVolumeList, cluster *cephv1.CephCluster, cephFSVolumesInUse map[string]struct{}, opNamespace string) []corev1.PersistentVolume {
var listCephFSPV []corev1.PersistentVolume
for _, pv := range listPVs.Items {
// Skip if pv is not provisioned by CSI
if pv.Spec.CSI == nil {
logger.Debugf("PV %q is not provisioned by CSI", pv.Name)
continue
}
if pv.Spec.CSI.Driver == fmt.Sprintf("%s.cephfs.csi.ceph.com", opNamespace) {
// Ignore PVs that support multinode access (RWX, ROX), since they can be mounted on multiple nodes.
if pvSupportsMultiNodeAccess(pv.Spec.AccessModes) {
continue
}
if pv.Spec.CSI.VolumeAttributes["staticVolume"] == "true" {
logger.Debugf("skipping, static PV %q", pv.Name)
continue
}
// Check if the volume is in use
if _, exists := cephFSVolumesInUse[pv.Spec.CSI.VolumeHandle]; exists {
listCephFSPV = append(listCephFSPV, pv)
}
}
}
return listCephFSPV
}
// pvSupportsMultiNodeAccess returns true if the PV access modes contain ReadWriteMany or ReadOnlyMany.
func pvSupportsMultiNodeAccess(accessModes []corev1.PersistentVolumeAccessMode) bool {
for _, accessMode := range accessModes {
switch accessMode {
case corev1.ReadOnlyMany, corev1.ReadWriteMany:
return true
}
}
return false
}
func (c *clientCluster) fenceRbdImage(
ctx context.Context, node *corev1.Node, cluster *cephv1.CephCluster,
clusterInfo *cephclient.ClusterInfo, rbdPV corev1.PersistentVolume,
) error {
logger.Debugf("rbd PV NAME %v", rbdPV.Spec.CSI.VolumeAttributes)
args := []string{"status", fmt.Sprintf("%s/%s", rbdPV.Spec.CSI.VolumeAttributes["pool"], rbdPV.Spec.CSI.VolumeAttributes["imageName"])}
cmd := cephclient.NewRBDCommand(c.context, clusterInfo, args)
cmd.JsonOutput = true
buf, err := cmd.Run()
if err != nil {
return pkgerror.Wrapf(err, "failed to list watchers for pool/imageName %s/%s.", rbdPV.Spec.CSI.VolumeAttributes["pool"], rbdPV.Spec.CSI.VolumeAttributes["imageName"])
}
ips, err := rbdStatusUnMarshal(buf)
if err != nil {
return pkgerror.Wrapf(err, "failed to unmarshal rbd status output")
}
if len(ips) == 0 {
logger.Infof("no active rbd clients found for rbd volume %q", rbdPV.Name)
return errActiveClientNotFound
}
err = c.createNetworkFence(ctx, rbdPV, node, cluster, ips, rbdDriver)
if err != nil {
return pkgerror.Wrapf(err, "failed to create network fence for node %q", node.Name)
}
return nil
}
func (c *clientCluster) fenceCephFSSubvolume(
ctx context.Context, node *corev1.Node, cluster *cephv1.CephCluster,
clusterInfo *cephclient.ClusterInfo, cephFSPV corev1.PersistentVolume,
) error {
logger.Infof("fencing cephfs subvolume %q on node %q", cephFSPV.Name, node.Name)
status, err := cephclient.StatusWithUser(c.context, clusterInfo)
if err != nil {
return pkgerror.Wrap(err, "failed to get ceph status for check active mds")
}
var activeMDS string
for _, fsRank := range status.Fsmap.ByRank {
if fsRank.Status == "up:active" {
activeMDS = fsRank.Name
}
}
args := []string{"tell", fmt.Sprintf("mds.%s", activeMDS), "client", "ls", "--format", "json"}
cmd := cephclient.NewCephCommand(c.context, clusterInfo, args)
cmd.JsonOutput = true
buf, err := cmd.Run()
if err != nil {
return fmt.Errorf("failed to list watchers for cephfs subvolumeName %s. %v", cephFSPV.Spec.CSI.VolumeAttributes["subvolumeName"], err)
}
ips, err := cephFSMDSClientMarshal(buf, node.Name, cephFSPV)
if err != nil {
return pkgerror.Wrap(err, "failed to unmarshal cephfs mds output")
}
if len(ips) == 0 {
logger.Infof("no active mds clients found for cephfs subvolume %q", cephFSPV.Name)
return errActiveClientNotFound
}
err = c.createNetworkFence(ctx, cephFSPV, node, cluster, ips, cephfsDriver)
if err != nil {
return pkgerror.Wrapf(err, "failed to create network fence for node %q", node.Name)
}
return nil
}
func cephFSMDSClientMarshal(output []byte, nodeName string, cephFSPV corev1.PersistentVolume) ([]string, error) {
type entity struct {
Addr struct {
Addr string `json:"addr"`
Nonce int `json:"nonce"`
} `json:"addr"`
}
type clientMetadata struct {
Root string `json:"root"`
Hostname string `json:"hostname"`
}
type cephFSData struct {
Entity entity `json:"entity"`
ClientMetadata clientMetadata `json:"client_metadata"`
}
var data []cephFSData
err := json.Unmarshal([]byte(output), &data)
if err != nil {
return []string{}, pkgerror.Wrapf(err, "failed to unmarshal cephFS data output")
}
watcherIPlist := []string{}
for _, d := range data {
if d.ClientMetadata.Hostname != "" {
if strings.Contains(nodeName, d.ClientMetadata.Hostname) && cephFSPV.Spec.CSI.VolumeAttributes["subvolumePath"] == d.ClientMetadata.Root {
logger.Infof("cephfs mds client ips to fence %v", d.Entity.Addr)
watcherIP := concatenateWatcherIp(d.Entity.Addr.Addr)
watcherIPlist = append(watcherIPlist, watcherIP)
break
}
}
}
return watcherIPlist, nil
}
func rbdStatusUnMarshal(output []byte) ([]string, error) {
type rbdStatus struct {
Watchers []struct {
Address string `json:"address"`
} `json:"watchers"`
}
var rbdStatusObj rbdStatus
err := json.Unmarshal([]byte(output), &rbdStatusObj)
if err != nil {
return []string{}, pkgerror.Wrapf(err, "failed to unmarshal rbd status output")
}
watcherIPlist := []string{}
for _, watcher := range rbdStatusObj.Watchers {
watcherIP := concatenateWatcherIp(watcher.Address)
watcherIPlist = append(watcherIPlist, watcherIP)
}
return watcherIPlist, nil
}
func concatenateWatcherIp(address string) string {
// address is in format `10.63.0.5:0/1254753579` for rbd and
// in the format '10.244.0.12:0' for cephfs
// split with separation ':0' to remove nounce and concatenating `/32` to define a network with only one IP address
watcherIP := strings.Split(address, ":0")[0] + "/32"
return watcherIP
}
func fenceResourceName(nodeName, driver, namespace string) string {
return fmt.Sprintf("%s-%s-%s", nodeName, driver, namespace)
}
func (c *clientCluster) createNetworkFence(ctx context.Context, pv corev1.PersistentVolume, node *corev1.Node, cluster *cephv1.CephCluster, cidr []string, driver string) error {
logger.Warningf("Blocking node IP %s", cidr)
secretName := pv.Annotations["volume.kubernetes.io/provisioner-deletion-secret-name"]
secretNameSpace := pv.Annotations["volume.kubernetes.io/provisioner-deletion-secret-namespace"]
if secretName == "" || secretNameSpace == "" {
storageClass, err := c.context.Clientset.StorageV1().StorageClasses().Get(ctx, pv.Spec.StorageClassName, metav1.GetOptions{})
if err != nil {
return pkgerror.Wrap(err, "failed to get storage class to fence volume")
}
secretName = storageClass.Parameters["csi.storage.k8s.io/provisioner-secret-name"]
secretNameSpace = storageClass.Parameters["csi.storage.k8s.io/provisioner-secret-namespace"]
}
networkFence := &addonsv1alpha1.NetworkFence{
ObjectMeta: metav1.ObjectMeta{
Name: fenceResourceName(node.Name, driver, cluster.Namespace),
Labels: map[string]string{
networkFenceLabel: string(cluster.GetUID()),
},
},
Spec: addonsv1alpha1.NetworkFenceSpec{
Driver: pv.Spec.CSI.Driver,
FenceState: addonsv1alpha1.Fenced,
Secret: addonsv1alpha1.SecretSpec{
Name: secretName,
Namespace: secretNameSpace,
},
Cidrs: cidr,
Parameters: map[string]string{
"clusterID": pv.Spec.CSI.VolumeAttributes["clusterID"],
},
},
}
err := c.client.Create(ctx, networkFence)
if err != nil && !errors.IsAlreadyExists(err) {
return err
}
logger.Infof("successfully created network fence CR for node %q", node.Name)
return nil
}
func (c *clientCluster) unfenceAndDeleteNetworkFence(ctx context.Context, node corev1.Node, cluster *cephv1.CephCluster, driver string) error {
networkFence := &addonsv1alpha1.NetworkFence{}
err := c.client.Get(ctx, types.NamespacedName{Name: fenceResourceName(node.Name, driver, cluster.Namespace)}, networkFence)
if err != nil && !errors.IsNotFound(err) {
return err
} else if errors.IsNotFound(err) {
return nil
}
logger.Infof("node %s does not have taint %s, unfencing networkFence CR", node.Name, corev1.TaintNodeOutOfService)
// Unfencing is required to unblock the node and then delete the network fence CR
networkFence.Spec.FenceState = addonsv1alpha1.Unfenced
err = c.client.Update(ctx, networkFence)
if err != nil {
logger.Errorf("failed to unFence network fence CR. %v", err)
return err
}
err = wait.PollUntilContextTimeout(ctx, 2*time.Second, 60*time.Second, true, func(ctx context.Context) (bool, error) {
err = c.client.Get(ctx, types.NamespacedName{Name: fenceResourceName(node.Name, driver, cluster.Namespace)}, networkFence)
if err != nil && !errors.IsNotFound(err) {
return false, err
}
if networkFence.Status.Message != addonsv1alpha1.UnFenceOperationSuccessfulMessage {
logger.Infof("waiting for network fence CR %q status to get result %q", networkFence.Name, addonsv1alpha1.UnFenceOperationSuccessfulMessage)
return false, err
}
logger.Infof("successfully unfenced %q network fence cr %q, proceeding with deletion", driver, networkFence.Name)
err = c.client.Delete(ctx, networkFence)
if err == nil || errors.IsNotFound(err) {
logger.Infof("successfully deleted network fence CR %s", networkFence.Name)
return true, nil
}
return false, nil
})
if err != nil {
return pkgerror.Wrapf(err, "timeout out deleting the %s network fence CR %s", driver, networkFence.Name)
}
return nil
}
// onDeviceCMUpdate is trigger when the hot plug config map is updated
func (c *clientCluster) onDeviceCMUpdate(oldObj, newObj runtime.Object) bool {
oldCm, ok := oldObj.(*corev1.ConfigMap)
+4 -81
View File
@@ -18,13 +18,10 @@ package cluster
import (
"context"
"fmt"
"os"
"strings"
"testing"
"github.com/coreos/pkg/capnslog"
addonsv1alpha1 "github.com/csi-addons/kubernetes-csi-addons/api/csiaddons/v1alpha1"
"github.com/pkg/errors"
cephv1 "github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
"github.com/rook/rook/pkg/client/clientset/versioned/scheme"
@@ -33,12 +30,9 @@ import (
exectest "github.com/rook/rook/pkg/util/exec/test"
"github.com/stretchr/testify/assert"
corev1 "k8s.io/api/core/v1"
v1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1"
apifake "k8s.io/apiextensions-apiserver/pkg/client/clientset/clientset/fake"
kerrors "k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apimachinery/pkg/types"
k8sFake "k8s.io/client-go/kubernetes/fake"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/client/fake"
@@ -47,7 +41,7 @@ import (
func getFakeClient(obj ...runtime.Object) client.Client {
// Register operator types with the runtime scheme.
scheme := scheme.Scheme
scheme.AddKnownTypes(cephv1.SchemeGroupVersion, &cephv1.CephCluster{}, &addonsv1alpha1.NetworkFence{}, &addonsv1alpha1.NetworkFenceList{})
scheme.AddKnownTypes(cephv1.SchemeGroupVersion, &cephv1.CephCluster{})
client := fake.NewClientBuilder().WithScheme(scheme).WithRuntimeObjects(obj...).Build()
return client
}
@@ -158,6 +152,7 @@ func TestOnK8sNode(t *testing.T) {
assert.False(t, b)
}
<<<<<<< HEAD
func TestHandleNodeFailure(t *testing.T) {
clusterns := "rook-ceph"
opns := "operator"
@@ -498,6 +493,8 @@ func TestFenceResourceName(t *testing.T) {
assert.Equal(t, FenceName, "fakenode-rbd-rook-ceph")
}
=======
>>>>>>> 20f350a71 (csi: remove automated node fencing code)
func TestOnDeviceCMUpdate(t *testing.T) {
// Set DEBUG logging
capnslog.SetGlobalLogLevel(capnslog.DEBUG)
@@ -575,77 +572,3 @@ func TestOnDeviceCMUpdate(t *testing.T) {
b = clientCluster.onDeviceCMUpdate(oldCM, newCM)
assert.True(t, b)
}
func Test_pvSupportMultiNodeAccess(t *testing.T) {
type args struct {
accessModes []corev1.PersistentVolumeAccessMode
}
tests := []struct {
name string
args args
want bool
}{
{
name: "single access mode, contains RWX",
args: args{
accessModes: []corev1.PersistentVolumeAccessMode{
corev1.ReadWriteMany,
},
},
want: true,
},
{
name: "single access mode, contains ROX",
args: args{
accessModes: []corev1.PersistentVolumeAccessMode{
corev1.ReadOnlyMany,
},
},
want: true,
},
{
name: "single access mode, contains only RWO",
args: args{
accessModes: []corev1.PersistentVolumeAccessMode{
corev1.ReadWriteOnce,
},
},
want: false,
},
{
name: "single access mode, contains only RWOP",
args: args{
accessModes: []corev1.PersistentVolumeAccessMode{
corev1.ReadWriteOncePod,
},
},
want: false,
},
{
name: "multiple access mode, contains RWX",
args: args{
accessModes: []corev1.PersistentVolumeAccessMode{
corev1.ReadWriteOnce,
corev1.ReadWriteMany,
},
},
want: true,
},
{
name: "multiple access mode, contains RWX and ROX",
args: args{
accessModes: []corev1.PersistentVolumeAccessMode{
corev1.ReadWriteMany,
corev1.ReadOnlyMany,
},
},
want: true,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got := pvSupportsMultiNodeAccess(tt.args.accessModes)
assert.Equal(t, tt.want, got)
})
}
}
-1
View File
@@ -46,7 +46,6 @@ import (
// - Bluestore running on devices
// Block
// - Mount/unmount a block device through the dynamic provisioner
// - Fencing of the block device
// - Read/write to the device
// File system
// - Create the file system via the CRD