mon: wait for the canary pods to be terminated

Wait for the mon-canary pods to be terminated before starting the actual
mons. When topology spread constraints are added to evenly spread the
mons, then pod-anti affinity does not allow mons to be created on the
same node where other mons are running, this also includes the
mon-canary pods. We delete the mon-canary pods and then start the actual
mon pods. Sometimes the termination of the canary pods might take more
time, so schedular won't schedular mon pods on that node.

Signed-off-by: Santosh Pillai <sapillai@redhat.com>
(cherry picked from commit 3137a857df)

pkg/operator/ceph/cluster/mon/mon.go

@@ -880,13 +880,10 @@ func (c *Cluster) removeCanaryDeployments(labelSelector string) {
 		return
 	}
 
-	// Delete the canary mons, but don't wait for them to exit
+	// Delete the canary mons
 	for _, canary := range canaryDeployments.Items {
 		logger.Infof("cleaning up canary monitor deployment %q", canary.Name)
-		var gracePeriod int64
-		propagation := metav1.DeletePropagationForeground
-		options := &metav1.DeleteOptions{GracePeriodSeconds: &gracePeriod, PropagationPolicy: &propagation}
-		if err := c.context.Clientset.AppsV1().Deployments(c.Namespace).Delete(c.ClusterInfo.Context, canary.Name, *options); err != nil {
+		if err := k8sutil.DeleteDeployment(c.ClusterInfo.Context, c.context.Clientset, c.Namespace, canary.Name); err != nil {
 			logger.Warningf("failed to delete canary monitor deployment %q. %v", canary.Name, err)
 		}
 	}