forked from rook/rook
Sébastien Han
ad24990473
We can now prioritize orchestrations on certain event. Today only two
events will cancel on-going orchestrations (if any):
* request for cluster deletion
* request for cluster upgrade
If one of the two are caught by the watcher we will cancel the on-going
orchestration.
For that we implemented a simple approach based on check points, where
we will check for a cancellation request in certain part of the
orchestration. Mainly before each mons/mgr/osds orchestration loops.
This solution is not perfect, but we are waiting for the
controller-runtime to release its 0.7 version which will embed context
support. With that we will be able to cancel reconciles more precisely
and rapidly.
Operator log example:
```
2020-11-24 13:54:59.499719 I | op-mon: parsing mon endpoints: a=10.109.126.120:6789
2020-11-24 13:54:59.499719 I | op-mon: parsing mon endpoints: a=10.109.126.120:6789
2020-11-25 12:59:12.986264 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph"
2020-11-25 13:07:33.776947 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{
CephVersion: v1.CephVersionSpec{
Image: "ceph/ceph:v15.2.5",
- AllowUnsupported: true,
+ AllowUnsupported: false,
},
DriveGroups: nil,
Storage: {UseAllNodes: true, Selection: {UseAllDevices: &true}},
... // 20 identical fields
}
2020-11-25 13:07:33.777039 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph"
2020-11-25 13:07:33.785088 I | op-mon: parsing mon endpoints: a=10.107.242.49:6789,b=10.109.71.30:6789,c=10.98.93.224:6789
2020-11-25 13:07:33.788626 I | ceph-cluster-controller: detecting the ceph image version for image ceph/ceph:v15.2.5...
2020-11-25 13:07:35.280789 I | ceph-cluster-controller: detected ceph image version: "15.2.5-0 octopus"
2020-11-25 13:07:35.280806 I | ceph-cluster-controller: validating ceph version from provided image
2020-11-25 13:07:35.285888 I | op-mon: parsing mon endpoints: a=10.107.242.49:6789,b=10.109.71.30:6789,c=10.98.93.224:6789
2020-11-25 13:07:35.287828 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config
2020-11-25 13:07:35.288082 I | cephclient: generated admin config in /var/lib/rook/rook-ceph
2020-11-25 13:07:35.621625 I | ceph-cluster-controller: cluster "rook-ceph": version "15.2.5-0 octopus" detected for image "ceph/ceph:v15.2.5"
2020-11-25 13:07:35.642688 I | op-mon: start running mons
2020-11-25 13:07:35.646323 I | op-mon: parsing mon endpoints: a=10.107.242.49:6789,b=10.109.71.30:6789,c=10.98.93.224:6789
2020-11-25 13:07:35.654070 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.107.242.49:6789","10.109.71.30:6789","10.98.93.224:6789"]}] data:a=10.107.242.49:6789,b=10.109.71.30:6789,c=10.98.93.224:6789 mapping:{"node":{"a":{"Name":"minikube","Hostname":"minikube","Address":"192.168.39.3"},"b":{"Name":"minikube","Hostname":"minikube","Address":"192.168.39.3"},"c":{"Name":"minikube","Hostname":"minikube","Address":"192.168.39.3"}}} maxMonId:2]
2020-11-25 13:07:35.868253 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config
2020-11-25 13:07:35.868573 I | cephclient: generated admin config in /var/lib/rook/rook-ceph
2020-11-25 13:07:37.074353 I | op-mon: targeting the mon count 3
2020-11-25 13:07:38.153435 I | op-mon: checking for basic quorum with existing mons
2020-11-25 13:07:38.178029 I | op-mon: mon "a" endpoint is [v2:10.107.242.49:3300,v1:10.107.242.49:6789]
2020-11-25 13:07:38.670191 I | op-mon: mon "b" endpoint is [v2:10.109.71.30:3300,v1:10.109.71.30:6789]
2020-11-25 13:07:39.477820 I | op-mon: mon "c" endpoint is [v2:10.98.93.224:3300,v1:10.98.93.224:6789]
2020-11-25 13:07:39.874094 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.107.242.49:6789","10.109.71.30:6789","10.98.93.224:6789"]}] data:a=10.107.242.49:6789,b=10.109.71.30:6789,c=10.98.93.224:6789 mapping:{"node":{"a":{"Name":"minikube","Hostname":"minikube","Address":"192.168.39.3"},"b":{"Name":"minikube","Hostname":"minikube","Address":"192.168.39.3"},"c":{"Name":"minikube","Hostname":"minikube","Address":"192.168.39.3"}}} maxMonId:2]
2020-11-25 13:07:40.467999 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config
2020-11-25 13:07:40.469733 I | cephclient: generated admin config in /var/lib/rook/rook-ceph
2020-11-25 13:07:41.071710 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config
2020-11-25 13:07:41.078903 I | cephclient: generated admin config in /var/lib/rook/rook-ceph
2020-11-25 13:07:41.125233 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed
2020-11-25 13:07:41.327778 I | op-k8sutil: updating deployment "rook-ceph-mon-a" after verifying it is safe to stop
2020-11-25 13:07:41.327895 I | op-mon: checking if we can stop the deployment rook-ceph-mon-a
2020-11-25 13:07:44.045644 I | op-k8sutil: finished waiting for updated deployment "rook-ceph-mon-a"
2020-11-25 13:07:44.045706 I | op-mon: checking if we can continue the deployment rook-ceph-mon-a
2020-11-25 13:07:44.045740 I | op-mon: waiting for mon quorum with [a b c]
2020-11-25 13:07:44.109159 I | op-mon: mons running: [a b c]
2020-11-25 13:07:44.474596 I | op-mon: Monitors in quorum: [a b c]
2020-11-25 13:07:44.478565 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed
2020-11-25 13:07:44.493374 I | op-k8sutil: updating deployment "rook-ceph-mon-b" after verifying it is safe to stop
2020-11-25 13:07:44.493403 I | op-mon: checking if we can stop the deployment rook-ceph-mon-b
2020-11-25 13:07:47.135524 I | op-k8sutil: finished waiting for updated deployment "rook-ceph-mon-b"
2020-11-25 13:07:47.135542 I | op-mon: checking if we can continue the deployment rook-ceph-mon-b
2020-11-25 13:07:47.135551 I | op-mon: waiting for mon quorum with [a b c]
2020-11-25 13:07:47.148820 I | op-mon: mons running: [a b c]
2020-11-25 13:07:47.445946 I | op-mon: Monitors in quorum: [a b c]
2020-11-25 13:07:47.448991 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed
2020-11-25 13:07:47.462041 I | op-k8sutil: updating deployment "rook-ceph-mon-c" after verifying it is safe to stop
2020-11-25 13:07:47.462060 I | op-mon: checking if we can stop the deployment rook-ceph-mon-c
2020-11-25 13:07:48.853118 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{
CephVersion: v1.CephVersionSpec{
- Image: "ceph/ceph:v15.2.5",
+ Image: "ceph/ceph:v15.2.6",
AllowUnsupported: false,
},
DriveGroups: nil,
Storage: {UseAllNodes: true, Selection: {UseAllDevices: &true}},
... // 20 identical fields
}
2020-11-25 13:07:48.853140 I | ceph-cluster-controller: upgrade requested, cancelling any ongoing orchestration
2020-11-25 13:07:50.119584 I | op-k8sutil: finished waiting for updated deployment "rook-ceph-mon-c"
2020-11-25 13:07:50.119606 I | op-mon: checking if we can continue the deployment rook-ceph-mon-c
2020-11-25 13:07:50.119619 I | op-mon: waiting for mon quorum with [a b c]
2020-11-25 13:07:50.130860 I | op-mon: mons running: [a b c]
2020-11-25 13:07:50.431341 I | op-mon: Monitors in quorum: [a b c]
2020-11-25 13:07:50.431361 I | op-mon: mons created: 3
2020-11-25 13:07:50.734156 I | op-mon: waiting for mon quorum with [a b c]
2020-11-25 13:07:50.745763 I | op-mon: mons running: [a b c]
2020-11-25 13:07:51.045108 I | op-mon: Monitors in quorum: [a b c]
2020-11-25 13:07:51.054497 E | ceph-cluster-controller: failed to reconcile. failed to reconcile cluster "rook-ceph": failed to configure local ceph cluster: failed to create cluster: CANCELLING CURRENT ORCHESTATION
2020-11-25 13:07:52.055208 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph"
2020-11-25 13:07:52.070690 I | op-mon: parsing mon endpoints: a=10.107.242.49:6789,b=10.109.71.30:6789,c=10.98.93.224:6789
2020-11-25 13:07:52.088979 I | ceph-cluster-controller: detecting the ceph image version for image ceph/ceph:v15.2.6...
2020-11-25 13:07:53.904811 I | ceph-cluster-controller: detected ceph image version: "15.2.6-0 octopus"
2020-11-25 13:07:53.904862 I | ceph-cluster-controller: validating ceph version from provided image
```
Closes: https://github.com/rook/rook/issues/6587
Signed-off-by: Sébastien Han <seb@redhat.com>
82 lines
3.7 KiB
AMPL
82 lines
3.7 KiB
AMPL
module github.com/rook/rook
|
|
|
|
go 1.13
|
|
|
|
require (
|
|
github.com/aws/aws-sdk-go v1.35.5
|
|
github.com/banzaicloud/k8s-objectmatcher v1.1.0
|
|
github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f
|
|
github.com/corpix/uarand v0.1.1 // indirect
|
|
github.com/davecgh/go-spew v1.1.1
|
|
github.com/ghodss/yaml v1.0.1-0.20190212211648-25d852aebe32
|
|
github.com/go-ini/ini v1.51.1
|
|
github.com/go-logr/logr v0.3.0 // indirect
|
|
github.com/go-sql-driver/mysql v1.5.0
|
|
github.com/google/go-cmp v0.5.2
|
|
github.com/google/uuid v1.1.1
|
|
github.com/hashicorp/vault/api v1.0.5-0.20200902155336-f9d5ce5a171a
|
|
github.com/icrowley/fake v0.0.0-20180203215853-4178557ae428
|
|
github.com/k8snetworkplumbingwg/network-attachment-definition-client v1.1.0
|
|
github.com/kube-object-storage/lib-bucket-provisioner v0.0.0-20201113145926-53eac50605b4
|
|
github.com/libopenstorage/secrets v0.0.0-20201006135900-af310b01fe47
|
|
github.com/miekg/dns v1.1.34 // indirect
|
|
github.com/openshift/cluster-api v0.0.0-20191129101638-b09907ac6668
|
|
github.com/openshift/machine-api-operator v0.2.1-0.20190903202259-474e14e4965a
|
|
github.com/pkg/errors v0.9.1
|
|
github.com/prometheus-operator/prometheus-operator v0.43.0
|
|
github.com/prometheus-operator/prometheus-operator/pkg/apis/monitoring v0.43.0
|
|
github.com/spf13/cobra v1.0.0
|
|
github.com/spf13/pflag v1.0.5
|
|
github.com/stretchr/testify v1.6.1
|
|
github.com/tevino/abool v1.2.0
|
|
github.com/yanniszark/go-nodetool v0.0.0-20191206125106-cd8f91fa16be
|
|
gopkg.in/ini.v1 v1.57.0
|
|
k8s.io/api v0.19.3
|
|
k8s.io/apiextensions-apiserver v0.19.3
|
|
k8s.io/apimachinery v0.19.3
|
|
k8s.io/apiserver v0.19.3
|
|
k8s.io/client-go v12.0.0+incompatible
|
|
k8s.io/cloud-provider v0.19.3
|
|
k8s.io/code-generator v0.19.3
|
|
k8s.io/kube-controller-manager v0.19.3
|
|
k8s.io/kubernetes v1.19.3
|
|
k8s.io/utils v0.0.0-20200729134348-d5654de09c73
|
|
sigs.k8s.io/controller-runtime v0.6.3
|
|
sigs.k8s.io/kustomize/kyaml v0.2.0
|
|
sigs.k8s.io/sig-storage-lib-external-provisioner/v6 v6.1.0
|
|
)
|
|
|
|
// This looks "horrible", but is due to the Rook including k8s.io/kubernetes directly which is not recommended,
|
|
// ee https://github.com/kubernetes/kubernetes/issues/79384#issuecomment-505725449
|
|
replace (
|
|
github.com/googleapis/gnostic => github.com/googleapis/gnostic v0.4.1
|
|
golang.org/x/text => golang.org/x/text v0.3.3
|
|
google.golang.org/grpc => google.golang.org/grpc v1.26.0
|
|
gopkg.in/square/go-jose.v2 => gopkg.in/square/go-jose.v2 v2.2.2
|
|
k8s.io/api => k8s.io/api v0.19.3
|
|
k8s.io/apiextensions-apiserver => k8s.io/apiextensions-apiserver v0.19.3
|
|
k8s.io/apimachinery => k8s.io/apimachinery v0.19.3
|
|
k8s.io/apiserver => k8s.io/apiserver v0.19.3
|
|
k8s.io/cli-runtime => k8s.io/cli-runtime v0.19.3
|
|
k8s.io/client-go => k8s.io/client-go v0.19.3
|
|
k8s.io/cloud-provider => k8s.io/cloud-provider v0.19.3
|
|
k8s.io/cluster-bootstrap => k8s.io/cluster-bootstrap v0.19.3
|
|
k8s.io/code-generator => k8s.io/code-generator v0.19.3
|
|
k8s.io/component-base => k8s.io/component-base v0.19.3
|
|
k8s.io/cri-api => k8s.io/cri-api v0.19.3
|
|
k8s.io/csi-translation-lib => k8s.io/csi-translation-lib v0.19.3
|
|
k8s.io/kube-aggregator => k8s.io/kube-aggregator v0.19.3
|
|
k8s.io/kube-controller-manager => k8s.io/kube-controller-manager v0.19.3
|
|
k8s.io/kube-openapi => k8s.io/kube-openapi v0.0.0-20201106092651-fd18780d1fff
|
|
k8s.io/kube-proxy => k8s.io/kube-proxy v0.19.3
|
|
k8s.io/kube-scheduler => k8s.io/kube-scheduler v0.19.3
|
|
k8s.io/kubectl => k8s.io/kubectl v0.19.3
|
|
k8s.io/kubelet => k8s.io/kubelet v0.19.3
|
|
k8s.io/legacy-cloud-providers => k8s.io/legacy-cloud-providers v0.19.3
|
|
k8s.io/metrics => k8s.io/metrics v0.19.3
|
|
k8s.io/node-api => k8s.io/node-api v0.19.3
|
|
k8s.io/sample-apiserver => k8s.io/sample-apiserver v0.19.3
|
|
k8s.io/sample-cli-plugin => k8s.io/sample-cli-plugin v0.19.3
|
|
k8s.io/sample-controller => k8s.io/sample-controller v0.19.3
|
|
)
|