updating to latest Kubernetes version 1.20.0 fix
security issues. In the current version, it allows
for the token leak in logs when logLevel >= 9.
Signed-off-by: subhamkrai <srai@redhat.com>
Update to the latest lib bucket provisioner code.
Fixes issue 6650
Modifies CRD for objectbucketclaims to fix an additional bug where an
ObjectBucket's 'ClaimRef' is lost due to the CRD validation being
specified incorrectly.
Changes OBC deletion/cleanup to delete the bucket before the user. A
user cannot be deleted without an unsafe purge option if the user has
buckets associated to it.
Does not reintroduce bug 6767 from previous fix for 6650
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
The legacy rgw deployments were removed in 1.0, so there is no more need
for the operator to keep checking for those!
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The operator namespace is not used in the StartOperatorSettingsWatch()
method. Instead, the method looks up the namespace from the
POD_NAMESPACE env var.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Currently, mgr deployment has two overlapping environment variables, ROOK_POD_IP.
This causes kube-apiserver to create error logs
Signed-off-by: binoue <banji-inoue@cybozu.co.jp>
Currently the reconcile of the clusterDisruption controller was triggered mainly due to the
ceph status update in the cephCluster CR. This PR adds reconciles the cluster when:
1. Reconcile when the cluster is created. (This will trigger the first reconcile)
2. Reconcile only when the clusterSpec is updated. (This will avoid triggers when cluster status is updated)
3. Reconcile for events on cephblockpool, cephfilesystem and cephObjectStore.
4. Reconcile for events on Main PDB and when `DisruptionsAllowed` is 0. (that is, when one of the OSD goes down).
5. Reconcile after 30 seconds when there is an active drain going on, that is, pdbStateMap has `draining-failure-domain` as not empty.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
The OSD configuration on local devices was being skipped if a
storageClassDeviceSet was specified. Both types of OSDs should
be allowed in the same cluster, which allows a cluster in the cloud
to also take advantage of local devices with different perf
properties.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The deviceClass property was being ignored when creating the
non-pvc OSDs. Now the deviceClass will be specified as a property
for individual devices, all devices on a node, or all OSDs in the
cluster, depending on the level where the config is applied in the
cluster CR.
Co-authored-by: shenjiatong <yshxxsjt715@gmail.com>
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The arbiter can only be configured with the stretch cluster if the
CRUSH map is balanced and there are two zones in the CRUSH map.
After the OSDs are configured, we wait for all the OSD pods to be
running and that the CRUSH map is balanced. If it takes more than
two minutes, we fail the reconcile and try again. This is only done
the first time the stretch cluster is configured. In future reconciles
we first check if the stretch cluster is already enabled before
enabling it again.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The mon timeouts and pod retries were removed back in v1.1,
this just removes some obsolete variables from the legacy
mon scheduling.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The operator schedules all the mons at the same time, then
serially starts each of the mons. If the operator was restarted
or otherwise failed before all the mons were started, the next
reconcile will schedule new mons again and ignore the scheduling
that was already computed. Since the last mon running is always
expected to be in quorum, we can reset the maxMonId to be
the highest mon currently created.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
If the kernel doesnt support mapping of
rbd image with deep-flatten feature,cephcsi
need to flatten the rbd image first and than
map the image on the node.
cephcsi first tries to add a task to flatten
the rbd image if its receives any permission
error it will try to call rbd CLI command which
is a blocking call.
This commit adds mgr caps to the csi-rbd-node user
so that cephcsi will add flatten task and return
immediate error to the kubelet, let kubelet retry
again,If we go with blocking rbd CLI call we may
end up having stale maps on the node in corner cases.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
The arbiter may have unique requirements for placement settings. Thus, if the arbiter
placement is specified in the cluster CR, that placement will be applied to the arbiter
mon and the other mons will retain their placement from the other mon or all
sections of the placement.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The node usage was used for the obsolete mon selection algorithm, which
was replaced long ago with placement via canary pods.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The TestCephBlockPoolController test is failing intermittently due
to a failure to update the rbd mirroring status on the pool CR.
Improve the test reliability by trying again several times if needed.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
When the cluster is external we want to expose manager service port
along with the endpoints.
This allows us to connect but Rook will keep on using 9283 as a facing
port for Prometheus and more.
```
monitoring:
enabled: true
...
...
externalMgrPrometheusPort: 9283
```
Signed-off-by: Sébastien Han <seb@redhat.com>
If the selector changes on the rgw service, during upgrade the
clients will briefly not be able to connect to the rgw pods.
After all the pods are updated with any new labels, the connections
would be restored, but we want to avoid that temporary outage.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Before cleaning up the cluster, we wait for all the daemon pods to be cleaned up. This fails when
a daemon is in pending state and has no NodeName. This PR ignores daemon pods that are not scheduled on any node.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
We can now collect logs directly into a side-car container.
A new CRD spec has been added:
spec:
logCollector:
enabled: true
periodicity: 24h
Every 24h we will rotate log files for each Ceph daemon.
Signed-off-by: Sébastien Han <seb@redhat.com>
We can now prioritize orchestrations on certain event. Today only two
events will cancel on-going orchestrations (if any):
* request for cluster deletion
* request for cluster upgrade
If one of the two are caught by the watcher we will cancel the on-going
orchestration.
For that we implemented a simple approach based on check points, where
we will check for a cancellation request in certain part of the
orchestration. Mainly before each mons/mgr/osds orchestration loops.
This solution is not perfect, but we are waiting for the
controller-runtime to release its 0.7 version which will embed context
support. With that we will be able to cancel reconciles more precisely
and rapidly.
Operator log example:
```
2020-11-24 13:54:59.499719 I | op-mon: parsing mon endpoints: a=10.109.126.120:6789
2020-11-24 13:54:59.499719 I | op-mon: parsing mon endpoints: a=10.109.126.120:6789
2020-11-25 12:59:12.986264 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph"
2020-11-25 13:07:33.776947 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{
CephVersion: v1.CephVersionSpec{
Image: "ceph/ceph:v15.2.5",
- AllowUnsupported: true,
+ AllowUnsupported: false,
},
DriveGroups: nil,
Storage: {UseAllNodes: true, Selection: {UseAllDevices: &true}},
... // 20 identical fields
}
2020-11-25 13:07:33.777039 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph"
2020-11-25 13:07:33.785088 I | op-mon: parsing mon endpoints: a=10.107.242.49:6789,b=10.109.71.30:6789,c=10.98.93.224:6789
2020-11-25 13:07:33.788626 I | ceph-cluster-controller: detecting the ceph image version for image ceph/ceph:v15.2.5...
2020-11-25 13:07:35.280789 I | ceph-cluster-controller: detected ceph image version: "15.2.5-0 octopus"
2020-11-25 13:07:35.280806 I | ceph-cluster-controller: validating ceph version from provided image
2020-11-25 13:07:35.285888 I | op-mon: parsing mon endpoints: a=10.107.242.49:6789,b=10.109.71.30:6789,c=10.98.93.224:6789
2020-11-25 13:07:35.287828 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config
2020-11-25 13:07:35.288082 I | cephclient: generated admin config in /var/lib/rook/rook-ceph
2020-11-25 13:07:35.621625 I | ceph-cluster-controller: cluster "rook-ceph": version "15.2.5-0 octopus" detected for image "ceph/ceph:v15.2.5"
2020-11-25 13:07:35.642688 I | op-mon: start running mons
2020-11-25 13:07:35.646323 I | op-mon: parsing mon endpoints: a=10.107.242.49:6789,b=10.109.71.30:6789,c=10.98.93.224:6789
2020-11-25 13:07:35.654070 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.107.242.49:6789","10.109.71.30:6789","10.98.93.224:6789"]}] data:a=10.107.242.49:6789,b=10.109.71.30:6789,c=10.98.93.224:6789 mapping:{"node":{"a":{"Name":"minikube","Hostname":"minikube","Address":"192.168.39.3"},"b":{"Name":"minikube","Hostname":"minikube","Address":"192.168.39.3"},"c":{"Name":"minikube","Hostname":"minikube","Address":"192.168.39.3"}}} maxMonId:2]
2020-11-25 13:07:35.868253 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config
2020-11-25 13:07:35.868573 I | cephclient: generated admin config in /var/lib/rook/rook-ceph
2020-11-25 13:07:37.074353 I | op-mon: targeting the mon count 3
2020-11-25 13:07:38.153435 I | op-mon: checking for basic quorum with existing mons
2020-11-25 13:07:38.178029 I | op-mon: mon "a" endpoint is [v2:10.107.242.49:3300,v1:10.107.242.49:6789]
2020-11-25 13:07:38.670191 I | op-mon: mon "b" endpoint is [v2:10.109.71.30:3300,v1:10.109.71.30:6789]
2020-11-25 13:07:39.477820 I | op-mon: mon "c" endpoint is [v2:10.98.93.224:3300,v1:10.98.93.224:6789]
2020-11-25 13:07:39.874094 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.107.242.49:6789","10.109.71.30:6789","10.98.93.224:6789"]}] data:a=10.107.242.49:6789,b=10.109.71.30:6789,c=10.98.93.224:6789 mapping:{"node":{"a":{"Name":"minikube","Hostname":"minikube","Address":"192.168.39.3"},"b":{"Name":"minikube","Hostname":"minikube","Address":"192.168.39.3"},"c":{"Name":"minikube","Hostname":"minikube","Address":"192.168.39.3"}}} maxMonId:2]
2020-11-25 13:07:40.467999 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config
2020-11-25 13:07:40.469733 I | cephclient: generated admin config in /var/lib/rook/rook-ceph
2020-11-25 13:07:41.071710 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config
2020-11-25 13:07:41.078903 I | cephclient: generated admin config in /var/lib/rook/rook-ceph
2020-11-25 13:07:41.125233 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed
2020-11-25 13:07:41.327778 I | op-k8sutil: updating deployment "rook-ceph-mon-a" after verifying it is safe to stop
2020-11-25 13:07:41.327895 I | op-mon: checking if we can stop the deployment rook-ceph-mon-a
2020-11-25 13:07:44.045644 I | op-k8sutil: finished waiting for updated deployment "rook-ceph-mon-a"
2020-11-25 13:07:44.045706 I | op-mon: checking if we can continue the deployment rook-ceph-mon-a
2020-11-25 13:07:44.045740 I | op-mon: waiting for mon quorum with [a b c]
2020-11-25 13:07:44.109159 I | op-mon: mons running: [a b c]
2020-11-25 13:07:44.474596 I | op-mon: Monitors in quorum: [a b c]
2020-11-25 13:07:44.478565 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed
2020-11-25 13:07:44.493374 I | op-k8sutil: updating deployment "rook-ceph-mon-b" after verifying it is safe to stop
2020-11-25 13:07:44.493403 I | op-mon: checking if we can stop the deployment rook-ceph-mon-b
2020-11-25 13:07:47.135524 I | op-k8sutil: finished waiting for updated deployment "rook-ceph-mon-b"
2020-11-25 13:07:47.135542 I | op-mon: checking if we can continue the deployment rook-ceph-mon-b
2020-11-25 13:07:47.135551 I | op-mon: waiting for mon quorum with [a b c]
2020-11-25 13:07:47.148820 I | op-mon: mons running: [a b c]
2020-11-25 13:07:47.445946 I | op-mon: Monitors in quorum: [a b c]
2020-11-25 13:07:47.448991 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed
2020-11-25 13:07:47.462041 I | op-k8sutil: updating deployment "rook-ceph-mon-c" after verifying it is safe to stop
2020-11-25 13:07:47.462060 I | op-mon: checking if we can stop the deployment rook-ceph-mon-c
2020-11-25 13:07:48.853118 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{
CephVersion: v1.CephVersionSpec{
- Image: "ceph/ceph:v15.2.5",
+ Image: "ceph/ceph:v15.2.6",
AllowUnsupported: false,
},
DriveGroups: nil,
Storage: {UseAllNodes: true, Selection: {UseAllDevices: &true}},
... // 20 identical fields
}
2020-11-25 13:07:48.853140 I | ceph-cluster-controller: upgrade requested, cancelling any ongoing orchestration
2020-11-25 13:07:50.119584 I | op-k8sutil: finished waiting for updated deployment "rook-ceph-mon-c"
2020-11-25 13:07:50.119606 I | op-mon: checking if we can continue the deployment rook-ceph-mon-c
2020-11-25 13:07:50.119619 I | op-mon: waiting for mon quorum with [a b c]
2020-11-25 13:07:50.130860 I | op-mon: mons running: [a b c]
2020-11-25 13:07:50.431341 I | op-mon: Monitors in quorum: [a b c]
2020-11-25 13:07:50.431361 I | op-mon: mons created: 3
2020-11-25 13:07:50.734156 I | op-mon: waiting for mon quorum with [a b c]
2020-11-25 13:07:50.745763 I | op-mon: mons running: [a b c]
2020-11-25 13:07:51.045108 I | op-mon: Monitors in quorum: [a b c]
2020-11-25 13:07:51.054497 E | ceph-cluster-controller: failed to reconcile. failed to reconcile cluster "rook-ceph": failed to configure local ceph cluster: failed to create cluster: CANCELLING CURRENT ORCHESTATION
2020-11-25 13:07:52.055208 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph"
2020-11-25 13:07:52.070690 I | op-mon: parsing mon endpoints: a=10.107.242.49:6789,b=10.109.71.30:6789,c=10.98.93.224:6789
2020-11-25 13:07:52.088979 I | ceph-cluster-controller: detecting the ceph image version for image ceph/ceph:v15.2.6...
2020-11-25 13:07:53.904811 I | ceph-cluster-controller: detected ceph image version: "15.2.6-0 octopus"
2020-11-25 13:07:53.904862 I | ceph-cluster-controller: validating ceph version from provided image
```
Closes: https://github.com/rook/rook/issues/6587
Signed-off-by: Sébastien Han <seb@redhat.com>
Rook's crashcollector pod posts entries to the ceph cluster when a crash occurs.
Over time the number cluster may hold crash entries needlessly.
To clean up old crash entries, this PR adds a field to the ceph cluster CR for the user to specify the number of days a crash entry should be kept for.
Providing a value for the field keepXDays creates a cronjob that runs every day at midnight, calling "ceph crash prune <keepXDays>".
Closes: https://github.com/rook/rook/issues/6332
Signed-off-by: Renan Campos <rcampos@redhat.com>
Since the predicate for the CephCluster object will soon move into its
own predidacte we need to export isDoNotReconcile so that it can be
consummed by the "cluster" package.
Signed-off-by: Sébastien Han <seb@redhat.com>
Since we moved to the controller-runtime, events are processed one by
one and so are reconciles. This means we won't have multiple
orchestrations happening at the same time. Thus removing this code.
Also removing one unused variable.
Signed-off-by: Sébastien Han <seb@redhat.com>
We must add the finalizer right after the object creation otherwise the
seerver will later return an error on update that the object has been
modified. Indeed, it has been by the task that updates the status when
the object is first created.
Signed-off-by: Sébastien Han <seb@redhat.com>
subhamkrai
Travis Nielsen
Blaine Gardner
Varsha Rao
binoue
Santosh Pillai
Madhu Rajanna
Blaine Gardner
Sébastien Han
Renan Campos
Arun Kumar Mohan