Commit Graph
566 Commits
Author SHA1 Message Date
Sébastien Han 4e0820f490 ceph: embed error in return
We were missing the error.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-02-15 17:09:37 +01:00
Blaine Gardner a95734160e ceph: create new osds on pvc before update existing
Add a new OSD provisioner status (reported by provisioner configmap)
that denotes that an OSD is preexisting and that OSD prepare does not
need to be run for the OSD. This only applies to OSDs on PVC currently
where existence of a deployment for the OSD indicates no further
provisioning needs to occur. New status is "preexisting".

Allow creating new OSDs on PVCs before updating existing ones by
allowing deferring OSDs during processing of OSD provisioning status
ConfigMaps. Deferral is identified by new "preexisting" status.

Add a unit test to ensure OSD on PVC provisioning occurs as expected
including deferring already-created OSDs to be updated after new PVCs
are created.

Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
2021-02-12 17:38:35 -07:00
Sébastien Han 1f4649f327 ceph: fix wrong payload path
Wrong file was used...

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-02-10 17:01:39 +01:00
Sébastien Han 1adebcd4d1 ceph: expose vault curl errors
If the Vault server returns an error, we must report it in the init
container. With this we can properly debug a failing init container.

Resulting container log:

```
oc logs rook-ceph-osd-0-dc6d4d4d-445dg -c encryption-kms-get-kek                                                                                                                                                                                                         ⏎
['error performing token check: Vault is sealed']%
```

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-02-10 12:45:16 +01:00
Sébastien Han 88cf490f00 ceph: silence the curl call
We don't need to output the progress.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-02-10 11:37:17 +01:00
Travis Nielsen 0dec1babd2 Merge pull request #7170 from leseb/bz-1922954
ceph: return an error if PVC is not clean
2021-02-09 08:01:11 -07:00
Sébastien Han 6dc15eb649 ceph: always override existing block for blkdevmapper
When the blkdevmapper init container initializes we must override any
existing special block. This will prevent re-using a different disk on
the OSD on PVC scenario.

Adding "--remove-destination" to the cp line will remove each existing
destination file before attempting to open it.
We **MUST** do this otherwise in environment where PVCs are dynamic, restarting the deployment will cause conflicts
When restarting the OSD, the PVC block might end up with a different Kernel disk allocation
For instance, prior to restart the block was mapped to 8:32 and when re-attached it was on 8:16
The previous "block" is still 8:32 so if we don't override it we will try to initialize on a disk that is not an OSD or worse another OSD
This is mainly because in https://github.com/rook/rook/commit/ae8dcf7cc3b51cf8ca7da22f48b7a58887536c4f we switched to use HostPath to store the OSD data
Since HostPath is not ephemeral, the block file must be re-hydrated each time the deployment starts

The test uses "==" instead of numbers since the stat command will print
an hex number and thus we won't always get integer, e,g: major/minor
value 202:13312 will be return ca:3400.

Example run of the new init container:

+ PVC_SOURCE=/set1-data-168h7m
+ PVC_DEST=/var/lib/ceph/osd/ceph-0/block
+ CP_ARGS=(--archive --dereference --verbose)
+ '[' -b /var/lib/ceph/osd/ceph-0/block ']'
++ stat -c %t%T /set1-data-168h7m
+ PVC_SOURCE_MAJ_MIN=ca6100
++ stat -c %t%T /var/lib/ceph/osd/ceph-0/block
PVC's source major/minor numbers changed
removed '/var/lib/ceph/osd/ceph-0/block'
'/set1-data-168h7m' -> '/var/lib/ceph/osd/ceph-0/block'
+ PVC_DEST_MAJ_MIN=ca4000
+ [[ ca6100 == \c\a\4\0\0\0 ]]
+ echo 'PVC'\''s source major/minor numbers changed'
+ CP_ARGS+=(--remove-destination)
+ cp --archive --dereference --verbose --remove-destination /set1-data-168h7m /var/lib/ceph/osd/ceph-0/block

Test procedure:

$ oc -n rook-ceph scale deployment rook-ceph-osd-0 --replicas=2
deployment.apps/rook-ceph-osd-0 scaled

Now we have 2 OSDs and one is crashlooping, which is expected:

rook-ceph-osd-0-66964f6d-2cwqq                              0/1     Init:CrashLoopBackOff   4          2m12s
rook-ceph-osd-0-66964f6d-pmhlq                              1/1     Running                 0          6m7s

The logs from the failing container is pretty clear:

+ PVC_SOURCE=/set1-data-168h7m
+ PVC_DEST=/var/lib/ceph/osd/ceph-0/block
+ CP_ARGS=(--archive --dereference --verbose)
+ '[' -b /var/lib/ceph/osd/ceph-0/block ']'
++ stat -c %t%T /set1-data-168h7m
+ PVC_SOURCE_MAJ_MIN=ca6100
++ stat -c %t%T /var/lib/ceph/osd/ceph-0/block
+ PVC_DEST_MAJ_MIN=ca6100
+ [[ ca6100 == \c\a\6\1\0\0 ]]
+ CP_ARGS+=(--no-clobber)
+ cp --archive --dereference --verbose --no-clobber /set1-data-168h7m /var/lib/ceph/osd/ceph-0/block
inferring bluefs devices from bluestore path
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.8/rpm/el8/BUILD/ceph-15.2.8/src/os/bluestore/BlueStore.cc: In function 'int BlueStore::expand_devices(std::ostream&)' thread 7f2a52d18240 time 2021-02-09T13:27:46.011397+0000
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.8/rpm/el8/BUILD/ceph-15.2.8/src/os/bluestore/BlueStore.cc: 7090: FAILED ceph_assert(r == 0)
2021-02-09T13:27:46.010+0000 7f2a52d18240 -1 bluestore(/var/lib/ceph/osd/ceph-0) _lock_fsid failed to lock /var/lib/ceph/osd/ceph-0/fsid (is another ceph-osd still running?)(11) Resource temporarily unavailable
 ceph version 15.2.8 (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus (stable)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f2a48e236e4]

 Acquiring the lock fails, so the OSD won't run.
 Now we scale back down:

$ oc -n rook-ceph scale deployment rook-ceph-osd-0 --replicas=1
deployment.apps/rook-ceph-osd-0 scaled

And we can see the last created OSD (the one crash looping) disappears:

rook-ceph-osd-0-66964f6d-2cwqq                              0/1     Terminating   5          3m6s
rook-ceph-osd-0-66964f6d-pmhlq                              1/1     Running       0          7m1s

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-02-09 14:28:39 +01:00
Sébastien Han 7c928f4196 ceph: return an error if PVC is not clean
With 5f57b6fd3a, we introduced a
regression in the validation code.
If the OSD PV comes from another cluster, GetCephVolumeRawOSDs must
return an error.
The prepare job will crash loop but at least with a useful and
comprehensive message.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-02-09 11:28:08 +01:00
Santosh Pillai f5808185e0 ceph: check for orchestration cancellation while OSDs are pending
completeOSDsForAllNodes() method waits for 20 minutes for all the OSDs to start on all the nodes. This PR checks if
cluster deletion request was sent while waiting for OSDs to start. If yes, then completeOSDsForAllNodes() method will returnwithout waiting for 20 minutes.

Signed-off-by: Santosh Pillai <sapillai@redhat.com>
2021-02-04 14:49:37 +05:30
Sébastien Han 8b65eb6126 ceph: do not enable ShareProcessNamespace if HostPID
If HostPID is enabled, this means the Ceph version is older than Octopus
and we are still running OSD with HostPID due to an internal
inter-process check.
If the log collector is enabled we need to enable ShareProcessNamespace
**only** if HostIPC is disabled.

Both HostPID and ShareProcessNamespace cannot be enabled.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-02-01 18:23:19 +01:00
Sébastien Han 260274cdb3 ceph: do not enable ShareProcessNamespace if HostIPC
If HostIPC is enabled, this means the Ceph version is older than Octopus
and we are still running OSD with HostIPC due to an internal
inter-process check.
If the log collector is enabled we need to enable ShareProcessNamespace
**only** if HostIPC is disabled.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-02-01 15:50:16 +01:00
Blaine Gardner 2caeb8bfe3 Merge pull request #6833 from sp98/configure-osd-upgrade-timeout
ceph: configure health check timeout for OSDs when upgrading
2021-01-28 09:46:12 -07:00
Santosh Pillai 4237aa7ecc ceph: configure health check timeout for OSDs when upgrading
added waitTimeoutForHealthyOSD in the cluster crd that defines the time (in minutes) the operator would wait before an OSD can be stopped for
upgrade. This PR also removes the ok-to-continue logic for osds as its already handled by ok-to-stop. The default value is 10 minutes.

Signed-off-by: Santosh Pillai <sapillai@redhat.com>
2021-01-27 11:13:05 +05:30
Travis Nielsen 570cba9b4e ceph: consistently order the osd args to avoid restarts
The settings for tuneDeviceClass and tuneFastDeviceClass were
stored as a map[string]string. When iterating over the map to
append the args, the args were being appended in an unpredictable
order. The order could change every reconcile, resulting in an
unnecessary OSD restart potentially every reconcile. Now we iterate
over a slice that will be ordered predictably and ensure OSDs
will not be restarted when not needed.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-01-26 17:23:01 -07:00
Travis Nielsen 4b43e7afa5 ceph: allow removal of arbitrary osds on pvcs and simplify pvc names
OSDs on PVCs previously were always replaced with PVCs of a given name.
For on-prem scenarios where the OSDs might need to be removed instead
of replaced, the operator now allows holes to exist in the PVC index
names as long as there are a sufficient number of PVCs to meet the
criteria for the deviceSet.count.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-01-26 14:38:12 -07:00
Jiffin Tony Thottan ee6a03dfb8 ceph: minor update init container script in generateVaultGetKEK
Replacing VAULT_KV_VERS with VAULT_BACKEND for specifing version of kv engine.

Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
2021-01-22 16:04:35 +05:30
Sébastien Han 093ed7dd62 Merge pull request #6984 from leseb/core-dumps-dir
ceph: set process working dir to /var/log/ceph
2021-01-20 10:39:42 +01:00
Satoru Takeuchi ae8dcf7cc3 ceph: fix osd corruption
Multiple OSD pods for the same OSD might run simultaneously becasue OSD pod
is managed by Deployment resource. However, OSD locking mechanism
doens't work because the lock file (fsid file) exist for each OSD pod.
It resutls in OSD corruption.

Closes: https://github.com/rook/rook/issues/6530

Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
2021-01-18 03:11:50 +00:00
shenjiatong 0d80cc396d ceph: exposing which node/osd is processing
it would be more firendly to cloud operators to
know which node is being processed during
upgrade

Signed-off-by: shenjiatong <yshxxsjt715@gmail.com>
2021-01-15 23:12:26 +08:00
Sébastien Han cb7d600841 ceph: set process working dir to /var/log/ceph
By setting the working directory of a Ceph daemon to the log direcor
(which is bindmounted to the host), we can ensure that the coredumps
will be available.

On CentOS 7 **only**, the kernel is configured with:

```
cat /proc/sys/kernel/core_pattern
core
```

which means that the coredumps will end up in the process working
directory and will be named "core".

On CentOS 8, everything is different and this won't work but still
CentOS will be able to consume this and changing the working directory
is harmless.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-01-15 16:06:42 +01:00
Sébastien Han 7558d37420 core: bump to controller-runtime 0.7.0 version
Now using https://github.com/kubernetes-sigs/controller-runtime/releases/tag/v0.7.0

Closes: https://github.com/rook/rook/issues/6689
Signed-off-by: Sébastien Han <seb@redhat.com>
2021-01-13 11:00:43 +01:00
subhamkrai 9109cc6b8c ceph: apply OSD resources and placement for device sets
`.spec.resources.osd` resources and `placement: all:` placements
parameters for the OSD pod did not reflect the value like
it has for other pods mon,mgr.

this commit will allow to set those parameters for the OSD pod also.

Signed-off-by: subhamkrai <srai@redhat.com>
2021-01-08 18:05:21 +05:30
binoue 2ecde7c9c6 ceph: remove deplicated env value
Currently, mgr deployment has two overlapping environment variables, ROOK_POD_IP.
This causes kube-apiserver to create error logs

Signed-off-by: binoue <banji-inoue@cybozu.co.jp>
2020-12-15 09:11:26 +09:00
Travis Nielsen 27b0295ac1 Merge pull request #6815 from travisn/osd-deviceclass
ceph: Apply deviceClass properly to new osds
2020-12-14 13:13:55 -07:00
Travis Nielsen 3ca3d7783b ceph: allow OSDs on both pvcs and devices
The OSD configuration on local devices was being skipped if a
storageClassDeviceSet was specified. Both types of OSDs should
be allowed in the same cluster, which allows a cluster in the cloud
to also take advantage of local devices with different perf
properties.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-12-11 14:21:47 -07:00
Travis Nielsenandshenjiatong a0a27bc426 ceph: apply deviceClass properly to new osds
The deviceClass property was being ignored when creating the
non-pvc OSDs. Now the deviceClass will be specified as a property
for individual devices, all devices on a node, or all OSDs in the
cluster, depending on the level where the config is applied in the
cluster CR.

Co-authored-by: shenjiatong <yshxxsjt715@gmail.com>
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-12-11 10:16:00 -07:00
Sébastien Han c6a87203ca ceph: add log collector
We can now collect logs directly into a side-car container.
A new CRD spec has been added:

spec:
  logCollector:
    enabled: true
    periodicity: 24h

Every 24h we will rotate log files for each Ceph daemon.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-12-01 16:30:17 +01:00
Sébastien Han ad24990473 ceph: ability to abort orchestration
We can now prioritize orchestrations on certain event. Today only two
events will cancel on-going orchestrations (if any):

* request for cluster deletion
* request for cluster upgrade

If one of the two are caught by the watcher we will cancel the on-going
orchestration.
For that we implemented a simple approach based on check points, where
we will check for a cancellation request in certain part of the
orchestration. Mainly before each mons/mgr/osds orchestration loops.

This solution is not perfect, but we are waiting for the
controller-runtime to release its 0.7 version which will embed context
support. With that we will be able to cancel reconciles more precisely
and rapidly.

Operator log example:

```
2020-11-24 13:54:59.499719 I | op-mon: parsing mon endpoints: a=10.109.126.120:6789
2020-11-24 13:54:59.499719 I | op-mon: parsing mon endpoints: a=10.109.126.120:6789
2020-11-25 12:59:12.986264 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph"
2020-11-25 13:07:33.776947 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff=  v1.ClusterSpec{
        CephVersion: v1.CephVersionSpec{
                Image:            "ceph/ceph:v15.2.5",
-               AllowUnsupported: true,
+               AllowUnsupported: false,
        },
        DriveGroups: nil,
        Storage:     {UseAllNodes: true, Selection: {UseAllDevices: &true}},
        ... // 20 identical fields
  }
2020-11-25 13:07:33.777039 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph"
2020-11-25 13:07:33.785088 I | op-mon: parsing mon endpoints: a=10.107.242.49:6789,b=10.109.71.30:6789,c=10.98.93.224:6789
2020-11-25 13:07:33.788626 I | ceph-cluster-controller: detecting the ceph image version for image ceph/ceph:v15.2.5...
2020-11-25 13:07:35.280789 I | ceph-cluster-controller: detected ceph image version: "15.2.5-0 octopus"
2020-11-25 13:07:35.280806 I | ceph-cluster-controller: validating ceph version from provided image
2020-11-25 13:07:35.285888 I | op-mon: parsing mon endpoints: a=10.107.242.49:6789,b=10.109.71.30:6789,c=10.98.93.224:6789
2020-11-25 13:07:35.287828 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config
2020-11-25 13:07:35.288082 I | cephclient: generated admin config in /var/lib/rook/rook-ceph
2020-11-25 13:07:35.621625 I | ceph-cluster-controller: cluster "rook-ceph": version "15.2.5-0 octopus" detected for image "ceph/ceph:v15.2.5"
2020-11-25 13:07:35.642688 I | op-mon: start running mons
2020-11-25 13:07:35.646323 I | op-mon: parsing mon endpoints: a=10.107.242.49:6789,b=10.109.71.30:6789,c=10.98.93.224:6789
2020-11-25 13:07:35.654070 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.107.242.49:6789","10.109.71.30:6789","10.98.93.224:6789"]}] data:a=10.107.242.49:6789,b=10.109.71.30:6789,c=10.98.93.224:6789 mapping:{"node":{"a":{"Name":"minikube","Hostname":"minikube","Address":"192.168.39.3"},"b":{"Name":"minikube","Hostname":"minikube","Address":"192.168.39.3"},"c":{"Name":"minikube","Hostname":"minikube","Address":"192.168.39.3"}}} maxMonId:2]
2020-11-25 13:07:35.868253 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config
2020-11-25 13:07:35.868573 I | cephclient: generated admin config in /var/lib/rook/rook-ceph
2020-11-25 13:07:37.074353 I | op-mon: targeting the mon count 3
2020-11-25 13:07:38.153435 I | op-mon: checking for basic quorum with existing mons
2020-11-25 13:07:38.178029 I | op-mon: mon "a" endpoint is [v2:10.107.242.49:3300,v1:10.107.242.49:6789]
2020-11-25 13:07:38.670191 I | op-mon: mon "b" endpoint is [v2:10.109.71.30:3300,v1:10.109.71.30:6789]
2020-11-25 13:07:39.477820 I | op-mon: mon "c" endpoint is [v2:10.98.93.224:3300,v1:10.98.93.224:6789]
2020-11-25 13:07:39.874094 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.107.242.49:6789","10.109.71.30:6789","10.98.93.224:6789"]}] data:a=10.107.242.49:6789,b=10.109.71.30:6789,c=10.98.93.224:6789 mapping:{"node":{"a":{"Name":"minikube","Hostname":"minikube","Address":"192.168.39.3"},"b":{"Name":"minikube","Hostname":"minikube","Address":"192.168.39.3"},"c":{"Name":"minikube","Hostname":"minikube","Address":"192.168.39.3"}}} maxMonId:2]
2020-11-25 13:07:40.467999 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config
2020-11-25 13:07:40.469733 I | cephclient: generated admin config in /var/lib/rook/rook-ceph
2020-11-25 13:07:41.071710 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config
2020-11-25 13:07:41.078903 I | cephclient: generated admin config in /var/lib/rook/rook-ceph
2020-11-25 13:07:41.125233 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed
2020-11-25 13:07:41.327778 I | op-k8sutil: updating deployment "rook-ceph-mon-a" after verifying it is safe to stop
2020-11-25 13:07:41.327895 I | op-mon: checking if we can stop the deployment rook-ceph-mon-a
2020-11-25 13:07:44.045644 I | op-k8sutil: finished waiting for updated deployment "rook-ceph-mon-a"
2020-11-25 13:07:44.045706 I | op-mon: checking if we can continue the deployment rook-ceph-mon-a
2020-11-25 13:07:44.045740 I | op-mon: waiting for mon quorum with [a b c]
2020-11-25 13:07:44.109159 I | op-mon: mons running: [a b c]
2020-11-25 13:07:44.474596 I | op-mon: Monitors in quorum: [a b c]
2020-11-25 13:07:44.478565 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed
2020-11-25 13:07:44.493374 I | op-k8sutil: updating deployment "rook-ceph-mon-b" after verifying it is safe to stop
2020-11-25 13:07:44.493403 I | op-mon: checking if we can stop the deployment rook-ceph-mon-b
2020-11-25 13:07:47.135524 I | op-k8sutil: finished waiting for updated deployment "rook-ceph-mon-b"
2020-11-25 13:07:47.135542 I | op-mon: checking if we can continue the deployment rook-ceph-mon-b
2020-11-25 13:07:47.135551 I | op-mon: waiting for mon quorum with [a b c]
2020-11-25 13:07:47.148820 I | op-mon: mons running: [a b c]
2020-11-25 13:07:47.445946 I | op-mon: Monitors in quorum: [a b c]
2020-11-25 13:07:47.448991 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed
2020-11-25 13:07:47.462041 I | op-k8sutil: updating deployment "rook-ceph-mon-c" after verifying it is safe to stop
2020-11-25 13:07:47.462060 I | op-mon: checking if we can stop the deployment rook-ceph-mon-c
2020-11-25 13:07:48.853118 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff=  v1.ClusterSpec{
        CephVersion: v1.CephVersionSpec{
-               Image:            "ceph/ceph:v15.2.5",
+               Image:            "ceph/ceph:v15.2.6",
                AllowUnsupported: false,
        },
        DriveGroups: nil,
        Storage:     {UseAllNodes: true, Selection: {UseAllDevices: &true}},
        ... // 20 identical fields
  }
2020-11-25 13:07:48.853140 I | ceph-cluster-controller: upgrade requested, cancelling any ongoing orchestration
2020-11-25 13:07:50.119584 I | op-k8sutil: finished waiting for updated deployment "rook-ceph-mon-c"
2020-11-25 13:07:50.119606 I | op-mon: checking if we can continue the deployment rook-ceph-mon-c
2020-11-25 13:07:50.119619 I | op-mon: waiting for mon quorum with [a b c]
2020-11-25 13:07:50.130860 I | op-mon: mons running: [a b c]
2020-11-25 13:07:50.431341 I | op-mon: Monitors in quorum: [a b c]
2020-11-25 13:07:50.431361 I | op-mon: mons created: 3
2020-11-25 13:07:50.734156 I | op-mon: waiting for mon quorum with [a b c]
2020-11-25 13:07:50.745763 I | op-mon: mons running: [a b c]
2020-11-25 13:07:51.045108 I | op-mon: Monitors in quorum: [a b c]
2020-11-25 13:07:51.054497 E | ceph-cluster-controller: failed to reconcile. failed to reconcile cluster "rook-ceph": failed to configure local ceph cluster: failed to create cluster: CANCELLING CURRENT ORCHESTATION
2020-11-25 13:07:52.055208 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph"
2020-11-25 13:07:52.070690 I | op-mon: parsing mon endpoints: a=10.107.242.49:6789,b=10.109.71.30:6789,c=10.98.93.224:6789
2020-11-25 13:07:52.088979 I | ceph-cluster-controller: detecting the ceph image version for image ceph/ceph:v15.2.6...
2020-11-25 13:07:53.904811 I | ceph-cluster-controller: detected ceph image version: "15.2.6-0 octopus"
2020-11-25 13:07:53.904862 I | ceph-cluster-controller: validating ceph version from provided image
```

Closes: https://github.com/rook/rook/issues/6587
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-11-25 15:32:04 +01:00
Sébastien Han f86f8300f9 Merge pull request #6568 from aruniiird/upgrade-dependencies-for-operator-sdk-1.x
Bump Controller Runtime version to 0.6
2020-11-19 17:45:18 +01:00
Arun Kumar Mohan 65d16bfc94 ceph: manual changes needed for kubernetes api updates
Fetched latest lib-bucket-provisioner changes as well.

Signed-off-by: Arun Kumar Mohan <amohan@redhat.com>
2020-11-18 21:14:01 +05:30
Santosh Pillai 8602b9c116 ceph: osd pdb reconciler changes
-creates a single PDB (max-unavailable=1) for all OSDs.  This PDB allows one OSD to go down at a given time.
-When a drain is detected, blocking PDBs (max-unavailable=0) will be created for each failure domain that is not being drained and the main PDB (max-unavilable=1) will be deleted. This will allow all the OSDs in the currently drained failure domain to be removed while blocking the deletion  of OSDs in other failure domains.
-Once the PGs are healthy again, the blocking PDBs will be deleted and the main PDB will be restored.
-Add PG healthcheck timeout
-Delete any legacy node drain pods and blocking OSD PDBs

Signed-off-by: Santosh Pillai <sapillai@redhat.com>
2020-11-18 18:25:07 +05:30
Sébastien Han e3d032dbc1 Merge pull request #6489 from travisn/stretch-mons
ceph: Stretch cluster configuration
2020-11-06 15:16:39 +01:00
Mateusz Gozdek 8ba3762fa4 docs: fix bunch of typos
Found by running the following command:

codespell -S .git,*.png,*.jpg -L \
aks,keyserver,atleast,dne,ser,ist,files\',ba,dum,iam,te -f -H

Signed-off-by: Mateusz Gozdek <mgozdekof@gmail.com>
2020-11-06 10:01:04 +01:00
Travis Nielsen 7c3cdbce50 ceph: use constants for k8s topology labels
Instead of directly defining our own constants, we should be using the k8s
constants for the well-known topology labels.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-11-05 09:23:58 -07:00
Travis Nielsen b91f4211c9 ceph: configure a stretched cluster
In clusters where only two datacenters (or similar failure domains)
are available, a different mon and osd approach is needed to deal
with the network partitions or some other reason for one of the failure
domains going down. The Ceph stretched cluster makes the mons aware
of the failure domains by configuring one as the arbiter in a third
zone, while keeping two replicas of the data in each of the data
zoens.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-11-05 09:17:52 -07:00
Sébastien Han b477d9c33d ceph: add support for VAULT_TLS_SERVER_NAME
We now support VAULT_TLS_SERVER_NAME which represents the name to use as
the SNI host when connecting via TLS.
This is typically needed when a single host has multiple certificates,
same as virtual hosts but for SSL certificates.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-11-03 14:59:15 +01:00
Sébastien Han 032e6b5d77 ceph: remove osd pvc when the cluster is gone
Just like we do for the monitors, we set an ownerref to the OSD PVC so
when the cluster is deleted we remove the PVC too.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-10-30 16:17:15 +01:00
Sébastien Han ea1d71cbfb ceph: add vault kms support for osd encryption
When the Ceph cluster runs on PVC and the OSDs are encrypted we can
store LUKS's Key Encryption Key inside a Key Management System. Today,
Rook only supports HashiCorp Vault: https://www.vaultproject.io/

The CephCluster has now a new "security" field which will plug onto the
KMS. Here is an example:

security:
  kms:
    tokenSecretName: <name of the secret containing a Vault token, used
    to authenticate>
    connectionDetails: < a map of strings containing connection
    information>

Refer to the ceph-cluster-crd documentation to lear more.

Closes: https://github.com/rook/rook/issues/6105
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-10-30 16:16:33 +01:00
Jonas Schäfer 7ace6ac255 ceph: make root= CRUSH label value configurable
The CRUSH map does not allow duplicate values (nor keys) for the
labels. That means that if the currently hardcoded label value
`default` occurs anywhere else in the tree (for example, because
some of the topology labels provided by the cloud provider use it),
the cluster cannot sucessfully spawn.

By giving the user control over the label value used for the root
CRUSH map label, it is possible to adapt the cluster to this
situation.

To be able to actually use the `default` value, the root=default
node which is created by default by ceph itself has to be removed;
since that node is accompanied by a replicated crush rule, we have
to remove that rule, too.

The integration tests are modified to sometimes use the custom
crushRoot (based on an arbitrary criterium) to get coverage across
the various suites. Separate tests could be added, but were not
deemed necessary at this point.

Fixes #4993.

Signed-off-by: Jonas Schäfer <jonas.schaefer@cloudandheat.com>
2020-10-29 09:22:41 +01:00
subhamkrai cb0ca66a6b ci: enable gosec linter in golangci-lint
golangci-lint linter gosec showing more errors
than gosec gh. This commit resolve
new errors. And, removing
nosec comments from autogenerated files.

Signed-off-by: subhamkrai <srai@redhat.com>
2020-10-19 11:50:06 +05:30
Sébastien Han b49b6393bc ceph: tune osd tcmalloc thread cache bytes
We can now tune the value of TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES by
setting an annotation on the CephCluster CRD like so:

```
annotation:
  osd:
    TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES: "134217728"
```

Alternatively, if nothing is specified Rook will read the value of the
file `/etc/sysconfig/ceph`.

Closes: https://github.com/rook/rook/issues/6134
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-10-07 18:49:18 +02:00
subhamkrai 364df7793f ceph: remove suffix _sdd for tuneFastDeviceClass
this commit remove _ssd  suffix from
tuneFastDeviceClass setting as with _ssd
won't affect disk.

Signed-off-by: subhamkrai <srai@redhat.com>
2020-10-06 20:54:52 +05:30
Travis Nielsen 3eddcf5477 Merge pull request #6283 from sp98/rook-IPv6-support
ceph: support IPv6 single-stack
2020-10-06 07:46:25 -06:00
subhamkrai c2c59ec47b ceph: tune fast device class for OSD on PVC in the Azure
on Azure, all SSDs are reporting as a rotational disk.
This commit will add `tuneFastDeviceClass` flag which will
tune for an SSD even if the cloud provider is reporting
rotational. We have a flag `tuneDeviceClass`
which works vice-versa.

Signed-off-by: subhamkrai <srai@redhat.com>
2020-10-05 20:19:55 +05:30
Sébastien Han 95ac992e48 ceph: check underlaying block status
During the initialization encrypted OSD initialization sequence, we check
for the presence of the encrypted container. If it exists we don't try
  to open it again since this will result in an error.
However, there is another case we need to handle, when the underlying
device is gone. For instance, if the pod/PV couple was drained and move
back, living the orphan dm. Once the pod comes back, the dm is still
present and perfectly matches. Unfortunately, the underlying disk is
different and thus the dm must be removed and the disk re-opened.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-10-05 15:47:29 +02:00
Santosh Pillai 70c5f70701 ceph: support IPv6 single-stack for ceph
Add --ms-bind-ipv6 true args to mom, osd, mgr, object, mds, rbd daemons.

Signed-off-by: Santosh Pillai <sapillai@redhat.com>
2020-10-02 01:03:03 +05:30
subhamkrai 0fddfcf307 ceph: handle golangci-lint linter errcheck error
this commit handle golangci-lint linter errcheck.

`errcheck` - Errcheck is a program for checking for
unchecked errors in go programs. These unchecked errors
can be critical bugs in some cases

To see only staticcheck linter output
`golangci-lint run --disable-all -E errcheck`

Signed-off-by: subhamkrai <srai@redhat.com>
2020-09-30 22:24:34 +05:30
Sébastien Han 61b2c6186d Merge pull request #6350 from leseb/fix-luks2-resize
ceph: disable keyring after opening encrypted block
2020-09-30 16:42:51 +02:00
Sébastien Han a7e78858ce ceph: disable keyring after opening encrypted block
On LUKS2 format, the resize operation requires the KEK to be passed. To
maintain a single simple init container for resize, we now open the
encrypted block with  "--disable-keyring" so that further operations
won't require the keyring.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-09-30 15:53:36 +02:00
Travis Nielsen caa51e4cb8 ceph: detect osd deployments on pvcs in a batch
OSD prepare jobs only need to be run if the OSD deployments on PVCs were
not already started. For efficiency, we now query them in a batch instead
of querying them one-by-one with a label selector, which proves to be very
slow in large clusters.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-09-28 12:52:17 -06:00