Commit Graph
445 Commits
Author SHA1 Message Date
Sébastien Han 3b5bbf28d5 Merge pull request #9003 from leseb/vault-k8s-auth
osd: add support for k8s with vault kms
2021-10-25 18:02:48 +02:00
Blaine Gardner 2f850b6ae6 Merge pull request #8613 from subhamkrai/remove-nautilus
ceph: remove ceph nautilus, ceph octopus to default
2021-10-25 09:09:18 -06:00
Yuichiro Ueno 3fd86f83ae core: add context parameter to opcontroller
This commit adds context parameter to utilities in opcontroller to
remove context.TODO use in opcontroller. By this, we can handle
cancellation of reconcilers in a fine-grained way.

Signed-off-by: Yuichiro Ueno <y1r.ueno@gmail.com>
2021-10-25 20:45:06 +09:00
Sébastien Han 18a4047679 osd: add support for k8s with vault kms
Rook cluster-wide encryption can now use the native Kubernetes
authentication to interact with vault KMS instead of using the token
method.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-10-21 13:59:28 +02:00
subhamkrai 0150966024 ceph: remove ceph nautilus, ceph octopus to default
since rook 1.8, ceph nautilus no longer supported,
ceph octopus will be the minimum ceph version.

Closes: https://github.com/rook/rook/issues/7908
Signed-off-by: subhamkrai <srai@redhat.com>
2021-10-20 14:45:12 +05:30
Sébastien Han 4017f94464 osd: use rook binary to fetch key encryption key
When deploying a cluster-wde encrypted cluster we now use the rook
binary to execute some code to fetch the key encryption key.

Using cURL all the time to fetch the key has its limitations. The
incoming integration with Kubernetes Authentication through service
accounts is leading the usage of cURL to its end.
The logic is really complex and prone to errors. Re-implementing the
Go logic into Bash is not a viable option.

Also, using the lib in a binary allows us to keep a consistent behavior
throughout the life cycle of our code.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-10-19 17:29:25 +02:00
Sébastien Han 121c2987e3 ceph: stop using tini
We don't need to use tini.
We don't have anything in the rook operator that would
either create zombie processes (no threads) or use
exec (to fork). The Go binary has a really good
signal handling mechanism.

Closes: https://github.com/rook/rook/issues/8794
Signed-off-by: Sébastien Han <seb@redhat.com>
2021-09-27 10:54:16 +02:00
Travis Nielsen 0a0b9c98bd build: remove obsolete flex driver
The flex driver has been fully deprecated and thus removed from Rook.
Before upgrading to v1.8, users will need to convert existing flex volumes
from flex to csi volumes.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-09-23 16:17:20 -06:00
Sébastien Han b89730d895 ceph: refactor operator initialization sequence
This commit is a large refactor on how the operator starts, stops and
how it starts various sub-components such as the ceph-csi driver. It
also refines the way we cancel orchestrations. We don't use breakpoints
anymore but send our self a SIGUP to reload our controller runtime
manager.
The reload will happen under different circonstances like:

* a new adminission controller secret is created/deleted/changed
* a CephCluster CR is edited

As mentioned earlier, the csi driver now has its own controller, just
like flex. It reacts to change in the operator config map for particular
ROOK_CSI_ fields.

A second new controller for the operator's general config has been
created, it manages:

* the logging level
* the ceph CLI command timeout
* the discovery daemon

The operator reacts much more rapidly to cancellation events by stopping
the manager's context and reloading it.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-09-17 16:57:12 +02:00
Hiroya Onoe 2ff5413b75 ceph: fix error message in UpdateNodeStatus
The error message in UpdateNodeStatus regards the second argument
as node name. However, it is a PVC name in OSD on PVC.

Signed-off-by: Hiroya Onoe <onoehiroya@gmail.com>
2021-09-02 02:43:16 +00:00
Travis Nielsen 42717d2fca Merge pull request #8566 from subhamkrai/merge-tolerations
ceph: merge toleration for osd/prepareOSD pod if specified both places
2021-08-26 08:55:27 -06:00
subhamkrai e1f232eeb7 ceph: merge toleration for osd/prepareOSD pod if specified both places
earlier, `ApplyToPodSpec()` was only taking one toleration and ignoring
tolerations from `placement.ALL()`.

this commit merge toleration for Mgr,Mon,Osd pod
example, for osd it will merge
spec.placement.all and
storageDeviceClassSets.Placement(in case of pvc) or
spec.placement.osd(in case of non-pvc's).

Signed-off-by: subhamkrai <srai@redhat.com>
2021-08-25 19:34:28 +05:30
parth-gr 77ba1ef11b core: convert util.NewSet() to sets.NewString()
Converting util.NewSet() instance to use sets.NewString() instance

Closes: https://github.com/rook/rook/issues/8479
Signed-off-by: parth-gr <paarora@redhat.com>
2021-08-25 13:13:05 +05:30
parth-gr 1601ba1c1c core: convert util.NewSet() to sets.NewString()
Converting util.NewSet() instance to use sets.NewString() instance

Closes: https://github.com/rook/rook/issues/8479
Signed-off-by: parth-gr <paarora@redhat.com>
2021-08-20 18:56:25 +05:30
subhamkrai 9838313398 ceph: add an option to enable/disable merge all placement
earlier, we had the issue for osd placement regarding
whether we should merge the placements or not.

Now, we have option `skipApplyAllPlacement` to
enable/disable osd placement. By default it is false
which means it will merge both placement.All() and
storageClassDeviceSets.Placement, when true
it will only apply storageClassDeviceSets.Placement.

Also, adding unit test.

Closes: https://github.com/rook/rook/issues/8135
Signed-off-by: subhamkrai <srai@redhat.com>
2021-08-03 19:47:40 +05:30
Sébastien Han d55fb8d13b Merge pull request #8351 from leseb/refact-exec-helpers
ceph: remove unnecessary exec helpers
2021-07-23 18:12:57 +02:00
Sébastien Han 6d77a9976c ceph: remove unnecessary exec helpers
Both `ExecuteCommandWithOutputFileTimeout()` and
`ExecuteCommandWithOutputFile()` generate unnecessary system calls by
creating/reading/removing files where the stream output of the command
can simply be used. So sticking with `ExecuteCommandWithOutput()` and
`ExecuteCommandWithCombinedOutput()` for reading outputs is sufficient.

Closes: https://github.com/rook/rook/issues/8343
Signed-off-by: Sébastien Han <seb@redhat.com>
2021-07-23 09:16:33 +02:00
Sébastien Han 6bce1ff3e9 ceph: move all of our docker.io reference to quay.io
Recently, the builds of `ceph/ceph` image moved to quay.io, see
https://github.com/ceph/ceph-build/pull/1883 for more details.
Current images will remain but new builds will happen on quay.io only.

This means that tags such as `v14.2`, `v15.2`,`v16.2` will need to
switch to quay.io to get updates.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-07-22 11:17:05 +02:00
subhamkrai d254c935e8 ceph: create new keyring for osd
when osd pods removed, they are not able
to add back due to missing `ceph auth`
or different `ceph auth`.

this commit mounts the secret `rook-ceph-admin-keyring`
inside the osd activate initcontainer which has
admin keyring.

Signed-off-by: subhamkrai <srai@redhat.com>
2021-07-19 19:26:06 +05:30
Satoru Takeuchi 763b076c75 ceph: mount dev directory in lvm mode osd on osd on lv-backed pvc
"/dev/" is only mounted on OSD on device. However, this directory
should also be mounted on OSD on LV-backed PVC.

Closes: https://github.com/rook/rook/issues/8264

Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
2021-07-13 12:21:41 +00:00
Blaine Gardner c22f545ebf ceph: block delete object store when buckets exist
Block deletion of CephObjectStore resources when buckets exist in the
object store.

Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
2021-06-29 14:31:39 -06:00
Sébastien Han bcd2511a72 ceph: do not leak key encryption key
When evaluating the potential "warning" key from the json output we were
not hiding stdout (1) from the output, so the key was leaked. Also the
warning check for the key was not perfect, it only checked for the
command to succeed, but we should introspect it since the "warning" key
is always present unlike the "error" one.
We now only print the warning or the error is present.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-06-22 15:04:13 +02:00
Sébastien Han e6929ce308 Merge pull request #7913 from abursavich/extract-kk
Remove dependency on k8s.io/kubernetes
2021-06-22 09:22:41 +02:00
Andy Bursavich 11f0add706 build: update golangci-lint version and nolint comment format
Signed-off-by: Andy Bursavich <abursavich@gmail.com>
2021-06-21 08:50:04 -07:00
Sébastien Han eb0c986196 ceph: use cacert if no client cert/key are present
We have seen issues with fullchain.pem files containing one self-signed
certificate. At the same time, the failing configuration was not
providing any client certificates or keys.

The error from curl was `"self signed certificate in the chain"`.

The openssl binary is not available so we cannot even c_rehash the
certificate directory when using `--capath`, however this worked already
with a fully signed fullchain.pem along with a client certificate and
key. So we cannot only use `--cacert` which will break that use case.

This is becoming really obscur on why using `--cacert` solves this
problem. I just can't wait to use the `vault` binary, that's all I can
say...

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-06-21 16:06:25 +02:00
Travis Nielsen dd3b4738bd ceph: activate osd container should use correct dataDirHostPath
The active osd container was always using /var/lib/rook for the
configuration of the OSD. If the dataDirHostPath is anything other
than the default, the config would show up in the wrong place.
Changing this back to the dataDirHostPath does not appear to affect
any upgrade scenarios as the activate container will simply start
extracting the config to the correct place.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-06-17 06:59:31 -06:00
Denis Egorenko 6dbb902674 ceph: set device class for already present osd deployments
Add ability to set device class for existing osd deployments,
based on its crush map device class.

Related-Issue: rook#8028
Signed-off-by: Denis Egorenko <degorenko@mirantis.com>
2021-06-15 13:31:37 +04:00
Blaine Gardner 21e290e003 ceph: implement dependencies for CephCluster
Implement the first step of `design/ceph/resource-dependencies.md` to
add dependency checking when deleting a CephCluster.

Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
2021-06-10 10:06:10 -06:00
Travis Nielsen 06fbf29b3c Merge pull request #8028 from degorenko/device-classes-resource-limits
ceph: add ability to set resource limit for OSDs based on device classes
2021-06-08 09:01:14 -06:00
Denis Egorenko 8a556fd847 ceph: add ability to set resource limit for OSDs based on device classes
Currently it is not possible to set resource limits based on device classes
for different OSDs. Now adding an ability to use predefined keys for main
cluster spec Resource section to reflect resource limits for different OSDs
with different device classes.

Closes: https://github.com/rook/rook/issues/8007
Signed-off-by: Denis Egorenko <degorenko@mirantis.com>
2021-06-08 17:46:41 +04:00
Sébastien Han 23777c859b ceph: remove potential new lines from the vault token variable
In some circonstances the VAULT_TOKEN env variable contains a new line.
We don't really know how and why. The saw this when upgrading from
Nautilus CentOS 7 to Octopus CentOS 8. So there might be something
different with bash and variable expansion.
Anyway, let's remove any potential new line and trailing spaces from
that var.

The error was splitting the curl line to something like:

```
+ curl --silent --show-error --request GET --header 'X-Vault-Token: s.aCHdo57tF19ddN1hQmGVyG8a
' --insecure https://vault.default.svc.cluster.local:8200/v1/rook/ver1/rook-ceph-osd-encryption-key-set1-data-0pztgf
curl: (92) HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)
```

Where a correct line will be:

```
+ curl --silent --show-error --request GET --header 'X-Vault-Token: s.aCHdo57tF19ddN1hQmGVyG8a' --insecure https://vault.default.svc.cluster.local:8200/v1/rook/ver1/rook-ceph-osd-encryption-key-set1-data-0pztgf
```

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-05-31 16:33:43 +02:00
Sébastien Han 26c0f8bcc1 ceph: only pass vault volume projection is necessary
We don't need to pass the Volume with projection for TLS when TLS is not enabled
Somehow when this happens and we try to update a deployment spec it fails with:

```
 ValidationError(Pod.spec.volumes[7].projected): missing required field "sources"
```

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-05-31 16:33:40 +02:00
Sébastien Han 0004869f66 ceph: add ceph cluster fsid to LUKS header
When configuring encrypted on-pvc clusters we now set the cluster fsid
in the LUKS header. If needed we can then determine if the encrypted
block is part of our cluster or not. We also attach the pvc_name in case
it might become useful.

Closes: https://github.com/rook/rook/issues/7991
Signed-off-by: Sébastien Han <seb@redhat.com>
2021-05-31 15:01:52 +02:00
Travis Nielsen 2a9c81ec55 Merge pull request #7951 from BlaineEXE/scan-disks-in-init-container-for-raw-osds-on-node
ceph: scan raw OSDs on nodes in OSD init container
2021-05-20 13:01:32 -06:00
Blaine Gardner 8f7f24b1fc ceph: scan raw OSDs on nodes in OSD init container
The 'ceph-volume raw list' command used to list OSDs on nodes by the
rook-ceph-osd-prepare- jobs return user-friendly device names (e.g.,
/dev/sda) rather than persistent device names. Friendly device names
often change when systems reboot, and even persistent device names can
change under certain circumstances (e.g., a disk controller is replaced
with a different one). To prevent OSD Pods created for raw OSDs on
nodes from crashing repeatedly if the device it was created with is
renamed, scan the list of OSDs on the host with 'ceph-volume raw list'
to find the right OSD to activate in the OSD pod's OSD activation init
container.

Fixes #7918

Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
2021-05-20 12:35:53 -06:00
Travis Nielsen b0a63711f5 build: refactor to consolidate the rook.io/v1 package
The rook.io/v1 package was only an internal implementation detail and
does not have any CRDs that rely on it. The CRD deserialization should
handle the change in internal types without any issue. This separation
gives more flexibility for the storage providers to implement exactly
what is needed for their storage provider instead of forcing to use the
same types and risk affecting another storage provider.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-05-18 19:55:37 -06:00
Satoru Takeuchi 6af014b798 ceph: fix hostpath of osd to avoid data corruption
A directory is created under dataDirHostPath for each OSD. In host-based cluster,
its name is constructed from kernel device name like _dev_sdb though user specifies
udev persistent device name. We should use unique name among all OSDs.

Closes: https://github.com/rook/rook/issues/7878

Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
2021-05-12 01:31:46 +00:00
Shachar Sharon 2a3cfc693b ceph: allow setting primary-affinity to osd
A user with unbalanced system (e.g., some OSDs use a partition which
resides on a device along-side OS partition) may want to have a
fine-grained control on the overall load of specific OSD. While osd
weight affects write-load, primary-affinity affects read-load.

This patch allow user to define osd's primary-affinity (similar to
'DeviceClass' and 'InitialWeight'), which is sent to the osd (via mon)
as part of deployment sequence, using 'ceph osd primary-affinity' command.

Valid values are within the range [0.0, 1.0). If not set, default value
is 1.

ROOK issue: https://github.com/rook/rook/issues/7773

Signed-off-by: Shachar Sharon <ssharon@redhat.com>
2021-05-06 17:46:15 +03:00
Travis Nielsen ce50eb2407 ceph: remove unused volumesource schema from cephcluster
The VolumeSource type was only used internally when loading
the storageClassDeviceSets. The volumeSource element is never
expected to be specified in the CephCluster CR, so it is now
removed from the schema. The VolumeSource type is renamed to
deviceSet for clarity.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-05-03 11:54:46 -06:00
Travis Nielsen 08f896e865 ceph: detect the osd topology affinity during upgrade
The nodeAffinity for portable OSDs is needed to ensure the OSDs are
tied to the same zone or other failure domain where the OSD was
prepared and where it is expected to run. During upgrade the topology
affinity was not being detected since the osd prepare job is skipped
in that case. Now the topology will be detected from the running
OSD daemon pod's node.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-04-27 08:51:51 -06:00
Nitin Goyal 99bc72d79f ceph: stop using updatePVCIfChanged and start using ExpandPVCIfRequired
delete updatePVCIfChanged method on "Cluster" from osd package and use
the common one ExpandPVCIfRequired from k8sutil pkg

Signed-off-by: Nitin Goyal <nigoyal@redhat.com>
2021-04-23 08:47:49 +05:30
Shachar Sharon 9766e9b8fb ceph: allow passing 'osd-crush-initial-weight'
Ceph support the option '--osd-crush-initial-weight' upon OSD start,
which sets an explicit weight (in TiB units) to specific OSD. Allow
passing this option all the way from the user (similar to
'DeviceClass'), for the special case where end users wants it cluster
to have non-even balance over specific OSDs (e.g., one of the OSDs is
placed over a partition alongside OS-partition).

ROOK issue: https://github.com/rook/rook/issues/7448

Signed-off-by: Shachar Sharon <ssharon@redhat.com>
2021-04-20 20:50:54 +03:00
Travis Nielsen b9cfddbaa7 ceph: fail osd creation if duplicate metadata section
The metadata name must be unique for a storageClassDeviceSet since
only one each of the sections data, metadata, and wal can be specified.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-04-14 15:21:04 -06:00
Lars Lehtonen 600a1498fe ceph: fix multiple imports
This fixes double imports within and beneath pkg/operator/ceph/cluster.

Signed-off-by: Lars Lehtonen <lars.lehtonen@gmail.com>
2021-04-14 01:51:35 -07:00
Sébastien Han c411c1182e Merge pull request #7518 from travisn/obsolete-osd-dirs
ceph: Remove obsolete osd dirs config
2021-04-06 11:17:55 +02:00
Travis Nielsen c577eb80ee ceph: remove obsolete osd dirs config
OSD directories were removed in v1.2, this is some more cleanup
of dir configuration that is no longer needed.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-04-01 12:42:07 -06:00
Sébastien Han 51e7310e09 ceph: add dualstack support on pacific
With Pacific comes the support for dualstack where ceph daemons can
listen on both ipv4 and ipv6 stacks.
A new field in the network spec has been added: `dualStack`

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-03-31 18:04:29 +02:00
Sébastien Han ec5ac91fe8 Merge pull request #7465 from leseb/type-duration
ceph: use metav1.Duration for time CRD field
2021-03-30 09:01:15 +02:00
Blaine Gardner 1df4336a68 Merge pull request #7386 from BlaineEXE/update-osds-in-parallel
ceph: Update osds in parallel
2021-03-29 16:32:07 -06:00
Blaine Gardner 795124b7a8 ceph: update osds in parallel
Update OSDs in parallel per the design in
design/ceph/update-osds-in-parallel.md

The max number of OSDs updated in parallel is currently fixed at 20.

Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
2021-03-29 10:55:28 -06:00