Commit Graph
547 Commits
Author SHA1 Message Date
sp98 2a9f6bfcc6 osd: use bash script if restart interval is > 0
If `FlappingRestartIntervalHours` for OSD is > 0 in the cephCluster CR,
then start OSD inside a bash script.

Signed-off-by: sp98 <sapillai@redhat.com>
(cherry picked from commit ffd69291f1)
2023-10-12 13:13:04 +00:00
sp98 e1811a89a7 osd: revert: make osd pod to sleep when osds are flapping"
This reverts commit 0e53c9ec61.

Signed-off-by: sp98 <sapillai@redhat.com>
2023-10-12 12:28:40 +05:30
sp98 0e53c9ec61 osd: revert: make osd pod to sleep when osds are flapping
This reverts commit 4eb9f62205.

Signed-off-by: sp98 <sapillai@redhat.com>
2023-10-05 21:26:35 +05:30
guoguangwu b81d618de5 core: import packages only once
Signed-off-by: guoguangwu <guoguangwu@magic-shield.com>
(cherry picked from commit 235ac293ff)
2023-09-21 21:30:45 +00:00
sp98 639b1ae25a osd: make osd pod to sleep when osds are flapping
When OSDs flap, ceph stops the OSD daemon if its marked down greater than
5 times in 600 seconds. But OSD pod restarts and marks the OSD `up` again.
This causes the PGs mapped to these OSDs to peer. While the PGs are peering,
IO to these PGs are blocked.

So we need to ensure that if ceph is marking OSD `down` due to flapping, OSD pod
should not restart to mark the OSDs `up` again.

This PR adds a sleep to the OSD pod if the container returned with a 0 exit code
Default behavior is to sleep for 6 hrs. But user can configure it from the
ceph cluster spec.

Signed-off-by: sp98 <sapillai@redhat.com>
(cherry picked from commit 4eb9f62205)
2023-09-15 18:24:50 +00:00
Travis Nielsen 41004d8d2f Merge pull request #12831 from rook/mergify/bp/release-1.12/pr-12791
core: restart ceph daemons when network updated (backport #12791)
2023-09-12 16:23:59 -06:00
Blaine Gardner 8b72dfa020 multus: detect network CIDRs via canary
Change how Rook detects network CIDRs for Multus networks. The IPAM
configuration is only defined as an arbitrary string JSON blob with a
"type" field and nothing more. Rook's detection of CIDRs for whereabouts
had already grown out of date since the initial implementation.
Additionally, Rook did not support DHCP IPAM, which is a reasonable
choice for users. And more, Rook did not support CNI plugin chaining,
which further complicates NADs. Based on the CNI spec, network chaning
can result in any changes to network CIDRs from the first-given plugin.

All these problems make it more and more difficult for Rook to support
Multus by inspecting the NAD itself to predict network CIDRs. Instead,
it is better for Rook to treat the CNI process as a black box. To
preserve legacy functionality of auto-detecting networks and to make
that as robust as possible, change to a canary-style architecture like
that used for Ceph mons, from which Rook will detect the network CIDRs
if possible.

Also allow users to specify overrides for CIDR ranges. This allows Rook
to still support esoteric and unexpected NAD or network configurations
where a CIDR range is not detectable or where the range detected would
be incomplete. Because it may be impossible for Rook to understand the
network CIDRs wholistically while residing only on a portion of the
network, this feature should have been present from Multus's inception.

Improving CIDR auto-detection and allowing users to specify overrides
for auto-detected CIDRs rounds out Rook's Multus support for CephCluster
(core/RADOS) installations. No further architectural changes should be
needed for CephClusters as regards application of public/cluster network
CIDRs for Multus networks.

Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
(cherry picked from commit 3c43268d0a)
2023-09-07 19:43:17 +00:00
sp98 ba91dec40f osd: replace existing OSDs to use new store
This follow up the #12507
- fixes replacing of encrypted OSDs.
- Updates OSD status at the end of reconcile

Signed-off-by: sp98 <sapillai@redhat.com>
(cherry picked from commit 11b8d10a5a)
2023-09-05 14:58:43 +00:00
subhamkrai 64fecc7564 core: restart ceph daemons when network updated
We need to restart all the ceph daemons whenever
cephCluster network settings are modified like
requiremsgr2, encryption and compression. This
required for Ceph to consider the new settings
it require new ceph daemons all over.

Signed-off-by: subhamkrai <srai@redhat.com>
(cherry picked from commit 29d2b6a071)
2023-09-01 14:43:26 +00:00
sp98 84b7b68582 osd: replace osd to use new backend store
If osd store is updated in the ceph cluster, then
delete OSDs one by one, cleanup disks and provision a new OSD on
the same disk

Signed-off-by: sp98 <sapillai@redhat.com>
(cherry picked from commit 886bb357ef)
2023-08-17 18:42:52 +00:00
Sheetal Pamecha 7a19548091 osd: use cp -a command for copying init-containers
remove copy-binaries cmd and related code

Signed-off-by: Sheetal Pamecha <spamecha@redhat.com>
(cherry picked from commit d86fa2abbc)
2023-07-31 18:05:28 +00:00
sp98 5d51214ad5 osd: configure backend store
Configure osd backend store from the cephCluster spec

Signed-off-by: sp98 <sapillai@redhat.com>
(cherry picked from commit c13d34f8d3)
2023-07-25 21:21:20 +00:00
YZ775 ed9da20d68 operator: add ceph image version label to PVC
This PR makes Rook operator to add a label on PVC that contains an image version of Ceph when creating an OSD.

Signed-off-by: YZ775 <yuzuki-mimura@cybozu.co.jp>
2023-06-19 08:19:46 +09:00
avanthakkar 541d091f9c core: use ROOK_CEPH_MON_HOST from config store in OSD pods too
Signed-off-by: avanthakkar <avanjohn@gmail.com>

Volume "ceph-daemons-sock-dir" is coming empty in case if dataDirHostPath,
which is the case for osd onPVC. Fix the volume creation by using the
ceph cluster spec dataDirHostPath, which allows to run socket commands
on osd containers.
2023-06-01 21:09:58 +05:30
Satoru Takeuchi 13e4464fed osd: supoprt expand lvm osd on pvc
The expansion of OSD on PVC is only supported in raw mode OSD.
It's nice to support lvm mode osd too.

Closes: https://github.com/rook/rook/issues/10835

Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
2023-05-16 00:23:09 +00:00
travisn db04b127b0 core: remove obsolete journal size config value
The journal size was only applicable to the filestore OSD
format which has not been supported by rook since v1.2.
Remove the remaining obsolete setting from the examples and
code.

Signed-off-by: travisn <tnielsen@redhat.com>
2023-04-24 15:54:53 -06:00
sp98 9e8e85d68b core: use clusterID for service nslookup
use ClusterID of the k8s cluster while doing nslookup of the
exported mon or osd service IP

Signed-off-by: sp98 <sapillai@redhat.com>
2023-04-11 15:24:08 +05:30
travisn cd94ed7ddc core: set key rotation default in code instead of crds
The defaults in CRDs cause an update and trigger a new
reconcile if the value is not set. For the key rotation
setting, the default was only being updated when the kms
feature was enabled. This update was causing the reconcile
to fail in the kms tests. With the default now in code,
the default can remain empty and no cr update will trigger
a new reconcile unexpectedly.

Signed-off-by: travisn <tnielsen@redhat.com>
2023-03-21 11:03:56 -06:00
Travis Nielsen fd2f5c8abe Merge pull request #11904 from parth-gr/golang-ci-lint-update
ci: update golangci-lint version as it fails for some k8s version in …
2023-03-16 10:52:13 -06:00
parth-gr 8e317ee074 ci: update golangci-lint version as it fails for some k8s version in 1.10
Closes: https://github.com/rook/rook/issues/11896

Signed-off-by: parth-gr <paarora@redhat.com>
2023-03-16 20:59:22 +05:30
Travis Nielsen c0fdcd3f40 Merge pull request #11823 from parth-gr/topology
osd: validate and remove duplicate topology
2023-03-15 12:39:17 -06:00
parth-gr 89764f0371 osd: remove deprecated k8s topology label
Signed-off-by: parth-gr <paarora@redhat.com>
2023-03-14 17:28:19 +05:30
parth-gr c22f5f36cb osd: validate and remove duplicate topology
remove the duplicate topology added to the node,
and the priority to keep duplicate would be give to
the one who is lowest in the hierarchy

Closes: https://github.com/rook/rook/issues/11800

Signed-off-by: parth-gr <paarora@redhat.com>
2023-03-09 21:34:38 +05:30
Rakshith R f39a32fcdb osd: add capability to reconcile key rotation cron jobs
This commits adds code to reconcile key rotation cron jobs.

Signed-off-by: Rakshith R <rar@redhat.com>
2023-03-08 12:07:26 +05:30
Rakshith R ef38aea2a4 osd: add util function to create key rotation cron job
This commit adds util function to create key rotation cron job
such as getKeyRotationContainer, getKeyRotationCronPodTemplateSpec
and makeKeyRotationCronJob.

Signed-off-by: Rakshith R <rar@redhat.com>
2023-03-08 12:07:26 +05:30
Rakshith R 71e011f731 osd: add rotate-key functionality to rook's key-management cmd
This commit adds functionality to be able to rotate
key encryption key of encrypted PVC backed OSDs.
Necessary changes such as adding update functionality
to kms and rbac changes are made as well.

Signed-off-by: Rakshith R <rar@redhat.com>
2023-03-08 12:07:26 +05:30
sp98 12bbc9b1a8 osd: add container ports only if mcs is enabled
Add container ports for OSD container only when multi cluster service is
enabled in the cluster spec.

Signed-off-by: sp98 <sapillai@redhat.com>
2023-03-01 09:07:08 +05:30
Travis Nielsen 6bcf9004a0 Merge pull request #11567 from cybozu-go/fix-10860
osd: handle device name change and device removel correctly
2023-02-27 11:54:30 -07:00
sp98 7292ac5927 core: export mon and OSD services
Signed-off-by: sp98 <sapillai@redhat.com>
2023-02-27 21:36:59 +05:30
Travis Nielsen d655c86b2f osd: create a service for each osd
The osd needs to bind to an ip that will allow both internal
and external access for rbd mirroring scenarios.
THe operator will create a service for each osd and
instruction the osd to bind to the pod ip while
publishing the clusterIP of the service as the
public IP.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
(cherry picked from commit 14e7225e0a18aff85ede1d476fcaddbbfbcbb732)
2023-02-27 21:05:48 +05:30
Satoru Takeuchi 3a44a394b5 osd: handle device name change and device removel correctly
If a kernel device name change happens and a block device file
in the OSD directory becomes dangling link, this OSD fails
to start continuously. This problem can be resolved by confirming
the validity of the device file and recreating it if necessary.

The original problem happened in host-based clusters. A similar
problem exists in PVC-based clusters. In this case, if a PV,
corresponds to an existing OSD, and points to a missing block
device file, the OSD pod fails to consume this PV. Although this
behavior is undesirable, I don't think this problem should be
handled in Rook. If doing so, we must re-create the existing PV.
Rook shouldn't do such work.

Closes: https://github.com/rook/rook/issues/10860

Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
2023-02-24 05:29:14 +00:00
parth-grandBlaine Gardner 69ae9569cd build: update k8s version to 1.26.1
Update the Kubernetes API version used to 1.26.1, and start testing
against Kubernetes version 1.26.1 in CI.

Co-authored-by: parth-gr <paarora@redhat.com>
Co-authored-by: Blaine Gardner <blaine.gardner@redhat.com>

Signed-off-by: parth-gr <paarora@redhat.com>
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
2023-02-23 20:18:47 -07:00
parth-gr a84daf9bf0 core: change io/ioutil package to use io and os package
few functions got change as they were deprecated
for ex: ioutil.Readfile change to os.Readfile
ioutil.TempFile change to os.CreateTemp
And fixed golang-ci-lint-issues

Signed-off-by: parth-gr <paarora@redhat.com>
2023-02-17 20:38:29 +05:30
Rakshith R 8b9ded1f77 csi: add option to enable read affinity for rbd volumes
This commit adds code to supply variables required
to enable read affinity for rbd volumes.
It leverages `read_from_replica=localize,crush_locations=key:value`
krbd map option from ceph to serve reads from the most local OSD.
refer:
https://docs.ceph.com/en/latest/man/8/rbd/#kernel-rbd-krbd-options

Signed-off-by: Rakshith R <rar@redhat.com>
2023-02-16 19:09:39 +05:30
Travis Nielsen 06eb431525 Merge pull request #11674 from travisn/osd-version-label
osd: Ensure rook version label is not set on pod
2023-02-15 12:21:09 -07:00
Avan Thakkar 460900756c core: add service monitor for ceph-exporter service
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
2023-02-15 15:44:56 +05:30
Travis Nielsen 1ba93b4047 osd: ensure rook version label is not set on pod
The rook-version tag must not be set on the pod spec labels
since it will result in the ceph daemons being restarted every
time there is a rook version update, even if the ceph version
or pod spec was not otherwise updated.

The rook-version tag was being added to the OSD pod spec
due to a shared pointer to the labels. The code intended
to only add the version label to the deployment labels,
but since the pod labels shared the map variable, the
pod unintentionally also had the version label added.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2023-02-14 16:10:53 -07:00
Avan Thakkar 2f8ee60374 core: introduce ceph-exporter
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
2023-02-06 02:47:09 -05:00
Yuma Ogami 7599f1bc6b osd: add expand-bluefs init container for host-based cluster
OSD is not resized in host-based clusters even though expanding
the backing block device. It's due to the lack of bluefs-bdev-expand
init container.

Closes: https://github.com/rook/rook/issues/10930
Signed-off-by: Yuma Ogami <yuma-ogami@cybozu.co.jp>
2023-01-03 05:03:03 +00:00
Yuma Ogami c6c264df84 osd: add unit test for chown-container-data-dir init container
Signed-off-by: Yuma Ogami <yuma-ogami@cybozu.co.jp>
2022-12-22 07:43:15 +00:00
Travis Nielsen 403335177a core: read mon secret from file instead of env var
Environment variables are not recommended for secrets in pods since
they can be easily leaked if the environemnt variables are logged.
By mounting the mon secret as a file, the mgr and osd prepare pods
can read the mon secret from a file for better security.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2022-12-14 16:24:44 -07:00
Travis Nielsen 4a70d3489d osd: replace dot with dash in pvc names
The volume names of a pod must follow the standard naming convention
which does not allow periods in the name. Since the device set name
is used as the pvc name and then the volume name to attach to the osd
prepare job, we must sanitize the name for the invalid character
and replace it.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2022-12-09 12:43:17 -07:00
Satoru Takeuchi 0e10d97431 Merge pull request #11331 from travisn/remove-env-vars
core: Remove unnecessary env vars from pod specs
2022-12-06 08:28:06 +09:00
Rakshith R bde286ef49 osd: re-open encrypted disk during osd-prepare-job if closed
This commit implements this corner case during osd-prepare job.
```
The encrypted block is not opened, this is an extreme corner case
The OSD deployment has been removed manually AND the node rebooted
So we need to re-open the block to re-hydrate the OSDInfo.

Handling this case would mean, writing the encryption key on a
temporary file, then call luksOpen to open the encrypted block and
then call ceph-volume to list against the opened encrypted block.
We don't implement this, yet and return an error.
```
When underlying PVC for osd are CSI provisioned, the encrypted device
is closed when PVC is unmounted due to osd pod being deleted.
Therefore, this may occur more frequently and needs to be handled.
This commit implements the fix for the same.

Signed-off-by: Rakshith R <rar@redhat.com>
2022-11-30 18:54:23 +05:30
Travis Nielsen 33e824a323 core: remove unnecessary env vars from pod specs
The fsid, username, and user secrets were from legacy and no
longer needed on the osd daemon or mon pods. This removes the
unnecessary env vars from the pod specs.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2022-11-18 16:39:22 -07:00
Shinya Hayashi 05875a3f4f osd: support loop devices for test clusters
A new variable is added to rook-ceph-operator-config
ConfigMap to allow using loop devices for osd.

This feature is intended to be used for testing purposes only.

Signed-off-by: Shinya Hayashi <shinya-hayashi@cybozu.co.jp>
2022-11-09 06:45:41 +00:00
Travis Nielsen d39258e0d2 Merge pull request #11159 from parth-gr/rotational-disk
osd: add deviceType label to an OSD
2022-11-03 08:55:53 -06:00
Travis Nielsen faf5eba8dc osd: reduce duplication of arguments
The osd arguments had a few duplicate lines that could be
factored out and thus simplified for code maintainability.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2022-10-26 16:50:04 -06:00
parth-gr 3bcc29e1ff osd: add deviceType label to an OSD
Closes: https://github.com/rook/rook/issues/11059

Signed-off-by: parth-gr <paarora@redhat.com>
2022-10-19 21:40:43 +05:30
Avan Thakkar 934aa91056 operator: make imagePullPolicy customizable for csi driver and ceph pods
Introduce a new env variable ROOK_CSI_IMAGE_PULL_POLICY in rook operator configmap which should be used to
customize the imagePullPolicy for the csi driver and imagePullPolicy property in cephVersionSpec for ceph pods.

Signed-off-by: Avan Thakkar <athakkar@redhat.com>
2022-09-27 11:57:43 +05:30