Commit Graph
1272 Commits
Author SHA1 Message Date
Travis Nielsen f6b3d53f5a Merge pull request #11849 from rook/mergify/bp/release-1.11/pr-11848
core: Add env var to exporter for ceph config (backport #11848)
2023-03-07 15:02:56 -07:00
avanthakkar 6accfb6380 core: add env var to exporter for ceph config
The exporter requires the mon config and keyring.
Use the CEPH_ARGS env var to specify the config
instead of expecting a ceph.conf to be generated
by another component on the host path.

Co-Authored-By: Travis Nielsen <tnielsen@redhat.com>
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
(cherry picked from commit b6013d9b11)
2023-03-07 21:22:17 +00:00
Redouane Kachach eaf09ad77a Revert "mgr: use k8s readiness probe to implement mgr HA"
This reverts commit dc76f81fea.

Signed-off-by: Redouane Kachach <rkachach@redhat.com>
(cherry picked from commit 0ae7867dd1)
2023-03-07 18:28:25 +00:00
Redouane Kachach 9a3e5c4e94 Revert "mgr: remove mgr sidecar"
This reverts commit ff75ec96b7.

Signed-off-by: Redouane Kachach <rkachach@redhat.com>
(cherry picked from commit 456a0c328c)
2023-03-07 18:28:25 +00:00
Redouane Kachach 4d56aca868 Revert "mgr: addressing review comments"
This reverts commit 9f37439a07.

Signed-off-by: Redouane Kachach <rkachach@redhat.com>
(cherry picked from commit adc79d4ee4)
2023-03-07 18:28:24 +00:00
Redouane Kachach 580a9ec5c3 Revert "ci: fix MultiClusterDeploySuite CI"
This reverts commit b464428978.

Signed-off-by: Redouane Kachach <rkachach@redhat.com>
(cherry picked from commit 0c652d28e2)
2023-03-07 18:28:24 +00:00
Redouane Kachach f6fa004343 Revert "mgr: wait for the mgr pod to be running"
This reverts commit 4d1017f805.

Signed-off-by: Redouane Kachach <rkachach@redhat.com>
(cherry picked from commit c31832095d)
2023-03-07 18:28:24 +00:00
sp98 5cb72de8d0 osd: add container ports only if mcs is enabled
Add container ports for OSD container only when multi cluster service is
enabled in the cluster spec.

Signed-off-by: sp98 <sapillai@redhat.com>
(cherry picked from commit 12bbc9b1a8)
2023-03-01 16:30:18 +00:00
Ben Gao 0cccf48a1f core: on crash pod ensure rook version label is not set
ceph crash pod gets rook version label added unintentionally, which is only for deployment label.
changes include:
1, make a copy of the labels for pod spec from deployment labels just before adding rook version label.
2, add unit test to make sure it works as designed.
This fix along with pr 11674 is to stop ceph daemons pod restarting unnecessarily after upgrading rook operator.

Signed-off-by: Ben Gao <bengao168@msn.com>
(cherry picked from commit ed6ed25257)
2023-02-28 18:25:38 +00:00
Satoru Takeuchi 0e4103a51c osd: handle device name change and device removel correctly
If a kernel device name change happens and a block device file
in the OSD directory becomes dangling link, this OSD fails
to start continuously. This problem can be resolved by confirming
the validity of the device file and recreating it if necessary.

The original problem happened in host-based clusters. A similar
problem exists in PVC-based clusters. In this case, if a PV,
corresponds to an existing OSD, and points to a missing block
device file, the OSD pod fails to consume this PV. Although this
behavior is undesirable, I don't think this problem should be
handled in Rook. If doing so, we must re-create the existing PV.
Rook shouldn't do such work.

Closes: https://github.com/rook/rook/issues/10860

Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
(cherry picked from commit 3a44a394b5)
2023-02-27 18:55:21 +00:00
sp98 c69c5a16fd core: export mon and OSD services
Signed-off-by: sp98 <sapillai@redhat.com>
(cherry picked from commit 7292ac5927)
2023-02-27 10:23:43 -07:00
Travis Nielsen 946fbcf66e osd: create a service for each osd
The osd needs to bind to an ip that will allow both internal
and external access for rbd mirroring scenarios.
THe operator will create a service for each osd and
instruction the osd to bind to the pod ip while
publishing the clusterIP of the service as the
public IP.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
(cherry picked from commit 14e7225e0a18aff85ede1d476fcaddbbfbcbb732)
(cherry picked from commit d655c86b2f)
2023-02-27 16:50:55 +00:00
parth-gr 4c92a2e93a build: update k8s version to 1.26.1
Update the Kubernetes API version used to 1.26.1, and start testing
against Kubernetes version 1.26.1 in CI.

Co-authored-by: parth-gr <paarora@redhat.com>
Co-authored-by: Blaine Gardner <blaine.gardner@redhat.com>

Signed-off-by: parth-gr <paarora@redhat.com>
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
(cherry picked from commit 69ae9569cd)
2023-02-24 22:13:45 +00:00
Redouane Kachach baa8a8988b mgr: addressing review comments
Signed-off-by: Redouane Kachach <rkachach@redhat.com>
(cherry picked from commit 9f37439a07)
2023-02-22 05:20:01 +00:00
Redouane Kachach 0207134140 mgr: remove mgr sidecar
With the new mgr HA implementation (based on readiness probe) we
don't need the mgr sidecar (live-watch) anymore. This commit is
intended to remove all the related code.

Signed-off-by: Redouane Kachach <rkachach@redhat.com>
(cherry picked from commit ff75ec96b7)
2023-02-22 05:20:00 +00:00
Travis Nielsen 60e0328ab2 mgr: wait for the mgr pod to be running
After an update, the mgr pod will be restarted. Since the
readiness probe will indicate the standby mgr pod is never
ready, we need to check simply for the running mgr pod
and not whether or not it is ready.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
(cherry picked from commit 4d1017f805)
2023-02-21 15:36:05 +00:00
parth-gr 63c66da813 core: change io/ioutil package to use io and os package
few functions got change as they were deprecated
for ex: ioutil.Readfile change to os.Readfile
ioutil.TempFile change to os.CreateTemp
And fixed golang-ci-lint-issues

Signed-off-by: parth-gr <paarora@redhat.com>
(cherry picked from commit a84daf9bf0)
2023-02-17 16:24:10 +00:00
Travis Nielsen 6343775767 Merge pull request #11695 from rook/mergify/bp/release-1.11/pr-11690
ci: fix MultiClusterDeploySuite CI (backport #11690)
2023-02-16 14:34:59 -07:00
Redouane Kachach 6f45ea1a98 ci: fix MultiClusterDeploySuite CI
This fixes the the MultiClusterDeploySuite CI failure. The operator
is timing out waiting for the mgr deployments to be
ready, according to the WaitForDeploymentToStart() method. After
the reconcile times out after about five minutes, the next
reconcile succeeds since the wait is only done for new mgr
deployments.

Closes: https://github.com/rook/rook/issues/11685

Signed-off-by: Redouane Kachach <rkachach@redhat.com>
(cherry picked from commit b464428978)
2023-02-16 18:20:58 +00:00
Rakshith R 21fa74f2d9 csi: add option to enable read affinity for rbd volumes
This commit adds code to supply variables required
to enable read affinity for rbd volumes.
It leverages `read_from_replica=localize,crush_locations=key:value`
krbd map option from ceph to serve reads from the most local OSD.
refer:
https://docs.ceph.com/en/latest/man/8/rbd/#kernel-rbd-krbd-options

Signed-off-by: Rakshith R <rar@redhat.com>
(cherry picked from commit 8b9ded1f77)
2023-02-16 15:00:32 +00:00
Travis Nielsen 06eb431525 Merge pull request #11674 from travisn/osd-version-label
osd: Ensure rook version label is not set on pod
2023-02-15 12:21:09 -07:00
Travis Nielsen 3dabc6dcb6 Merge pull request #11317 from avanthakkar/introduce-ceph-exporter
core: introduce ceph-exporter
2023-02-15 11:59:05 -07:00
Travis Nielsen b9be333651 Merge pull request #11678 from Madhu-1/fix-v2-map
core: set rbd default map options
2023-02-15 11:49:04 -07:00
Travis Nielsen b008c5753f Merge pull request #11643 from rkachach/fix_issue_11640
mgr: use k8s readiness probe to implement mgr HA
2023-02-15 09:51:48 -07:00
Madhu Rajanna 0ab6c34bf2 core: set rbd default map options
Set the rbd map default option to ms_mode=prefer_crc
when v2 ports are used and encryption is disabled.
If we dont pass/set the ms_mode in the map options
the kernel client will assume its using v1 ports
and map operation will fail.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2023-02-15 12:46:40 +01:00
Avan Thakkar 460900756c core: add service monitor for ceph-exporter service
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
2023-02-15 15:44:56 +05:30
Redouane Kachach dc76f81fea mgr: use k8s readiness probe to implement mgr HA
The idea behind this change is to use the readiness probe to implement
the mgr HA mechanism. In the current ceph mgr implementation only the
active instance offers the command 'mgr_status' through the admin
socket. We use this command combined with a Readiness Exec Probe to
detect which manager is active. Kubernetes will automatically mark
it as ready and redirect any service traffic to the active instance.

Closes: https://github.com/rook/rook/issues/11640
Closes: https://github.com/rook/rook/issues/11638
Signed-off-by: Redouane Kachach <rkachach@redhat.com>
2023-02-15 10:12:16 +01:00
Travis Nielsen 1ba93b4047 osd: ensure rook version label is not set on pod
The rook-version tag must not be set on the pod spec labels
since it will result in the ceph daemons being restarted every
time there is a rook version update, even if the ceph version
or pod spec was not otherwise updated.

The rook-version tag was being added to the OSD pod spec
due to a shared pointer to the labels. The code intended
to only add the version label to the deployment labels,
but since the pod labels shared the map variable, the
pod unintentionally also had the version label added.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2023-02-14 16:10:53 -07:00
Travis Nielsen 5e0cf95228 Merge pull request #11627 from travisn/require-msgrv2
core: Option to require msgrv2 even without encryption
2023-02-10 10:58:22 -07:00
Travis Nielsen e7320ad63b mon: allow mons on msgr1 even if v2 settings required
Mons that are configured on msgr1 will be using the v1 port
in the csi driver since only one port is allowed in the configmap.
To allow mons to be started on v2 while keeping quorum with the
v1 mons, the v1 mons must continue running on the v1 port.
So even if msgr2 is required, allow existing v1 mons to continue
running on that port.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2023-02-10 09:38:09 -07:00
Travis Nielsen ab3209c140 helm: add labels and annotations to config overrides configmap
The default configmap does not have any labels or annotations
applied. If the helm chart is later updated to configure the
ceph config overrides, the configmap will fail to be created
by the chart since it doesn't have the needed label and helm
annotations so helm will know it can take over the resource
management.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2023-02-10 07:33:35 -07:00
Avan Thakkar 2f8ee60374 core: introduce ceph-exporter
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
2023-02-06 02:47:09 -05:00
Madhu Rajanna d1a86f6a63 core: unset messenger2 configuration
Unset encryption configuration and default
rbd map option when the encryption is not
enabled or it disabled after sometime.

Unset compression option also when the compression
is disabled.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2023-01-18 11:05:42 +01:00
Madhu Rajanna bc41df4dd8 core: set rbd_default_map_options for encryption
Set rbd_default_map_options to secure when encryption
is enabled or else set it to prefer-crc.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2023-01-12 10:02:15 +01:00
Travis Nielsen aa2c31f154 Merge pull request #11493 from parth-gr/external-errors
external: Dont call telemetry if the cluster is external
2023-01-03 10:14:50 -07:00
Yuma Ogami 7599f1bc6b osd: add expand-bluefs init container for host-based cluster
OSD is not resized in host-based clusters even though expanding
the backing block device. It's due to the lack of bluefs-bdev-expand
init container.

Closes: https://github.com/rook/rook/issues/10930
Signed-off-by: Yuma Ogami <yuma-ogami@cybozu.co.jp>
2023-01-03 05:03:03 +00:00
parth-gr a52e2f1794 external: dont call telemetry if the cluster is external
closes: https://github.com/rook/rook/issues/11470
Signed-off-by: parth-gr <paarora@redhat.com>
2023-01-02 20:41:40 +05:30
Yuma Ogami c6c264df84 osd: add unit test for chown-container-data-dir init container
Signed-off-by: Yuma Ogami <yuma-ogami@cybozu.co.jp>
2022-12-22 07:43:15 +00:00
Travis Nielsen 403335177a core: read mon secret from file instead of env var
Environment variables are not recommended for secrets in pods since
they can be easily leaked if the environemnt variables are logged.
By mounting the mon secret as a file, the mgr and osd prepare pods
can read the mon secret from a file for better security.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2022-12-14 16:24:44 -07:00
Travis Nielsen e90f7ce022 Merge pull request #11414 from travisn/sanitize-deviceset-name
osd: Replace dot with dash in storageClassDeviceSet names since used as volume names
2022-12-12 09:57:14 -07:00
Travis Nielsen b693a5cca5 core: refactor crash collector for more node daemons
The crash collector controller is designed for watching nodes where
ceph daemons are running, and ensuring a special daemon is running
on that node to provide additional support for ceph on that node.
The crash collector is the first example of a daemon that should be
running on all the ceph daemon nodes. The next example of such a
node daemon will be the ceph exporter that will listen for the
ceph metrics as described in the design doc.
https://github.com/rook/rook/blob/master/design/ceph/ceph-exporter.md

Now the crash collector controller is renamed to the node daemon controller
so the ceph exporter daemon can also be managed by the same controller.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2022-12-09 16:57:05 -07:00
Travis Nielsen 4a70d3489d osd: replace dot with dash in pvc names
The volume names of a pod must follow the standard naming convention
which does not allow periods in the name. Since the device set name
is used as the pvc name and then the volume name to attach to the osd
prepare job, we must sanitize the name for the invalid character
and replace it.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2022-12-09 12:43:17 -07:00
Satoru Takeuchi 0e10d97431 Merge pull request #11331 from travisn/remove-env-vars
core: Remove unnecessary env vars from pod specs
2022-12-06 08:28:06 +09:00
Rakshith R bde286ef49 osd: re-open encrypted disk during osd-prepare-job if closed
This commit implements this corner case during osd-prepare job.
```
The encrypted block is not opened, this is an extreme corner case
The OSD deployment has been removed manually AND the node rebooted
So we need to re-open the block to re-hydrate the OSDInfo.

Handling this case would mean, writing the encryption key on a
temporary file, then call luksOpen to open the encrypted block and
then call ceph-volume to list against the opened encrypted block.
We don't implement this, yet and return an error.
```
When underlying PVC for osd are CSI provisioned, the encrypted device
is closed when PVC is unmounted due to osd pod being deleted.
Therefore, this may occur more frequently and needs to be handled.
This commit implements the fix for the same.

Signed-off-by: Rakshith R <rar@redhat.com>
2022-11-30 18:54:23 +05:30
Travis Nielsen 33e824a323 core: remove unnecessary env vars from pod specs
The fsid, username, and user secrets were from legacy and no
longer needed on the osd daemon or mon pods. This removes the
unnecessary env vars from the pod specs.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2022-11-18 16:39:22 -07:00
Sebastian Hasler 120c7dddcb operator: fix a bug where crash collectors in wrong namespaces were deleted
Fixes #11321

Signed-off-by: Sebastian Hasler <sebastian.hasler@stuvus.uni-stuttgart.de>
2022-11-17 18:12:56 +01:00
Travis Nielsen 3a07f3ac98 Merge pull request #10887 from travisn/duplicate-mon-endpoint
mon: Remove out of quorum mons from ceph.conf
2022-11-09 10:16:07 -07:00
Shinya Hayashi 05875a3f4f osd: support loop devices for test clusters
A new variable is added to rook-ceph-operator-config
ConfigMap to allow using loop devices for osd.

This feature is intended to be used for testing purposes only.

Signed-off-by: Shinya Hayashi <shinya-hayashi@cybozu.co.jp>
2022-11-09 06:45:41 +00:00
Travis Nielsen b6e8ea2b50 mon: remove out of quorum mons from ceph.conf
The mons that are out of quorum may cause ceph commands to timeout
or fail unnecessarily trying to connect to a mon that is no longer
online. Now the mon health check will update the mon endpoints configmap
when a mon is detected out of quorum. This also means that if the
operator is restarted during a mon failover, the failed mon will no
longer remain in the ceph.conf, thus allowing the quorum to be more
likely to respond to the mons that are still in quorum.

The update for mons out of quorum only applies if other mons are in
quorum. If quorum is down, the configmap will not keep track of the offline
mons since too many are offline.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2022-11-08 11:36:37 -07:00
Travis Nielsen 1291885501 mon: use the assimilate mode for setall commands
The monstore has a SetAll method for applying multiple settings
that was applying them individually. Now we use the assimilate
config option to make this more efficient.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2022-11-03 10:23:10 -06:00