Commit Graph
239 Commits
Author SHA1 Message Date
zc 1c58c90177 mgr: failed to update the port of mgr's dashboard
if port was used by other daemon,the dashboard will start failed and the status of the ceph will be err,then we update the port to a unused port,the operator can not deal with it

Signed-off-by: zc <zc@zcdeMacBook-Pro.local>
(cherry picked from commit caf7b8c753)
2023-04-06 13:59:53 +00:00
xiaobaowen 6e171c3218 mgr: disable the prometheus module by crds
Signed-off-by: xiaobaowen <xiaobaowen@deeproute.ai>
(cherry picked from commit e7254c2a1b)
2023-03-31 14:17:33 +00:00
Redouane Kachach eaf09ad77a Revert "mgr: use k8s readiness probe to implement mgr HA"
This reverts commit dc76f81fea.

Signed-off-by: Redouane Kachach <rkachach@redhat.com>
(cherry picked from commit 0ae7867dd1)
2023-03-07 18:28:25 +00:00
Redouane Kachach 9a3e5c4e94 Revert "mgr: remove mgr sidecar"
This reverts commit ff75ec96b7.

Signed-off-by: Redouane Kachach <rkachach@redhat.com>
(cherry picked from commit 456a0c328c)
2023-03-07 18:28:25 +00:00
Redouane Kachach 4d56aca868 Revert "mgr: addressing review comments"
This reverts commit 9f37439a07.

Signed-off-by: Redouane Kachach <rkachach@redhat.com>
(cherry picked from commit adc79d4ee4)
2023-03-07 18:28:24 +00:00
Redouane Kachach 580a9ec5c3 Revert "ci: fix MultiClusterDeploySuite CI"
This reverts commit b464428978.

Signed-off-by: Redouane Kachach <rkachach@redhat.com>
(cherry picked from commit 0c652d28e2)
2023-03-07 18:28:24 +00:00
Redouane Kachach f6fa004343 Revert "mgr: wait for the mgr pod to be running"
This reverts commit 4d1017f805.

Signed-off-by: Redouane Kachach <rkachach@redhat.com>
(cherry picked from commit c31832095d)
2023-03-07 18:28:24 +00:00
Redouane Kachach baa8a8988b mgr: addressing review comments
Signed-off-by: Redouane Kachach <rkachach@redhat.com>
(cherry picked from commit 9f37439a07)
2023-02-22 05:20:01 +00:00
Redouane Kachach 0207134140 mgr: remove mgr sidecar
With the new mgr HA implementation (based on readiness probe) we
don't need the mgr sidecar (live-watch) anymore. This commit is
intended to remove all the related code.

Signed-off-by: Redouane Kachach <rkachach@redhat.com>
(cherry picked from commit ff75ec96b7)
2023-02-22 05:20:00 +00:00
Travis Nielsen 60e0328ab2 mgr: wait for the mgr pod to be running
After an update, the mgr pod will be restarted. Since the
readiness probe will indicate the standby mgr pod is never
ready, we need to check simply for the running mgr pod
and not whether or not it is ready.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
(cherry picked from commit 4d1017f805)
2023-02-21 15:36:05 +00:00
Redouane Kachach 6f45ea1a98 ci: fix MultiClusterDeploySuite CI
This fixes the the MultiClusterDeploySuite CI failure. The operator
is timing out waiting for the mgr deployments to be
ready, according to the WaitForDeploymentToStart() method. After
the reconcile times out after about five minutes, the next
reconcile succeeds since the wait is only done for new mgr
deployments.

Closes: https://github.com/rook/rook/issues/11685

Signed-off-by: Redouane Kachach <rkachach@redhat.com>
(cherry picked from commit b464428978)
2023-02-16 18:20:58 +00:00
Travis Nielsen 3dabc6dcb6 Merge pull request #11317 from avanthakkar/introduce-ceph-exporter
core: introduce ceph-exporter
2023-02-15 11:59:05 -07:00
Avan Thakkar 460900756c core: add service monitor for ceph-exporter service
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
2023-02-15 15:44:56 +05:30
Redouane Kachach dc76f81fea mgr: use k8s readiness probe to implement mgr HA
The idea behind this change is to use the readiness probe to implement
the mgr HA mechanism. In the current ceph mgr implementation only the
active instance offers the command 'mgr_status' through the admin
socket. We use this command combined with a Readiness Exec Probe to
detect which manager is active. Kubernetes will automatically mark
it as ready and redirect any service traffic to the active instance.

Closes: https://github.com/rook/rook/issues/11640
Closes: https://github.com/rook/rook/issues/11638
Signed-off-by: Redouane Kachach <rkachach@redhat.com>
2023-02-15 10:12:16 +01:00
Avan Thakkar 2f8ee60374 core: introduce ceph-exporter
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
2023-02-06 02:47:09 -05:00
Travis Nielsen 403335177a core: read mon secret from file instead of env var
Environment variables are not recommended for secrets in pods since
they can be easily leaked if the environemnt variables are logged.
By mounting the mon secret as a file, the mgr and osd prepare pods
can read the mon secret from a file for better security.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2022-12-14 16:24:44 -07:00
Travis Nielsen 33e824a323 core: remove unnecessary env vars from pod specs
The fsid, username, and user secrets were from legacy and no
longer needed on the osd daemon or mon pods. This removes the
unnecessary env vars from the pod specs.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2022-11-18 16:39:22 -07:00
Travis Nielsen f79ff42e0c mgr: set the public dashboard port differently from the target port
If the port is greater than 1024, for backward compatibility the port and
targetPort should be the same value. If the port is less than 1024,
the internal port must use a higher port number. In that case, the internal
port will be the default port numbers and only the public port will be
the desired port in the cluster CR.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2022-10-05 11:37:37 -06:00
Avan Thakkar 934aa91056 operator: make imagePullPolicy customizable for csi driver and ceph pods
Introduce a new env variable ROOK_CSI_IMAGE_PULL_POLICY in rook operator configmap which should be used to
customize the imagePullPolicy for the csi driver and imagePullPolicy property in cephVersionSpec for ceph pods.

Signed-off-by: Avan Thakkar <athakkar@redhat.com>
2022-09-27 11:57:43 +05:30
Travis Nielsen b8dac76d18 mgr: skip updating service if there is no selector
When the mgr services are updated by the operator to set the
active mgr properly, skip services that do not currently
have a selector set.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2022-07-12 16:30:20 -06:00
Travis Nielsen dad97f3425 core: remove support for ceph octopus
With octopus coming to end of life, we remove support from
Rook for deploying Ceph Octopus and assume a min version of
Pacific v16. Any checks for octopus or earlier are removed
from the reconciles since they are obsolete.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2022-07-07 15:03:26 -06:00
Travis Nielsen 28957b3ff1 mon: disable insecure global ids for new deployments
In new deployments we always want to disable the insecure global id.
If the mgr deployment is newly created, we know it's a new cluster
and the mons are in quorum, so we can go ahead and disable the
insecure global IDs.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2022-06-24 12:26:00 -06:00
Alexander Trost 8686296e17 core: remove double imported packages
This removes double package imports. Example:
```
"github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
cephv1 "github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
```
Only one is now being used as shown in go-staticcheck ST1019

Signed-off-by: Alexander Trost <galexrt@googlemail.com>
2022-04-25 13:51:45 +02:00
subhamkrai 24802c559e core: fix golangci linter
fix golangci linter

Signed-off-by: subhamkrai <srai@redhat.com>
2022-04-04 20:59:31 +05:30
Travis Nielsen 6c50530737 Merge pull request #9685 from weirdwiz/context-remove
Remove context.TODO and add context parameter
2022-03-30 12:20:30 -05:00
Divyansh Kamboj 9008409f87 core: add context parameter to functions
This commit adds context parameter to various functions, and remove the
usage of context.TODO.

Closes: https://github.com/rook/rook/issues/8701
Signed-off-by: Divyansh Kamboj <dkamboj@redhat.com>
2022-03-22 08:19:07 +05:30
Travis Nielsen 4edcff04f7 monitoring: create prometheus rules with helm chart
The prometheus rules had been previously created if the cephcluster CR
setting monitoring.enabled was set to true. The rules were not customizable
and therefore not flexible enough. Now the rules are installed by the helm
chart. To customize the rules, a post-processor can be applied to the helm
chart.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2022-03-21 14:47:52 -06:00
parth-gr 413662d475 core: change livenessProbe file name to probes
Added unit test for probes changes and
did re-naming of liveness probes file
as it will contain multiple types of probes

Signed-off-by: parth-gr <paarora@redhat.com>
2022-02-23 17:20:07 +05:30
parth-gr 1679af4a02 mds: override mds startup liveness probes from CephFilesystem CR
Rook code allows overriding the probes for mds from the CephCluster,
but this should really come from the CephFilesystem CR.

Closes: https://github.com/rook/rook/issues/9519

Signed-off-by: parth-gr <paarora@redhat.com>
2022-02-22 20:36:40 +05:30
Satoru Takeuchi 2d8f9b67c6 core: eliminate unnecessary code
We can eliminate the OSD creation code for Ceph < v15.2.0 because Nautilus(v14.2.z)
is no longer supported. In other words, we can now expect that all newly-created
OSDs on PVC are raw mode.

Please note that the existing lvm mode OSDs on PVCs still be supported after this change.

Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
2022-02-04 13:59:01 +00:00
Eng Zer Jun ed63571530 test: use T.TempDir to create temporary test directory
The directory created by `T.TempDir()` and is automatically removed when
the test and all its subtests complete.

Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-01-24 23:52:54 +08:00
Travis Nielsen 11127f54ca mgr: update services when active mgr changes
When the active mgr changes, there may be user-created services
that also need to have their selector updated to point to the
new active mgr. Now the operator will update any service in the
rook operator namespace that has the label app=rook-ceph-mgr
to point to the active mgr.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2022-01-13 07:25:24 -07:00
Blaine Gardner c07d89d9ea core: rgw: allow specifying daemon startup probes
Allow specifying daemon startup probes where we also allow configuring
liveness probes. Startup probes allow Rook to tolerate when Ceph daemons
occasionally take a long time to start up while not also making
Kubernetes liveness probes slower to detect runtime failures of daemons.

Startup probes are beta in Kubernetes 1.18, so we should not enable
probes by default for earlier Kubernetes versions.

Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
2021-12-21 15:12:41 -07:00
Sébastien Han 5c1e459a6a mgr: run active-watch as root and privileged
For now, we must run the container with UID 0 and privileged for
multiple reasons:

* the rook binary writes ceph config to /var/lib/rook which is owned by
  root
* it's difficult to use /etc/ceph since it will conflict with the
  rook-ceph-override configmap AND is also owned by root since it's a
  mounted configmap.
* using /etc/ceph might be possible but has other issues with rook's
  exec package since the ceph config is built from /var/lib/rook

Closes: https://github.com/rook/rook/issues/9385
Signed-off-by: Sébastien Han <seb@redhat.com>
2021-12-13 14:02:33 +01:00
Sébastien Han 9d345cafdf Merge pull request #9392 from y1r/add-context-k8sutil-prometheus
core: add context parameter to k8sutil prometheus
2021-12-13 10:06:09 +01:00
Yuichiro Ueno 03e976b52c core: add context parameter to k8sutil service
This commit adds context parameter to k8sutil service functions. By
this, we can handle cancellation during API call of service resource.

Signed-off-by: Yuichiro Ueno <y1r.ueno@gmail.com>
2021-12-11 11:40:28 +09:00
Yuichiro Ueno c6c8dca0a7 core: add context parameter to k8sutil prometheus
This commit adds context parameter to k8sutil prometheus functions. By
this, we can handle cancellation during API call of prometheus resource.

Signed-off-by: Yuichiro Ueno <y1r.ueno@gmail.com>
2021-12-11 11:29:24 +09:00
parth-gr 0a86d26b2e core: create rook resources with k8s recommended labels
Adding Recommended Labels on the resources created by rook
    and using Recommended Labels in the helm chart,
    for better visuals and management of k8s object

Closes: https://github.com/rook/rook/issues/8400
Signed-off-by: parth-gr <paarora@redhat.com>
2021-12-07 18:16:44 +05:30
Travis Nielsen 801b5d65a7 Merge pull request #9180 from LittleFox94/8502-monitoring-labels
#8502: Make monitoring object labels overridable
2021-11-23 09:44:06 -07:00
Travis Nielsen 1afd322650 core: ensure cluster name is available on cluster info
The cluster info is important context for the cluster controller to
create the cluster, and all the fields must be properly set.
A test cluster name was being set temporarily, resulting in
mons incorrectly getting the wrong cluster CR name. There is no
known issue from the temporary value, it was just exposed by
https://github.com/rook/rook/pull/8678 setting the value to a label.

Now the functions are more clearly named so only unit and
integration tests should be using the test value for the cluster
name where it is not important.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-11-18 15:30:26 -07:00
Mara Sophie Grosch 00a5debbab monitoring: allow overriding monitoring labels
The templates for the mgr-generated ServiceMonitor and PrometheusRule objects included the labels
prometheus and team, making it impossible to override them as user.

This adds a new method `OverwriteApplyToObjectMeta` to
pkg/apis/ceph.rook.io/v1.Labels, which, contrary to the existing
`ApplyToObjectMeta` method, overwrites existing labels.

Closes: https://github.com/rook/rook/issues/8502
Signed-off-by: Mara Sophie Grosch <littlefox@lf-net.org>
2021-11-17 17:32:05 +01:00
Yuichiro Ueno 0b575703c7 core: add context parameter to k8sutil deployment
This commit adds context parameter to k8sutil deployment functions. By
this, we can handle cancellation during API call of deployment resource.

Signed-off-by: Yuichiro Ueno <y1r.ueno@gmail.com>
2021-11-13 14:58:13 +09:00
Sébastien Han bccc84efa2 mgr: do not set the balancer mode on pacific
On Pacific, Ceph's default is "upmap", so we should let it be
like this.
This lets the user change the mode is desired.
On Octopus though, Rook continues to force the mode to "upmap".

Closes: https://github.com/rook/rook/issues/9062
Signed-off-by: Sébastien Han <seb@redhat.com>
2021-10-29 15:43:29 +02:00
subhamkrai 0150966024 ceph: remove ceph nautilus, ceph octopus to default
since rook 1.8, ceph nautilus no longer supported,
ceph octopus will be the minimum ceph version.

Closes: https://github.com/rook/rook/issues/7908
Signed-off-by: subhamkrai <srai@redhat.com>
2021-10-20 14:45:12 +05:30
Sébastien Han 8da68bfb78 mon: run ceph commands to mon with timeout
If the mons are not in quorum yet the commands interacting with mon
config store will stale for a very long time.

Closes: https://github.com/rook/rook/issues/8928
Signed-off-by: Sébastien Han <seb@redhat.com>
2021-10-08 11:57:16 +02:00
Travis Nielsen 0a0b9c98bd build: remove obsolete flex driver
The flex driver has been fully deprecated and thus removed from Rook.
Before upgrading to v1.8, users will need to convert existing flex volumes
from flex to csi volumes.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-09-23 16:17:20 -06:00
Juan Miguel Olmo Martínez 7fbd9f2225 ceph: do not use http for mgr liveness probe
When private/public network have been defined in the Ceph rook cluster it is not
possible to configure properly the ip address of the liveness probe
for the manager.
Changes in the manager in Pacific introduced this regression.

This change replaces the http probe by a command probe, avoiding thus to
determine what is going to be the ip address of the manager before launching
the pod.

fixes: https://github.com/rook/rook/issues/8510

Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
2021-09-20 13:28:51 +00:00
Sébastien Han b89730d895 ceph: refactor operator initialization sequence
This commit is a large refactor on how the operator starts, stops and
how it starts various sub-components such as the ceph-csi driver. It
also refines the way we cancel orchestrations. We don't use breakpoints
anymore but send our self a SIGUP to reload our controller runtime
manager.
The reload will happen under different circonstances like:

* a new adminission controller secret is created/deleted/changed
* a CephCluster CR is edited

As mentioned earlier, the csi driver now has its own controller, just
like flex. It reacts to change in the operator config map for particular
ROOK_CSI_ fields.

A second new controller for the operator's general config has been
created, it manages:

* the logging level
* the ceph CLI command timeout
* the discovery daemon

The operator reacts much more rapidly to cancellation events by stopping
the manager's context and reloading it.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-09-17 16:57:12 +02:00
parth-gr e50d83771e ceph: add pdb for mgr
Currently PDB for MGR is not been created,
Creating the PDB for MGR if its count is 2

Closes: https://github.com/rook/rook/issues/8275
Signed-off-by: parth-gr <paarora@redhat.com>
2021-09-03 14:30:58 +05:30
Sébastien Han d4dd0577f9 Merge pull request #8529 from sp98/id-mapping
ceph: add ClusterID and PoolID mappings between local and peer cluster
2021-08-31 17:32:14 +02:00