Commit Graph
30 Commits
Author SHA1 Message Date
Chiman Jain 2034be1f37 build: migrate gopkg.in/yaml to go.yaml.in/yaml
Signed-off-by: Chiman Jain <chimanjain15@gmail.com>
2026-07-22 20:05:39 +05:30
Joshua Hoblitt 8a72d80127 test: exercise multisite chart values in CephHelmSuite
Configure the CephHelmSuite object store as an RGW multisite zone so the
chart's CephObjectRealm, CephObjectZoneGroup, and CephObjectZone templates
are exercised on a Helm install. The operator must reconcile the realm,
zone group, zone, and store to Ready, which the existing install check
already gates on via the RGW pod count.

The realm/zone group/zone and shared-pool layout mirror the fixture in
tests/integration/object/util/sharedstore: a zone backed by explicit
CephBlockPools referenced through sharedPools.poolPlacements. The rgw pools
are appended to the block pools already configured for CSI, and are torn
down along with the multisite CRs when the default storage CRs are removed.

Gate the multisite configuration behind a new UseMultisiteObjectStore
setting enabled only for CephHelmSuite. The other Helm-based suites keep the
plain object store; the upgrade suite in particular installs an older base
chart that lacks the multisite templates and cannot resolve a zone-based
store.

Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
2026-07-09 09:59:18 -07:00
subhamkrai a3e02e607d csi: update csi-operator version to v1.0.4
Updating csi-operator to latest v1.0.4 and
updating the required doc changes as well.

Signed-off-by: subhamkrai <srai@redhat.com>
2026-07-07 07:56:55 -06:00
Joshua Hoblitt cff8b7f311 ci: reduce CephUpgradeSuite flakes
The CephUpgradeSuite workflow failed ~31% of PR runs vs the ~19%
broken-PR baseline of the other integration suites. Classifying the
failing step of every failed run from the last ~400 PR runs and the
logs of every failure on non-dependabot branches shows the excess
comes from a handful of too-tight waits, unretried network fetches,
and environment races rather than from the upgrade logic itself:

- The biggest class (21 of 44 genuine test-phase failure runs):
  "giving up waiting for deployment(s) with label
  app=rook-ceph-{rgw,osd},ceph-version!=..." during the
  Squid->Tentacle upgrade. The operator updates daemons sequentially
  (mons, mgr, osds with PG health gates between them, mds, rgw), so
  each later daemon's fixed 275s wait also absorbs the time spent on
  every daemon before it; rgw, last in the sequence, failed most
  often. The mon wait was already extended for slow image pulls in
  98bd03a42; extend the same retry count to all the daemon waits.

- ~7 runs: "snapshot controller is not ready" in the Helm upgrade
  path. WaitForSnapshotController(30) allows only 150s, and the logs
  show the deployment still converging (readyreplicas 1 < replicas 2)
  on the final poll. Raise to 90 retries.

- InstallOrUpgradeHelmRepoChart ran helm with no retry; an observed
  failure fetched the ceph-csi-drivers chart tarball from GitHub
  release assets and got a 504. Retry up to 5 times, as
  InstallLocalHelmChart already does.

- Raise the go test timeouts (2400s->3600s rook, 1800s->2700s helm).
  Failed runs ended in "panic: test timed out" during teardown, which
  aborts cleanup and log collection; the longer daemon waits above
  also need the headroom.

- The "setup cluster resources" composite action (shared by all
  suites) accounted for half of all failed jobs. The one steady class
  there (8 distinct runs across 7 different days): minikube exits with
  K8S_FAIL_CONNECT (code 40) when the requested kubernetes version is
  missing from its bundled version list, because it then validates the
  version with an anonymous GitHub API request (GITHUB_TOKEN is not
  honored on that code path), and anonymous requests from shared
  runner IPs are regularly rate-limited. minikube 1.38.x predates
  v1.35.5, so only the v1.35.5 jobs hit this class. Pass --force to
  minikube start to skip that check: the kubernetes versions used in
  CI are pinned constants already validated by the PRs that bump
  them, and an invalid version would still fail fast at the kubeadm
  download. Also seen twice: dpkg failing on a corrupt cri-dockerd .deb
  because curl ran without --fail and saved an HTTP error page as the
  package; add --fail and retries.

- On runners without the /mnt resource disk, the fallback OSD disk is
  an iSCSI (LIO) device and use_local_disk_for_integration_test
  returned early on those runners, skipping the udev nowatch
  workaround for the device re-probe storms of rook issue 8975. A burn-in
  failure on this PR captured the consequence with full kernel
  forensics: ~100 udev change events on the OSD disk, and ceph-volume
  activate wedged in uninterruptible sleep on the block device lock
  (blkdev_llseek -> rwsem_down_write_slowpath) for 18+ minutes during
  the ceph version upgrade while the cluster's only OSD stayed down.
  Install the nowatch rule before the early return so it applies to
  every runner type, and before the disk is first written rather than
  after.

- One burn-in round failed before the upgrade even began: the
  pre-upgrade PVC create on the Helm path expired WaitUntilPVCIsBound
  (RetryLoop, 275s) while the CSI provisioner pods were still
  starting. Set RETRY_MAX=110 for this workflow to double the
  framework's base wait budget, instead of extending individual waits
  one flake at a time.

- minikube also exits with GUEST_START (code 80) when its internal 6
  minute node-ready wait expires on slow runners (5 runs in the
  dataset, two more observed while burning in this PR, one of them on
  another suite). Pass --wait-timeout=15m to minikube start.

Not addressed (small or episodic classes): OSDs never coming up on
initial deploy (3 runs, possibly the same udev/iSCSI wedge), the
filesystem not becoming active (2 runs), and several day-clustered
minikube incidents (exit codes 65/67/90).

Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
2026-06-11 15:27:02 -07:00
subhamkrai 34c5ad7049 build: update csi-operator to v1.0.1
Updating csi-operator to latest v1.0.1 and
updating the required doc changes as well.

Signed-off-by: subhamkrai <srai@redhat.com>
2026-05-27 18:05:56 +05:30
subhamkraiandTravis Nielsen 06452991bf csi: update csi-operator to v1.0.0
this commit update the csi-operator to latest v1.0.0.
we need to manually patch the csi drivers with the new serviceAccount
name that are being creating based on the latest ceph-csi-operator
release v1.0.0 where serviceAccount are being separate from csi-operator
chart.

co-authored-by: Travis Nielsen <tnielsen@redhat.com>
Signed-off-by: subhamkrai <srai@redhat.com>
2026-05-21 21:52:32 +05:30
subhamkrai 4eefad42e8 csi: move csi management to admin
Going forward, admin will manage the csi operator
CR's and rook will only manage Ceph Connection cr
and client Profile cr.

The old csi driver is completely removed from Rook
and can no longer be used starting in Rook v1.20.

The upgrade guide will contain the needed transition steps
for managing the csi operator settings.

Signed-off-by: subhamkrai <srai@redhat.com>
2026-04-29 14:24:26 -06:00
subhamkraiandTravis Nielsen c57a47f774 ci: run csi-operator only in canary and upgrade suite
this commit add check to only run the csi-operator in
all the canary tests and upgrade suite only, other suite
like smoke and object will still test csi-driver.

Also, adding changes to make CI happy.

Signed-off-by: subhamkrai <srai@redhat.com>
Co-Authored-by: Travis Nielsen <tnielsen@redhat.com>
Signed-off-by: subhamkrai <srai@redhat.com>
2025-08-19 11:23:14 +05:30
subhamkrai 8bd4b38399 test: add pathType with ingress dashboard host
Recentally, helm related ci started failing due to
```
testutil: failed to install chart. result=. walk.go:75: found symbolic link in path: /home/runner/work/rook/rook/deploy/charts/rook-ceph-cluster/charts/library resolves to /home/runner/work/rook/rook/deploy/charts/library. Contents of linked file included and used
I0714 09:44:18.963620   18291 warnings.go:110] "Warning: path /ceph-dashboard(/|$)(.*) cannot be used with pathType Prefix"
Error: INSTALLATION FAILED: 1 error occurred:
	* admission webhook "validate.nginx.ingress.kubernetes.io" denied the request: ingress contains invalid paths: path /ceph-dashboard(/|$)(.*) cannot be used with pathType Prefix, err=failed to run helm command on args [install --create-namespace rook-ceph-cluster /home/runner/work/rook/rook/deploy/charts/rook-ceph-cluster --namespace helm-ns -f values-test.yaml] : . walk.go:75: found symbolic link in path: /home/runner/work/rook/rook/deploy/charts/rook-ceph-cluster/charts/library resolves to /home/runner/work/rook/rook/deploy/charts/library. Contents of linked file included and used
I0714 09:44:18.963620   18291 warnings.go:110] "Warning: path /ceph-dashboard(/|$)(.*) cannot be used with pathType Prefix"
Error: INSTALLATION FAILED: 1 error occurred:
```
as default `pathType` is `Prefix`, we need to change it to
`pathType:ImplementationSpecific`.

Signed-off-by: subhamkrai <srai@redhat.com>
2025-07-15 12:50:20 +05:30
Carlos Barria 030ada45b0 core: fix golangci-lint check QF1003 QF1002
Refactored multiple conditional if-else blocks into switch statements to improve readability and maintainability. Also removed outdated staticcheck rule comments (QF1002, QF1003) from .golangci.yaml.

Signed-off-by: Carlos Barria <cbarria@yahoo.com>
2025-05-27 13:49:49 -04:00
Joshua Hoblitt 3cb343f62a core: run gofumpt on all files
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
2025-03-26 10:41:48 -07:00
Travis Nielsen 810de394e7 helm: add enforce host network setting
The ROOK_ENFORCE_HOST_NETWORK option was implemented recently
and now we add the helm setting to expose this new setting
in the rook chart.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2024-10-02 15:14:46 -06:00
ee8bcad49d rgw: add support for keystone auth + swift/s3
For the specification see:
<https://github.com/rook/rook/blob/master/design/ceph/object/swift-and-keystone-integration.md>

* extend the API object specs for swift and keystone integration

* adapt rgw to the new go-ceph version

  - The parameter lists of the API call have changes, as parameters
    ignored by the RGW Admin Ops API are no longer serialized, therefore
    the mock has to be adapted.

  - There is now validation for the user keys that are passed to the
    User get API, therefore things failed when we had empty keys in our
    User proxy object.

* expand the reconcile loop for the swift and keystone integration

* fix minor mistakes in design document

* add env var to pass extra args to minikube

  Minikube decides CPU cores and memory automatically based on the
  available resources on the machine which may be insufficient to
  run rook. This commit adds an environment variable to add arbitrary
  arguments to the minikube command, so both can be specified if
  desired.

* integration tests for swift and keystone

  The new integration of swift or s3 and keystone support by rook
  does not have any integration tests yet.

  This commit introduces integration tests for swift and keystone. The
  tests are done against a minimal keystone setup (keystone container
  image from Yaook-project (https://yaook.cloud), sqlite as database
  backend, cert-manager and trust-manager for test certificate setup).

  To prevent hardcoded credentials, passwords are generated
  by the tests. The integration tests use the openstack client
  (keystone- and swift-functionality) (https://docs.openstack.org/
  python-openstackclient/ latest/). This was a concious design decision
  to use client tooling as close as possible to the end user instead of
  using other go-libraries (such as gophercloud).

* add documentation on swift and keystone

  Currently there is no documentation on the use of Swift to access
  an object store as well as the use of OpenStack keystone for
  authentication.

  This commit adds documentation on the use of Swift and OpenStack
  keystone, as well as CRD-related documentation and an example setup.

* add integration tests for S3 via keystone

  This commit introduces integration tests for s3 and keystone. The
  tests are run against the same minimal keystone setup that the tests
  for swift and keystone use.

  The integration tests use the aws s3 client to use client tooling as
  close as possible to the end user instead of using other go-libraries.

Co-authored-by: Jan Klippel <jan.klippel@uhurutec.com>
Co-authored-by: Silvio Ankermann <silvio.ankermann@cloudandheat.com>
Signed-off-by: Sebastian Riese <sebastian.riese@cloudandheat.com>
Signed-off-by: Jan Klippel <jan.klippel@uhurutec.com>
Signed-off-by: Silvio Ankermann <silvio.ankermann@cloudandheat.com>
2024-08-08 14:26:21 +02:00
subhamkrai d8766ff871 ci: remove ceph SVGs in helm test
Both the helm tests are failing because,
```
2023-12-19 08:47:06.136640 E | ceph-file-controller: failed to reconcile CephFilesystem "helm-ns/ceph-filesystem-test". CephFilesystem "helm-ns/ceph-filesystem-test" will not be deleted until all dependents are removed: CephFilesystemSubVolumeGroups: [ceph-filesystem-test-csi]
```
So, let's remove the ceph SVGs before removing Ceph filesystem.

Signed-off-by: subhamkrai <srai@redhat.com>
2023-12-19 17:05:21 +05:30
subhamkrai 28cc1ebc55 core: remove webhook & controller-runtime from apis
This commits removes controller-runtime dependencies
from the apis dir and to achieve that we are removing
webhook.

Signed-off-by: subhamkrai <srai@redhat.com>
2023-12-01 14:15:40 +05:30
Loïc Antoine-GombeaudandTravis Nielsen 59f6cd00ff helm: fix toolbox command, default to cluster image
Signed-off-by: Loïc Antoine-Gombeaud <lantoinegombeaud@gestform.com>
Co-authored-by: Travis Nielsen <tnielsen@redhat.com>
2023-01-04 14:23:37 -07:00
Travis Nielsen a6dfb77827 build: update min version to k8s 1.19
Rook will support the most recent seven K8s releases,
so we update the min version to 1.19 for Rook v1.10.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2022-08-30 15:15:20 -06:00
Josh Soref 6e7b8767f3 core: fix spelling
* another
* are
* availability
* available
* bootstrap
* boundaries
* ceph
* certificate
* class
* codifies
* consuming
* corrupted
* createor
* csi
* deployments
* exceeded
* execute
* filesystem
* healthiness
* heuristics
* immediately
* insecure
* installed
* isolated
* maintained
* maximum
* minute
* monitor
* new
* nginx
* nonexistent
* not
* occurs
* omitempty
* operator
* orchestration
* persistentvolumes
* placement
* preexisting
* prometheus
* protecting
* provisioner
* purposes
* reconcile
* regex
* related
* requests
* returns
* rubbish
* running
* schedulable
* schedule
* serviceaccount
* simulating
* snapshots
* statement
* static
* tenants
* the
* unavailable
* volumeattachment
* waiting
* with
* wrapper
* zonegroup

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
2022-07-07 18:10:47 -04:00
Travis Nielsen d26b6cf403 build: update min version to k8s 1.17
Support is removed for k8s for various limitations such as
priority classes not working and csi driver feature
incompleteness and missing snapshots. Documentation is updated
with the new min version of k8s 1.17 and also the tests are
updated to run on the min version of 1.17.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2022-04-25 15:32:09 -06:00
Travis Nielsen fb86955f01 core: examples set default priority class names
By default, we should set the priority class to one of the built-in
priority class names to ensure that pods critical to the storage
will be able to remain running when resources are low. Otherwise,
critical rook pods could be evicted and affect many other pods
that rely on the storage to continue functioning. The options have
been available in the CRs, but until now we have just not set the
defaults in the examples. Critical rook components are now set to
the priority class system-node-critical if they are generally pinned
to a node, and system-cluster-critical if they are critical to the storage.
Some pods such as the operator and crash collector do not have a
default priority class set in the examples since they don't affect
the data path.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2022-04-19 16:09:34 -06:00
Jiffin Tony Thottan fc2b8012c6 object: use us-east-1 for aws go lang sdk
The aws go lang sdk needs value for region, it is set differently in
various part of current code. With PR the value is always `us-east-1` so
that it will work RGW server without any issues.

This reverts commit 280c29f330.

Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
2022-03-24 14:37:39 +05:30
Travis Nielsen 4edcff04f7 monitoring: create prometheus rules with helm chart
The prometheus rules had been previously created if the cephcluster CR
setting monitoring.enabled was set to true. The rules were not customizable
and therefore not flexible enough. Now the rules are installed by the helm
chart. To customize the rules, a post-processor can be applied to the helm
chart.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2022-03-21 14:47:52 -06:00
Yuval Manor ff7a5c2d2d helm: enable resource defaults on rook components
This PR assign default values to the resources of all Rook and Ceph components

Closes: https://github.com/rook/rook/issues/9858
Signed-off-by: Yuval Manor <yuvalman958@gmail.com>
2022-03-21 21:23:35 +02:00
Travis Nielsen 741f211a13 helm: apply operator settings in configmap instead of deployment
Some operator settings can be applied dynamically in a configmap
instead of requiring the operator to restart. While these settings
may be infrequently updated, applying these settings in the configmap
can avoid an unnecessary operator restart. This will also make the
operator deployment more consistent with the non-helm install.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2022-01-20 14:41:28 -07:00
Travis Nielsen 8fb758fee9 test: implement helm upgrade integration test
The helm tests were previously only for new installs, and did
not have an upgrade path. Now the upgrade path is tested
to give confidence in the helm upgrades.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2022-01-12 07:19:42 -07:00
Tom Hellier ba44602477 helm: allow further configurability of ingress version
The ingress api version changed when it went to v1, and this has caused some upheaval
throughout the kubernetes ecosystem. This commit uses a common method of deciding which
ingress api to use, and allows the optional override of the kubernetes version
presented to helm using the helm build-in capabilities.
also add an ingress into the helm integration tests so any regressions to how ingresses
are handled in the future are caught easier.

Closes rook#9174

Signed-off-by: Tom Hellier <me@tomhellier.com>
2021-11-22 10:03:02 +00:00
Travis Nielsen 9ff0753514 test: run all integration tests against the local build
The integration tests must always be run against the local
build of rook, and an image should never be pulled from dockerhub.
To prevent pulling a release or master tag, the local build
will use a tag specific to the build and not ever published
elsewhere.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
(cherry picked from commit a8a40428b0)
2021-09-22 07:39:42 -06:00
Sébastien Han b89730d895 ceph: refactor operator initialization sequence
This commit is a large refactor on how the operator starts, stops and
how it starts various sub-components such as the ceph-csi driver. It
also refines the way we cancel orchestrations. We don't use breakpoints
anymore but send our self a SIGUP to reload our controller runtime
manager.
The reload will happen under different circonstances like:

* a new adminission controller secret is created/deleted/changed
* a CephCluster CR is edited

As mentioned earlier, the csi driver now has its own controller, just
like flex. It reacts to change in the operator config map for particular
ROOK_CSI_ fields.

A second new controller for the operator's general config has been
created, it manages:

* the logging level
* the ceph CLI command timeout
* the discovery daemon

The operator reacts much more rapidly to cancellation events by stopping
the manager's context and reloading it.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-09-17 16:57:12 +02:00
Tom Hellier 89ab4f90ec ceph: adds helm functionality for ingress, and ceph storage crds
This commit adds an ingress resource to the rook-ceph-cluster helm chart, allowing
ingress to the ceph-dashboard service. It also adds the ability to define the
various storage types that you can run on ceph inside kubernetes.

Closes https://github.com/rook/rook/issues/8384

Signed-off-by: Tom Hellier <me@tomhellier.com>
2021-07-28 21:51:56 +01:00
Henry Zhang ecf6bdcfae ceph: add build and tests for rook-ceph-cluster Helm chart
Adds rook-ceph-cluster chart to chart build, add tests.
Pulled out some common functionality between the Helm and non-Helm installers

Signed-off-by: Henry Zhang <me@henry.dev>
2021-05-27 01:08:27 -07:00