The CSI operator is installed unconditionally by the integration test
installer, so the EnableCsiOperator setting no longer selects between
CSI management modes and is consumed nowhere in the test framework.
Remove the setting and the three suites' usages of it.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
The package-level VersionAtLeast function had no callers anywhere in the
repo (the K8sHelper.VersionAtLeast method remains in use), and
IsKubectlErrorNotFound's only caller was provisioners.go, which is
deleted in this PR. Verified with deadcode and repo-wide grep.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
The CephUpgradeSuite workflow failed ~31% of PR runs vs the ~19%
broken-PR baseline of the other integration suites. Classifying the
failing step of every failed run from the last ~400 PR runs and the
logs of every failure on non-dependabot branches shows the excess
comes from a handful of too-tight waits, unretried network fetches,
and environment races rather than from the upgrade logic itself:
- The biggest class (21 of 44 genuine test-phase failure runs):
"giving up waiting for deployment(s) with label
app=rook-ceph-{rgw,osd},ceph-version!=..." during the
Squid->Tentacle upgrade. The operator updates daemons sequentially
(mons, mgr, osds with PG health gates between them, mds, rgw), so
each later daemon's fixed 275s wait also absorbs the time spent on
every daemon before it; rgw, last in the sequence, failed most
often. The mon wait was already extended for slow image pulls in
98bd03a42; extend the same retry count to all the daemon waits.
- ~7 runs: "snapshot controller is not ready" in the Helm upgrade
path. WaitForSnapshotController(30) allows only 150s, and the logs
show the deployment still converging (readyreplicas 1 < replicas 2)
on the final poll. Raise to 90 retries.
- InstallOrUpgradeHelmRepoChart ran helm with no retry; an observed
failure fetched the ceph-csi-drivers chart tarball from GitHub
release assets and got a 504. Retry up to 5 times, as
InstallLocalHelmChart already does.
- Raise the go test timeouts (2400s->3600s rook, 1800s->2700s helm).
Failed runs ended in "panic: test timed out" during teardown, which
aborts cleanup and log collection; the longer daemon waits above
also need the headroom.
- The "setup cluster resources" composite action (shared by all
suites) accounted for half of all failed jobs. The one steady class
there (8 distinct runs across 7 different days): minikube exits with
K8S_FAIL_CONNECT (code 40) when the requested kubernetes version is
missing from its bundled version list, because it then validates the
version with an anonymous GitHub API request (GITHUB_TOKEN is not
honored on that code path), and anonymous requests from shared
runner IPs are regularly rate-limited. minikube 1.38.x predates
v1.35.5, so only the v1.35.5 jobs hit this class. Pass --force to
minikube start to skip that check: the kubernetes versions used in
CI are pinned constants already validated by the PRs that bump
them, and an invalid version would still fail fast at the kubeadm
download. Also seen twice: dpkg failing on a corrupt cri-dockerd .deb
because curl ran without --fail and saved an HTTP error page as the
package; add --fail and retries.
- On runners without the /mnt resource disk, the fallback OSD disk is
an iSCSI (LIO) device and use_local_disk_for_integration_test
returned early on those runners, skipping the udev nowatch
workaround for the device re-probe storms of rook issue 8975. A burn-in
failure on this PR captured the consequence with full kernel
forensics: ~100 udev change events on the OSD disk, and ceph-volume
activate wedged in uninterruptible sleep on the block device lock
(blkdev_llseek -> rwsem_down_write_slowpath) for 18+ minutes during
the ceph version upgrade while the cluster's only OSD stayed down.
Install the nowatch rule before the early return so it applies to
every runner type, and before the disk is first written rather than
after.
- One burn-in round failed before the upgrade even began: the
pre-upgrade PVC create on the Helm path expired WaitUntilPVCIsBound
(RetryLoop, 275s) while the CSI provisioner pods were still
starting. Set RETRY_MAX=110 for this workflow to double the
framework's base wait budget, instead of extending individual waits
one flake at a time.
- minikube also exits with GUEST_START (code 80) when its internal 6
minute node-ready wait expires on slow runners (5 runs in the
dataset, two more observed while burning in this PR, one of them on
another suite). Pass --wait-timeout=15m to minikube start.
Not addressed (small or episodic classes): OSDs never coming up on
initial deploy (3 runs, possibly the same udev/iSCSI wedge), the
filesystem not becoming active (2 runs), and several day-clustered
minikube incidents (exit codes 65/67/90).
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
Delete three files from tests/framework that are entirely dead code; no
symbol in any of them is referenced anywhere in the repo, including the
build-tagged integration suites (verified with deadcode and repo-wide
grep):
- clients/read_write.go: abandoned NFS test client (ReadWriteOperation
and its methods)
- installer/provisioners.go: hostpath PV helpers orphaned when the
cassandra and nfs installers were removed
- installer/device.go: IsAdditionalDeviceAvailableOnCluster, unused
since the upgrade test rework
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
Remove the AWS SDK v1 (github.com/aws/aws-sdk-go)
dependency entirely. All S3 operations now use AWS SDK v2
exclusively.
- Remove the v1 Client field from S3Agent struct and
rename ClientV2 to Client
- Remove v1 session/client initialization from NewS3Agent
- Update all call sites referencing ClientV2
- Convert integration tests to use v2 API calling
conventions (context parameter) and smithy error handling
- Remove aws-sdk-go v1.55.8 from go.mod
Signed-off-by: Oded Viner <oviner@redhat.com>
this commit update the csi-operator to latest v1.0.0.
we need to manually patch the csi drivers with the new serviceAccount
name that are being creating based on the latest ceph-csi-operator
release v1.0.0 where serviceAccount are being separate from csi-operator
chart.
co-authored-by: Travis Nielsen <tnielsen@redhat.com>
Signed-off-by: subhamkrai <srai@redhat.com>
migrate the notification provisioner and s3ext packages from
aws sdk v1 to v2. the custom DeleteBucketNotification call is
rewritten to manually build and sign the HTTP request using the
v2 v4 signer, since this ceph-specific API has no sdk equivalent.
also fix t.Skipped() -> t.Skip() in notification integration test.
Signed-off-by: Oded Viner <oviner@redhat.com>
Going forward, admin will manage the csi operator
CR's and rook will only manage Ceph Connection cr
and client Profile cr.
The old csi driver is completely removed from Rook
and can no longer be used starting in Rook v1.20.
The upgrade guide will contain the needed transition steps
for managing the csi operator settings.
Signed-off-by: subhamkrai <srai@redhat.com>
The upgrade integration tests for the v1.20 release need to
validate the upgrade starting from v1.19 to the latest.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
the rook-ceph-nvmeof service account is not listed in the
securitycontextconstraints users, causing cephnvmeofgateway
pods to fail on openshift. add the service account to the
scc in the helm charts, go scc builder, and test namespace
substitution.
Signed-off-by: Oded Viner <oviner@redhat.com>
Updated the following csi sidecars to their latest available versions:
- csi-attacher: v4.11.0
- csi-snapshotter: v8.5.0
- csi-resizer: v2.1.0
- csi-provisioner: v6.1.1
- csi-node-driver-registrar: v2.16.0
Signed-off-by: Praveen M <m.praveen@ibm.com>
The upgrade test is expected to check from the previous minor
release to the current master or latest release. With the
release of 1.19 coming soon, the upgrade test will now
check from v1.18.x.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
For Rook v1.19,removing support for Ceph Reef. It is at end of life.
Users on Reef can continue to use Rook v1.18.x or older.
The latest two Ceph versions Squid and Tentacle are supported.
Signed-off-by: subhamkrai <srai@redhat.com>
Use a sync map for the clusterMap in the cluster controller
and fix other issues to allow for concurrent cluster
reconcile.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
this commit add check to only run the csi-operator in
all the canary tests and upgrade suite only, other suite
like smoke and object will still test csi-driver.
Also, adding changes to make CI happy.
Signed-off-by: subhamkrai <srai@redhat.com>
Co-Authored-by: Travis Nielsen <tnielsen@redhat.com>
Signed-off-by: subhamkrai <srai@redhat.com>
For the integration tests, now we start upgrading from v1.17.x
to the master branch, to test that the upgrades are stable
to what will become the 1.18 release.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Recentally, helm related ci started failing due to
```
testutil: failed to install chart. result=. walk.go:75: found symbolic link in path: /home/runner/work/rook/rook/deploy/charts/rook-ceph-cluster/charts/library resolves to /home/runner/work/rook/rook/deploy/charts/library. Contents of linked file included and used
I0714 09:44:18.963620 18291 warnings.go:110] "Warning: path /ceph-dashboard(/|$)(.*) cannot be used with pathType Prefix"
Error: INSTALLATION FAILED: 1 error occurred:
* admission webhook "validate.nginx.ingress.kubernetes.io" denied the request: ingress contains invalid paths: path /ceph-dashboard(/|$)(.*) cannot be used with pathType Prefix, err=failed to run helm command on args [install --create-namespace rook-ceph-cluster /home/runner/work/rook/rook/deploy/charts/rook-ceph-cluster --namespace helm-ns -f values-test.yaml] : . walk.go:75: found symbolic link in path: /home/runner/work/rook/rook/deploy/charts/rook-ceph-cluster/charts/library resolves to /home/runner/work/rook/rook/deploy/charts/library. Contents of linked file included and used
I0714 09:44:18.963620 18291 warnings.go:110] "Warning: path /ceph-dashboard(/|$)(.*) cannot be used with pathType Prefix"
Error: INSTALLATION FAILED: 1 error occurred:
```
as default `pathType` is `Prefix`, we need to change it to
`pathType:ImplementationSpecific`.
Signed-off-by: subhamkrai <srai@redhat.com>
With the tentacle release approaching, let's start
running the Rook tests against the tentacle devel
images to catch if any issues.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Refactored multiple conditional if-else blocks into switch statements to improve readability and maintainability. Also removed outdated staticcheck rule comments (QF1002, QF1003) from .golangci.yaml.
Signed-off-by: Carlos Barria <cbarria@yahoo.com>
The Kubernetes CSI sidecars have had several releases that were not
included in deployments by Rook yet, update them to the versions that
are available today:
- csi-attacher:v4.8.1
- csi-provisioner:v5.2.0
- csi-resizer:v1.13.2
- csi-snapshotter:v8.2.1
This change is important, because Ceph-CSI will implement the new
Controller.GetSnapshot CSI procedure. A bug in csi-lib-utils causes a
panic when a ControllerCapability is provided, but not (yet) known to
the CSI sidecars. The updated sidecars consume a version of
csi-lib-utils with a fix for that panic.
See-also: kubernetes-csi/csi-lib-utils#188
Signed-off-by: Niels de Vos <ndevos@ibm.com>
When upgrading from one Ceph version to another, the new image to
upgrade to can take a long time to pull in some cases before the upgrade
can even begin. For example, some ceph-ci images regularly take 6+
minutes to pull, which exceeds the timeout waiting for mons to be ready.
Extend the timeout for mons to account for these cases.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
this PR updates the Prometheus Operator URL references from
version v0.71.1 to the latest release v0.81.0 in documentation
and integration test scripts. This ensures we are aligned with
the latest features and improvements from
the Prometheus Operator project.
Signed-off-by: Oded Viner <oviner@redhat.com>
userSecretRef and passwordSecretRef fields are added to allow the Kafka
endpoint username and password to be supplied from a Kubernetes Secret,
rather than exposed as plaintext as part of the endpoint URI. If the
endpoint URI has HTTP basic auth user-id and user-pass components, they
are overridden by userSecretRef and passwordSecretRef.
Squid added bucket topic attributes for configuring the user-name and
password for pushing notifications to Kafka as an alternative to
encoding credentials into the URI. However, these attributes are not
supported under Reef. Thus, this initial implementation relies on URI
mangling for compatible with both Reef and Squid. Future work could
using Ceph version detection and switch to the new attributes to be used
with Squid and/or the implementation could be converted exclusively to
use the new attributes once Rook has dropped support for Reef.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
The tests sometimes see that the mons run out of space
in the CI, therefore, we reduce the space required
from 30% down to 10%.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The upgrade tests in master have been upgrading from 1.15 to
master. In anticipation of the v1.17 release, we change
the upgrade tests to start from v1.16 to test if there
are any regressions in the supported upgrades.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
One ore more OSDs can be down and PGs can still be active+clean (data
was rebalanced to other available OSDs). This PR sets
maxunavailable=1+downOSDs to allow one healthy to be drained.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
Create the CSI operator in the go integration test suites
to test the new CSI operator scenarios. For the upgrade
test suite, enable the CSI operator after the upgrade
to verify the working cluster after it is enabled.
Signed-off-by: subhamkrai <srai@redhat.com>
Implement an allow list mechanism that disables potentially unsafe OBC
fields by default. OBC fields beyond `maxObjects` and `maxSize` don't
neatly fit into the OBC framework as it was originally envisioned and
implemented.
Some of the newly added configs could allow users to cause confusion for
themselves. Others might allow users to hijack others buckets. Some
might allow bricking the entire S3 store.
Out of an abundance of safety, allow-list the known-safe options by
default, and require administrators to enable potentially troublesome
options via the new operator-level config
`ROOK_OBC_ALLOW_ADDITIONAL_CONFIG_FIELDS`.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
The ci was using a pretty old version og golangci-lint.
This updates to the latest version.
Additionally, it silences some
gosec integer conversion overflow false positves
and fixes some real errors of this category
and string format errors found by golangci-lint, while at it.
Co-authored-by: Blaine Gardner <b.blaine.gardner@gmail.com>
Co-authored-by: Travis Nielsen <tnielsen@redhat.com>
Signed-off-by: Michael Adam <obnox@samba.org>
Ceph has changed the image build process to only require a
dockerfile and stop using the ceph-container repo. The
daily images are pushed to the quay.io/ceph-ci/ceph repo,
so the Rook CI will now start using those images.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The upgrade tests in master have been upgrading from 1.14 to
master. In anticipation of the v1.16 release, we change
the upgrade tests to start from v1.15 to test if there
are any regressions in the supported upgrades.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The devel images have been fairly stable for Rook to test
against, but on occasion there are regressions from
Ceph development that affect the Rook CI. For stability during
Rook development, use the latest stable version of ceph for
PRs, master, and release tests. The daily CI will still use
the devel images from Ceph.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The mon canaries may be created even when the mon daemons
are not created thereafter during the integration tests.
Therefore, the integration tests need to also query a label
specific to the mon daemon so the canaries are not a distraction
to the test.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The v1beta1 cron jobs have been obsolete since K8s 1.21,
and Rook has not supported that version of K8s
for many moons, so we can remove the obsolete code
for the handling of v1beta1 cron jobs for crash pruning.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Given that Ceph Quincy (v17) is past end of life,
remove Quincy from the supported Ceph versions,
examples, and documentation.
Supported versions now include only Reef and Squid.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The ROOK_ENFORCE_HOST_NETWORK option was implemented recently
and now we add the helm setting to expose this new setting
in the rook chart.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The docker.io image prefix is expected to be prepended
to the image names in the test images. This was missed
in 14550 related to some CI tests, which was now causing
the CI failures in the 1.15 branch where the search and
replace was missing the new docker.io prefix.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
(cherry picked from commit 3045076db8)
Travis Nielsen
Joshua Hoblitt
Oded Viner
subhamkrai
Praveen M
Michael AdamandTravis Nielsen
Carlos Barria
Niels de Vos
Blaine Gardner
Santosh Pillai
Steven Kreitzer
