Removed the leading v from the Helm OCI chart version tag so that helm pull can auto-resolve the latest version without specifying the tag explicitly.
Signed-off-by: Prabhala Tara Aasrita <taraasrita@ibm.com>
before the csi op, the unblish grpc call was not present,
and it has no op
But with csi op, the unpublish grpc is called before the
volumeattachment get remove, and the unpublish call need a
unpublish secret, currently it is getting the secret from the
client profile, but the client profile secret is hardcoded to default
So to override this secret we are adding the secret name in the
storageclass parameter
Signed-off-by: parth-gr <partharora1010@gmail.com>
build_rook() captures the make output into $o and inspects it for
transient network errors (connection reset, INTERNAL_ERROR, 503, 500)
to decide whether to retry the build. The capture was stdout-only,
but go and make write those errors to stderr, so $o never matched any
retry case. A transient module-proxy failure (e.g. proxy.golang.org
returning INTERNAL_ERROR while downloading modules) therefore fell
through to the "valid failure" branch and failed the job in the build
step, before the integration test even ran.
Redirect stderr into the captured output (2>&1) so the existing retry
logic actually triggers on network failures.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
The raw-disk-with-object, osd-with-metadata-device, and pvc canary jobs
used loopback devices to supply an extra block device beyond the single
spare disk a GitHub runner provides. Loop devices behave differently
from real block devices -- Rook gates them behind
ROOK_CEPH_ALLOW_LOOP_DEVICES and special-cases them in the OSD daemon --
so exercising core scenarios on them is less realistic than using real
SCSI disks.
Provision the extra device(s) over iSCSI (LIO) instead, reusing the
existing create_extra_disk helper: it now creates N LUNs, the new
find_extra_block_devs and find_second_block_dev helpers enumerate them,
and deploy_cluster gains a two_raw_disks mode. The three jobs now run
entirely on real disks, and the canary CI no longer creates loop devices
or relies on the loop-device test helpers.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
These shell scripts under tests/scripts are referenced nowhere in the
repository -- no workflow, Makefile, Go test, documentation, or
.mergify.yml invokes them:
- tests/scripts/multi-node/ (build-rook.sh, rpm-system-prerequisites.sh,
config.rb): the abandoned Vagrant multi-node VM harness, pinned to the
long-EOL centos/7 box.
- tests/scripts/kubeadm-install.sh: an old kubeadm/apt installer for a
pinned Kubernetes version, no longer wired into any test path.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
Three shell functions defined under tests/scripts are never invoked --
not internally, not from another script, and not dispatched by name from
any CI workflow:
- deploy_csi_hostnetwork_disabled_cluster (github-action-helper.sh)
- wait_for_ceph_csi_configmap_to_be_updated (github-action-helper.sh)
- test_demo_pool (validate_cluster.sh): the daemon validation loop never
passes a "pool" daemon, so this branch is unreachable.
Removing them orphans no other helpers; each only called functions that
remain in use elsewhere.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
CephMgrSuite (ceph_mgr_test.go) exercised the Rook orchestrator mgr
module through `ceph orch` commands. Remove the suite together with its
GitHub Actions workflow (integration-test-mgr-suite.yaml), whose sole
purpose was to run this suite, gated behind the run-mgr-suite label and
a nightly cron.
No other code references the deleted suite. Its helpers (logBytesInfo,
the defaultTries constant, and the host/service/serviceStatus types)
were all local to the file, so nothing is left orphaned.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
When the operator starts, there is no need to handle the
node watcher immediately while the controller runtime
cache is being initialized. The cluster will anyway be
fully reconciled and create new OSDs if needed. The node
watcher is only intended for triggering a reconcile when
a node is added or updated at a later time after a sync
would anyway be triggered.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
currently k8s add the default eviction time as 300 sec
setting it to 5 sec for the quick re-schedule
Signed-off-by: parth-gr <partharora1010@gmail.com>
The bucket sync checkpoint confirms the bucket's data sync markers are
caught up at the RADOS level, but the reading RGW can briefly serve 404
for the freshly synced object until it refreshes its view. The post-
checkpoint read was given only a 60s budget, which is occasionally too
short; give it 180s so a transient RGW serving lag does not fail the
test.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
The reusable canary workflow declares a default ceph_images value that is never used: both callers always pass an explicit image list. A stale default is a drift hazard because a new caller could silently exercise the wrong image. Drop the default and mark the input required so callers must choose deliberately.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
Canary jobs upload their logs under an artifact named by job and ceph image, but the job matrix also varies kubernetes-version. When more than one kubernetes version is exercised the upload-artifact step fails on a duplicate artifact name. Append the kubernetes version so each matrix cell produces a unique artifact.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
The validate-rgw-endpoint step gave each validation loop only 15
seconds. A single attempt of the external script can exceed that
budget when the RGW is slow to respond right after the object store
was created, killing the step before any retry could happen. This
showed up as sporadic canary job failures with exit code 124 and no
diagnostics.
Validation failures are also reported on stderr while the script exits
0, so the until loop never actually retried them. Retry until the
script succeeds with no stderr output, give the loops a generous
timeout, and print the captured stderr when validation does not
succeed in time.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
The rgw endpoint validation issued an HTTP request to the admin ops
api without a timeout. If the RGW is accepting connections but slow to
respond, for example right after the object store was created, the
script hangs indefinitely instead of reporting a validation error that
the caller could retry on.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
Installing the external-snapshotter CRDs and controller runs several
kubectl commands against raw.githubusercontent.com manifest URLs with
no retry, and getManifestFromURL did not retry either, nor did it check
the HTTP status code, so an error page could be fed to kubectl as a
manifest. Transient fetch failures showed up in TestCephSmokeSuite as
NFS test failures during snapshot CRD installation.
Retry the URL-based kubectl invocations and the manifest download, fail
on non-200 responses, and bump the snapshot controller wait in the file
test from 75s to 450s to match the block test since the
snapshot-controller image pull can take longer in CI.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
When the deletion condition never appeared within the retry window,
assertObjectStoreDeletion recorded the failure with assert and then
dereferenced the nil condition, panicking and aborting the whole test
binary without teardown or operator log collection. Observed in a
TestCephSmokeSuite run as a SIGSEGV right after
'delete_object_store ... Should not be: 10'.
Use require for the timeout and a nil check before dereferencing the
condition, and extend the retry window from 30s to the standard retry
budget since the operator can take longer to set the deletion condition
when its reconcile queue is busy.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
retryBlockImageCountCheck aborted the whole wait on the first rbd ls
failure. The listing runs through the toolbox under a timeout and can
fail transiently right after the pool is created (observed as exit
status 124 while the pool PGs were still being created), turning a
recoverable delay into an immediate test failure.
Keep retrying on listing errors and double the retry budget: when the
first provisioning attempt fails, the external-provisioner sidecar
backs off exponentially and its next attempt can be several minutes
out, which the previous 275s budget could not ride out.
This commit also bumps the snapshot controller wait in the block test
from 75s to 450s; the snapshot-controller image pull can take longer
than 75s in CI.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
The csi-operator creates the driver deployments asynchronously, so the
ctrlplugin pods may still be pulling images when the ceph daemons are
already up and the integration tests start. If a PVC is created before
the provisioner is ready, the first provisioning attempt fails (the
mounted csi config may also not be visible to the driver yet) and the
external-provisioner sidecar enters exponential backoff, delaying the
volume creation by several minutes and timing out the test wait.
Observed in TestCephSmokeSuite runs as 'timed out waiting for image
count to reach 1' with the rbd ctrlplugin pod cold-starting mid-test
and the first CreateVolume failing with 'failed to fetch monitor list
using clusterID'. In one run the second provisioning attempt landed two
seconds after the test wait expired.
Wait for the ctrlplugin deployments of the enabled drivers to be ready
before declaring the cluster installed.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
The keystone auth suite installs cert-manager and trust-manager with
helm --wait and then immediately applies Issuer, Certificate, and
Bundle resources. helm --wait returns when the webhook deployments
report ready, but the webhook service endpoints may not be programmed
on the apiserver's node yet, so the apply is rejected with 'failed
calling webhook ... connect: connection refused' and the suite fails
before any test runs. This is currently the dominant failure mode of
the keystone suite, seen on master and on PRs that cannot have caused
it (dependabot github-actions bumps, mergify backports).
Add utils.Eventually, a timeout-based polling primitive intended as
the shared replacement for hand-rolled retry loops in test code:
- cond is func(ctx) error rather than func() bool, so each failed
attempt logs its reason via t.Logf and the final timeout error wraps
the last attempt's error.
- cond runs on the calling goroutine and receives a context carrying
the overall deadline, so cooperative operations stop at the deadline
instead of overrunning it. It is never run on another goroutine: a
hung attempt would keep executing concurrently with its retry, and
require's FailNow is unsafe off the test goroutine.
- utils.AttemptTimeout decorates a cond with a per-attempt deadline
for operations that can hang but would succeed if canceled and
retried. The bound is cooperative; conds that cannot honor a context
must be bounded at the operation level instead.
Expose it as a K8sHelper.ApplyWithRetry method that retries kubectl
apply until the webhooks accept connections, and use it for the
keystone setup applies instead of failing the suite on the first
attempt.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
actionlint only understands workflow files, so the .github/actionlint.yaml
ignore keeps the composite action definitions out of `make lint.workflows`.
Apply the same shellcheck fixes there anyway, found by running shellcheck over
their run: blocks directly:
- quote unquoted variable expansions (SC2086): the $GITHUB_ENV writes in
collect-logs, tmate_debug and upterm_debug, and ${ARCH} in the cri-dockerd
install in integration-test-setup-cluster-resources
- split `export BLOCK=$(...)` into a separate assignment and export so a
failing command substitution is not masked (SC2155) in
encryption-pvc-kms-ibm-kp
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>