find_extra_block_dev piped find_extra_block_devs into head -1. Since
find_extra_block_devs prints every eligible disk and ends with echo
"$devs", head closes the pipe after the first line and the producer's
echo races into SIGPIPE (echo: write error: Broken pipe), failing the
command substitution under pipefail/errexit.
The race only fires reliably now that the runners present two data disks
(sdb and sdc), so find_extra_block_devs emits multiple lines; with a
single disk the one-line output almost never triggered it.
Read the producer to completion with mapfile via process substitution
and return the first element, mirroring find_second_block_dev, so there
is no downstream head to close the pipe early.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit 7079c9bf62)
shouldReconcileChangedNode only diffed the node spec, so changes to node
labels, including rook's own topology.rook.io/* labels, were never detected.
Relabeling a node therefore did not trigger a CephCluster reconcile until some
other condition eventually did.
Trigger a reconcile from the node update predicate when one of the OSD topology
labels rook cares about changes. The whitelist reuses the canonical label set
from topology.GetDefaultTopologyLabels() (the hostname, the Kubernetes
region/zone labels, and the topology.rook.io/* labels), so a reconcile is
scoped to topology changes instead of firing on any label or annotation change.
The UpdateFunc returns true directly for these changes because onK8sNode()
returns false for a node that is already an OSD host, which would otherwise
swallow the relabel.
Resolves#17652
Signed-off-by: Anas Khan <anas@anaskhan.me>
(cherry picked from commit 47b4167b70)
The integration and canary suites ran on a single-node minikube
`driver: none` cluster, where the kubelet ran directly on the GitHub
runner, so the host docker daemon doubled as the cluster runtime and
host block devices and host paths were directly visible to pods. Replace
that with a kind cluster.
Every suite creates its cluster through the shared
integration-test-setup-cluster-resources composite action, so the
conversion is centralized there and converts the smoke, object, helm,
keystone, multi-cluster, upgrade, on-release, nightly, encryption-KMS and
all canary jobs at once.
- Replace the setup-minikube step with helm/kind-action, selecting the
kubernetes version via the kindest/node image tag and creating a
single-node cluster from a new kind config (kind pinned to v0.32.0 for
reproducibility).
- Drop the cri-dockerd install; kind nodes use their built-in containerd.
- Add a kind config that bind-mounts the host /dev, /var/lib/rook and
/run/udev into the node so the existing host-based disk-prep helpers
(use_local_disk*, create_partitions_for_osds, blockDevicePV.sh,
localPathPV.sh, ...) keep working unchanged: devices and partitions
created on the host appear in the node and in the OSD pods that
hostPath-mount the node /dev, and ceph-volume can read the host udev
database it needs to inventory disks.
- Prepare the kind node for the host-level operations rook runs against the
underlying host: remount /sys read-write so CSI's kernel RBD mapping
(`rbd map --device-type krbd`, which writes /sys/bus/rbd) works, and install
lvm2 and cryptsetup, which rook runs in the node's mount namespace to
provision LVM- and encryption-backed OSDs. kindest/node images provide none
of this; the minikube driver:none runner host did.
- Route the Service and pod CIDRs from the runner to the kind node so
host-side tests (the `go test` process runs on the runner) can reach
in-cluster ClusterIPs, e.g. an S3 request to the RGW service. With minikube
driver:none the runner already shared the cluster network.
- Load locally built images into the cluster. Under minikube `driver: none`
the built image was already in the cluster runtime; under kind it must be
imported, so build_rook and create_helm_tag now import their images into
each node's containerd through a new load_image_into_cluster helper (via the
node's ctr, which avoids the kind/kindest-node containerd-config version
skew that breaks `kind load docker-image`).
- Point Vault's kubernetes-auth at the in-cluster API endpoint
(kubernetes.default.svc) instead of the kubeconfig server URL: kind exposes
that as https://127.0.0.1:<port>, unreachable from the in-cluster Vault pod,
so OSD encryption-key retrieval via k8s-auth failed.
- Replace the remaining direct minikube references in the canary workflow: a
`minikube kubectl` call and the external-cluster topology values.
- Adapt host-name assumptions that only held under driver:none: resolve the
disk-cleanup job by the k8s node name rather than the runner hostname, and
let kind-action ignore post-job cluster-teardown failures (the runner is
ephemeral; nvme/multus devices can wedge `docker rm` of the node).
- Update stale comments that described the CI environment as minikube.
- Move the multus integration test's kind config under tests/config too, so
both kind cluster configs live in one place.
create-dev-cluster.sh and other local-dev tooling are intentionally left on
minikube.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit d59abb1956)
Add a step to the release process, when creating a new release branch,
to add a row for the new release series to the support matrix in the
Maintenance and Support page. This keeps the matrix current as new
minor versions are released.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit 9e35153876)
Rename the Release Cycle page to "Maintenance and Support" and add a
table of the Kubernetes and Ceph versions supported by each Rook minor
release series (1.12 through 1.20).
The per-series numbers are sourced from each release branch's
prerequisites/quickstart guide (Kubernetes) and Ceph upgrade guide
(Ceph), giving users a single place to check version compatibility
across releases. A redirect preserves the previous release-cycle URL.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit a346690ae3)
Fix duplicate words, incorrect articles (a/an), it's/its, and other small
grammar mistakes in Go comments and user-facing messages across pkg/, cmd/,
and tests/.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit c0eb360041)
The MGR watch-active sidecar needs Kubernetes API access to
read configmaps and monitor active-standby state. The API
server is host-networked and cannot be targeted by
pod/namespace selectors, making egress restrictions
impractical.
Switch the rook-ceph-mgr NetworkPolicy from egress-only to
ingress-only:
- Allow MON/OSD/MDS/tools pods on ports 6800-7300
- Allow Prometheus scraping on port 9283
- Remove egress rules that blocked API server access and
caused CrashLoopBackOff in the watch-active container
Signed-off-by: Oded Viner <oviner@redhat.com>
(cherry picked from commit 9cec9780fd)
find_extra_block_devs reads the runner's current extra disks with
`lsblk ... | egrep -v "($boot_dev|loop|nbd)"` to decide how many more to
provision. When the runner has no extra disk, that egrep matches nothing
and exits 1; under the script's `set -eo pipefail` the assignment aborts
the function before it ever calls create_extra_disk, so callers get zero
disks and fail with "expected >= N extra disks, found 0" (or a bare /dev/
path) on those runners.
Guard that first read with `|| true` (as the adjacent device-count line
already is) so an empty result means "no extra disks yet" and the
provisioning path runs. The second read, after create_extra_disk, is left
strict on purpose: if provisioning produced no disk, egrep's non-zero exit
should abort there rather than silently return nothing.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit 029b13b000)
the label could be different based on differnt cluster
setup so no need for label vased verification of csi-op.
Signed-off-by: subhamkrai <srai@redhat.com>
(cherry picked from commit afff9dbdf4)
Add a conventions doc and conversion playbook so the remaining old-style
object tests (store lifecycle, notifications, COSI, bucket rw/quota/policy/
lifecycle) can be converted to this pattern mechanically.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit 8ee5d92022)
The object store user (caps/keys/opmask), bucket-owner, and bucket topic
tests already shared a CephObjectStore but polled with count-based
utils.Retry behind a wide entry signature. Move them onto the wait4 toolkit
and a slim (t, k8sh, store) signature: each is an ordered t.Run script using
typed clients and watch-based waits, with per-package check helpers for
repeated verification. Extend the shared store fixture to provision the
realm/zonegroup/zone pools and serve TLS, build its rgw admin and SNS clients
through the consolidated util/client package (replacing the separate
admin/sns/s3 packages), and run every package in both the TLS and non-TLS
passes (now separate parallel jobs).
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit eb404e6677)
Add the shared helpers the object test packages build on:
- wait4: watch-based Assert/Require waiters (Create/Delete/Condition/Absent)
that block on Kubernetes watch events rather than fixed-interval polling,
plus Eventually for non-k8s state (rgw admin/S3/SNS), a pod-log waiter, and
the readiness predicates used with them.
- secrets: verification of the Secret references CRDs publish in status.
- fixture: create-with-t.Cleanup helpers for pure-cleanup resources, and
constructors for ObjectBucketClaim test resources.
- client: rgw admin and SNS client builders, S3 credentials/endpoint, and the
store's TLS cert plus a verification-skipping HTTP client.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit b59101d8e8)
Correct duplicate words, incorrect articles, subject/verb agreement, and
other small grammar mistakes across the user-facing Documentation/ pages
and the design/ records. No content or behavior changes.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit d395d1916b)
Several godoc comments led with a stale or incorrect identifier, left
over from renames, exported/unexported changes, copy-paste between
sibling declarations, or plain typos. As a result the documented name no
longer matched the function, method, type, or var it describes. Correct
each leading word to the name of the declaration it documents.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit 49612461a4)
this commit implement api to configure
ceph health warning mute/unmute.
Signed-off-by: subhamkrai <srai@redhat.com>
(cherry picked from commit 9a6c5842bc)
adding api changes to mute/unmute ceph warnings
based on user configuration. The field is map[string]string
key is ceph warning and value is duration how longs
the warnings will be muted.
Signed-off-by: subhamkrai <srai@redhat.com>
(cherry picked from commit ff5f3baa58)
The validate-rgw-endpoint step gave each validation loop only 15
seconds. A single attempt of the external script can exceed that
budget when the RGW is slow to respond right after the object store
was created, killing the step before any retry could happen. This
showed up as sporadic canary job failures with exit code 124 and no
diagnostics.
Validation failures are also reported on stderr while the script exits
0, so the until loop never actually retried them. Retry until the
script succeeds with no stderr output, give the loops a generous
timeout, and print the captured stderr when validation does not
succeed in time.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit 0c374266e5)
The rgw endpoint validation issued an HTTP request to the admin ops
api without a timeout. If the RGW is accepting connections but slow to
respond, for example right after the object store was created, the
script hangs indefinitely instead of reporting a validation error that
the caller could retry on.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit 459c4bebe9)
build_rook() captures the make output into $o and inspects it for
transient network errors (connection reset, INTERNAL_ERROR, 503, 500)
to decide whether to retry the build. The capture was stdout-only,
but go and make write those errors to stderr, so $o never matched any
retry case. A transient module-proxy failure (e.g. proxy.golang.org
returning INTERNAL_ERROR while downloading modules) therefore fell
through to the "valid failure" branch and failed the job in the build
step, before the integration test even ran.
Redirect stderr into the captured output (2>&1) so the existing retry
logic actually triggers on network failures.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit 04734d7150)
before the csi op, the unblish grpc call was not present,
and it has no op
But with csi op, the unpublish grpc is called before the
volumeattachment get remove, and the unpublish call need a
unpublish secret, currently it is getting the secret from the
client profile, but the client profile secret is hardcoded to default
So to override this secret we are adding the secret name in the
storageclass parameter
Signed-off-by: parth-gr <partharora1010@gmail.com>
(cherry picked from commit 88e31185dc)
The raw-disk-with-object, osd-with-metadata-device, and pvc canary jobs
used loopback devices to supply an extra block device beyond the single
spare disk a GitHub runner provides. Loop devices behave differently
from real block devices -- Rook gates them behind
ROOK_CEPH_ALLOW_LOOP_DEVICES and special-cases them in the OSD daemon --
so exercising core scenarios on them is less realistic than using real
SCSI disks.
Provision the extra device(s) over iSCSI (LIO) instead, reusing the
existing create_extra_disk helper: it now creates N LUNs, the new
find_extra_block_devs and find_second_block_dev helpers enumerate them,
and deploy_cluster gains a two_raw_disks mode. The three jobs now run
entirely on real disks, and the canary CI no longer creates loop devices
or relies on the loop-device test helpers.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit 4b7d769102)
Installing the external-snapshotter CRDs and controller runs several
kubectl commands against raw.githubusercontent.com manifest URLs with
no retry, and getManifestFromURL did not retry either, nor did it check
the HTTP status code, so an error page could be fed to kubectl as a
manifest. Transient fetch failures showed up in TestCephSmokeSuite as
NFS test failures during snapshot CRD installation.
Retry the URL-based kubectl invocations and the manifest download, fail
on non-200 responses, and bump the snapshot controller wait in the file
test from 75s to 450s to match the block test since the
snapshot-controller image pull can take longer in CI.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit c85643692e)
When the deletion condition never appeared within the retry window,
assertObjectStoreDeletion recorded the failure with assert and then
dereferenced the nil condition, panicking and aborting the whole test
binary without teardown or operator log collection. Observed in a
TestCephSmokeSuite run as a SIGSEGV right after
'delete_object_store ... Should not be: 10'.
Use require for the timeout and a nil check before dereferencing the
condition, and extend the retry window from 30s to the standard retry
budget since the operator can take longer to set the deletion condition
when its reconcile queue is busy.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit 76ec455402)
retryBlockImageCountCheck aborted the whole wait on the first rbd ls
failure. The listing runs through the toolbox under a timeout and can
fail transiently right after the pool is created (observed as exit
status 124 while the pool PGs were still being created), turning a
recoverable delay into an immediate test failure.
Keep retrying on listing errors and double the retry budget: when the
first provisioning attempt fails, the external-provisioner sidecar
backs off exponentially and its next attempt can be several minutes
out, which the previous 275s budget could not ride out.
This commit also bumps the snapshot controller wait in the block test
from 75s to 450s; the snapshot-controller image pull can take longer
than 75s in CI.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit 12a2c0f905)
The csi-operator creates the driver deployments asynchronously, so the
ctrlplugin pods may still be pulling images when the ceph daemons are
already up and the integration tests start. If a PVC is created before
the provisioner is ready, the first provisioning attempt fails (the
mounted csi config may also not be visible to the driver yet) and the
external-provisioner sidecar enters exponential backoff, delaying the
volume creation by several minutes and timing out the test wait.
Observed in TestCephSmokeSuite runs as 'timed out waiting for image
count to reach 1' with the rbd ctrlplugin pod cold-starting mid-test
and the first CreateVolume failing with 'failed to fetch monitor list
using clusterID'. In one run the second provisioning attempt landed two
seconds after the test wait expired.
Wait for the ctrlplugin deployments of the enabled drivers to be ready
before declaring the cluster installed.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit 53ab1c5136)
The bucket sync checkpoint confirms the bucket's data sync markers are
caught up at the RADOS level, but the reading RGW can briefly serve 404
for the freshly synced object until it refreshes its view. The post-
checkpoint read was given only a 60s budget, which is occasionally too
short; give it 180s so a transient RGW serving lag does not fail the
test.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit e9b59476a6)
After the sync-readiness gate was added, rgw-multisite-testing still
fails ~25% of master-push runs, now in the new "wait for multisite
sync to be established" step. Diagnosis from the failure diagnostics:
the secondary zone never initializes metadata sync. For the whole
600s its sync status reports "metadata sync failed to read sync
status: (2) No such file or directory".
The master pushes each new RGW configuration period to the secondary
RGW, but until the realm system user is resolvable locally on the
secondary that push is received unauthenticated and rejected with
HTTP 403. The master then re-pushes the same period unchanged every
30s, so a secondary that starts out unable to resolve the user never
recovers. In passing runs the same push later authenticates as
realm-a-system-user and returns 200, at which point the zone converges;
in failing runs it is 403 for the entire window (zero successful
pushes). The generated keys are clean and identical on both clusters,
so this is a fresh-zone bootstrap race in the gateway, not a key
problem.
Rather than wait out a doomed timeout, nudge the secondary to converge
when its metadata sync has not initialized within a couple of minutes:
pull the latest period directly (outbound auth works, so this does not
depend on the rejected inbound push) and restart the secondary RGW so
it re-runs metadata sync init against the now-ready master. This is
what ceph's own multisite QA does (restart zone gateways after the
period is committed). Healthy setups pass the first check in seconds,
so the nudge only runs when sync is genuinely stuck.
Also guard wait_for_sync_status against matching empty output from a
failed exec, and pass the realm/zonegroup/zone to the diagnostics
sync-status dump so the primary cluster no longer reports the unrelated
default zone.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit 325d314dd7)
When the operator starts, there is no need to handle the
node watcher immediately while the controller runtime
cache is being initialized. The cluster will anyway be
fully reconciled and create new OSDs if needed. The node
watcher is only intended for triggering a reconcile when
a node is added or updated at a later time after a sync
would anyway be triggered.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
(cherry picked from commit 7803fac10d)
mergify[bot]
Joshua Hoblitt
Anas Khan
Joshua Hoblitt
Oded Viner
Travis Nielsen
subhamkrai
parth-gr
kavirakesh14