The 'extra' block device attached to GH actions runners has changed size twice in 3 months. The previous strategy of detecting the disk by size is becoming harder to maintain. Additionally, the block size with recent changes (75G) is now the same as the boot device (also 75G), making the method inexact. The method can now be summarized as, "find the boot disk and choose the disk that isn't the boot disk to be the 'extra' one used." Prior to this, we used a one-liner based on `lsblk`. While we could still make this a one-liner, the method is now updated to 2 effective lines, plus debug text output to stderr to help if we need to debug further in the future. Of note, the 'extra' disk has a mount point of "/mnt", but it is unclear whether this is a reliable heuristic for detecting the extra disk. For years now, GH action runners have had only 2 disks. Therefore, it seems slightly more likely that a heuristic to "choose the non-boot disk" will be a more robust long-term solution. If this strategy proves to be unreliable in the future, it may be wise to consider whether "the device with a partition mounted to '/mnt'" would be a good alternative. Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
82 lines
4.3 KiB
YAML
82 lines
4.3 KiB
YAML
name: RGW Multisite Test
|
|
description: Reusable workflow to test RGW multisite integration
|
|
|
|
runs:
|
|
using: "composite"
|
|
steps:
|
|
- name: setup cluster resources
|
|
uses: ./.github/workflows/canary-test-config
|
|
|
|
- name: install additional deps for object testing
|
|
shell: bash --noprofile --norc -eo pipefail -x {0}
|
|
run: |
|
|
sudo apt-get install -y s3cmd
|
|
|
|
- name: use local disk into two partitions
|
|
shell: bash --noprofile --norc -eo pipefail -x {0}
|
|
run: |
|
|
export BLOCK="/dev/$(tests/scripts/github-action-helper.sh find_extra_block_dev)"
|
|
tests/scripts/github-action-helper.sh use_local_disk
|
|
tests/scripts/create-bluestore-partitions.sh --disk "$BLOCK" --osd-count 2
|
|
sudo lsblk
|
|
|
|
- name: deploy first cluster rook
|
|
shell: bash --noprofile --norc -eo pipefail -x {0}
|
|
run: |
|
|
tests/scripts/github-action-helper.sh deploy_first_rook_cluster
|
|
kubectl create -f deploy/examples/object-multisite-test.yaml
|
|
# wait for multisite store to be created
|
|
tests/scripts/github-action-helper.sh wait_for_rgw rook-ceph
|
|
|
|
- name: prep second cluster pull realm config
|
|
shell: bash --noprofile --norc -eo pipefail -x {0}
|
|
run: |
|
|
cd deploy/examples/
|
|
IP_ADDR=$(kubectl -n rook-ceph get svc rook-ceph-rgw-multisite-store -o jsonpath="{.spec.clusterIP}")
|
|
yq w -i -d1 object-multisite-pull-realm-test.yaml spec.pull.endpoint http://${IP_ADDR}:80
|
|
BASE64_ACCESS_KEY=$(kubectl -n rook-ceph get secrets realm-a-keys -o jsonpath="{.data.access-key}")
|
|
BASE64_SECRET_KEY=$(kubectl -n rook-ceph get secrets realm-a-keys -o jsonpath="{.data.secret-key}")
|
|
sed -i 's/VzFjNFltMVdWRTFJWWxZelZWQT0=/'"$BASE64_ACCESS_KEY"'/g' object-multisite-pull-realm-test.yaml
|
|
sed -i 's/WVY1MFIxeExkbG84U3pKdlRseEZXVGR3T3k1U1dUSS9KaTFoUVE9PQ==/'"$BASE64_SECRET_KEY"'/g' object-multisite-pull-realm-test.yaml
|
|
|
|
- name: deploy second cluster rook
|
|
shell: bash --noprofile --norc -eo pipefail -x {0}
|
|
run: |
|
|
tests/scripts/github-action-helper.sh deploy_second_rook_cluster
|
|
kubectl create -f deploy/examples/object-multisite-pull-realm-test.yaml
|
|
# wait for realms to be pulled and zone-b-multisite-store to be created
|
|
tests/scripts/github-action-helper.sh wait_for_rgw rook-ceph-secondary
|
|
|
|
- name: wait for ceph cluster 1 to be ready
|
|
shell: bash --noprofile --norc -eo pipefail -x {0}
|
|
run: |
|
|
tests/scripts/validate_cluster.sh osd 1
|
|
kubectl -n rook-ceph get pods
|
|
|
|
- name: write an object to one cluster, read from the other
|
|
shell: bash --noprofile --norc -eo pipefail -x {0}
|
|
run: |
|
|
tests/scripts/github-action-helper.sh test_multisite_object_replication
|
|
|
|
# if this test fails, it could mean the RGW `period get` or `period update` output has changed
|
|
- name: RGW configuration period should be committed on first reconcile and not be committed on second reconcile
|
|
shell: bash --noprofile --norc -eo pipefail -x {0}
|
|
run: |
|
|
ns_name_primary_object_store='"rook-ceph/multisite-store"' # double quotes intended
|
|
ns_name_secondary_object_store='"rook-ceph-secondary/zone-b-multisite-store"' # double quotes intended
|
|
committed_msg="committing changes to RGW configuration period for CephObjectStore"
|
|
tests/scripts/github-action-helper.sh verify_operator_log_message "${committed_msg} ${ns_name_primary_object_store}"
|
|
tests/scripts/github-action-helper.sh verify_operator_log_message "${committed_msg} ${ns_name_secondary_object_store}"
|
|
ns_name_primary_object_zone='"rook-ceph/zone-a"' # double quotes intended
|
|
ns_name_secondary_object_zone='"rook-ceph-secondary/zone-b"' # double quotes intended
|
|
tests/scripts/github-action-helper.sh restart_operator
|
|
not_committed_msg="there are no changes to commit for RGW configuration period for CephObjectStore"
|
|
tests/scripts/github-action-helper.sh wait_for_operator_log_message "${not_committed_msg} ${ns_name_primary_object_zone}" 600
|
|
tests/scripts/github-action-helper.sh wait_for_operator_log_message "${not_committed_msg} ${ns_name_secondary_object_zone}" 600
|
|
|
|
- name: collect common logs
|
|
if: always()
|
|
shell: bash --noprofile --norc -eo pipefail -x {0}
|
|
run: |
|
|
tests/scripts/collect-logs.sh
|