Commit Graph
86 Commits
Author SHA1 Message Date
subhamkrai 61ba7ec45a ci: it test to run on multiple k8s version for master/release
this commit enables IT to run on multiple k8s version
and for master/release, all the 5 k8s version
'v1.15.12','v1.16.15','v1.18.15','v1.19.7','v1.20.0'
similar to jenkins.

Signed-off-by: subhamkrai <srai@redhat.com>
2021-03-22 15:55:01 +05:30
Sébastien Han c43058a98c Merge pull request #7355 from leseb/more-disks-ci
ceph: add more osds to the ci
2021-03-18 10:56:19 +01:00
subhamkrai 07df8220d8 ci: run gh action on multiple k8s version
this commit enables gh action to run

Flex test on only 'v1.15.12',
Smoke and Helm tests on both 'v1.15.12','v1.20.2',
Mgr, Multicluster and upgrade tests on only 'v1.20.2'

Signed-off-by: subhamkrai <srai@redhat.com>
2021-03-18 09:03:56 +05:30
Travis Nielsen bdbf264e68 ceph: integration tests in github actions skip cleanup
The integration tests run independently in the github actions so there is
no need to cleanup from every test. The cleanup is still needed in the
Jenkins tests where all the suites run serially.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-03-17 11:26:11 -06:00
Travis Nielsen 32c628108c Merge pull request #7416 from leseb/debug-ci
ci: debug runners on failures
2021-03-16 08:57:32 -06:00
Sébastien Han ab50e75e08 Merge pull request #7402 from subhamkrai/yaml-ignore
ceph: ignore monitoring folder from yamllint
2021-03-16 10:46:17 +01:00
Sébastien Han 88c0eb57cd ci: debug runners on failures
If one of the suite fails, it is sometimes handy to ssh into the
instance to precisely understand what's going on.
So by applying a debug label so as "debug-smoke-suite" you will be able
to ssh into the runner to debug. By default it looks
like the runner will be cancelled automatically after 6 hours as per the
official doc: https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idtimeout-minutes

I've had a different experience, and somehow 45min seems to be the limit
but I might be wrong.

Once the label is applied, go into the check and the once you are
interested in and search the step "setup tmate session for debugging".
It should output lines with the connection details.

To get the list of all labels see: https://github.com/rook/rook/labels?q=debug-
Example run is available here: https://github.com/rook/rook/pull/7416/checks?check_run_id=2120103989

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-03-16 10:43:38 +01:00
Sébastien Han 31db03fece ceph: auto-gen crds
Finally! We can now stop editing manually our crds definition. Simply
run `make crds-gen`. These two files:

* `cluster/examples/kubernetes/ceph/crds.yaml`
* `cluster/charts/rook-ceph/templates/resources.yaml`

will be autogenerated for us.

We rely on the controller-gen tool, it reads our API definitions from
`pkg/apis/` and produces the CRD files accordingly.

It uses "markers" to add extra yaml fields to the CRD, for instance we
have a lot fields with:

```
nullable: true
x-kubernetes-preserve-unknown-fields: true
```

The corresponding API type markers are:

```
// +kubebuilder:pruning:PreserveUnknownFields
// +nullable
```

For more API convention see
https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#optional-vs-required
and for the markers see: https://book.kubebuilder.io/reference/markers/crd-validation.html

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-03-16 09:16:24 +01:00
subhamkrai acfbc78392 ceph: ignore the monitoring folder from yamllint
this commit adds the monitoring folder to
yamllint ignore list.

Signed-off-by: subhamkrai <srai@redhat.com>
2021-03-16 12:33:37 +05:30
Blaine Gardner 6324746c18 Merge pull request #7409 from BlaineEXE/fix-ci
ci: update minikube github action to fix apt-get
2021-03-15 10:39:26 -06:00
Blaine Gardner b744a08c54 ci: update minikube github action to fix apt-get
Fix an apt-get update issue by bumping to the latest (fixed) version of
manusa/actions-setup-minikube (v2.3.1).

Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
2021-03-15 10:10:34 -06:00
Satoru Takeuchi 4766954dfc ceph: suppress golanglint-ci complaints
Suppress the following complaints.

```
$ golangci-lint run -E gosec
pkg/operator/test/client.go:244:13: G404: Use of weak random number generator (math/rand instead of crypto/rand) (gosec)
        randIdx := rand.Intn(len(nodes.Items))
                   ^
tests/framework/installer/ceph_manifests_v1.5.go:44:19: G107: Potential HTTP request made with variable url (gosec)
        response, err := http.Get(url)
                         ^
pkg/daemon/ceph/client/pool.go:182:6: ineffectual assignment to stats (ineffassign)
        var stats = new(PoolStatistics)
            ^
pkg/operator/k8sutil/pod_test.go:34:2: ineffectual assignment to container (ineffassign)
        container, err := GetMatchingContainer([]v1.Container{}, expectedName)
        ^
```

Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
2021-03-12 15:34:27 +00:00
subhamkrai b46e31dbee ci: add gh action for yaml linter
this commit adds gh action for yaml linter.

Signed-off-by: subhamkrai <srai@redhat.com>
2021-03-11 08:23:18 +05:30
Sébastien Han cc822a531c ceph: add more osds to the ci
We now create 3 disks which will give us 2 OSDs and potentially help us
to catch more errors in our scenarios, especially on iterations.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-03-09 15:24:35 +01:00
Blaine Gardner bc9c5f2057 ceph: start using latest Ceph v15 in integration
Do not use old Ceph v15.2.7 in integration that was working around the
issue of ceph-volume partition support being removed in v15.2.8.

Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
(cherry picked from commit 99cee7648cfaf72b02d62e281a2ac3a18bce4132)
2021-03-03 14:06:19 +01:00
Sébastien Han 5fccaf5e62 Merge pull request #7328 from leseb/ubuntu-18-ci
ceph: stick with ubuntu-18.04 on the CI
2021-03-01 18:44:41 +01:00
Sébastien Han 0adf7624d9 ceph: stick with ubuntu-18.04 on the CI
Github just switched to use Ubuntu 20.04 for the ubuntu-latest and it is
breaking some of our scenarios so far. Especially scenario creating
partitions.

Closes: https://github.com/rook/rook/issues/7327
Signed-off-by: Sébastien Han <seb@redhat.com>
2021-03-01 18:20:42 +01:00
Sébastien Han bd9010e7fc Merge pull request #7296 from subhamkrai/fix-codegen
ci: fix codegen err
2021-03-01 11:35:06 +01:00
subhamkrai 9b6bf82925 ci: update checkout version
this commit update checkout version.

Signed-off-by: subhamkrai <srai@redhat.com>
2021-03-01 10:30:00 +05:30
subhamkrai fd6f6d4953 ci: replace cd with working-directory
this commit replace `cd` with `working-directory`
in gh action.

Signed-off-by: subhamkrai <srai@redhat.com>
2021-03-01 10:30:00 +05:30
subhamkrai afed5bf109 ci: fix codegen err
codegen is currently broken in master.
this commit fix the broken codegen by fixing
the correct working directory in gh action.

Signed-off-by: subhamkrai <srai@redhat.com>
2021-03-01 10:29:26 +05:30
Arun Kumar Mohan bc4dd24306 ceph: making changes to fix CI test failures
Changed the random port creation's max value from 255 to 254, as the IP
with value 255 fails.
Re-raise/propagate the 'ExecutionFailureException' exception so that we
don't end-up in ZERO exit value.

Signed-off-by: Arun Kumar Mohan <amohan@redhat.com>
2021-02-24 23:54:42 +05:30
Arun Kumar Mohan 72b88b4267 ceph: adding the test-data file into the image fs
Signed-off-by: Arun Kumar Mohan <amohan@redhat.com>
2021-02-24 23:54:42 +05:30
Satoru Takeuchi 2b073c3bcd ceph: add the integration test of osd on lv-backed pvc
Add missing OVC on LV-backed PVC test. Although we already have
the pvc test case, the code path of lv-backed pvc is a bit different
from raw device.

Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
2021-02-09 10:29:11 +00:00
subhamkrai 6d66f0a6f2 ci: update canary-integration-test
this commit update canary-integration-test
to remove `object-openshift.yaml` file from
`dry-run`.

Signed-off-by: subhamkrai <srai@redhat.com>
2021-02-01 13:51:16 +05:30
Travis Nielsen 050abd14b6 bot: update stale bot to a github action
The stale bot has stopped working. Now we use a github
action to mark and close the stale issues.
Issues will be marked as stale after 90 days
and closed after one week. PRs will be marked
as stale after 90 days and closed after two weeks.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-01-26 08:53:19 -07:00
Jiffin Tony Thottan 968b002a6f test: validation test for RGW vault authentication
Extending existing deploy-validate-vault.sh to include RGW as well

Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
2021-01-22 10:44:17 +05:30
Sébastien Han 94d9c69361 ceph: add kubectl-check-ownerreferences in ci
After each non-pvc and pvc scenario we run the
kubectl-check-ownerreferences tool to analyze potential incorrect owner
references. We had issues in the past with incorrect owner ref so this
tool might help us ensure we are doing things right.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-01-20 10:51:11 +01:00
Sébastien Han 2c021fb357 ci: bump golang version to 1.15
The rest of the CI already uses 1.15 so let's homogenize that.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-01-18 08:47:31 +01:00
subhamkrai fcd1529b23 ci: add option to download artifacts in canary test
this commit add option to download artifacts in
canary integration test for better debug.

Signed-off-by: subhamkrai <srai@redhat.com>
2021-01-12 21:06:30 +05:30
Sébastien Han adaa90b668 ci: split the test into their own files
If each has its own file we can restart them independently in case of
intermittent failures.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-01-08 16:21:36 +01:00
Marc Nuri 801ee22887 ci: upgrade actions-setup-minikube to v2.2.0
Signed-off-by: Marc Nuri <marc@marcnuri.com>
2020-12-20 07:58:51 +01:00
Travis Nielsen 114a949cfb ceph: test latest ceph on raw devices and previous ceph on partitions
The latest ceph-volume does not support creating OSDs on partitions.
The github actions only have a partition available, so we will run
the github tests on v14.2.12 and v15.2.7 that still support partitions,
while the Jenkins environment has a raw device available where we
can run the latest versions of Ceph in the tests.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-12-17 18:21:16 -07:00
subhamkrai 702569ba21 ci: add yaml validation check
this commit add yaml validation check
by adding `--dry-run` and add path
'.github/workflows/**' to Canary integration
tests.

Signed-off-by: subhamkrai <srai@redhat.com>
2020-12-10 09:49:33 +05:30
Sébastien Han 3da9f2212f bot: fix codespell issue
Make the bot happy.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-11-24 11:41:09 +01:00
Sébastien Han f86f8300f9 Merge pull request #6568 from aruniiird/upgrade-dependencies-for-operator-sdk-1.x
Bump Controller Runtime version to 0.6
2020-11-19 17:45:18 +01:00
Sébastien Han d276705904 Merge pull request #6582 from leseb/mergify-auto-merge
bot: auto-merge pull request under conditions and do not run ci when no code changes
2020-11-19 16:09:50 +01:00
Sébastien Han cd08296f08 ci: skip codespell on go.sum
We don't need to lint go.sum since it's auto generated.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-11-18 18:37:56 +01:00
subhamkrai 49a3511532 ci: manifests changes to run gh action
to run the integration test there needs
to be a few changes in manifests like using
`deviceFilter` and other related changes.

Signed-off-by: subhamkrai <srai@redhat.com>
2020-11-18 09:31:07 +05:30
subhamkrai c240e70f55 ci: run integration test on github actions
as we are planning to move toward Github actions
for integration test. This commit runs ceph
suites using Github action.

Signed-off-by: subhamkrai <srai@redhat.com>
2020-11-18 09:11:48 +05:30
Sébastien Han b011fd579b ci: add build job
Run the build **only** when no go files have changed since the other job
will build Rook anyway.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-11-16 18:05:11 +01:00
Sébastien Han 2d1cf09d8d ci: only run the canary test on code changes
If no Go files changed we do not run the CI.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-11-16 18:05:07 +01:00
Travis Nielsen bf1d3888f4 ceph: factor out CRDs from the common.yaml and helm chart
The CRDs need to be installed and updated separately from the
rest of the manifests. Helm charts also do not have a way to update
CRDs, therefore, any updates to the CRD schema must be done separately
from the helm chart. Now the CRDs are created in a new crd.yaml that must
be created along with common.yaml and before creating the operator.

The helm chart still contains the CRDs for initial creation, though
the helm chart will never update or remove them.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-11-11 12:16:20 -07:00
Mateusz Gozdek 8912e491d8 ci: .github/workflows: add codespell pipeline
As a follow up to #6563 which fixed all the typos found in the
repository, to avoid adding more obvious typos.

Signed-off-by: Mateusz Gozdek <mgozdekof@gmail.com>
2020-11-06 12:43:29 +01:00
Mateusz Gozdek 8ba3762fa4 docs: fix bunch of typos
Found by running the following command:

codespell -S .git,*.png,*.jpg -L \
aks,keyserver,atleast,dne,ser,ist,files\',ba,dum,iam,te -f -H

Signed-off-by: Mateusz Gozdek <mgozdekof@gmail.com>
2020-11-06 10:01:04 +01:00
Arun Kumar Mohan 20e7c4fbb9 ceph: changes to run unittests for external script
Signed-off-by: Arun Kumar Mohan <amohan@redhat.com>
2020-11-03 19:24:10 +05:30
Sébastien Han 3403776c3e ci: add more osd on pvc scenario
Add the following scenario:

* simple osd on pvc
* osd on pvc with db device
* osd on pvc with wal device
* encrypted simple osd on pvc
* encrypted osd on pvc with db device
* encrypted osd on pvc with wal device

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-10-30 16:17:10 +01:00
Sébastien Han ea1d71cbfb ceph: add vault kms support for osd encryption
When the Ceph cluster runs on PVC and the OSDs are encrypted we can
store LUKS's Key Encryption Key inside a Key Management System. Today,
Rook only supports HashiCorp Vault: https://www.vaultproject.io/

The CephCluster has now a new "security" field which will plug onto the
KMS. Here is an example:

security:
  kms:
    tokenSecretName: <name of the secret containing a Vault token, used
    to authenticate>
    connectionDetails: < a map of strings containing connection
    information>

Refer to the ceph-cluster-crd documentation to lear more.

Closes: https://github.com/rook/rook/issues/6105
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-10-30 16:16:33 +01:00
subhamkrai a3cb168020 ci: increase golangci linter timeout
golangci linter sometimes failing due to
the timeout. So, increasing it to 6m from 4m.

Signed-off-by: subhamkrai <srai@redhat.com>
2020-10-21 22:38:03 +05:30
subhamkrai cb0ca66a6b ci: enable gosec linter in golangci-lint
golangci-lint linter gosec showing more errors
than gosec gh. This commit resolve
new errors. And, removing
nosec comments from autogenerated files.

Signed-off-by: subhamkrai <srai@redhat.com>
2020-10-19 11:50:06 +05:30