The integration test for Cassandra waited for Pods to become Running
before trying to connect to the Cassandra cluster. However, the check
should be that the cluster is Ready, not that the container state is Running.
If we only check for the Running state, we bump into a scenario where:
- The Cassandra container has started (thus is in Running state)
- The Cassandra server has not stated yet
and then tests will fail.
Signed-off-by: Yannis Zarkadas <yanniszark@arrikto.com>
Rook's testing utilities include a function for loading the default
kubeconfig for a cluster. This function requires maintainance effort and
also doesn't support authentication methods like Basic Authentication.
Instead of adding support for it, drop the config loader and use the one
provided by the controller-runtime library.
Signed-off-by: Yannis Zarkadas <yanniszark@arrikto.com>
Work around issue https://github.com/rook/rook/issues/7573
and make sure integration tests check for regressions.
Eventually we should use the RADOS Gateway admin REST API, but for now
we need to work around an issue where the built version of
'radosgw-admin' has incompatibilities with the RADOS Gateway version
running in the Ceph cluster.
Of note, Rook built on the Ceph Pacific image will not support
some 'radosgw-admin' commands to Ceph Nautilus (v14) or Octopus (v15)
clusters.
Further complicating matters, the flag used for the workaround changes
between Ceph v16.2.0 and v16.2.1 (both Pacific).
This bug needs to be treated a little differently than most of the ways
Rook handles different commands for different Ceph versions because this
is based on the Ceph version that is installed in the container with the
Rook operator primarily and not the version of Ceph running in the
cluster.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
The integration tests pick up the manifests directly from the examples
folder, including with the release tag. In the release branch the Jenkins
build still uses the master build when building locally before tagging
it with the release tag. So the tests need to use the master tag.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
(cherry picked from commit 456e2f3a79)
With the Ceph Pacific release coming this week we add support
in Rook for Pacific with the Rook v1.6 release coming soon.
The integration tests will now run across nautilus, octopus,
and pacific to cover all supported Ceph versions. The default
examples still specify Octopus until there is more bake time
for Pacific.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Sometimes the ci fails to find the object store user, this could be
because the user is created before the object store. So first it fails
since there is no object store and stay in a retry back-off.
Then the object store is created and always need a bit of time, at the
same time the object-user is still waiting and the next retry is longer.
Creating the object store first and then the user hopefully will help
resolve that so we don't have to increase the retry for the object-user
presence.
The error:
```
2021-03-26T08:39:34.7283769Z 2021-03-26 08:39:34.727946 I | testutil: All 2 pod(s) with label rook_object_store=teststore are running
2021-03-26T08:39:34.7284744Z 2021-03-26 08:39:34.727972 I | testutil: Running kubectl [apply -f -]
2021-03-26T08:39:35.1110681Z service/rgw-external-teststore created
2021-03-26T08:39:35.1142392Z 2021-03-26 08:39:35.112479 I | integrationTest: Check that RGW pods are Running
2021-03-26T08:39:35.1181683Z 2021-03-26 08:39:35.117763 I | testutil: 2 of 1 pods with label app=rook-ceph-rgw were found
2021-03-26T08:39:35.1251276Z 2021-03-26 08:39:35.124933 I | testutil: 2 of 1 pods with label app=rook-ceph-rgw were found
2021-03-26T08:39:35.1348603Z 2021-03-26 08:39:35.133276 I | integrationTest: RGW pods are running
2021-03-26T08:39:35.1350090Z 2021-03-26 08:39:35.133290 I | integrationTest: Object store created successfully
2021-03-26T08:39:35.1351373Z 2021-03-26 08:39:35.133295 I | integrationTest: Waiting 5 seconds for the object user to be created
2021-03-26T08:39:40.1340659Z 2021-03-26 08:39:40.133401 I | integrationTest: Checking to see if the user secret has been created
2021-03-26T08:39:40.1342715Z 2021-03-26 08:39:40.133440 D | exec: Running command: kubectl get -n smoke-ns secrets -l rook_object_store=teststore -l user=rook-user
2021-03-26T08:39:40.2618726Z 2021-03-26 08:39:40.261412 I | clients: Unable to find user secret
2021-03-26T08:39:40.2619705Z 2021-03-26 08:39:40.261433 I | integrationTest: (0) secret check sleeping for 5 seconds ...
2021-03-26T08:39:45.2621870Z 2021-03-26 08:39:45.261641 D | exec: Running command: kubectl get -n smoke-ns secrets -l rook_object_store=teststore -l user=rook-user
2021-03-26T08:39:45.3863424Z 2021-03-26 08:39:45.386048 I | clients: Unable to find user secret
2021-03-26T08:39:45.3864586Z 2021-03-26 08:39:45.386075 I | integrationTest: (1) secret check sleeping for 5 seconds ...
2021-03-26T08:39:50.3869496Z 2021-03-26 08:39:50.386292 D | exec: Running command: kubectl get -n smoke-ns secrets -l rook_object_store=teststore -l user=rook-user
2021-03-26T08:39:50.4991166Z 2021-03-26 08:39:50.498862 I | clients: Unable to find user secret
2021-03-26T08:39:50.4991987Z 2021-03-26 08:39:50.498881 I | integrationTest: (2) secret check sleeping for 5 seconds ...
2021-03-26T08:39:55.4998176Z 2021-03-26 08:39:55.499038 D | exec: Running command: kubectl get -n smoke-ns secrets -l rook_object_store=teststore -l user=rook-user
2021-03-26T08:39:55.6075640Z 2021-03-26 08:39:55.607337 I | clients: Unable to find user secret
2021-03-26T08:39:55.6076472Z 2021-03-26 08:39:55.607355 I | integrationTest: (3) secret check sleeping for 5 seconds ...
2021-03-26T08:40:00.6080687Z 2021-03-26 08:40:00.607466 D | exec: Running command: kubectl get -n smoke-ns secrets -l rook_object_store=teststore -l user=rook-user
2021-03-26T08:40:00.8415649Z 2021-03-26 08:40:00.841309 I | clients: Unable to find user secret
2021-03-26T08:40:00.8417179Z 2021-03-26 08:40:00.841331 I | integrationTest: (4) secret check sleeping for 5 seconds ...
2021-03-26T08:40:05.8419407Z 2021-03-26 08:40:05.841385 D | exec: Running command: kubectl get -n smoke-ns secrets -l rook_object_store=teststore -l user=rook-user
2021-03-26T08:40:05.9732221Z 2021-03-26 08:40:05.972910 I | clients: Unable to find user secret
2021-03-26T08:40:05.9733777Z 2021-03-26 08:40:05.972932 I | integrationTest: (5) secret check sleeping for 5 seconds ...
2021-03-26T08:40:10.9734134Z 2021-03-26 08:40:10.973067 D | exec: Running command: kubectl get -n smoke-ns secrets -l rook_object_store=teststore -l user=rook-user
2021-03-26T08:40:11.1686581Z 2021-03-26 08:40:11.168373 I | clients: Unable to find user secret
2021-03-26T08:40:11.1687925Z ceph_base_object_test.go:87:
2021-03-26T08:40:11.1688924Z Error Trace: ceph_base_object_test.go:87
2021-03-26T08:40:11.1689961Z ceph_smoke_test.go:132
2021-03-26T08:40:11.1691113Z Error: Should be true
2021-03-26T08:40:11.1692167Z Test: TestCephSmokeSuite/TestObjectStorage_SmokeTest
2021-03-26T08:40:11.1693751Z 2021-03-26 08:40:11.168502 D | ceph-object-controller: getting s3 user "rook-user"
2021-03-26T08:40:11.1695580Z 2021-03-26 08:40:11.168514 D | exec: Running command: kubectl exec -i rook-ceph-tools-78cdfd976c-2nbp2 -n smoke-ns -- timeout 15 radosgw-admin user info --uid rook-user --rgw-realm= --rgw-zonegroup= --rgw-zone=
2021-03-26T08:40:11.6090406Z ceph_base_object_test.go:89:
2021-03-26T08:40:11.6093015Z Error Trace: ceph_base_object_test.go:89
2021-03-26T08:40:11.6093968Z ceph_smoke_test.go:132
2021-03-26T08:40:11.6095116Z Error: Received unexpected error:
2021-03-26T08:40:11.6095798Z failed to get user info: warn: s3 user not found
2021-03-26T08:40:11.6096724Z github.com/rook/rook/pkg/operator/ceph/object.GetUser
2021-03-26T08:40:11.6097795Z /home/runner/work/rook/rook/pkg/operator/ceph/object/user.go:98
2021-03-26T08:40:11.6098764Z github.com/rook/rook/tests/framework/clients.(*ObjectUserOperation).GetUser
2021-03-26T08:40:11.6099814Z /home/runner/work/rook/rook/tests/framework/clients/object_user.go:45
2021-03-26T08:40:11.6100792Z github.com/rook/rook/tests/integration.runObjectE2ETest
2021-03-26T08:40:11.6101858Z /home/runner/work/rook/rook/tests/integration/ceph_base_object_test.go:88
2021-03-26T08:40:11.6103298Z github.com/rook/rook/tests/integration.(*SmokeSuite).TestObjectStorage_SmokeTest
2021-03-26T08:40:11.6104371Z /home/runner/work/rook/rook/tests/integration/ceph_smoke_test.go:132
2021-03-26T08:40:11.6105104Z reflect.Value.call
2021-03-26T08:40:11.6105856Z /opt/hostedtoolcache/go/1.15.10/x64/src/reflect/value.go:476
2021-03-26T08:40:11.6106662Z reflect.Value.Call
2021-03-26T08:40:11.6107412Z /opt/hostedtoolcache/go/1.15.10/x64/src/reflect/value.go:337
2021-03-26T08:40:11.6108217Z github.com/stretchr/testify/suite.Run.func1
2021-03-26T08:40:11.6109104Z /home/runner/go/pkg/mod/github.com/stretchr/testify@v1.6.1/suite/suite.go:158
2021-03-26T08:40:11.6109844Z testing.tRunner
2021-03-26T08:40:11.6110578Z /opt/hostedtoolcache/go/1.15.10/x64/src/testing/testing.go:1123
2021-03-26T08:40:11.6111271Z runtime.goexit
2021-03-26T08:40:11.6112037Z /opt/hostedtoolcache/go/1.15.10/x64/src/runtime/asm_amd64.s:1374
2021-03-26T08:40:11.6112995Z Test: TestCephSmokeSuite/TestObjectStorage_SmokeTest
```
Signed-off-by: Sébastien Han <seb@redhat.com>
If a CI test fails we don't want to cleanup anything and leave the
cluster in the state it is. This will help debugging the CI.
Signed-off-by: Sébastien Han <seb@redhat.com>
The integration tests run independently in the github actions so there is
no need to cleanup from every test. The cleanup is still needed in the
Jenkins tests where all the suites run serially.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The integration tests have long been painful to maintain with
settings in various places and copied to multiple types,
inconsistent variable names, and otherwise difficult to maintain
code. Now the settings for a test suite are all in one place and
they remain in the same settings type throughout the test.
The multi-cluster suite is also refactored to use the same install
and uninstall helpers as the other suites.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The rbac required for multus was present in role which makes
only the network-attachment-definitions in the rook cluster namespace
accessible to the operator. Moved it to clusterrole for cluster wide
access to NAD.
Signed-off-by: rohan47 <rohgupta@redhat.com>
It's better to validate ownerReferences when setting them. In addition, we should use
controllerrutil.Set{Controller,Owner}Reference, that have such validation, as possible.
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
Finally! We can now stop editing manually our crds definition. Simply
run `make crds-gen`. These two files:
* `cluster/examples/kubernetes/ceph/crds.yaml`
* `cluster/charts/rook-ceph/templates/resources.yaml`
will be autogenerated for us.
We rely on the controller-gen tool, it reads our API definitions from
`pkg/apis/` and produces the CRD files accordingly.
It uses "markers" to add extra yaml fields to the CRD, for instance we
have a lot fields with:
```
nullable: true
x-kubernetes-preserve-unknown-fields: true
```
The corresponding API type markers are:
```
// +kubebuilder:pruning:PreserveUnknownFields
// +nullable
```
For more API convention see
https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#optional-vs-required
and for the markers see: https://book.kubebuilder.io/reference/markers/crd-validation.html
Signed-off-by: Sébastien Han <seb@redhat.com>
This is a follow up to:
https://github.com/rook/rook/pull/7264
which added pool quotas. It was stated that it would be nice to
be able to set pool max_bytes quotas using standard k8s quantity
formats rather than an integer representing the number of bytes.
I was too busy at the time to make the changes. This PR adds the
functionality.
Signed-off-by: Frank Ritchie <12985912+fritchie@users.noreply.github.com>
If there are multiple mgrs running, the operator will automatically
add pod antiaffinity across hosts by default. For stretch clusters,
antiaffinity across the stretch failure domain (e.g. zones)
will be added. Required antiaffinity will be created unless the
test setting allowMultiplePerNode is set to true.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The mgr daemon may be failed over by ceph if the active mgr is not
responding and the standby mgr is available. If the active mgr changes
the services for the dashboard and metrics will be updated with a
label selector for the new active mgr. The services cannot direct
traffic to the standby mgr or else they will be incorrectly redirected
to the active mgr.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The GRPC metrics exposed by both the provisioner pod
and the node plugin pod on some port. Provisioner pod
is running on the pod network and the daemonset pods
run on the host network. sometimes starting the GRPC
metrics by default can lead to node plugin pods
crashloopback state this is due to the port conflict.
Moreover, the GRPC metrics are not for the user it's
for the one which will help to debug the time taken
by cephcsi to serve each GRPC call. Enabling it by
default won't be a good idea. So the plan is to
disable it by default, If someone faces any issue it
can be enabled later at some point in time.
closes#7378
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
We now create 3 disks which will give us 2 OSDs and potentially help us
to catch more errors in our scenarios, especially on iterations.
Signed-off-by: Sébastien Han <seb@redhat.com>
The integration tests will now read the manifests from the same source
that is used in the examples for installation. No longer is there a need
for a copy of the CRDs in the integration test code.
The tests will read the manifests directly from disk under a path
relative to the rook root. The upgrade test will read the manifests from
a github url where the base version has been published.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Since the 1.5 releaase we need the integration test to upgrade from
1.5 to master instead of from 1.4 to master.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
this commit removes file `config-admission-controller.sh`
as we don't need this anymore and update the doc to point
`tests/scripts/deploy_admission_controller.sh` to start
admission controller.
Signed-off-by: subhamkrai <srai@redhat.com>
this commit adds new crd to admission webhooks to check
if one of the port or securePort are set to non-zero
value in objectstore crd.
If both are set to 0 in crd we'll see error like
```
Error from server (invalid create: either of port or securePort
fields should be not be zero): error when creating "object-test.yaml":
admission webhook
"cephobjectstore-wh-rook-ceph-admission-controller-rook-ceph.rook.io"
denied the request: invalid create: either of port or securePort fields
should be not be zero.
```
Signed-off-by: subhamkrai <srai@redhat.com>
we are now using cert-manager for certificates
instead of a self-signed certificate for admission
webhooks.
Signed-off-by: subhamkrai <srai@redhat.com>
This reverts commit 114a949cfb since
14.2.14 is out and has a fix to use raw mode instead of LVM mode on
partitions.
Signed-off-by: Sébastien Han <seb@redhat.com>
Do not use old Ceph v15.2.7 in integration that was working around the
issue of ceph-volume partition support being removed in v15.2.8.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
(cherry picked from commit 99cee7648cfaf72b02d62e281a2ac3a18bce4132)
Even though CRs for bucket health check was added, CRDs were missing from
crds.yaml file. So the health check always enabled for RGW.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
The vault env variable VAULT_BACKEND is read by the secret library so we
must not change it. This is reverting
ee6a03dfb8 with a few extra fixes for the
rgw code.
Signed-off-by: Sébastien Han <seb@redhat.com>
codegen is currently broken in master.
this commit fix the broken codegen by fixing
the correct working directory in gh action.
Signed-off-by: subhamkrai <srai@redhat.com>
bipuladh
Yannis Zarkadas
Blaine Gardner
Travis Nielsen
Sébastien Han
subhamkrai
rohan47
Satoru Takeuchi
Frank Ritchie
Madhu Rajanna
Santosh Pillai