`CephObjectStore->gateway->type` is not used. Rook has only supported s3-like
interface and hasn't had no code which handles `type` field.
In addition, this field was removed from CRD in the following commit.
ceph: auto-gen crds
31db03fece
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
master build are failing due to bad
permission on build script. this
commit giv +rxw permission to the file.
Signed-off-by: subhamkrai <srai@redhat.com>
The multicluster test was never removing the finalizer of the external
cluster since the core cluster was being removed first. Now we remove
the external cluster first to ensure the finalizer will be removed
properly instead of forcefully by the test.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
We are moving towards github action.
This commit allows pushing build images
to Docker hub using gh action for master
build similar to what Jenkins does.
And, also removing `Publish if Master` step
from the Jenkinsfil file.
Signed-off-by: subhamkrai <srai@redhat.com>
This commit updates external-snapshotter version to
v4.0.0 which supports snapshots v1.
Rook now defaults to enabling RBD and CephFS snapshotter
for K8s >= v1.17 and disabling it for K8s <= v1.16.
Supporting changes in documents and examples yaml files
are made.
Signed-off-by: Rakshith R <rar@redhat.com>
Because of the recent CRD changes made, the mon count had a minimum of
1, making the configuration of the external cluster impossible. The
operator would fail to add the finalizer:
```
2021-04-29 16:57:38.429451 E | ceph-cluster-controller: failed to reconcile. failed to add finalizer: failed to add finalizer "cephcluster.ceph.rook.io" on "test-external": CephCluster.ceph.rook.io "test-external" is invalid: spec.mon.count: Invalid value: 0: spec.mon.count in body should be greater than or equal to 1
```
We now fixed the CRD as well as adding the CR status.
Signed-off-by: Sébastien Han <seb@redhat.com>
Try to mitigate the following error from the CI:
```
+ sudo vgcreate test-rook-vg /dev/sdb
Can't open /dev/sdb exclusively. Mounted filesystem?
Error: Process completed with exit code 5.
```
Now we retry up to 3 times to create the volume group.
Closes: https://github.com/rook/rook/issues/7487
Signed-off-by: Sébastien Han <seb@redhat.com>
We now the generate the CRDs as well as the CSV on a single GOARCH only:
amd64. This helps us avoiding builds colliding with each others.
Closes: https://github.com/rook/rook/issues/7655
Signed-off-by: Sébastien Han <seb@redhat.com>
RGW supports only v2 version of kv engine, current integration test is written for kv v1,
hence seperating existing test into two.
Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
we have lots of duplicate codes for
github action. this commit refactor
them to a bash script which will minimize
duplicate codes.
Signed-off-by: subhamkrai <srai@redhat.com>
Add an integration regression test for Ceph Object Storage Object Bucket
Claim (OBC) to verify that the OBC stays in "Bound" state after being
created. The claim should not fall back into "Pending" (or any other)
state soon after being created.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
We have seen new cases where retrying to lock the device 3 times is not
enough. It's the same race we had experienced where Ceph tries to
acquire a lock on the device but systemd-udevd does the same too.
Retrying 20 times every 0.1sec seems to mitigate that issue in the CI.
Signed-off-by: Sébastien Han <seb@redhat.com>
We generate vendor directory in building process. It doesn't reduce
build time. In addition, we have encountered some vendoring-related
problems. So let's stop vendoring.
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
Since Rook 1.6 is using raw mode for simple OSD scenarios we can use v14
again without having issue with ceph-volume.
We keep the upgraade test with v14.2.12 since Rook 1.5 does not have the
raw code to deploy OSDs.
Closes: https://github.com/rook/rook/issues/7669
Signed-off-by: Sébastien Han <seb@redhat.com>
In case ssl enabled for RGW, bucket healthcheck won't work since access is denied from the server.
In that case configure S3 agent using "InsecureSkipVerify: true" option.
Fixes: 7288
Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
currently, the build action is not failing if there
is/are any modified files after the build. this commit
will validate build with `git status` after `make build`
command will fail if any changed file found.
Signed-off-by: subhamkrai <srai@redhat.com>
The integration test for Cassandra waited for Pods to become Running
before trying to connect to the Cassandra cluster. However, the check
should be that the cluster is Ready, not that the container state is Running.
If we only check for the Running state, we bump into a scenario where:
- The Cassandra container has started (thus is in Running state)
- The Cassandra server has not stated yet
and then tests will fail.
Signed-off-by: Yannis Zarkadas <yanniszark@arrikto.com>
Rook's testing utilities include a function for loading the default
kubeconfig for a cluster. This function requires maintainance effort and
also doesn't support authentication methods like Basic Authentication.
Instead of adding support for it, drop the config loader and use the one
provided by the controller-runtime library.
Signed-off-by: Yannis Zarkadas <yanniszark@arrikto.com>
Work around issue https://github.com/rook/rook/issues/7573
and make sure integration tests check for regressions.
Eventually we should use the RADOS Gateway admin REST API, but for now
we need to work around an issue where the built version of
'radosgw-admin' has incompatibilities with the RADOS Gateway version
running in the Ceph cluster.
Of note, Rook built on the Ceph Pacific image will not support
some 'radosgw-admin' commands to Ceph Nautilus (v14) or Octopus (v15)
clusters.
Further complicating matters, the flag used for the workaround changes
between Ceph v16.2.0 and v16.2.1 (both Pacific).
This bug needs to be treated a little differently than most of the ways
Rook handles different commands for different Ceph versions because this
is based on the Ceph version that is installed in the container with the
Rook operator primarily and not the version of Ceph running in the
cluster.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
The integration tests pick up the manifests directly from the examples
folder, including with the release tag. In the release branch the Jenkins
build still uses the master build when building locally before tagging
it with the release tag. So the tests need to use the master tag.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
(cherry picked from commit 456e2f3a79)
With the Ceph Pacific release coming this week we add support
in Rook for Pacific with the Rook v1.6 release coming soon.
The integration tests will now run across nautilus, octopus,
and pacific to cover all supported Ceph versions. The default
examples still specify Octopus until there is more bake time
for Pacific.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Sometimes the ci fails to find the object store user, this could be
because the user is created before the object store. So first it fails
since there is no object store and stay in a retry back-off.
Then the object store is created and always need a bit of time, at the
same time the object-user is still waiting and the next retry is longer.
Creating the object store first and then the user hopefully will help
resolve that so we don't have to increase the retry for the object-user
presence.
The error:
```
2021-03-26T08:39:34.7283769Z 2021-03-26 08:39:34.727946 I | testutil: All 2 pod(s) with label rook_object_store=teststore are running
2021-03-26T08:39:34.7284744Z 2021-03-26 08:39:34.727972 I | testutil: Running kubectl [apply -f -]
2021-03-26T08:39:35.1110681Z service/rgw-external-teststore created
2021-03-26T08:39:35.1142392Z 2021-03-26 08:39:35.112479 I | integrationTest: Check that RGW pods are Running
2021-03-26T08:39:35.1181683Z 2021-03-26 08:39:35.117763 I | testutil: 2 of 1 pods with label app=rook-ceph-rgw were found
2021-03-26T08:39:35.1251276Z 2021-03-26 08:39:35.124933 I | testutil: 2 of 1 pods with label app=rook-ceph-rgw were found
2021-03-26T08:39:35.1348603Z 2021-03-26 08:39:35.133276 I | integrationTest: RGW pods are running
2021-03-26T08:39:35.1350090Z 2021-03-26 08:39:35.133290 I | integrationTest: Object store created successfully
2021-03-26T08:39:35.1351373Z 2021-03-26 08:39:35.133295 I | integrationTest: Waiting 5 seconds for the object user to be created
2021-03-26T08:39:40.1340659Z 2021-03-26 08:39:40.133401 I | integrationTest: Checking to see if the user secret has been created
2021-03-26T08:39:40.1342715Z 2021-03-26 08:39:40.133440 D | exec: Running command: kubectl get -n smoke-ns secrets -l rook_object_store=teststore -l user=rook-user
2021-03-26T08:39:40.2618726Z 2021-03-26 08:39:40.261412 I | clients: Unable to find user secret
2021-03-26T08:39:40.2619705Z 2021-03-26 08:39:40.261433 I | integrationTest: (0) secret check sleeping for 5 seconds ...
2021-03-26T08:39:45.2621870Z 2021-03-26 08:39:45.261641 D | exec: Running command: kubectl get -n smoke-ns secrets -l rook_object_store=teststore -l user=rook-user
2021-03-26T08:39:45.3863424Z 2021-03-26 08:39:45.386048 I | clients: Unable to find user secret
2021-03-26T08:39:45.3864586Z 2021-03-26 08:39:45.386075 I | integrationTest: (1) secret check sleeping for 5 seconds ...
2021-03-26T08:39:50.3869496Z 2021-03-26 08:39:50.386292 D | exec: Running command: kubectl get -n smoke-ns secrets -l rook_object_store=teststore -l user=rook-user
2021-03-26T08:39:50.4991166Z 2021-03-26 08:39:50.498862 I | clients: Unable to find user secret
2021-03-26T08:39:50.4991987Z 2021-03-26 08:39:50.498881 I | integrationTest: (2) secret check sleeping for 5 seconds ...
2021-03-26T08:39:55.4998176Z 2021-03-26 08:39:55.499038 D | exec: Running command: kubectl get -n smoke-ns secrets -l rook_object_store=teststore -l user=rook-user
2021-03-26T08:39:55.6075640Z 2021-03-26 08:39:55.607337 I | clients: Unable to find user secret
2021-03-26T08:39:55.6076472Z 2021-03-26 08:39:55.607355 I | integrationTest: (3) secret check sleeping for 5 seconds ...
2021-03-26T08:40:00.6080687Z 2021-03-26 08:40:00.607466 D | exec: Running command: kubectl get -n smoke-ns secrets -l rook_object_store=teststore -l user=rook-user
2021-03-26T08:40:00.8415649Z 2021-03-26 08:40:00.841309 I | clients: Unable to find user secret
2021-03-26T08:40:00.8417179Z 2021-03-26 08:40:00.841331 I | integrationTest: (4) secret check sleeping for 5 seconds ...
2021-03-26T08:40:05.8419407Z 2021-03-26 08:40:05.841385 D | exec: Running command: kubectl get -n smoke-ns secrets -l rook_object_store=teststore -l user=rook-user
2021-03-26T08:40:05.9732221Z 2021-03-26 08:40:05.972910 I | clients: Unable to find user secret
2021-03-26T08:40:05.9733777Z 2021-03-26 08:40:05.972932 I | integrationTest: (5) secret check sleeping for 5 seconds ...
2021-03-26T08:40:10.9734134Z 2021-03-26 08:40:10.973067 D | exec: Running command: kubectl get -n smoke-ns secrets -l rook_object_store=teststore -l user=rook-user
2021-03-26T08:40:11.1686581Z 2021-03-26 08:40:11.168373 I | clients: Unable to find user secret
2021-03-26T08:40:11.1687925Z ceph_base_object_test.go:87:
2021-03-26T08:40:11.1688924Z Error Trace: ceph_base_object_test.go:87
2021-03-26T08:40:11.1689961Z ceph_smoke_test.go:132
2021-03-26T08:40:11.1691113Z Error: Should be true
2021-03-26T08:40:11.1692167Z Test: TestCephSmokeSuite/TestObjectStorage_SmokeTest
2021-03-26T08:40:11.1693751Z 2021-03-26 08:40:11.168502 D | ceph-object-controller: getting s3 user "rook-user"
2021-03-26T08:40:11.1695580Z 2021-03-26 08:40:11.168514 D | exec: Running command: kubectl exec -i rook-ceph-tools-78cdfd976c-2nbp2 -n smoke-ns -- timeout 15 radosgw-admin user info --uid rook-user --rgw-realm= --rgw-zonegroup= --rgw-zone=
2021-03-26T08:40:11.6090406Z ceph_base_object_test.go:89:
2021-03-26T08:40:11.6093015Z Error Trace: ceph_base_object_test.go:89
2021-03-26T08:40:11.6093968Z ceph_smoke_test.go:132
2021-03-26T08:40:11.6095116Z Error: Received unexpected error:
2021-03-26T08:40:11.6095798Z failed to get user info: warn: s3 user not found
2021-03-26T08:40:11.6096724Z github.com/rook/rook/pkg/operator/ceph/object.GetUser
2021-03-26T08:40:11.6097795Z /home/runner/work/rook/rook/pkg/operator/ceph/object/user.go:98
2021-03-26T08:40:11.6098764Z github.com/rook/rook/tests/framework/clients.(*ObjectUserOperation).GetUser
2021-03-26T08:40:11.6099814Z /home/runner/work/rook/rook/tests/framework/clients/object_user.go:45
2021-03-26T08:40:11.6100792Z github.com/rook/rook/tests/integration.runObjectE2ETest
2021-03-26T08:40:11.6101858Z /home/runner/work/rook/rook/tests/integration/ceph_base_object_test.go:88
2021-03-26T08:40:11.6103298Z github.com/rook/rook/tests/integration.(*SmokeSuite).TestObjectStorage_SmokeTest
2021-03-26T08:40:11.6104371Z /home/runner/work/rook/rook/tests/integration/ceph_smoke_test.go:132
2021-03-26T08:40:11.6105104Z reflect.Value.call
2021-03-26T08:40:11.6105856Z /opt/hostedtoolcache/go/1.15.10/x64/src/reflect/value.go:476
2021-03-26T08:40:11.6106662Z reflect.Value.Call
2021-03-26T08:40:11.6107412Z /opt/hostedtoolcache/go/1.15.10/x64/src/reflect/value.go:337
2021-03-26T08:40:11.6108217Z github.com/stretchr/testify/suite.Run.func1
2021-03-26T08:40:11.6109104Z /home/runner/go/pkg/mod/github.com/stretchr/testify@v1.6.1/suite/suite.go:158
2021-03-26T08:40:11.6109844Z testing.tRunner
2021-03-26T08:40:11.6110578Z /opt/hostedtoolcache/go/1.15.10/x64/src/testing/testing.go:1123
2021-03-26T08:40:11.6111271Z runtime.goexit
2021-03-26T08:40:11.6112037Z /opt/hostedtoolcache/go/1.15.10/x64/src/runtime/asm_amd64.s:1374
2021-03-26T08:40:11.6112995Z Test: TestCephSmokeSuite/TestObjectStorage_SmokeTest
```
Signed-off-by: Sébastien Han <seb@redhat.com>
If a CI test fails we don't want to cleanup anything and leave the
cluster in the state it is. This will help debugging the CI.
Signed-off-by: Sébastien Han <seb@redhat.com>
The integration tests run independently in the github actions so there is
no need to cleanup from every test. The cleanup is still needed in the
Jenkins tests where all the suites run serially.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The integration tests have long been painful to maintain with
settings in various places and copied to multiple types,
inconsistent variable names, and otherwise difficult to maintain
code. Now the settings for a test suite are all in one place and
they remain in the same settings type throughout the test.
The multi-cluster suite is also refactored to use the same install
and uninstall helpers as the other suites.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The rbac required for multus was present in role which makes
only the network-attachment-definitions in the rook cluster namespace
accessible to the operator. Moved it to clusterrole for cluster wide
access to NAD.
Signed-off-by: rohan47 <rohgupta@redhat.com>
It's better to validate ownerReferences when setting them. In addition, we should use
controllerrutil.Set{Controller,Owner}Reference, that have such validation, as possible.
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
Finally! We can now stop editing manually our crds definition. Simply
run `make crds-gen`. These two files:
* `cluster/examples/kubernetes/ceph/crds.yaml`
* `cluster/charts/rook-ceph/templates/resources.yaml`
will be autogenerated for us.
We rely on the controller-gen tool, it reads our API definitions from
`pkg/apis/` and produces the CRD files accordingly.
It uses "markers" to add extra yaml fields to the CRD, for instance we
have a lot fields with:
```
nullable: true
x-kubernetes-preserve-unknown-fields: true
```
The corresponding API type markers are:
```
// +kubebuilder:pruning:PreserveUnknownFields
// +nullable
```
For more API convention see
https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#optional-vs-required
and for the markers see: https://book.kubebuilder.io/reference/markers/crd-validation.html
Signed-off-by: Sébastien Han <seb@redhat.com>
This is a follow up to:
https://github.com/rook/rook/pull/7264
which added pool quotas. It was stated that it would be nice to
be able to set pool max_bytes quotas using standard k8s quantity
formats rather than an integer representing the number of bytes.
I was too busy at the time to make the changes. This PR adds the
functionality.
Signed-off-by: Frank Ritchie <12985912+fritchie@users.noreply.github.com>
If there are multiple mgrs running, the operator will automatically
add pod antiaffinity across hosts by default. For stretch clusters,
antiaffinity across the stretch failure domain (e.g. zones)
will be added. Required antiaffinity will be created unless the
test setting allowMultiplePerNode is set to true.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Travis Nielsen
Sébastien Han
Satoru Takeuchi
subhamkrai
Rakshith R
Jiffin Tony Thottan
Blaine Gardner
bipuladh
Yannis Zarkadas
rohan47
Frank Ritchie