Commit Graph
204 Commits
Author SHA1 Message Date
Sébastien Han 7c8dc4bc1a ci: fix multisite test
We just need to wait a little for the object to be replicated to the
other gateway. A simple retry solves this.

Closes: https://github.com/rook/rook/issues/8671
Signed-off-by: Sébastien Han <seb@redhat.com>
2021-09-22 10:23:02 +02:00
Sébastien Han 8786b40d64 rgw: do not create the rgw ops user on the secondary cluster
If the cluster where the rgw is started is secondary and not primary,
trying to create the admin ops user will fail with:

```
Please run the command on master zone.
Performing this operation on non-master zone
leads to inconsistent metadata between zones
```

So we need to force the creation regardless, it is fine the creation will
return UserAlreadyExist and then we just read the current user.

Closes: https://github.com/rook/rook/issues/8671
Signed-off-by: Sébastien Han <seb@redhat.com>
2021-09-21 16:58:03 +02:00
subhamkrai 038031ddbd ci: dry-run is deprecated and replaced with --dry-run=client
--dry-run is deprecated and replaced with --dry-run=client

Signed-off-by: subhamkrai <srai@redhat.com>
2021-09-14 16:13:27 +05:30
Sébastien Han 0e961c4bdc ci: stop build 3 times
This time for real.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-09-09 17:51:03 +02:00
Sébastien Han 73c340ded7 ci: fix pod list
We should not use .items[0].metadata.name if the array length is 0. This
is the case when nothing has been initialized yet. Instead, we should
use .items[*].metadata.name, the wildcard ensures to always return 0
even if nothing is present yet.

Fixes: https://github.com/rook/rook/issues/8676
Signed-off-by: Sébastien Han <seb@redhat.com>
2021-09-09 17:24:56 +02:00
Blaine Gardner 7febd4e303 ci: add debugging to release build script
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
2021-09-01 13:58:21 -06:00
parth-gr 827ee1d879 ceph: auto grow OSDs size on PVCs
When an OSD reaches OSD_NEARFULL state,
we have to manually increase the PVC volume claim
or manually increase the count of OSDs in the device set

Added a script auto-grow-storage.sh which will
i)automatically increase claim volume
ii)automatically add number of OSDs

Closes: https://github.com/rook/rook/issues/6101
Signed-off-by: parth-gr <paarora@redhat.com>
2021-09-01 12:57:42 +05:30
subhamkrai ebe21c8f68 ci: add action for shellcheck linter
we are adding new linter for shellcheck.
As we are writing more shell scripts this
will help maintain quality.

Also, doing all the changes required in
bash files to pass this shellcheck.

Closes: https://github.com/rook/rook/issues/8431
Signed-off-by: subhamkrai <srai@redhat.com>
2021-08-26 10:03:32 +05:30
parth-gr 33bf21ba69 ceph: update CSR from v1beta1 to v1
This commit update the certificates.k8s.io to use version v1
Updated to v1 as v1beta1 certificates.k8s.io is deprecated in v1.19+

Closes: https://github.com/rook/rook/issues/8308
Signed-off-by: parth-gr <paarora@redhat.com>
2021-08-12 20:27:05 +05:30
Ali Maredia b6404ae2ab ceph: rgw multisite intgration testing
This test:
- starts up 2 minikube clusters
- create 2 ceph clusters
- creates object multisite CRDs on each cluster and
syncs the clusters
- writes an object to cluster 1 and reads it on
cluster 2

This commit also adds new functions in
github-action-helper.sh that aid in the multisite
test and the multi cluster mirroring test.

Signed-off-by: Ali Maredia <amaredia@redhat.com>
2021-08-11 13:25:43 -04:00
Travis Nielsen 529ec0ff7e Merge pull request #8461 from subhamkrai/automate-release-action
ci: create tag and print release notes
2021-08-06 09:41:11 -06:00
subhamkrai 8572fdb58b ci: create tag and print release notes
since we moved to github action for the release,
we maunally have to push the tag to build image.

with this change,
first, this action will set some
environment variable which will be consume by
the script to print release notes.
then,
it will ask for 1 input from
github UI(in Action tab)
tag: where we'll give the tag for the release

after creating tag it will run the script and
print release notes.

**NOTE** please verify the environment var
and release notes after the successful action run.

.

Signed-off-by: subhamkrai <srai@redhat.com>
2021-08-06 16:00:25 +05:30
Sébastien Han d7ce570644 Merge pull request #8495 from leseb/fix-retry
ci: break if no errors on build
2021-08-06 08:41:16 +02:00
Sébastien Han 6582fdce97 ci: break if no errors on build
No wonder why the build has been taking longer... We were building 3
times.
We just got at least 2 minutes back of build time thanks to this fix.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-08-05 15:40:48 +02:00
subhamkraiandTravis Nielsen 12de60e4a0 ceph: get pr tile and user for release note
this commit include script which will help
the person who is creating release note
to get all the pr number with title and owner
in required order for the release notes. In the
past, it was manual step.

Co-authored-by: Travis Nielsen <tnielsen@redhat.com>
Signed-off-by: subhamkrai <srai@redhat.com>
2021-08-04 20:19:43 +05:30
Sébastien Han 99e00dea1e ceph: auto detect vault k/v version
Rook will now auto detect the kv version of the vault server. This
allows users not having to pass the VAULT_BACKEND configuration in the
CephCluster CR.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-07-29 10:16:31 +02:00
Travis Nielsen 87596c6f8e build: detect git version instead of setting to 0
The build version should be detected with the git describe
command, but if the VERSION var is already set, it will be
used instead of being detected from git. During the official
builds or even local builds, we just want to let the makefile
detect the git version.

The full git history is added to enable proper detection of the
version. Without the full history only a hash is returned for
the version, which then results in an invalid semantic version
error for the helm chart.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-07-28 07:25:10 -06:00
Travis Nielsen 7830095809 build: promote the build with tag suffix if needed
The beta releases should have the tag suffx included in
the helm chart and release version tags.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
(cherry picked from commit a1d37b4efc)
2021-07-26 17:34:25 -06:00
Travis Nielsen 67a3822e95 build: publish and promote with correct branch name
The tagged builds will run git checkout on the correct tag,
though the publish and promote actions expect a branch name
to be available due to internal build assumptions. Now the
publish looks up the branch name from the tag and passes
it to the publish command as expected.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
(cherry picked from commit d8c5f459de)
2021-07-26 17:34:10 -06:00
subhamkrai e291afad07 ci: missing function call in script
in pr 8328, function call was missing
so action for pushing the image is not
starting.

Signed-off-by: subhamkrai <srai@redhat.com>
2021-07-23 22:05:15 +05:30
subhamkrai 492a8176e0 ci: action for release build
this commit adds github action for
release builds and also modify the
helper script `build-release.sh`.

Closes: https://github.com/rook/rook/issues/8041
Signed-off-by: subhamkrai <srai@redhat.com>
2021-07-23 13:09:30 +05:30
Sébastien Han db40a31bb2 ci: change yq path to /usr/local/bin/yq
So it will override any existing installation.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-07-21 14:15:07 +02:00
Sébastien Han 741dd25efb Merge pull request #8278 from leseb/retry-ci
ci: retry on network errors
2021-07-08 08:52:28 +02:00
Sébastien Han ff8d24a296 ci: retry on network errors
Seen today in https://github.com/rook/rook/pull/8272/checks?check_run_id=3010477951

```
github.com/rook/rook/pkg/apis/ceph.rook.io/v1 imports
	github.com/hashicorp/vault/api: github.com/hashicorp/vault@v1.4.2: reading https://proxy.golang.org/github.com/hashicorp/vault/@v/v1.4.2.zip: 503 Service Unavailable
	server response: Service Unavailable
github.com/rook/rook/pkg/daemon/ceph/osd/kms imports
	github.com/libopenstorage/secrets/vault imports
	github.com/hashicorp/vault/command/agent/auth: github.com/hashicorp/vault@v1.4.2: reading https://proxy.golang.org/github.com/hashicorp/vault/@v/v1.4.2.zip: 503 Service Unavailable
	server response: Service Unavailable
github.com/rook/rook/pkg/daemon/ceph/osd/kms imports
	github.com/libopenstorage/secrets/vault imports
	github.com/hashicorp/vault/command/agent/auth/kubernetes: github.com/hashicorp/vault@v1.4.2: reading https://proxy.golang.org/github.com/hashicorp/vault/@v/v1.4.2.zip: 503 Service Unavailable
	server response: Service Unavailable
make: *** [go.mod.check] Error 1
```

Also https://github.com/rook/rook/pull/8272/checks?check_run_id=3010482113

```
go get: sigs.k8s.io/controller-tools@v0.4.1 updating to
	sigs.k8s.io/controller-tools@v0.5.1-0.20210420220833-f284e2e8098c requires
	k8s.io/apimachinery@v0.20.2 requires
	github.com/json-iterator/go@v1.1.10: verifying go.mod: github.com/json-iterator/go@v1.1.10/go.mod: reading https://sum.golang.org/tile/8/0/x005/043: stream error: stream ID 1099; INTERNAL_ERROR
make[3]: *** [/home/runner/work/rook/rook/.cache/tools/linux_amd64/controller-gen-f284e2e8098cb0193c2b0c7c1c651ae75496539b] Error 1
make[2]: *** [generate-csv-ceph-templates] Error 1
```

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-07-07 18:00:01 +02:00
subhamkrai bc1fabb6e4 ci: update helm version to 3.6.2
this commit update helm version to `3.6.2`

Signed-off-by: subhamkrai <srai@redhat.com>
2021-07-07 20:09:26 +05:30
Sébastien Han 830b36c039 Merge pull request #7920 from thotz/tlscitest
test: ci test for TLS objectstore
2021-07-05 11:23:40 +02:00
Sébastien Han b578f916e7 ceph: add fs mirror config
Similarly to block volume replication, Ceph is capable of replicating the
content of a Ceph Filesystem from one cluster to another.
For this, during the 1.6 cycle, we introduced a new CRD called
CephFilesystemMirror which effectively deploys a cephfs-mirror daemon.
However, configuring peers to enable replication between two clusters
had to be done manually.
Also various bug fix made it in Ceph eventually and the minimum required
version for this to work is to run on Ceph Pacific 16.2.5 at least.

So the automatic configuration of Ceph Filesystem peers is now possible.

By editing the CephFilesystem CRD, you can now turn on mirroring:

```yaml
  mirroring:
    enabled: false
    # list of Kubernetes Secrets containing the peer token
    # for more details see: https://docs.ceph.com/en/latest/dev/cephfs-mirroring/#bootstrap-peers
    peers:
      secretNames:
        - secondary-cluster-peer
```

Also, the mirroring status is displayed in the CR status:

```
status:
  info:
    fsMirrorBootstrapPeerSecretName: fs-peer-token-myfs
  mirroringStatus:
    daemonsStatus:
    - daemon_id: 4186
      filesystems:
      - filesystem_id: 2
        name: myfs
    lastChecked: "2021-07-01T14:16:29Z"
  phase: Ready
  snapshotScheduleStatus:
    lastChecked: "2021-07-01T14:16:29Z"
    snapshotSchedules:
    - fs: myfs
      path: /
      rel_path: /
      retention: {}
      schedule: 24h
```

Closes: https://github.com/rook/rook/issues/7063
Signed-off-by: Sébastien Han <seb@redhat.com>
2021-07-01 17:35:19 +02:00
Jiffin Tony Thottan 9e3cf68d04 test: ci test for TLS objectstore
Extend the object store smoke test to include TLS configurations.

Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
2021-07-01 18:50:27 +05:30
Sébastien Han 9738131010 ci: retry on network build failures
We have seen cases where the build might fail due to a small network
hiccup so let's retry up to 3 times if this is the case.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-06-29 09:18:56 +02:00
Sébastien Han e6734304cd ci: format helper using standard 2 spaces indent
Bash common indent is 2 spaces so let's use it.
See: https://google.github.io/styleguide/shellguide.html

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-06-29 09:18:56 +02:00
Travis Nielsen 054004ba76 Merge pull request #7946 from leseb/ci-arm64-run
ci: add arm64 runner
2021-06-02 10:13:31 -06:00
Sébastien Han ba8940800f Merge pull request #8005 from leseb/luks-label
ceph: add ceph cluster fsid to LUKS header
2021-06-02 14:52:56 +02:00
Sébastien Han b178c8ae2c ci: add arm64 hosted runner
Thanks to @xin3liang we now have 2 hosted runners on ARM64. So let's add
a small integration test for it.
The action will run every day at midnight.

Successful run: https://github.com/rook/rook/pull/7946/checks?check_run_id=2726403458

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-06-02 11:54:43 +02:00
Sébastien Han 0004869f66 ceph: add ceph cluster fsid to LUKS header
When configuring encrypted on-pvc clusters we now set the cluster fsid
in the LUKS header. If needed we can then determine if the encrypted
block is part of our cluster or not. We also attach the pvc_name in case
it might become useful.

Closes: https://github.com/rook/rook/issues/7991
Signed-off-by: Sébastien Han <seb@redhat.com>
2021-05-31 15:01:52 +02:00
Sébastien Han 7c42b61f3c ci: add more debug logs to failed job
Gather more logs.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-05-31 14:16:46 +02:00
subhamkrai 7f0179c1ba ceph: update cert-manager version
this commit update cert-manager to v1.3.1

Signed-off-by: subhamkrai <srai@redhat.com>
2021-05-31 11:27:02 +05:30
subhamkrai 9773634cc1 ceph: wait for cainjector to inject the CA certificates
adding 20s sleep to let cainjector to inject the CA
certificates into cert-manager webhook.

Signed-off-by: subhamkrai <srai@redhat.com>
2021-05-31 11:27:02 +05:30
subhamkrai c738982503 ceph: add more log to admission controller
adding more logs in case there are errors
starting admission controller.

Signed-off-by: subhamkrai <srai@redhat.com>
2021-05-31 11:27:02 +05:30
Blaine Gardner 285dad339b ceph: update references to v15.2.11 to v16.2.4
Ceph v16 will be the default in Rook v1.7. Update references in the
master branch from v15 (currently v15.2.11) to v16 (currently v16.2.4).

Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
2021-05-26 12:30:18 -06:00
subhamkrai f988e2e517 ci: update action api token
this commit update the api token
to use correct secret.

Signed-off-by: subhamkrai <srai@redhat.com>
2021-05-07 11:24:15 +05:30
subhamkrai 4d66b7067b ci: right permission to build script
master build are failing due to bad
permission on build script. this
commit giv +rxw permission to the file.

Signed-off-by: subhamkrai <srai@redhat.com>
2021-05-07 06:57:08 +05:30
subhamkrai 4683254419 ci: push build images to Docker hub using gh action
We are moving towards github action.
This commit allows pushing build images
to Docker hub using gh action for master
build similar to what Jenkins does.

And, also removing `Publish if Master` step
from the Jenkinsfil file.

Signed-off-by: subhamkrai <srai@redhat.com>
2021-05-06 12:45:24 +05:30
Sébastien Han 0c806588d2 Merge pull request #7790 from leseb/retry-vg
ci: retry vgcreate on failure
2021-04-29 19:01:35 +02:00
Sébastien Han c63a42a661 ci: retry vgcreate on failure
Try to mitigate the following error from the CI:

```
+ sudo vgcreate test-rook-vg /dev/sdb
  Can't open /dev/sdb exclusively.  Mounted filesystem?
Error: Process completed with exit code 5.

```

Now we retry up to 3 times to create the volume group.

Closes: https://github.com/rook/rook/issues/7487
Signed-off-by: Sébastien Han <seb@redhat.com>
2021-04-29 18:24:48 +02:00
Sébastien Han 0f7128ece9 build: fix parallel build
We now the generate the CRDs as well as the CSV on a single GOARCH only:
amd64. This helps us avoiding builds colliding with each others.

Closes: https://github.com/rook/rook/issues/7655
Signed-off-by: Sébastien Han <seb@redhat.com>
2021-04-29 17:31:22 +02:00
Jiffin Tony Thottan 3cacd8db32 test: add seperate test for rgw vault integration
RGW supports only v2 version of kv engine, current integration test is written for kv v1,
hence seperating existing test into two.

Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
2021-04-28 14:40:08 +05:30
subhamkrai 7974d46b20 ci: improve github action writing
we have lots of duplicate codes for
github action. this commit refactor
them to a bash script which will minimize
duplicate codes.

Signed-off-by: subhamkrai <srai@redhat.com>
2021-04-27 20:15:52 +05:30
Satoru Takeuchi 8a48dbb00d ceph: stop vendoring
We generate vendor directory in building process. It doesn't reduce
build time. In addition, we have encountered some vendoring-related
problems. So let's stop vendoring.

Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
2021-04-21 02:01:56 +00:00
subhamkrai 9305f12ecf ci: make build not failing
currently, the build action is not failing if there
is/are any modified files after the build. this commit
will validate build with `git status` after `make build`
command will fail if any changed file found.

Signed-off-by: subhamkrai <srai@redhat.com>
2021-04-20 13:10:33 +05:30
Travis Nielsen 665856b90a ceph: enable pacific as a supported ceph version
With the Ceph Pacific release coming this week we add support
in Rook for Pacific with the Rook v1.6 release coming soon.
The integration tests will now run across nautilus, octopus,
and pacific to cover all supported Ceph versions. The default
examples still specify Octopus until there is more bake time
for Pacific.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-04-07 06:59:04 -06:00