Commit Graph
927 Commits
Author SHA1 Message Date
parth-gr b28455245d ci: fix for CephObjectStores flakiness
Integration test CephSmokeSuite fails frequently
A quick fix for it by reordering storeName,
running tlsteststore before teststore

Closes: https://github.com/rook/rook/issues/8309
Signed-off-by: parth-gr <paarora@redhat.com>
2021-08-09 14:08:32 +05:30
Travis Nielsen 529ec0ff7e Merge pull request #8461 from subhamkrai/automate-release-action
ci: create tag and print release notes
2021-08-06 09:41:11 -06:00
subhamkrai 8572fdb58b ci: create tag and print release notes
since we moved to github action for the release,
we maunally have to push the tag to build image.

with this change,
first, this action will set some
environment variable which will be consume by
the script to print release notes.
then,
it will ask for 1 input from
github UI(in Action tab)
tag: where we'll give the tag for the release

after creating tag it will run the script and
print release notes.

**NOTE** please verify the environment var
and release notes after the successful action run.

.

Signed-off-by: subhamkrai <srai@redhat.com>
2021-08-06 16:00:25 +05:30
Sébastien Han d7ce570644 Merge pull request #8495 from leseb/fix-retry
ci: break if no errors on build
2021-08-06 08:41:16 +02:00
Sébastien Han 6582fdce97 ci: break if no errors on build
No wonder why the build has been taking longer... We were building 3
times.
We just got at least 2 minutes back of build time thanks to this fix.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-08-05 15:40:48 +02:00
subhamkraiandTravis Nielsen 12de60e4a0 ceph: get pr tile and user for release note
this commit include script which will help
the person who is creating release note
to get all the pr number with title and owner
in required order for the release notes. In the
past, it was manual step.

Co-authored-by: Travis Nielsen <tnielsen@redhat.com>
Signed-off-by: subhamkrai <srai@redhat.com>
2021-08-04 20:19:43 +05:30
Sébastien Han 99e00dea1e ceph: auto detect vault k/v version
Rook will now auto detect the kv version of the vault server. This
allows users not having to pass the VAULT_BACKEND configuration in the
CephCluster CR.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-07-29 10:16:31 +02:00
Travis Nielsen c2a551123f Merge pull request #8401 from TomHellier/add-additional-rook-ceph-cluster-helm-features
ceph: Add additional helm chart functionality for ingresses and defining the ceph storage crds
2021-07-28 16:14:48 -06:00
Tom Hellier 89ab4f90ec ceph: adds helm functionality for ingress, and ceph storage crds
This commit adds an ingress resource to the rook-ceph-cluster helm chart, allowing
ingress to the ceph-dashboard service. It also adds the ability to define the
various storage types that you can run on ceph inside kubernetes.

Closes https://github.com/rook/rook/issues/8384

Signed-off-by: Tom Hellier <me@tomhellier.com>
2021-07-28 21:51:56 +01:00
Travis Nielsen 643eddf261 Merge pull request #8291 from jmolmo/enable_mgr_tests
ceph: enable again the Rook orchestrator mgr test
2021-07-28 09:27:46 -06:00
Juan Miguel Olmo Martínez 9edff582c3 ceph: enable again the Rook orchestrator mgr test
Enable again the mgr test:
- Now is more reliable and robust the start of the test.
- Minor fixes to adapt the <service ls>  to the new name of the crash daemon
deployed by rook

- The creation of OSDs is disabled in the orchestrator, so i have removed
this test until we will have the functionality ready again in the orchestrator
part

My plan is to provide in the orchestrator two different ways to create OSDs:
- Creation of OSD using specific devices if discovery daemon is running
- Creation of OSds using PVS (if we have LSO/other LS operator running)

Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
2021-07-28 16:27:42 +02:00
Travis Nielsen 87596c6f8e build: detect git version instead of setting to 0
The build version should be detected with the git describe
command, but if the VERSION var is already set, it will be
used instead of being detected from git. During the official
builds or even local builds, we just want to let the makefile
detect the git version.

The full git history is added to enable proper detection of the
version. Without the full history only a hash is returned for
the version, which then results in an invalid semantic version
error for the helm chart.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-07-28 07:25:10 -06:00
Travis Nielsen 7830095809 build: promote the build with tag suffix if needed
The beta releases should have the tag suffx included in
the helm chart and release version tags.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
(cherry picked from commit a1d37b4efc)
2021-07-26 17:34:25 -06:00
Travis Nielsen 67a3822e95 build: publish and promote with correct branch name
The tagged builds will run git checkout on the correct tag,
though the publish and promote actions expect a branch name
to be available due to internal build assumptions. Now the
publish looks up the branch name from the tag and passes
it to the publish command as expected.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
(cherry picked from commit d8c5f459de)
2021-07-26 17:34:10 -06:00
Travis Nielsen 61b85a2d3a Merge pull request #8372 from subhamkrai/add-missing-function
ci: missing function call in script
2021-07-23 10:55:10 -06:00
subhamkrai e291afad07 ci: missing function call in script
in pr 8328, function call was missing
so action for pushing the image is not
starting.

Signed-off-by: subhamkrai <srai@redhat.com>
2021-07-23 22:05:15 +05:30
Travis Nielsen 069bc484ee Merge pull request #8328 from subhamkrai/action-for-release-build
ci: action for release build
2021-07-23 10:21:30 -06:00
Sébastien Han c8ee5674dd Merge pull request #8103 from leseb/rm-jenkins
ci: remove jenkins from master branch
2021-07-23 18:20:10 +02:00
Sébastien Han 0811359d28 Merge pull request #8358 from leseb/move-to-quay
ceph: move all of our docker.io reference to quay.io
2021-07-23 18:03:53 +02:00
subhamkrai 492a8176e0 ci: action for release build
this commit adds github action for
release builds and also modify the
helper script `build-release.sh`.

Closes: https://github.com/rook/rook/issues/8041
Signed-off-by: subhamkrai <srai@redhat.com>
2021-07-23 13:09:30 +05:30
Travis Nielsen 07ddcda0ce ceph: update test to watch for v1 cronjob
The v1beta1 cronjob is deprecated and needs to use v1 for K8s 1.16 and newer

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-07-22 11:25:55 -06:00
Sébastien Han 6bce1ff3e9 ceph: move all of our docker.io reference to quay.io
Recently, the builds of `ceph/ceph` image moved to quay.io, see
https://github.com/ceph/ceph-build/pull/1883 for more details.
Current images will remain but new builds will happen on quay.io only.

This means that tags such as `v14.2`, `v15.2`,`v16.2` will need to
switch to quay.io to get updates.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-07-22 11:17:05 +02:00
Sébastien Han db40a31bb2 ci: change yq path to /usr/local/bin/yq
So it will override any existing installation.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-07-21 14:15:07 +02:00
Travis Nielsen d1f02f22f3 ceph: test upgrades from v1.6 to master
With v1.7 approaching, the upgrade integration test will now test from
v1.6.x to the latest master, which will effectively become the v1.7
release soon.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-07-20 11:03:40 -06:00
Travis NielsenandNicolaj Græsholt f114db952b nfs: generate the crds from types for v1
The CRDs v1 requires the full schema for all settings, so we now
generate the CRDs for nfs for full fidelity of all settings.

Co-authored-by: Nicolaj Græsholt <figaw@hotmail.com>
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-07-20 08:18:05 -06:00
Travis Nielsen 093cf6dbbd cassandra: generate the crds from types for v1
The CRDs v1 requires the full schema for all settings, so we now
generate the CRDs for cassandra for full fidelity of all the
settings.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-07-20 08:18:05 -06:00
Travis Nielsen 6ad9190f55 ceph: factor out test manifest helpers for providers
The ceph provider had been refactored to read the crds
and operator manifests from a file instead of copying the
manifests into the test code. Now the helpers are refactored
to allow the other providers to also reduce the manifest
duplication in tests.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-07-20 08:17:29 -06:00
Sébastien Han 741dd25efb Merge pull request #8278 from leseb/retry-ci
ci: retry on network errors
2021-07-08 08:52:28 +02:00
Sébastien Han eaa6e7732c Merge pull request #8272 from leseb/exec-in-pod
ceph: proxy ceph command when multus is configured
2021-07-07 21:36:54 +02:00
Sébastien Han bd58790c31 ceph: proxy ceph commands when multus is configured
When the CephCluster is configured with Multus and multiple networks are
used to deploy Ceph some commands are failing to be executed from the
Operator. These commands, in particular, `radosgw-admin` ones need access
to the "ceph public network" to talk to OSDs. Unfortunately, the
Rook-Ceph Operator does not have the network annotations and thus
doesn't have the networks available and cannot reach OSDs. So the commands end
up hanging and eventually time out.
Applying the annotations to the Operator pod is possible but will result
in restarting the operator too and this should be avoided at all costs.
Also, applying the annotations beforehand is not possible since the
Multus declaration is in the CephCluster specification. So we would have
no idea what to do.

So the current approach runs a new sidecar container in the mgr pod to
act as a proxy for "some" ceph commands, only the `radosgw-admin` ones
for multi-site setup. This is a small container with admin access
running idle waiting for commands to be executed. In a sense, it is
similar to the toolbox but we didn't want to clearly expose it, so
running as a sidecar is quite nice.

Proxying command is obviously not always recommended since we add an
extra hop in the network path. Now each request has to go from the
operator pod to the API server to the remote pod to Ceph. Previously,
the command only goes from the operator to Ceph.

It's worth noting that external mode is not impacted since no rgw pod
is configured. This scenario is flexible and allows us to scale
pretty well since any CephCluster with Multus will see its mgr sidecar
deployed and can then talk to Ceph. We are not limited.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-07-07 19:08:32 +02:00
Sébastien Han ff8d24a296 ci: retry on network errors
Seen today in https://github.com/rook/rook/pull/8272/checks?check_run_id=3010477951

```
github.com/rook/rook/pkg/apis/ceph.rook.io/v1 imports
	github.com/hashicorp/vault/api: github.com/hashicorp/vault@v1.4.2: reading https://proxy.golang.org/github.com/hashicorp/vault/@v/v1.4.2.zip: 503 Service Unavailable
	server response: Service Unavailable
github.com/rook/rook/pkg/daemon/ceph/osd/kms imports
	github.com/libopenstorage/secrets/vault imports
	github.com/hashicorp/vault/command/agent/auth: github.com/hashicorp/vault@v1.4.2: reading https://proxy.golang.org/github.com/hashicorp/vault/@v/v1.4.2.zip: 503 Service Unavailable
	server response: Service Unavailable
github.com/rook/rook/pkg/daemon/ceph/osd/kms imports
	github.com/libopenstorage/secrets/vault imports
	github.com/hashicorp/vault/command/agent/auth/kubernetes: github.com/hashicorp/vault@v1.4.2: reading https://proxy.golang.org/github.com/hashicorp/vault/@v/v1.4.2.zip: 503 Service Unavailable
	server response: Service Unavailable
make: *** [go.mod.check] Error 1
```

Also https://github.com/rook/rook/pull/8272/checks?check_run_id=3010482113

```
go get: sigs.k8s.io/controller-tools@v0.4.1 updating to
	sigs.k8s.io/controller-tools@v0.5.1-0.20210420220833-f284e2e8098c requires
	k8s.io/apimachinery@v0.20.2 requires
	github.com/json-iterator/go@v1.1.10: verifying go.mod: github.com/json-iterator/go@v1.1.10/go.mod: reading https://sum.golang.org/tile/8/0/x005/043: stream error: stream ID 1099; INTERNAL_ERROR
make[3]: *** [/home/runner/work/rook/rook/.cache/tools/linux_amd64/controller-gen-f284e2e8098cb0193c2b0c7c1c651ae75496539b] Error 1
make[2]: *** [generate-csv-ceph-templates] Error 1
```

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-07-07 18:00:01 +02:00
subhamkrai bc1fabb6e4 ci: update helm version to 3.6.2
this commit update helm version to `3.6.2`

Signed-off-by: subhamkrai <srai@redhat.com>
2021-07-07 20:09:26 +05:30
Sébastien Han 830b36c039 Merge pull request #7920 from thotz/tlscitest
test: ci test for TLS objectstore
2021-07-05 11:23:40 +02:00
Sébastien Han baaea4a1ea Merge pull request #7604 from leseb/cephfs-mirror-peer-config
ceph: add filesystem mirror peers configuration
2021-07-05 10:36:30 +02:00
Travis Nielsen 7cec49b2d8 ceph: run helm tests without admission controller
The helm tests frequently hit an issue with the admission controller.
The admission controller is also enabled in other test suites, so
it seems safe enough to disable for the helm test suite.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-07-02 15:28:57 -06:00
Sébastien Han b578f916e7 ceph: add fs mirror config
Similarly to block volume replication, Ceph is capable of replicating the
content of a Ceph Filesystem from one cluster to another.
For this, during the 1.6 cycle, we introduced a new CRD called
CephFilesystemMirror which effectively deploys a cephfs-mirror daemon.
However, configuring peers to enable replication between two clusters
had to be done manually.
Also various bug fix made it in Ceph eventually and the minimum required
version for this to work is to run on Ceph Pacific 16.2.5 at least.

So the automatic configuration of Ceph Filesystem peers is now possible.

By editing the CephFilesystem CRD, you can now turn on mirroring:

```yaml
  mirroring:
    enabled: false
    # list of Kubernetes Secrets containing the peer token
    # for more details see: https://docs.ceph.com/en/latest/dev/cephfs-mirroring/#bootstrap-peers
    peers:
      secretNames:
        - secondary-cluster-peer
```

Also, the mirroring status is displayed in the CR status:

```
status:
  info:
    fsMirrorBootstrapPeerSecretName: fs-peer-token-myfs
  mirroringStatus:
    daemonsStatus:
    - daemon_id: 4186
      filesystems:
      - filesystem_id: 2
        name: myfs
    lastChecked: "2021-07-01T14:16:29Z"
  phase: Ready
  snapshotScheduleStatus:
    lastChecked: "2021-07-01T14:16:29Z"
    snapshotSchedules:
    - fs: myfs
      path: /
      rel_path: /
      retention: {}
      schedule: 24h
```

Closes: https://github.com/rook/rook/issues/7063
Signed-off-by: Sébastien Han <seb@redhat.com>
2021-07-01 17:35:19 +02:00
Jiffin Tony Thottan 9e3cf68d04 test: ci test for TLS objectstore
Extend the object store smoke test to include TLS configurations.

Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
2021-07-01 18:50:27 +05:30
Blaine Gardner 037945aabc Merge pull request #8112 from BlaineEXE/dependencies-2-son-of-dependencies
ceph: block delete object store when buckets and/or users exist
2021-06-30 12:45:58 -06:00
Blaine Gardner c22f545ebf ceph: block delete object store when buckets exist
Block deletion of CephObjectStore resources when buckets exist in the
object store.

Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
2021-06-29 14:31:39 -06:00
Sébastien Han 9738131010 ci: retry on network build failures
We have seen cases where the build might fail due to a small network
hiccup so let's retry up to 3 times if this is the case.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-06-29 09:18:56 +02:00
Sébastien Han e6734304cd ci: format helper using standard 2 spaces indent
Bash common indent is 2 spaces so let's use it.
See: https://google.github.io/styleguide/shellguide.html

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-06-29 09:18:56 +02:00
Andy Bursavich 11f0add706 build: update golangci-lint version and nolint comment format
Signed-off-by: Andy Bursavich <abursavich@gmail.com>
2021-06-21 08:50:04 -07:00
Andy Bursavich 7ab1baa794 test: extract utils dependency on k8s.io/kubernetes
Signed-off-by: Andy Bursavich <abursavich@gmail.com>
2021-06-21 08:36:08 -07:00
subhamkrai 0ff452fdd1 ceph: remove unnecessary file
this commit removes `mysql_helper.go`,
unnecessary file.

Signed-off-by: subhamkrai <srai@redhat.com>
2021-06-15 15:28:13 +05:30
Travis Nielsen 44d31f148e ceph: update the toolbox in the upgrade test
The upgrade test needs to also update the toolbox to ensure that it will not
be denied access to the cluster when the insecure connections are disabled
by the operator.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-06-10 10:05:30 -06:00
Sébastien Han 18b1477352 ci: remove jenkins from master branch
Thank you Jenkins for all those years, but it's to retire.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-06-10 15:34:15 +02:00
Sébastien Han fccbadd837 ci: only run ceph-csi volume replication on k8s >= 1.16
Prior to 1.16 we do not injec the volumereplication CRDs so the
csi-rbdplugin-provisioner will fail to fetch the CRD.

Closes: https://github.com/rook/rook/issues/8039
Signed-off-by: Sébastien Han <seb@redhat.com>
2021-06-10 15:13:05 +02:00
Travis Nielsen 054004ba76 Merge pull request #7946 from leseb/ci-arm64-run
ci: add arm64 runner
2021-06-02 10:13:31 -06:00
Sébastien Han ba8940800f Merge pull request #8005 from leseb/luks-label
ceph: add ceph cluster fsid to LUKS header
2021-06-02 14:52:56 +02:00
Sébastien Han b178c8ae2c ci: add arm64 hosted runner
Thanks to @xin3liang we now have 2 hosted runners on ARM64. So let's add
a small integration test for it.
The action will run every day at midnight.

Successful run: https://github.com/rook/rook/pull/7946/checks?check_run_id=2726403458

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-06-02 11:54:43 +02:00