Commit Graph
7173 Commits
Author SHA1 Message Date
Sébastien Han eaa6e7732c Merge pull request #8272 from leseb/exec-in-pod
ceph: proxy ceph command when multus is configured
2021-07-07 21:36:54 +02:00
Sébastien Han bd58790c31 ceph: proxy ceph commands when multus is configured
When the CephCluster is configured with Multus and multiple networks are
used to deploy Ceph some commands are failing to be executed from the
Operator. These commands, in particular, `radosgw-admin` ones need access
to the "ceph public network" to talk to OSDs. Unfortunately, the
Rook-Ceph Operator does not have the network annotations and thus
doesn't have the networks available and cannot reach OSDs. So the commands end
up hanging and eventually time out.
Applying the annotations to the Operator pod is possible but will result
in restarting the operator too and this should be avoided at all costs.
Also, applying the annotations beforehand is not possible since the
Multus declaration is in the CephCluster specification. So we would have
no idea what to do.

So the current approach runs a new sidecar container in the mgr pod to
act as a proxy for "some" ceph commands, only the `radosgw-admin` ones
for multi-site setup. This is a small container with admin access
running idle waiting for commands to be executed. In a sense, it is
similar to the toolbox but we didn't want to clearly expose it, so
running as a sidecar is quite nice.

Proxying command is obviously not always recommended since we add an
extra hop in the network path. Now each request has to go from the
operator pod to the API server to the remote pod to Ceph. Previously,
the command only goes from the operator to Ceph.

It's worth noting that external mode is not impacted since no rgw pod
is configured. This scenario is flexible and allows us to scale
pretty well since any CephCluster with Multus will see its mgr sidecar
deployed and can then talk to Ceph. We are not limited.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-07-07 19:08:32 +02:00
Sébastien Han d6dd557961 Merge pull request #8276 from subhamkrai/update-helm-ci-version
ci: update helm version to 3.6.2
2021-07-07 17:28:50 +02:00
subhamkrai bc1fabb6e4 ci: update helm version to 3.6.2
this commit update helm version to `3.6.2`

Signed-off-by: subhamkrai <srai@redhat.com>
2021-07-07 20:09:26 +05:30
Sébastien Han 65ab66d3f4 Merge pull request #8269 from thotz/healthcheckadminops
ceph: remove creation of adminops client in checkObjectStoreHealth()
2021-07-06 16:57:11 +02:00
Sébastien Han 93937fb0b5 Merge pull request #8243 from thotz/rgwk8stlssupport
ceph: add support for tls certs via k8s tls secrets for rgw
2021-07-06 16:56:33 +02:00
Sébastien Han e3efdb374b Merge pull request #8242 from subhamkrai/helm-suite-failing
ci: in helm chart field should not be empty
2021-07-06 16:56:08 +02:00
Jiffin Tony Thottan 1665ad6ea7 ceph: add support for tls certs via k8s tls secrets for rgw
With this PR the RGW can accept TLS certs as K8s TLS secrets

Fixes: 2079
Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
2021-07-06 16:30:16 +05:30
subhamkrai 34313c9dc9 ci: in helm chart field should not be empty
ci is complaining due to empty field in
values.yaml with error
```
coalesce.go:199: warning: destination for config is a table. Ignoring non-table value <nil>
```
this commit make those field empty map/list.

Closes: https://github.com/rook/rook/issues/8239
Signed-off-by: subhamkrai <srai@redhat.com>
2021-07-06 14:28:58 +05:30
Jiffin Tony Thottan 6c2c95284c ceph: remove creation of adminops client in checkObjectStoreHealth()
The adminops client is already created in newBucketChecker(), use that
instance in checkObjectStoreHealth() than creating a new one.

Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
2021-07-06 11:40:11 +05:30
Sébastien Han 830b36c039 Merge pull request #7920 from thotz/tlscitest
test: ci test for TLS objectstore
2021-07-05 11:23:40 +02:00
Sébastien Han c4d15f4de2 Merge pull request #8254 from henryzhangsta/helm-network-optional
ceph: network configuration should be optional in Helm chart
2021-07-05 11:23:09 +02:00
Sébastien Han 97f1cc3fd2 Merge pull request #8244 from thotz/obcfetchcorrectport
ceph: fetch rgw port from cephobjectstore crd for obc
2021-07-05 11:21:48 +02:00
Sébastien Han 23f3f17d24 Merge pull request #8250 from travisn/nfs-error-on-scaledown
ceph: Scaling down nfs deployment was failing
2021-07-05 11:19:11 +02:00
Sébastien Han baaea4a1ea Merge pull request #7604 from leseb/cephfs-mirror-peer-config
ceph: add filesystem mirror peers configuration
2021-07-05 10:36:30 +02:00
Sébastien Han 6c408e04dc Merge pull request #8253 from travisn/stable-helm-tests
ceph: Run helm tests without admission controller
2021-07-05 10:30:19 +02:00
Sébastien Han 6be28011a4 Merge pull request #8208 from thotz/enabledebugforadminOps
ceph: enable debug for adminops client if the rook loglevel <= debug
2021-07-05 10:22:50 +02:00
Sébastien Han d59ae25d2b Merge pull request #8215 from sp98/disable-mirroring
ceph: ability to disable pool mirroring
2021-07-05 10:22:24 +02:00
Jiffin Tony Thottan e4a9b4bac1 ceph: fetch rgw port from cephobjectstore crd for obc
Currently `setObjectStorePort()` queries port from RGW service and
picks up the first port from the list, other places of the code
prefers for TLS, if it is enabled so for OBC's we end up having
`https` endpoint with non-secure port.

Fixes: #8141
Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
2021-07-05 10:55:05 +05:30
Henry Zhang a5227bf126 ceph: network configuration should be optional in Helm chart
adds a default empty dict for the network when deciding whether toolbox
should also be on the host network to catch the case when the entire
`network` block is empty

Signed-off-by: Henry Zhang <me@henry.dev>
2021-07-04 01:05:08 -07:00
Travis Nielsen 7cec49b2d8 ceph: run helm tests without admission controller
The helm tests frequently hit an issue with the admission controller.
The admission controller is also enabled in other test suites, so
it seems safe enough to disable for the helm test suite.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-07-02 15:28:57 -06:00
Travis Nielsen 4b1cf51ef1 ceph: scaling down nfs deployment was failing
When scaling down the nfs deployment gracefully, the wrong error code was checked,
resulting in a failed scaling down even though the condition was expected.
Now we check properly if the deployment was removed properly.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-07-02 11:52:03 -06:00
Travis Nielsen 0a09d844ad Merge pull request #8237 from travisn/retry-after-cancel
ceph: Retry reconcile immediately after cancellation
2021-07-02 10:11:24 -06:00
Travis Nielsen 8778e3cab5 Merge pull request #8238 from subhamkrai/oom-killed
ceph: operator pod killed due to resource limit
2021-07-02 10:11:04 -06:00
Travis Nielsen fc5402b3e5 Merge pull request #8236 from travisn/debug-log-k8s-version
ceph: Debug log the k8s version instead of info
2021-07-02 09:23:39 -06:00
subhamkraiandSébastien Han 4241ce7e66 ceph: operator pod killed due to resource limit
when operator pod resources are set(which are
default in case of helm), operator pod are
killed due to using concurrency while creating
object store.

this commit checks if operator resources are set
or not. if set then we'll *not* create object store
in concurrency or if not set then we'll create in
concurrency.

Closes: https://github.com/rook/rook/issues/8149
Co-authored-by: Sébastien Han <seb@redhat.com>
Signed-off-by: subhamkrai <srai@redhat.com>
2021-07-02 20:18:58 +05:30
Travis Nielsen c39c1c7ddf ceph: retry reconcile immediately after cancellation
If the reconcile is cancelled due to a CR update, we want to retry the next
reconcile immediately rather than wait for the exponential backoff timeout
if the reconcile was already failing. The wait can easily be minutes
if the reconcile was in this state, which makes it appear the operator
is ignoring the request to start a new reconcile.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-07-02 08:23:25 -06:00
Travis Nielsen 889a082734 Merge pull request #8029 from Yuggupta27/update-csidriver
ceph: update CSIDriver object from betav1 to v1
2021-07-01 23:46:36 -06:00
Santosh Pillai 05b0ae09bb ceph: disable mirroring
The PR disables mirroring on a pool when PoolSpec.Mirroring.Enabled
is set to false
- disable mirroring on the pool for `Mirroring.Mode == pool`
- Add warning for the user to disable mirroring manually for `Mirroring.Mode == image`
- Stop the mirroring health checker goroutine
- Reset the mirroring health check status in the CR.

Signed-off-by: Santosh Pillai <sapillai@redhat.com>
2021-07-02 10:35:38 +05:30
Yug d513dc4bc5 ceph: update CSIDriver object from betav1 to v1
Update the CSIDriver object to v1 from betav1 as it will be
deprecated in v1.19+ and become unavailable in v1.22+

Signed-off-by: Yug <yuggupta27@gmail.com>
2021-07-02 10:19:53 +05:30
Travis Nielsen aaa939f686 ceph: debug log the k8s version instead of info
The info log is rather verbose with all the filtering of the K8s version
like the following:
returning version v1.19.0 instead of v1.19.0+43983cd

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-07-01 17:09:20 -06:00
Travis Nielsen af18727359 Merge pull request #8227 from leseb/silence-errors
ceph: silence harmless errors
2021-07-01 10:12:32 -06:00
Sébastien Han b578f916e7 ceph: add fs mirror config
Similarly to block volume replication, Ceph is capable of replicating the
content of a Ceph Filesystem from one cluster to another.
For this, during the 1.6 cycle, we introduced a new CRD called
CephFilesystemMirror which effectively deploys a cephfs-mirror daemon.
However, configuring peers to enable replication between two clusters
had to be done manually.
Also various bug fix made it in Ceph eventually and the minimum required
version for this to work is to run on Ceph Pacific 16.2.5 at least.

So the automatic configuration of Ceph Filesystem peers is now possible.

By editing the CephFilesystem CRD, you can now turn on mirroring:

```yaml
  mirroring:
    enabled: false
    # list of Kubernetes Secrets containing the peer token
    # for more details see: https://docs.ceph.com/en/latest/dev/cephfs-mirroring/#bootstrap-peers
    peers:
      secretNames:
        - secondary-cluster-peer
```

Also, the mirroring status is displayed in the CR status:

```
status:
  info:
    fsMirrorBootstrapPeerSecretName: fs-peer-token-myfs
  mirroringStatus:
    daemonsStatus:
    - daemon_id: 4186
      filesystems:
      - filesystem_id: 2
        name: myfs
    lastChecked: "2021-07-01T14:16:29Z"
  phase: Ready
  snapshotScheduleStatus:
    lastChecked: "2021-07-01T14:16:29Z"
    snapshotSchedules:
    - fs: myfs
      path: /
      rel_path: /
      retention: {}
      schedule: 24h
```

Closes: https://github.com/rook/rook/issues/7063
Signed-off-by: Sébastien Han <seb@redhat.com>
2021-07-01 17:35:19 +02:00
Travis Nielsen 5dde16fcf7 Merge pull request #8180 from subhamkrai/mds-resource-limit
ceph: give resource request preference for mds_cache_memory_limit
2021-07-01 09:08:17 -06:00
subhamkrai d41878cb9e ceph: give resource request preference for mds_cache_memory_limit
earlier, for mds, when resource limit is not defined it has
default limit of `"mds_cache_memory_limit": "4294967296"`
even if resouce.request is defined.

Now, resource request will be given preference when both
request and limit is defined and if only limit is defined
it will be applied or otherwise.

Closes: https://github.com/rook/rook/issues/8143
Signed-off-by: subhamkrai <srai@redhat.com>
2021-07-01 19:54:07 +05:30
Sébastien Han 8c589a3785 ceph: silence harmless errors
Let's not look at misleading errors if the operator is still
initializing.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-07-01 15:37:08 +02:00
Jiffin Tony Thottan 9e3cf68d04 test: ci test for TLS objectstore
Extend the object store smoke test to include TLS configurations.

Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
2021-07-01 18:50:27 +05:30
Sébastien Han b68cac34cb Merge pull request #8222 from sp98/update-block-pool-channels
ceph: update blockPoolChannel before starting the mirror monitoring
2021-07-01 14:10:17 +02:00
Santosh Pillai 8d48a9e10a ceph: update blockPoolChannel before starting the mirror monitoring
update blockPoolChannel monitoringRunning field to `true` before
starting the goroutine to monitor mirroring

Signed-off-by: Santosh Pillai <sapillai@redhat.com>
2021-07-01 15:27:37 +05:30
Sébastien Han 048b75c254 Merge pull request #8226 from leseb/fix-secetion-pending-note
docs: add news feature in the features section
2021-07-01 11:38:27 +02:00
Sébastien Han 647cccf1d0 docs: add news feature in the features section
New features were in the "breaking changes" section.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-07-01 10:57:48 +02:00
Sébastien Han 2baac8372f Merge pull request #8172 from sp98/hybrid-storage-pools
ceph: add support for hybrid Storage Pools
2021-07-01 09:15:16 +02:00
Jiffin Tony Thottan 560f44b07e ceph: enable debug for adminops client if the rook loglevel <= debug
The debug for adminOps client can be enabled by setting `Debug` flag.
In this PR, it is enabled for OBC and cephobjectstore not healthchecker.
Also add similar changes while calling `NewS3Agent` in the bucket
provisioner code path.

Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
2021-07-01 11:36:51 +05:30
Santosh Pillai 4041868e2e ceph: hybrid Storage Pools
Creates hybrid crush rule for choosing Primary OSD for
high performing SSD devices and remaining OSD for low performance HDD devices.

Signed-off-by: Santosh Pillai <sapillai@redhat.com>
2021-07-01 11:16:54 +05:30
Blaine Gardner 037945aabc Merge pull request #8112 from BlaineEXE/dependencies-2-son-of-dependencies
ceph: block delete object store when buckets and/or users exist
2021-06-30 12:45:58 -06:00
Travis Nielsen 03f6dddcfd Merge pull request #8170 from subhamkrai/update-helm-version
ci: remove read access from kube config
2021-06-30 11:29:35 -06:00
Blaine Gardner 53459bbb44 Merge pull request #8195 from BlaineEXE/ignore-atari-partitions
ceph: ignore atari partitions when provisioning OSDs
2021-06-30 11:26:56 -06:00
Blaine Gardner 7f516b9e3d ceph: ignore atari partitions when scanning disks
Ceph bluestore raw disks can sometimes appear as though they have Atari
(AHDI) partitions on them. If a disk has Atari partitions, we just
ignore them as though they don't exist. This should be a safe assumption
since the hardware was last manufactured in 1992 and likely can't run
Kubernetes.

If we don't ignore the Atari partitions, Rook can create a new OSD on a
disk that is already running an OSD, corrupting the first and possibly
also the latest OSD. This can happen an arbitrary number of times per
disk.

More info: https://github.com/rook/rook/issues/7940

Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
2021-06-30 09:42:33 -06:00
Travis Nielsen c77a9a9aa8 Merge pull request #8219 from leseb/bz-1964467-upstream
ceph: do not require rgw 'instances' spec
2021-06-30 07:24:00 -06:00
Sébastien Han e25e0e8103 ceph: do not require rgw 'instances' spec
When the cluster is external we don't have any instances of rgw running
so let's not have this as a required field in the spec.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-06-30 11:24:09 +02:00