When the CephCluster is configured with Multus and multiple networks are
used to deploy Ceph some commands are failing to be executed from the
Operator. These commands, in particular, `radosgw-admin` ones need access
to the "ceph public network" to talk to OSDs. Unfortunately, the
Rook-Ceph Operator does not have the network annotations and thus
doesn't have the networks available and cannot reach OSDs. So the commands end
up hanging and eventually time out.
Applying the annotations to the Operator pod is possible but will result
in restarting the operator too and this should be avoided at all costs.
Also, applying the annotations beforehand is not possible since the
Multus declaration is in the CephCluster specification. So we would have
no idea what to do.
So the current approach runs a new sidecar container in the mgr pod to
act as a proxy for "some" ceph commands, only the `radosgw-admin` ones
for multi-site setup. This is a small container with admin access
running idle waiting for commands to be executed. In a sense, it is
similar to the toolbox but we didn't want to clearly expose it, so
running as a sidecar is quite nice.
Proxying command is obviously not always recommended since we add an
extra hop in the network path. Now each request has to go from the
operator pod to the API server to the remote pod to Ceph. Previously,
the command only goes from the operator to Ceph.
It's worth noting that external mode is not impacted since no rgw pod
is configured. This scenario is flexible and allows us to scale
pretty well since any CephCluster with Multus will see its mgr sidecar
deployed and can then talk to Ceph. We are not limited.
Signed-off-by: Sébastien Han <seb@redhat.com>
It's better to create PDB of RGW if ObjectStore's `instances` field is 2.
In the current implementation, at least three desired RGW instances are needed
to create this. If `instances` is 1, we can still safely skip the creation
of PDB because it's useless.
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
The adminops client is already created in newBucketChecker(), use that
instance in checkObjectStoreHealth() than creating a new one.
Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
Currently `setObjectStorePort()` queries port from RGW service and
picks up the first port from the list, other places of the code
prefers for TLS, if it is enabled so for OBC's we end up having
`https` endpoint with non-secure port.
Fixes: #8141
Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
When scaling down the nfs deployment gracefully, the wrong error code was checked,
resulting in a failed scaling down even though the condition was expected.
Now we check properly if the deployment was removed properly.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
when operator pod resources are set(which are
default in case of helm), operator pod are
killed due to using concurrency while creating
object store.
this commit checks if operator resources are set
or not. if set then we'll *not* create object store
in concurrency or if not set then we'll create in
concurrency.
Closes: https://github.com/rook/rook/issues/8149
Co-authored-by: Sébastien Han <seb@redhat.com>
Signed-off-by: subhamkrai <srai@redhat.com>
If the reconcile is cancelled due to a CR update, we want to retry the next
reconcile immediately rather than wait for the exponential backoff timeout
if the reconcile was already failing. The wait can easily be minutes
if the reconcile was in this state, which makes it appear the operator
is ignoring the request to start a new reconcile.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The PR disables mirroring on a pool when PoolSpec.Mirroring.Enabled
is set to false
- disable mirroring on the pool for `Mirroring.Mode == pool`
- Add warning for the user to disable mirroring manually for `Mirroring.Mode == image`
- Stop the mirroring health checker goroutine
- Reset the mirroring health check status in the CR.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
Update the CSIDriver object to v1 from betav1 as it will be
deprecated in v1.19+ and become unavailable in v1.22+
Signed-off-by: Yug <yuggupta27@gmail.com>
The info log is rather verbose with all the filtering of the K8s version
like the following:
returning version v1.19.0 instead of v1.19.0+43983cd
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Similarly to block volume replication, Ceph is capable of replicating the
content of a Ceph Filesystem from one cluster to another.
For this, during the 1.6 cycle, we introduced a new CRD called
CephFilesystemMirror which effectively deploys a cephfs-mirror daemon.
However, configuring peers to enable replication between two clusters
had to be done manually.
Also various bug fix made it in Ceph eventually and the minimum required
version for this to work is to run on Ceph Pacific 16.2.5 at least.
So the automatic configuration of Ceph Filesystem peers is now possible.
By editing the CephFilesystem CRD, you can now turn on mirroring:
```yaml
mirroring:
enabled: false
# list of Kubernetes Secrets containing the peer token
# for more details see: https://docs.ceph.com/en/latest/dev/cephfs-mirroring/#bootstrap-peers
peers:
secretNames:
- secondary-cluster-peer
```
Also, the mirroring status is displayed in the CR status:
```
status:
info:
fsMirrorBootstrapPeerSecretName: fs-peer-token-myfs
mirroringStatus:
daemonsStatus:
- daemon_id: 4186
filesystems:
- filesystem_id: 2
name: myfs
lastChecked: "2021-07-01T14:16:29Z"
phase: Ready
snapshotScheduleStatus:
lastChecked: "2021-07-01T14:16:29Z"
snapshotSchedules:
- fs: myfs
path: /
rel_path: /
retention: {}
schedule: 24h
```
Closes: https://github.com/rook/rook/issues/7063
Signed-off-by: Sébastien Han <seb@redhat.com>
earlier, for mds, when resource limit is not defined it has
default limit of `"mds_cache_memory_limit": "4294967296"`
even if resouce.request is defined.
Now, resource request will be given preference when both
request and limit is defined and if only limit is defined
it will be applied or otherwise.
Closes: https://github.com/rook/rook/issues/8143
Signed-off-by: subhamkrai <srai@redhat.com>
update blockPoolChannel monitoringRunning field to `true` before
starting the goroutine to monitor mirroring
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
The debug for adminOps client can be enabled by setting `Debug` flag.
In this PR, it is enabled for OBC and cephobjectstore not healthchecker.
Also add similar changes while calling `NewS3Agent` in the bucket
provisioner code path.
Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
Creates hybrid crush rule for choosing Primary OSD for
high performing SSD devices and remaining OSD for low performance HDD devices.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
When the Operator set the livenessProbe of the POD, it overwrote the contents of the shared
desiredProbe. Therefore, the wrongly configured livenessProbe was applied to other PODs,
resulting in a CrashLoop. In this PR, I created another instance to avoid overwriting the
desiredProbe.
Closes: https://github.com/rook/rook/issues/8196
Signed-off-by: Yuji Ito <llamerada.jp@gmail.com>
In pkg/operator/k8sutil/network.go:ApplyMultus(), be more explicit about
the intended behavior in both var names, code comments, and unit tests.
Additionally fix a logic bug that could cause errors in the future if
more "cluster network apps" are identified in the future.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
We know append additional information to the rbd-mirror bootstrap peer
token. It is useful for disaster recovery scenario where the other
cluster is reading the peer token and needs to know the pool_id as well
as the namespace.
Signed-off-by: Sébastien Han <seb@redhat.com>
In a production cluster we never should have two mons running on
the same node. However, if two mons have ended up on the same node,
whether from a bug or some other unintentional event, the operator
will evict one of them and failover to a new mon. This only applies
when allowMultiplePerNode is false.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
When evaluating the potential "warning" key from the json output we were
not hiding stdout (1) from the output, so the key was leaked. Also the
warning check for the key was not perfect, it only checked for the
command to succeed, but we should introspect it since the "warning" key
is always present unlike the "error" one.
We now only print the warning or the error is present.
Signed-off-by: Sébastien Han <seb@redhat.com>
In pkg/operator/k8sutil/network.go:ApplyMultus(), sort the string
annotations before applying them so that Rook will avoid the situation
where a Deployment or Pod will need updated repeatedly even when no
Multus network configuration has changed.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
Sébastien Han
Satoru Takeuchi
Jiffin Tony Thottan
Travis Nielsen
subhamkraiandSébastien Han
Santosh Pillai
Yug
Blaine Gardner
Yuji Ito
Rakshith R
Andy Bursavich