Rook currently knows rbd pools only if they defined through CephBlockPool, so it overrides the config
mgr/prometheus/rbd_stats_pools which may have already been set for pools external to Rook. Check config
before running set command and if there some existing pools append them to enableStatsForCephBlockPools.
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
if Multus is enabled the clusterinfo should be updated with
network as multus as to run the ceph cmds in remote
executor
Signed-off-by: parth-gr <paarora@redhat.com>
`rbd pool init` cmd initializes pool for rbd images.
This commit makes modification to init only
rbd application pools, since it is not required
by other pools like ".nfs",".mgr" & "mgr_devicehealth".
Signed-off-by: Rakshith R <rar@redhat.com>
During deletion of a CephBlockPool CR, the underlying ceph
pool was not being deleted. During the deletion sequence
this is due to the spec being refreshed upon a call to
check for dependents in ReportDeletionNotBlockedDueToDependents().
The pool name was then coming up blank during the deletion
request and rook of course then wasn't finding the pool
to delete.
Now Rook properly sets the pool name in the named spec
every time it is converted internally from a CephBlockPool
type to a NamedPoolSpec.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The CSI package needs to load clusterInfo, today this code is in the mon
package which makes the call of LoadClusterInfo impossible without
having a circular import.
Signed-off-by: Sébastien Han <seb@redhat.com>
This introduces a new CRD to add the ability
to create rados namespace for a given
ceph block pool. Typically the name of the pool
is the name of the blockpool created by rook.
Closes: #7035
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
In Quincy, the device_health_metrics pool is renamed to .mgr.
In the example cluster-test.yaml, we therefore update the
example to create the pool with the new name.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
adding observedGeneration field in the cephcluster cr
status for having better control on reconciling,
as observedGeneration field will be updated by the controller
Closes: https://github.com/rook/rook/issues/9673
Signed-off-by: parth-gr <paarora@redhat.com>
The reconcile was skipping updating most pool properties for
object stores. The implementation of pools between the file,
object, and pool controllers had some duplicate code, so
this change also factors out the common code for better
reuse in a single place. Anytime a pool is created or updated,
it will now consistently update all the pool properties
that are expected to be modifiable.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The built-in pools device_health_metrics and .nfs created by ceph
need to be configured for replicas, failure domain, etc.
To support this, we allow the pool to be created as a CR.
Since K8s does not support underscores in the resource names
the operator must translate this special pool name into
the name expected by ceph.
This also sets the basis for allowing filesystem data
pools to specify the desired pool name instead of requiring
a generated name.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
When CephBlockPool, CephFilesystem, or CephObjectStore resources are
deleted after removing their finalizer, the code path to stop monitoring
was not stopping monitoring since a non-present resource does not have a
name and namespace attached. When the object is deleted, ensure the
internal representation used to stop monitoring has a name and namespace
to fix the issue.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
Clean up the code used to stop health checkers for all controllers
(pool, file, object). Health checkers should now be stopped when
removing the finalizer for a forced deletion when the CephCluster does
not exist. This prevents leaking a running health checker for a resource
that is going to be imminently removed.
Also tidy the health checker stopping code so that it is similar for all
3 controllers. Of note, the object controller now uses namespace and
name for the object health checker, which would create a problem for
users who create a CephObjectStore with the same name in different
namespaces.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
This commit adds context parameter to k8sutil deployment functions. By
this, we can handle cancellation during API call of deployment resource.
Signed-off-by: Yuichiro Ueno <y1r.ueno@gmail.com>
The cluster CR can be forcefully deleted and cleanup the
cluster resources if the yes-really-destroy-data policy
is set on the CR. In this case, the other controllers should
treat the cluster CR as not existing and allow the finalizers
to be removed on those resources if they are requested for
deletion.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
This commit adds context parameter to utilities in opcontroller to
remove context.TODO use in opcontroller. By this, we can handle
cancellation of reconcilers in a fine-grained way.
Signed-off-by: Yuichiro Ueno <y1r.ueno@gmail.com>
This is done in order to prevent deadlock when parallel
PVC create requests are issued on a new uninitialized
rbd block pool due to https://tracker.ceph.com/issues/52537.
Fixes: #8696
Signed-off-by: Rakshith R <rar@redhat.com>
This commit is a large refactor on how the operator starts, stops and
how it starts various sub-components such as the ceph-csi driver. It
also refines the way we cancel orchestrations. We don't use breakpoints
anymore but send our self a SIGUP to reload our controller runtime
manager.
The reload will happen under different circonstances like:
* a new adminission controller secret is created/deleted/changed
* a CephCluster CR is edited
As mentioned earlier, the csi driver now has its own controller, just
like flex. It reacts to change in the operator config map for particular
ROOK_CSI_ fields.
A second new controller for the operator's general config has been
created, it manages:
* the logging level
* the ceph CLI command timeout
* the discovery daemon
The operator reacts much more rapidly to cancellation events by stopping
the manager's context and reloading it.
Signed-off-by: Sébastien Han <seb@redhat.com>
The ceph-block-pool controller was missing the network spec from the
cephcluster that contains all the details about networking.
So when the pool was deleting on multus the controller was not proxying
the rbd command to the mgr pod but executed the command in the operator
pod which does not have the network annotation and then connect to the
ceph cluster.
Signed-off-by: Sébastien Han <seb@redhat.com>
During disaster recovery/migration of a cluster, as part of the failover, the
kubernetes artifacts like deployment, PVC, PV, etc will be restored to a new
cluster by the admin. Even if the kubernetes objects are restored the
corresponding RBD/CephFS subvolume cannot be retrieved during CSI operations as
the clusterID and poolID are not the same in both clusters
This PR creates a mapping between Cluster ID and RBD Pool ID between
local cluster and peer cluster.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
The mon config had two different implementations that have evolved
over the lifetime of the project. This is a simple refactor to remove
the SetConfig() option and stick with the MonStore as a single
implementation for updating the mon store.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
They are scenarios where the mirroring information want to be shared
between clusters prior to creating pool. Because the bootstrap peer
import command needs a pool name to operate this is not suitable. So
additionally now each time the cluster is reconciled and on any new
clusters a new secret will be created that contains a boostrap peer
token. It can be exchanged with another cluster.
Signed-off-by: Sébastien Han <seb@redhat.com>
Instead of having the RBDMirroringPeerSpec in the RBDMirror CRD we want
to move it on the CephBlockPool CRD so that each pool can have its own peer.
This enables pool to re-use an existing peer secret if it points to the
same cluster peer.So we don't need to duplicate secret peer anymore.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
The PR disables mirroring on a pool when PoolSpec.Mirroring.Enabled
is set to false
- disable mirroring on the pool for `Mirroring.Mode == pool`
- Add warning for the user to disable mirroring manually for `Mirroring.Mode == image`
- Stop the mirroring health checker goroutine
- Reset the mirroring health check status in the CR.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
Similarly to block volume replication, Ceph is capable of replicating the
content of a Ceph Filesystem from one cluster to another.
For this, during the 1.6 cycle, we introduced a new CRD called
CephFilesystemMirror which effectively deploys a cephfs-mirror daemon.
However, configuring peers to enable replication between two clusters
had to be done manually.
Also various bug fix made it in Ceph eventually and the minimum required
version for this to work is to run on Ceph Pacific 16.2.5 at least.
So the automatic configuration of Ceph Filesystem peers is now possible.
By editing the CephFilesystem CRD, you can now turn on mirroring:
```yaml
mirroring:
enabled: false
# list of Kubernetes Secrets containing the peer token
# for more details see: https://docs.ceph.com/en/latest/dev/cephfs-mirroring/#bootstrap-peers
peers:
secretNames:
- secondary-cluster-peer
```
Also, the mirroring status is displayed in the CR status:
```
status:
info:
fsMirrorBootstrapPeerSecretName: fs-peer-token-myfs
mirroringStatus:
daemonsStatus:
- daemon_id: 4186
filesystems:
- filesystem_id: 2
name: myfs
lastChecked: "2021-07-01T14:16:29Z"
phase: Ready
snapshotScheduleStatus:
lastChecked: "2021-07-01T14:16:29Z"
snapshotSchedules:
- fs: myfs
path: /
rel_path: /
retention: {}
schedule: 24h
```
Closes: https://github.com/rook/rook/issues/7063
Signed-off-by: Sébastien Han <seb@redhat.com>
update blockPoolChannel monitoringRunning field to `true` before
starting the goroutine to monitor mirroring
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
In the latest Ceph releases starting with v16.2.1, all clients are recommended
to be updated so they will have a security fix to connect with a secure
global ID. A health warning will be raised if any insecure clients are connected
and another health warning is raised if insecure clients are still allowed.
Rook will now disable allowing the insecure clients if the health warning
is not being raised to indicate that there are insecure clients still connected.
This means that upgraded clusters will not have this disabled until all the
daemons are updated.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
We have been having many issues with external mode with Ceph version
mismatching. The operator would have a Ceph version different than the
external cluster. The `radosgw-admin` was used to interact with S3
users, even a small version delta would cause the command to coredump.
After checking with the rgw core team it appears Rook was misusing the
CLI and the admin ops API should be used instead.
So this patch is the first introduction of go-ceph in Rook to consume
the rgw admin ops API instead of the `radosgw-admin` CLI, **only** for
user management in this initial commit.
Later we can do more such as bucket operation, zone management etc.
Closes: https://github.com/rook/rook/issues/7924
Signed-off-by: Sébastien Han <seb@redhat.com>
Saw this today in the logs:
```
2021-06-02 13:02:54.787838 I | ceph-block-pool-controller: deleting pool "testpool"
2021-06-02 13:02:56.178094 I | cephclient: no images/snapshosts present in pool "testpool"
2021-06-02 13:02:56.178125 I | cephclient: purging pool "testpool" (id=17)
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1cef49f]
goroutine 1607 [running]:
github.com/rook/rook/pkg/operator/ceph/pool.(*mirrorChecker).checkMirroringHealth(0xc001a98ba0, 0x8, 0xc000d34c60)
/home/runner/work/rook/rook/pkg/operator/ceph/pool/health.go:115 +0xdf
github.com/rook/rook/pkg/operator/ceph/pool.(*mirrorChecker).checkMirroring(0xc001a98ba0, 0xc0016fd500)
/home/runner/work/rook/rook/pkg/operator/ceph/pool/health.go:82 +0x147
created by github.com/rook/rook/pkg/operator/ceph/pool.(*ReconcileCephBlockPool).reconcile
/home/runner/work/rook/rook/pkg/operator/ceph/pool/controller.go:298 +0xee5
```
Essentially, it's intermittent but when deleting the pool the
healthcheck kicked in and fetched the mirroring status, which returned
empty. THe subsequent code tried to access content of a nil pointer,
hence the error.
So now we stop monitoring first, then we proceed with the deletion.
Signed-off-by: Sébastien Han <seb@redhat.com>
If the monitors addresses change, we must update the peer secret with
the new addresses. For this, each time the configmap
"rook-ceph-mon-endpoints" is updated, we trigger a reconcile on the
CephBlockPool CRs, which will update the token secret.
Signed-off-by: Sébastien Han <seb@redhat.com>
If the ceph cli outputs an error with "error calling conf_read_file"
this means that the operator has not written its ceph configuration
file. Thus ceph cli commands will fail, so we can just ignore that since
the operator will soon write this file in its initialization sequence.
Signed-off-by: Sébastien Han <seb@redhat.com>
We must add the finalizer right after the object creation otherwise the
seerver will later return an error on update that the object has been
modified. Indeed, it has been by the task that updates the status when
the object is first created.
Signed-off-by: Sébastien Han <seb@redhat.com>
Now, we can schedule snapshots on pools from the CephBlockPool CR when
the pool is mirrored.
It can be enabled like this:
```
mirroring:
enabled: true
mode: pool
snapshotSchedules:
- interval: 24h # daily snapshots
startTime: 14:00:00-05:00
```
Multiple schedules are supported since snapshotSchedules is a list.
Signed-off-by: Sébastien Han <seb@redhat.com>
travisn
Avan Thakkar
parth-gr
zhucan
Rakshith R
shalev.p
Josh Soref
Sébastien Han
Madhu Rajanna
Divyansh Kamboj
subhamkrai
Blaine Gardner
Yuichiro Ueno
Santosh Pillai