The rbd mirror daemon logs to two files, one for the
lodal mirror daemon and one log from the other cluster
where the mirroring is taking place. The log rotation
was only rotating one of the logs. Now we make the
log rotation more general to catch all of the rbd mirror
logs from different clusters.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
With octopus coming to end of life, we remove support from
Rook for deploying Ceph Octopus and assume a min version of
Pacific v16. Any checks for octopus or earlier are removed
from the reconciles since they are obsolete.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The CSI package needs to load clusterInfo, today this code is in the mon
package which makes the call of LoadClusterInfo impossible without
having a circular import.
Signed-off-by: Sébastien Han <seb@redhat.com>
adding observedGeneration field in the cephcluster cr
status for having better control on reconciling,
as observedGeneration field will be updated by the controller
Closes: https://github.com/rook/rook/issues/9673
Signed-off-by: parth-gr <paarora@redhat.com>
Prior to this, we were comparing a pointer (the memory address) with a
struct. This was obviously always failing and returned false. We must
dereference the pointer to access the data contained at that memory
location.
Closes: https://github.com/rook/rook/issues/9544
Signed-off-by: Sébastien Han <seb@redhat.com>
Adding Recommended Labels on the resources created by rook
and using Recommended Labels in the helm chart,
for better visuals and management of k8s object
Closes: https://github.com/rook/rook/issues/8400
Signed-off-by: parth-gr <paarora@redhat.com>
This commit adds context parameter to k8sutil job functions. By this, we
can handle cancellation during API call of job resource.
Signed-off-by: Yuichiro Ueno <y1r.ueno@gmail.com>
The cluster CR can be forcefully deleted and cleanup the
cluster resources if the yes-really-destroy-data policy
is set on the CR. In this case, the other controllers should
treat the cluster CR as not existing and allow the finalizers
to be removed on those resources if they are requested for
deletion.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
This commit adds context parameter to utilities in opcontroller to
remove context.TODO use in opcontroller. By this, we can handle
cancellation of reconcilers in a fine-grained way.
Signed-off-by: Yuichiro Ueno <y1r.ueno@gmail.com>
This commit is a large refactor on how the operator starts, stops and
how it starts various sub-components such as the ceph-csi driver. It
also refines the way we cancel orchestrations. We don't use breakpoints
anymore but send our self a SIGUP to reload our controller runtime
manager.
The reload will happen under different circonstances like:
* a new adminission controller secret is created/deleted/changed
* a CephCluster CR is edited
As mentioned earlier, the csi driver now has its own controller, just
like flex. It reacts to change in the operator config map for particular
ROOK_CSI_ fields.
A second new controller for the operator's general config has been
created, it manages:
* the logging level
* the ceph CLI command timeout
* the discovery daemon
The operator reacts much more rapidly to cancellation events by stopping
the manager's context and reloading it.
Signed-off-by: Sébastien Han <seb@redhat.com>
Instead of having the RBDMirroringPeerSpec in the RBDMirror CRD we want
to move it on the CephBlockPool CRD so that each pool can have its own peer.
This enables pool to re-use an existing peer secret if it points to the
same cluster peer.So we don't need to duplicate secret peer anymore.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
Both `ExecuteCommandWithOutputFileTimeout()` and
`ExecuteCommandWithOutputFile()` generate unnecessary system calls by
creating/reading/removing files where the stream output of the command
can simply be used. So sticking with `ExecuteCommandWithOutput()` and
`ExecuteCommandWithCombinedOutput()` for reading outputs is sufficient.
Closes: https://github.com/rook/rook/issues/8343
Signed-off-by: Sébastien Han <seb@redhat.com>
Recently, the builds of `ceph/ceph` image moved to quay.io, see
https://github.com/ceph/ceph-build/pull/1883 for more details.
Current images will remain but new builds will happen on quay.io only.
This means that tags such as `v14.2`, `v15.2`,`v16.2` will need to
switch to quay.io to get updates.
Signed-off-by: Sébastien Han <seb@redhat.com>
Since https://github.com/rook/rook/pull/7180 rook does not deployment
multiple deployments of the rbd-mirror daemon but uses replicas. Users
have upgraded from subsequent 1.6.x release and have that supported so
we can remove this code that was scale up/down the number of
deployments.
Closes: https://github.com/rook/rook/issues/7183
Signed-off-by: Sébastien Han <seb@redhat.com>
Similarly to block volume replication, Ceph is capable of replicating the
content of a Ceph Filesystem from one cluster to another.
For this, during the 1.6 cycle, we introduced a new CRD called
CephFilesystemMirror which effectively deploys a cephfs-mirror daemon.
However, configuring peers to enable replication between two clusters
had to be done manually.
Also various bug fix made it in Ceph eventually and the minimum required
version for this to work is to run on Ceph Pacific 16.2.5 at least.
So the automatic configuration of Ceph Filesystem peers is now possible.
By editing the CephFilesystem CRD, you can now turn on mirroring:
```yaml
mirroring:
enabled: false
# list of Kubernetes Secrets containing the peer token
# for more details see: https://docs.ceph.com/en/latest/dev/cephfs-mirroring/#bootstrap-peers
peers:
secretNames:
- secondary-cluster-peer
```
Also, the mirroring status is displayed in the CR status:
```
status:
info:
fsMirrorBootstrapPeerSecretName: fs-peer-token-myfs
mirroringStatus:
daemonsStatus:
- daemon_id: 4186
filesystems:
- filesystem_id: 2
name: myfs
lastChecked: "2021-07-01T14:16:29Z"
phase: Ready
snapshotScheduleStatus:
lastChecked: "2021-07-01T14:16:29Z"
snapshotSchedules:
- fs: myfs
path: /
rel_path: /
retention: {}
schedule: 24h
```
Closes: https://github.com/rook/rook/issues/7063
Signed-off-by: Sébastien Han <seb@redhat.com>
We know append additional information to the rbd-mirror bootstrap peer
token. It is useful for disaster recovery scenario where the other
cluster is reading the peer token and needs to know the pool_id as well
as the namespace.
Signed-off-by: Sébastien Han <seb@redhat.com>
We have been having many issues with external mode with Ceph version
mismatching. The operator would have a Ceph version different than the
external cluster. The `radosgw-admin` was used to interact with S3
users, even a small version delta would cause the command to coredump.
After checking with the rgw core team it appears Rook was misusing the
CLI and the admin ops API should be used instead.
So this patch is the first introduction of go-ceph in Rook to consume
the rgw admin ops API instead of the `radosgw-admin` CLI, **only** for
user management in this initial commit.
Later we can do more such as bucket operation, zone management etc.
Closes: https://github.com/rook/rook/issues/7924
Signed-off-by: Sébastien Han <seb@redhat.com>
The rook.io/v1 package was only an internal implementation detail and
does not have any CRDs that rely on it. The CRD deserialization should
handle the change in internal types without any issue. This separation
gives more flexibility for the storage providers to implement exactly
what is needed for their storage provider instead of forcing to use the
same types and risk affecting another storage provider.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
After the peer is imported via a token, ceph internally stores the
information of the remote cluster. So the rbd-mirror daemon does not
need to have files (key and config) for the remote site.
This means lighter code for us, as well no more restarts for the
rbd-mirror daemon once a peer is added to the configuration.
Signed-off-by: Sébastien Han <seb@redhat.com>
When a private docker registry is used and an image pull secret is specified in the chart, the pods with default Service Account fail to pull the image due to authentication issues.
Added rook-ceph-default service account and modify the pods specifications by adding the serviceAccountName.
Closes: https://github.com/rook/rook/issues/6673
Co-authored-by: Tareq Sharafy <tareq.sha@gmail.com>
Signed-off-by: parth-gr <partharora1010@gmail.com>
In the case of PVC,
We are giving lower priority to all placement.
We want deviceSet placement to applied and
override in case of overlapping settings and
we are merging nodeAffinity if applied in both
all placement and deviceSet.
In case of non-PVC,
we apply spec.placement
Signed-off-by: subhamkrai <srai@redhat.com>
Co-authored-by: Travis Nielsen <tnielsen@redhat.com>
It's better to validate ownerReferences when setting them. In addition, we should use
controllerrutil.Set{Controller,Owner}Reference, that have such validation, as possible.
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
Finally! We can now stop editing manually our crds definition. Simply
run `make crds-gen`. These two files:
* `cluster/examples/kubernetes/ceph/crds.yaml`
* `cluster/charts/rook-ceph/templates/resources.yaml`
will be autogenerated for us.
We rely on the controller-gen tool, it reads our API definitions from
`pkg/apis/` and produces the CRD files accordingly.
It uses "markers" to add extra yaml fields to the CRD, for instance we
have a lot fields with:
```
nullable: true
x-kubernetes-preserve-unknown-fields: true
```
The corresponding API type markers are:
```
// +kubebuilder:pruning:PreserveUnknownFields
// +nullable
```
For more API convention see
https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#optional-vs-required
and for the markers see: https://book.kubebuilder.io/reference/markers/crd-validation.html
Signed-off-by: Sébastien Han <seb@redhat.com>
we don't want to merge the node affinity if specified in both
storageClassDeviceSet and placement. Now, we are passing new
boolean argument in `ApplyToPodSpec` which will be false in
case of osd and prepare pods because we don't want to overlap
placement of osd on PVC's and non-PVC's.
Signed-off-by: subhamkrai <srai@redhat.com>
For the rbd-mirror daemon, We now rely on the "replica" setting in the Deployment spec when the
desired instance count is higher than 1 instead of creating another
deployment.
So now, we don't need to the number of deployments anymore and let
kubernetes handles the number of pods.
Signed-off-by: Sébastien Han <seb@redhat.com>
If the ceph cli outputs an error with "error calling conf_read_file"
this means that the operator has not written its ceph configuration
file. Thus ceph cli commands will fail, so we can just ignore that since
the operator will soon write this file in its initialization sequence.
Signed-off-by: Sébastien Han <seb@redhat.com>
With Ceph Pacific, Rook can now deploy the cephfs-mirror daemon.
This initial commit covers the deployment of a single daemon only.
Multiple mirror daemons is currently untested.
Only a single mirror daemon is recommended.
The configuration of peers will come in a later PR since the mgr module
is still pending upstream: https://github.com/ceph/ceph/pull/39050
The same goes for integration tests, they will get added later once we
start testing on Pacific.
Closes: https://github.com/rook/rook/issues/7002
Signed-off-by: Sébastien Han <seb@redhat.com>
If the CephCluster CR spec is updated to activate the logCollector, we
must reflect that change onto child CRDs, like the mds and rgw since
their configurationn would be impacted too.
Now we watch for the CephCluster object changes from the object/file
controllers and react upon the appropriate event.
Closes: https://github.com/rook/rook/issues/7022
Signed-off-by: Sébastien Han <seb@redhat.com>
By setting the working directory of a Ceph daemon to the log direcor
(which is bindmounted to the host), we can ensure that the coredumps
will be available.
On CentOS 7 **only**, the kernel is configured with:
```
cat /proc/sys/kernel/core_pattern
core
```
which means that the coredumps will end up in the process working
directory and will be named "core".
On CentOS 8, everything is different and this won't work but still
CentOS will be able to consume this and changing the working directory
is harmless.
Signed-off-by: Sébastien Han <seb@redhat.com>
We can now collect logs directly into a side-car container.
A new CRD spec has been added:
spec:
logCollector:
enabled: true
periodicity: 24h
Every 24h we will rotate log files for each Ceph daemon.
Signed-off-by: Sébastien Han <seb@redhat.com>
Found by running the following command:
codespell -S .git,*.png,*.jpg -L \
aks,keyserver,atleast,dne,ser,ist,files\',ba,dum,iam,te -f -H
Signed-off-by: Mateusz Gozdek <mgozdekof@gmail.com>
golangci-lint linter gosec showing more errors
than gosec gh. This commit resolve
new errors. And, removing
nosec comments from autogenerated files.
Signed-off-by: subhamkrai <srai@redhat.com>
this commit handle golangci-lint linter staticcheck error.
`staticcheck` - Staticcheck is a go vet on steroids,
applying a ton of static analysis checks.
To see only `staticcheck` linter output
`golangci-lint run --disable-all -E staticcheck`
Signed-off-by: subhamkrai <srai@redhat.com>
Travis Nielsen
Joseph Lee
Josh Soref
Sébastien Han
parth-gr
subhamkrai
Yuichiro Ueno
Blaine Gardner
Santosh Pillai
parth-grandTareq Sharafy
Satoru Takeuchi
Arun Kumar Mohan
Mateusz Gozdek
subhamkrai