Rook now exports both ceph and csi SCCs to run on openshift so any
program pulling rook can inject the right SCCs.
In the meantime, default SCC has been reinforced to be less permissive.
Documentation has been reworked and a new path for openshift
installations has been created. Rook still provides the YAML files for
manual installations.
Closes: #8713
Signed-off-by: Sébastien Han <seb@redhat.com>
When private/public network have been defined in the Ceph rook cluster it is not
possible to configure properly the ip address of the liveness probe
for the manager.
Changes in the manager in Pacific introduced this regression.
This change replaces the http probe by a command probe, avoiding thus to
determine what is going to be the ip address of the manager before launching
the pod.
fixes: https://github.com/rook/rook/issues/8510
Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
If the CephObjectStore health checker fails to be created, return a
reconcile failure so that the reconcile will be run again and Rook will
retry creating the health checker. This also means that Rook will not
list the CephObjectStore as ready if the health checker can't be
started.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
This commit is a large refactor on how the operator starts, stops and
how it starts various sub-components such as the ceph-csi driver. It
also refines the way we cancel orchestrations. We don't use breakpoints
anymore but send our self a SIGUP to reload our controller runtime
manager.
The reload will happen under different circonstances like:
* a new adminission controller secret is created/deleted/changed
* a CephCluster CR is edited
As mentioned earlier, the csi driver now has its own controller, just
like flex. It reacts to change in the operator config map for particular
ROOK_CSI_ fields.
A second new controller for the operator's general config has been
created, it manages:
* the logging level
* the ceph CLI command timeout
* the discovery daemon
The operator reacts much more rapidly to cancellation events by stopping
the manager's context and reloading it.
Signed-off-by: Sébastien Han <seb@redhat.com>
We don't need to use github.com/ghodss/yaml since
"k8s.io/apimachinery/pkg/util/yaml" provides the same functionality and
we already import it.
Signed-off-by: Sébastien Han <seb@redhat.com>
Rook checks for down OSDs by checking the `ReadyReplicas` count
in the OSD deployement. When an OSD pod goes into CBLO due to
disk failure, there is a delay before this `ReadyReplicas` count
becomes 0. The deplay is very small but may result in rook missing
OSD down event. As a result no blocking PDBs will be created and
only default PDB with `AllowedDisruptions` count as 0 is available.
This PR tries to solve this. The OSD pdb reconciler will be
reconciled again if `AllowedDisruptions` count in the main
PDB is 0.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
correct typo in logging, it was showing `ok-to-stop`
instead of `ok-to-continue` when 'continueUpgradeAfterChecksEvenIfNotHealthy' is true
Co-Authored-by: Zeaone <zeaone@ZeaonedeMacBook-Pro.local>
Signed-off-by: subhamkrai <srai@redhat.com>
Sometimes the error does not tell much, so as `exit status 1` and
printing the output along returning the error is useful.
For instance, I saw a job failing with no osd and the prepare job had
those lines:
```
exec: Running command: lsblk /dev/sdb1 --bytes --nodeps --pairs ....
inventory: skipping device "sdb1". exit status 1
```
We need to understand more about the lsblk issue.
Signed-off-by: Sébastien Han <seb@redhat.com>
The ceph-block-pool controller was missing the network spec from the
cephcluster that contains all the details about networking.
So when the pool was deleting on multus the controller was not proxying
the rbd command to the mgr pod but executed the command in the operator
pod which does not have the network annotation and then connect to the
ceph cluster.
Signed-off-by: Sébastien Han <seb@redhat.com>
We just need to wait longer when the status is not ready. We needed
another sleep otherwise the status was never nil and the loop went too
fast. See:
```
2021-09-02 16:29:11.372249 I | integrationTest:
2021-09-02 16:29:11.374427 I | integrationTest:
2021-09-02 16:29:11.377764 I | integrationTest:
2021-09-02 16:29:11.379950 I | integrationTest:
2021-09-02 16:29:11.382084 I | integrationTest:
2021-09-02 16:29:11.385383 I | integrationTest:
2021-09-02 16:29:11.388499 I | integrationTest:
2021-09-02 16:29:11.391301 I | integrationTest:
2021-09-02 16:29:11.393545 I | integrationTest:
2021-09-02 16:29:11.396249 I | integrationTest:
```
Signed-off-by: Sébastien Han <seb@redhat.com>
Signed-off-by: Sébastien Han <seb@redhat.com>
Typical clusters only require 3 or maybe 5 mons. If someone really wants
to go crazy they can add more mons, but over 9 mons really should not be
used.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
While an even number of mons can cause lower availability of
mon quorum, it also can provide higher durability for the cluster.
Mon quorum can be restored from a single mon according to the
disaster recovery guide, so there may be scenarios where
an even number of mons may be preferable.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
When the op is initializing, the ceph config is not ready and thus we
should reconcile. This avoids a double reconcile.
Signed-off-by: Sébastien Han <seb@redhat.com>
When setting a default values with kubebuilder flags it seems that the
kube operator intercepts the API call, mutate the CR and inject it.
This results in two events, one for creation and one for edition and we
end up with 2 reconciles. Let's avoid this.
Signed-off-by: Sébastien Han <seb@redhat.com>
The error message in UpdateNodeStatus regards the second argument
as node name. However, it is a PVC name in OSD on PVC.
Signed-off-by: Hiroya Onoe <onoehiroya@gmail.com>
During disaster recovery/migration of a cluster, as part of the failover, the
kubernetes artifacts like deployment, PVC, PV, etc will be restored to a new
cluster by the admin. Even if the kubernetes objects are restored the
corresponding RBD/CephFS subvolume cannot be retrieved during CSI operations as
the clusterID and poolID are not the same in both clusters
This PR creates a mapping between Cluster ID and RBD Pool ID between
local cluster and peer cluster.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
Passing struct by value essentially gives you a copy, so when modified
within a function, the scope is then reduced to that function. Using
pointers solves that you mutate the struct as many times as you want from
anywhere.
As a result, the auto-detection of the Vault KV backend was not working
correctly.
Also, added a ton of unit tests for Vault.
Signed-off-by: Sébastien Han <seb@redhat.com>
If we call OwnerInfo.SetOwnerReference for an object multiple times,
it results in OwnerReference duplication.
Signed-off-by: Yuzuki Mimura <yuzuki725.m@gmail.com>
Co-authored-by: Hiroya Onoe <onoehiroya@gmail.com>
When mirroring is enabled on the filesystem, the status was not
being set on the filesystem. Now the reconcile will ensure the
status is updated on the CR whether or not mirroring is enabled.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
earlier, `ApplyToPodSpec()` was only taking one toleration and ignoring
tolerations from `placement.ALL()`.
this commit merge toleration for Mgr,Mon,Osd pod
example, for osd it will merge
spec.placement.all and
storageDeviceClassSets.Placement(in case of pvc) or
spec.placement.osd(in case of non-pvc's).
Signed-off-by: subhamkrai <srai@redhat.com>
This handles the scenario where the OSDs have been created but not yet
started due to a wrong CR configuration.
For instance, when OSDs are encrypted and Vault is used to store
encryption keys, if the KV version is incorrect during the cluster
initialization the OSDs will fail to start and stay in CLBO until the
CR is updated again with the correct KV version so that it can start.
For this scenario, if the CRUSH map has no host registered yet it's fair
to assume the initialization broke and we need to fix it. So when don't
need to call ok-to-stop since it will always fail and eventually force
pass but let's not wait for nothing.
Signed-off-by: Sébastien Han <seb@redhat.com>
The mon config had two different implementations that have evolved
over the lifetime of the project. This is a simple refactor to remove
the SetConfig() option and stick with the MonStore as a single
implementation for updating the mon store.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Our admission webhooks will now run as part of the Operator container
and not an additional deployment. This has the advantage of consuming
fewer resources in the cluster and not having to manage affinities and
tolerations. This only drawback is that the Secret containing the
certificates is not mounted anymore and the content needs to be written
inside the Operator. This is not practical since we also need to watch
for the Secret content to change. Meaning that the certificates have
been renewed and the webhook server needs to use them.
A new approach is on its way to hopefully simplify this last issue and
implement a watcher for the Secret.
In the meantime, users need to use the cert-manager or renew
certificates manually. Additionally, they must update the
ValidatingWebhookConfiguration object with the new CA bundle.
Signed-off-by: Sébastien Han <seb@redhat.com>
Thanks to Golang 1.16, we can now embed files in the Go binary. This
means we don't need to add the CSI templates files to the container
image. They are added in the Go binary at build time.
The existing location of the template must be in the package calling it.
So they moved to pkg/operator/ceph/csi/template. All the files have been
symlinked back to cluster/examples/kubernetes/ceph/csi/template.
Closes: https://github.com/rook/rook/issues/7609
Signed-off-by: Sébastien Han <seb@redhat.com>
Sébastien Han
Juan Miguel Olmo Martínez
Blaine Gardner
Santosh Pillai
Jonas Zeiger
Blaine Gardner
subhamkraiandZeaone
Jiffin Tony Thottan
Travis Nielsen
JrCs
parth-gr
Hiroya Onoe
YZ775andHiroya Onoe
Madhu Rajanna