Some log messages indicate when the upgrade checks will be
performed. If the Ceph image hasn't changed, the upgrade checks
are skipped. The messages were confusing if running a Rook
upgrade.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
If the cluster has previously been initialized and the operator
is restarted, the operator needs to perform the first orchestration
to confirm basic cluster health, then start watching for CR events.
If there was a single mon pod in a failed or pending pod state, the
orchestration was blocked even if just one mon was in this state.
The scenario is when a node is lost and a mon and the operator happen
to be running on that node. The operator will restart, but the mon
will not restart because it is tied to the node that died (unless running
on a PVC). The mon would remain in a pending state while attempting
to schedule on the node that is down indefinitely and the operator would
be blocked indefinitely. Now we will just log the error and attempt
to continue.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Some testing on OpenShift has shown that enabling Hostnetworking causes
issues with ceph-volume creating OSD.
This is a pure Red Hat OpenShift downstream issue.
This is meant to be temporary while we are waiting for the Selinux team
to get back to us.
Signed-off-by: Sébastien Han <seb@redhat.com>
In Octopus, this commit
https://github.com/ceph/ceph/pull/30859/commits/c09b82a80a392ccd0da7677c7b424ce5cd3fa5d6
moved the quorum_status command to a socket call. Thus, the `ceph
quorum_status` command does not exist anymore, to use it we need to do
`ceph daemon mon.$ID quorum_status`. However, this command only works
locally so it won't work if we run it from the operator.
The `ceph mon_status` command is a general command that run from
anywhere and returns the mon map, let's use this one on the operator
code.
This is not a breaking change.
Signed-off-by: Sébastien Han <seb@redhat.com>
If mon deployments need to be created, the operator should
go ahead and create all of them instead of waiting for quorum
in between checking each mon. In particular, if a cluster is
being restored in a disaster recovery situation, none of the mons
deployments will exist. All of them should be created before
checking for quorum.
Signed-off-by: travisn <tnielsen@redhat.com>
Following the Ceph OSD rework, my test environment regularly failed to
start OSDs, as they would start before the postStart lifecyle hook which
was chown'ing the log and data directories could finish. Pods failed to
start for an arbitrarily long time before starting successfully. This
was a manifestation of a race condition identified in Ceph monitors in
https://github.com/rook/rook/pull/3594#discussion_r312279176
Create a Ceph operator-level method to create an init container that
will chown the necessary directories for all Ceph daemons (any daemon
which runs as the ceph:ceph user-and-group). Also add this container to
expectations for Ceph pod templates in unit tests.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
We now differentiate the cases where:
* we only consume the external cluster
* we consume the external cluster as well as creating stateless
resources in Kubernetes (bootstrap mds,rgw, nfs)
This is mostly controlled via the image property spec. If not defined,
not extra CRs won't be able to be created.
Now the external cluster feature supports Ceph cluster as of Luminous 12.2.
Signed-off-by: Sébastien Han <seb@redhat.com>
On rare occasion, the ceph checks might not be 100% correct and the user
might decide to force an upgrade anyway.
Correctly, if we are in an upgrade, we perform `ok-to-stop` check for
each daemon, if one of them fails, we retry and eventually fail.
They are corner-cases where users would like to force the upgrade anyway.
Enhance the new CR property: "skipUpgradeChecks: true".
Closes: https://github.com/rook/rook/issues/3872
Signed-off-by: Sébastien Han <seb@redhat.com>
The CSI secret was only being created for new clusters.
Upgraded clusters did not have the secret generated. To support
upgraded clusters to use CSI, we always ensure the csi secret
exists when the orchestration runs.
Signed-off-by: travisn <tnielsen@redhat.com>
Several commands were not using the cli finalizer function which builds
up the right argument to run a ceph command. This was working on the
operator since it used to have an /etc/ceph/ceph.conf file so we could
ran the command without specifying the ceph conf and key paths.
However, we recently removed that and it silently broke this "already"
broken code :).
Signed-off-by: Sébastien Han <seb@redhat.com>
After further consideration and working with the feature for a few
weeks, the `configOverride` functionality seems to be more complexity
than is worth it for the expected benefit. Users still have the most
preferred option of setting configs in Ceph's config database using
`ceph cofig set` or the dashboard, and the override ConfigMap still
exists for fallback scenarios. The config database can still be used
by Rook for setting necessary configs, but the user has no way of
telling Rook to do so.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
Reduce the duplication of detecting the version of the external cluster
by refactoring to a single method that retrieves and validates the
version
Signed-off-by: travisn <tnielsen@redhat.com>
Add various upgrade tests so that the local cluster does not fail if
an upgrade is available on the external cluster.
Also, add better upgrades handling on update and on add. We now rely as
much as possible on a unique function that works both local and external
cluster 'detectAndValidateCephVersion()'. So the code will act the same
way wether a CR update is triggered or if the operator is restarted.
The health check now reports any topology changes on the external
cluster such as odd number of mons and low number. It will also detect
if the external cluster has been upgraded and will log that information,
helping the admin in his/her decision to trigger an image upgrade.
Finally, we prevent the creation of new CRs if the external cluster
version has a higher major number than the local one.
Closes: https://github.com/rook/rook/issues/3731
Signed-off-by: Sébastien Han <seb@redhat.com>
Allow users to specify overrides in the CephCluster CRD. The override
ConfigMap still exists for emergency situations and is mounted into
daemon pods directly instead of being merged into a Rook-created config
file.
Rook Ceph no longer uses a config file for managing daemons with the
exception of the OSDs which still generate a config in an init
container.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
From now on, Rook will only perform check before upgrades when there is
an actual upgrade. So if the Ceph image changed and a new version is
desired Rook will go through all the daemons and update them one by one
and perform checks in between.
Closes: https://github.com/rook/rook/issues/3583
Signed-off-by: Sébastien Han <seb@redhat.com>
this patch adds two new related features. first, it removes the
requirement that all monitor deployments be pinned to nodes using node
selector. this is allowed when not using host networking and when PVC
storage is used. the second feature introduced in this patch is the
elimination of the Rook scheduler for placing monitors that continue to
use the node selector by using canary pods to that are scheduled by k8s,
after which Rook uses the scheduling results to pin the monitors to
nodes.
fixes: #3667
Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
ceph: Use the new NetworkSpec api
ceph: Change all tests to use the NetworkSpec
ceph: Remove TypeMeta field from NetworkSpec
ceph: Change IsHost network func signature
Signed-off-by: giovanism <giovanism@outlook.co.id>
saveMonConfig already calls WriteConnectionConfig so there is no need to
call WriteConnectionConfig again after saveMonConfig is done.
Signed-off-by: Sébastien Han <seb@redhat.com>
With this change, the admin can specify a single mon and the operator
will add any other mons to both ClusterInfo and the configmap of mon
endpoints.
It will periodically check for changes too and update both ClusterInfo
and the configmap accordingly.
Signed-off-by: Sébastien Han <seb@redhat.com>
By exporting cluster to Cluster we can now reach it from the controller
health check so that it can be populated with 'cluster' data, see the
next commit for better understanding.
Signed-off-by: Sébastien Han <seb@redhat.com>
When configuring an external cluster the orchestration of discover and all the ceph
daemons will be skipped. The operator will still watch for the creation of
crds for this namespace. When a filesystem, object, object user, or
ganesha crd are created, the operator will simply print an error
to the log.
Signed-off-by: travisn <tnielsen@redhat.com>
Signed-off-by: Sébastien Han <seb@redhat.com>
With the 1.1 release we should not be creating bluestore OSDs
with rook's legacy partitioning scheme. All new bluestore OSDs
should be created with ceph-volume. Therefore, the miminum
version allowed is a version of mimic supporting ceph-volume.
This commit also adds consistent version checking between
the Add and Update methods for the operator handling the
CR events.
Signed-off-by: travisn <tnielsen@redhat.com>
Instead of requring csi instances for every ceph cluster that rook
manages, maintain one config map for all ceph clusters. This allows
for one set of csi drivers per rook operator.
The config is intialized as empty when Ceph CSI components are first
created and then updated every time a cluster's mons change.
As before clusters are identified by the k8s namespace the ceph cluster
is running in.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
Create a mutex at the rook operator level that will be used when
updating the shared-between-clusters config map that Ceph CSI consumes.
Pass the mutex down to the function that will update the CM.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
Add a new function that, given existing config string, key, and current
mons, will return an updated version of the Ceph CSI config string
having updated the data with the new mons.
Add a unit test function for the update function.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
when the root of a filesystem mode pvc is mounted as the base data directory
for the monitor it will not be empty: `lost+found` from ext4 will be present.
this causes the mon mkfs to fail because it expects an empty directory.
the solution is to mount under a directory in the source volume.
Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
the life cycle post-start hook is used to chown the ceph daemon log
directory. however, hooks are not guaranteed to run before a container
entry point.
"Kubernetes sends the postStart event immediately after the Container is
created. There is no guarantee, however, that the postStart handler is
called before the Container’s entrypoint is called"
Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
The CSI driver requires the ceph key for administrating
the cluster. The key is stored in a secret and specified in the
storage class. The operator will now automatically generate
a secret named rook-ceph-csi that can be specified by
default in the storage class instead of requiring the
admin to create it.
Signed-off-by: travisn <tnielsen@redhat.com>
a helper flag indicating the exitence of an empty zone (empty = no
monitor pods) is computed when checking the health of the cluster in
order to determine if mons from overloaded zones have some new location
they can be scheduled. the computation of this flag did not check that
the empty zone contains valid nodes that pods can be scheduled on to.
this lead to a mon creation loop.
fixes: #3545
Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
Before each pod starts, the daemon changes the ownership of
/var/log/ceph which is bindmounted on a hostPath.
On OpenShift envionments, the container must be privileged in order to
successfully change the owernship.
Closes: https://github.com/rook/rook/issues/3485
Signed-off-by: Sébastien Han <seb@redhat.com>
OpenShift environment requires privileged containers when interacting
with the host filesystem, typically via dataDirHostPath.
The init container that changes the ownership of dataDirHostPath was
missing the privileged flag and users were getting permission denied
errors on the chown command.
Fixes: https://github.com/rook/rook/issues/3485
Signed-off-by: Sébastien Han <seb@redhat.com>
At the beginning of every container startup we now change the ownership
of every known ceph directories to ceph:ceph.
This will allow daemons to start properly as well as allowing daemon to
writes their logs on files.
This is needed because ceph daemon now run as 'ceph' instead of 'root'
Signed-off-by: Sébastien Han <seb@redhat.com>
We do not determine the daemon type and name by string parsing anymore
but passing that info directly in the update call, it's more reliable.
Also removing a bunch of unused functions.
Signed-off-by: Sébastien Han <seb@redhat.com>
Travis Nielsen
d-luu
Sébastien Han
RAJAT SINGH
Blaine Gardner
Noah Watkins
giovanism
John Mulligan