The host name of an OSD in the CRUSH map should be the real
host name for non-portable OSDs. It was incorrectly being set
to the PVC name. Now the non-portable OSDs based on PVCs will
corretly have the host name set to the node name.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The minio operator has not had community support nor
any updates since being added to Rook. Support is being
removed from Rook due to this lack of community interest.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
A new CRD option `continueUpgradeAfterChecksEvenIfNotHealthy` is added.
When upgrading, Rook goes OSD by OSD and then waits for PGs to be clean
before proceeding to the next OSD. Currently, Rook waits for 5 hours but
there might be circumstances where PGs need more time to settle.
Thus setting `continueUpgradeAfterChecksEvenIfNotHealthy` to true will
pursue the upgrade process, even if PGs are not 100% active+clean.
Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1786029
Signed-off-by: Sébastien Han <seb@redhat.com>
Add support for the k8s 1.17 official zone/region failure domain labels.
Our earlier guess that the official labels would be
<failure-domain.kubernetes.io> was wrong; the official is now
<topology.kubernetes.io>.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
The operator kit had more utility originally when the operator
was creating and managing the TPRs and CRDs directly. Since
the CRDs are now created from a manifest and no longer by the
operators, the utility of operator kit is limited to the
controller watcher. Since we are moving to the controller runtime
we simplify the code to make the transition smoother. Now
there is only a simple WatchCR method that will need to be
replaced as we maek that transition.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
This commit adds status field for ceph related CRs which will be useful for knowing the ceph component status without checking the logs.
Signed-off-by: Ashish Ranjan <ashishranjan738@gmail.com>
Reduce from 5 minutes to less than 50 seconds the time needed to restart a
mgr and osds(PVCs based),rgw,mds,rbd,nfs and toolbox pod that was running
in a k8s "NotReady" Node.
A new Rook Operator setting has been added to allow the user to set the
time used in pod's toleration <node.kubernetes.io/unreachable>:
ROOK_UNREACHABLE_NODE_TOLERATION_SECONDS = <5>
New setting added in release notes
Addressed @travisn,@leseb, and @blaineEXE suggestions
Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
An owner reference cannot be set with BlockOwnerDeletion: true
if the operator does not have privs to add a finalizer to the parent.
The discover resources don't need to block deletion based on the
parent so we remove that property from the owner reference.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The app name was not being set in k8sutil's cmdreporter code, resulting
in jobs being labeled `app=` with no name. Change cmdreporter's `New`
method to validate the contents of the config struct and not the inputs
to the `New` method to help prevent an issue like this from creeping up
in the future.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
The topology of the cluster should be based on the node labels rather
than a setting in the cluster CR. This allows a much richer and more
dynamic topology to be configured. The location will now be ignored
if specified in the cluster CR.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
We removed the topologyAware CRD option since it was redundant with the
use of an OSD being backed by a PVC.
So now, if an OSD is backed by a PVC we assume the topology aware
decision and will discover zone and region labels on that host.
Signed-off-by: Sébastien Han <seb@redhat.com>
The node labels were already supported for zones and regions to add to
the CRUSH map. Now all layers of the CRUSH map will be supported
with the new labels such as topology.rook.io/rack.
The labels will be detected at the osd startup time
similar to the zone and region labels already being detected.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Add support for setting Toleration and NodeAffinity
to CSI Provisioner deployment or statefulset and
plugin daemonset through ENV variables.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
function AddNodeAffinity was not adding any node
affinity instead it was forming the nodeaffinity
object. renamed it to GenerateNodeAffinity for more
meaningful
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
previously we are not updating the service
if it's already present. with this change,
we will update the service. this does not
trigger any error message if service
already present.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
The node selector for running OSDs on PVCs was using the node name
rather than the node's hostname. In clusters where the node name
is different from the hostname this would cause the osd daemons
to be in pending state indefinitely. Now the node selector will
use the hostname as expected.
Signed-off-by: travisn <tnielsen@redhat.com>
On rare occasion, the ceph checks might not be 100% correct and the user
might decide to force an upgrade anyway.
Correctly, if we are in an upgrade, we perform `ok-to-stop` check for
each daemon, if one of them fails, we retry and eventually fail.
They are corner-cases where users would like to force the upgrade anyway.
Enhance the new CR property: "skipUpgradeChecks: true".
Closes: https://github.com/rook/rook/issues/3872
Signed-off-by: Sébastien Han <seb@redhat.com>
From now on, Rook will only perform check before upgrades when there is
an actual upgrade. So if the Ceph image changed and a new version is
desired Rook will go through all the daemons and update them one by one
and perform checks in between.
Closes: https://github.com/rook/rook/issues/3583
Signed-off-by: Sébastien Han <seb@redhat.com>
Deployment behaves better when a node gets disconnected
from the rest of the cluster - new provisioner leader
is elected in ~15 seconds, while it may take up to
5 minutes for StatefulSet to start a new replica.
if kube version is 1.13.x deploy provisioner as statefulset.
if kube version is higher than 1.14+ deploy provisioner
as deployment.
Refer: kubernetes-csi/external-provisioner@52d1fbc
Refer: ceph/ceph-csi#497
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
- Added code to support StorageClassDeviceSet spec provided in the cluster-on-pvc.yaml
- The code reads the StorageClassDeviceSet spec and creates pvc based on the ‘count’ field for each device set.
- OSD prepare job is started for each PVC which activates the ceph-volume on each PVC
- Finally OSD is started on each of the PVC device.
Co-authored-by: rohan47 <rohgupta@redhat.com>
Co-authored-by: Ashish Ranjan <aranjan@redhat.com>
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
We do not determine the daemon type and name by string parsing anymore
but passing that info directly in the update call, it's more reliable.
Also removing a bunch of unused functions.
Signed-off-by: Sébastien Han <seb@redhat.com>
When a cluster is updated with a different image version, this triggers
a serialized restart of all the pods. Prior to this commit, no safety
check were performed and rook was hoping for the best outcome.
Now before doing restarting a daemon we check it can be restarted. Once
it's restarted we also check we can pursue with the rest of the
platform. For instance, with monitors we check that they are in quorum,
for OSD we check that PGs are clean and for MDS we make sure they are
all active.
Fixes: https://github.com/rook/rook/issues/2889
Signed-off-by: Sébastien Han <seb@redhat.com>
Add missing docstrings for container image
functions in pod.go, noting the behavior
when getting a container image from a pod
which only has a single container defined.
Signed-off-by: Kristoffer Grönlund <kgronlund@suse.com>
Follow-up to #3436:
The validation module has a defined constant for
label value max length. It makes sense to use it
here.
Signed-off-by: Kristoffer Grönlund <kgronlund@suse.com>
this patch introduces failure-domain aware scheduling for monitor pods.
previously the only policy was to avoid scheduling monitors on the same
nodes. the new policy tries to spread monitors across zones and nodes.
NOTE: this patch only enforces the well-known k8s "zone"
failure-domain label (i.e. it does not look for region labels). this
decision is made for two reasons. first, eliminating explicit
scheduling is an important goal and will be done as part of upcoming
work. this will enable all supported failure-domain labels to be
handled. second, the "zone" domain is likely more appropriate for a
larger portion of users (as opposed to the geographical distinction
made with region), and supporting multiple failure domain types would
add significant complexity for a temporary resolution to failure
domain scheduling.
at a high-level this patch introduces a data structure `NodeUsage` that
is a pairing of a v1.Node and metadata relevant to scheduling: number of
monitor pods on the node, and its status w.r.t. to being scheduable for
new monitor pods.
each scheduling event computes a unified data structure that organizes
for each node a `NodeUsage` structure into per-failure-zone groups.
subsequent algorithms rely soley on this unified structure to make
scheduling decisions.
there are two entry points to the changes made:
1) during orchestration mon:Cluster:assignMons is invoked to make a
place new monitor pods onto nodes. the core scheduling logic is now
self-contained in mon:scheduleMonitor which implements node and zone
aware scheduling policies.
2) periodic health checks resolve any conflicts that may occur due to
changes to the cluster (e.g. new nodes, configuration changes, etc...).
the two existing checks: mons on invalid nodes, and overloaded nodes are
both handled.
fixes: #2603
Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
Add a wrapper for the cmd-reporter function of the 'rook' binary to
k8sutil so that it's easier for operators to use the functionality.
Operators still need to have some idea of how CmdReporter works. It runs
the requested command in a job, but if the calling operator needs the
job to have special configuration, the operator must request the pointer
to the job struct from CmdReporter to modify it before calling the
CmdReporter's Run method.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
Travis Nielsen
Sébastien Han
Blaine Gardner
Ashish Ranjan
Juan Miguel Olmo Martínez
Song Gao
Madhu Rajanna
rohan47
giovanism
Noah Watkins
AllenZMC
Nick McKinney
Mateusz Gozdek
Kristoffer Grönlund
Umanga Chapagain