By setting the working directory of a Ceph daemon to the log direcor
(which is bindmounted to the host), we can ensure that the coredumps
will be available.
On CentOS 7 **only**, the kernel is configured with:
```
cat /proc/sys/kernel/core_pattern
core
```
which means that the coredumps will end up in the process working
directory and will be named "core".
On CentOS 8, everything is different and this won't work but still
CentOS will be able to consume this and changing the working directory
is harmless.
Signed-off-by: Sébastien Han <seb@redhat.com>
We can now collect logs directly into a side-car container.
A new CRD spec has been added:
spec:
logCollector:
enabled: true
periodicity: 24h
Every 24h we will rotate log files for each Ceph daemon.
Signed-off-by: Sébastien Han <seb@redhat.com>
this commit handle golangci-lint linter staticcheck error.
`staticcheck` - Staticcheck is a go vet on steroids,
applying a ton of static analysis checks.
To see only `staticcheck` linter output
`golangci-lint run --disable-all -E staticcheck`
Signed-off-by: subhamkrai <srai@redhat.com>
This allows users to specify additional labels to be added to the components
created by the Rook Ceph operator.
Additionally the rbd mirrors are now getting their annotations and
labels set.
Signed-off-by: Alexander Trost <galexrt@googlemail.com>
The operator will now allow the resource limits to be applied below the
recommended minimums. In small clusters, even the recommended minimums
may not be necessary. A warning is still printed to the operator log,
but we allow the configuration to continue.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The labels cannot be updated on a deployment match selector.
In v1.4 a new label was added for ceph_daemon_type that is intended
to be on the pod labels, but cannot be applied to the match selectors.
Therefore, we suppress any new labels that are added to the daemons.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Rename PodLabels function to CephDaemonAppLabels for more clarity about
what the function's purpose is.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
this commit handles all the gosec g601
error code (i.e Implicit memory aliasing
of items from a range statement).
Signed-off-by: subhamkrai <subhamkumarrai03@gmail.com>
To provide more context for executing commands in a ceph cluster,
the full clusterInfo is now passed to the ceph execution commands.
All information about the cluster will now be available throughout
all the areas of the operator. The namespace, ceph credentials,
mon endpoints, and other info is a core part of that cluster info.
Arguments passed through the controllers are also simplified for
mons, mgr, osds, and other daemons where the parameters had
become too complex.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The ceph config and client packages are conceptually the same.
To avoid circular dependencies in some cases, we simplify by combining
the packages into the client package.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
ceph mds deactivate is not valid command for Nautilus, changing the mechnaism of upgrading so that it can be done in the way Nautilus does it.
Closes: https://github.com/rook/rook/issues/5278
Signed-off-by: RAJAT SINGH <rajasing@redhat.com>
Remove specific mds config added to the mon configuration database when deleting a CephObjectstore
or when scaling down number of mds
Also remove mds CephX key
Signed-off-by: n.fraison <n.fraison@criteo.com>
You can now use Rook along with Multus. Multus must be up and running
and the right ressources must exist such as NetworkAttachmentDefinition
CR.
The Cluster CR spec has new fields to work with multus:
network:
provider: multus (or 'host' for hostNetworking)
selectors:
public: NetworkAttachmentDefinition name
cluster: NetworkAttachmentDefinition name
If only a single NetworkAttachmentDefinition is provided Rook will use
both anyway for the Ceph traffic.
Please refer to the doc to learn more.
Closes: https://github.com/rook/rook/issues/4716
Signed-off-by: Sébastien Han <seb@redhat.com>
This type was a string already and was just making us doing string()
calls all the time to it's not worth it.
Signed-off-by: Sébastien Han <seb@redhat.com>
Now Kubernetes will perform liveness checks on mon, mds and osd daemons.
The command will:
* call the socket (check for existence)
* execute a command and check the return code (success if 0)
This handles the case where the daemon is stuck locally and
unresponsive. It's unlikely but not impossible.
These checks bring more robustness to the implementation.
rbd-mirror and nfs have been leftover for the following reason. The
rbd-mirror socket name is different from other daemons (could be fixed
though): /run/ceph/ceph-client.rbd-mirror.a.1.94362516231272.asok also,
the command to call would need to be changed from "status" to "rbd
mirror status" so we can keep this for a later.
The nfs ganesha has no socket only a PID file which doesn't mean much.
No PID means the process does not run so Kubernetes will already handle
this and the pod will crash loop.
Signed-off-by: Sébastien Han <seb@redhat.com>
The pools had some legacy structs that translated between the
ceph.v1 types used by the CRDs and the internal implementation
of the pools. This simplifies the pool implementation by removing
the intermediate model and leaving us only with the ceph v1
pool types and no unnecessary translation.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The CRD watcher has been replaced by the new controller-runtime
framework.
This brings robustness in our operator, meaning that any resources that
are modified will be reconciled into the desired state.
Closes: https://github.com/rook/rook/issues/4940
Signed-off-by: Sébastien Han <seb@redhat.com>
As part of the transition to support Ceph release **as of** Nautilus, we
left over that portion of code.
We don't need it anymore.
Signed-off-by: Sébastien Han <seb@redhat.com>
Now, the CephBlockPool CRD is managed with the controller-runtime.
So the watcher is outside of the main controller reconciliation loop of
CephCluster which brings numerous benefit such as:
* having its own reconciliation loop
* won't block anything from the main CephCluster controller loop
* fast than waiting for CephCluster loop to completion
Partially close: https://github.com/rook/rook/issues/1981
Signed-off-by: Sébastien Han <seb@redhat.com>
This commit is to handle all those unhandled errors which raises the gosec warning.
Fixed G104: Unhandled Errors are handled now
Signed-off-by: Nizamudeen <nia@redhat.com>
We now always perform ok-to-stop change when the deployment is going to
be updated. This could be for various reasons like change in the spec or
a new Ceph image.
In any case, if Ceph resources are going to be restarted this must be
done carefully.
Also trimmed the isUpgrade variable, the upcoming commit will remove the
need of that variable.
Keeping isUpgrade from the controller code since we need it to detect
whether an orchestration upgrade should be triggered or not (based on
the ceph cluster status).
Also, we keep it on the child controllers so we don't go into an
orchestration phase if the cluster image hasn't changed. The child does
not need to be updated/go into its orchestration phase because the
cluster CR changed.
Closes: https://github.com/rook/rook/issues/4379
Signed-off-by: Sébastien Han <seb@redhat.com>
We now force the mds daemon to join the fs that was created by Rook.
This can always be changed by using the ceph config store.
Closes: https://github.com/rook/rook/issues/4707
Signed-off-by: Sébastien Han <seb@redhat.com>
With this modification is guaranteed that at least 1 MDS pod is going to be
placed in each of the available zones in the k8s cluster.
This would be improved in the future using:
<Pod Topology Spread Constraints> (still in alpha state since kubernetes V.1.16)
This modification obtain the number of zones in the k8s cluster and appends an
<antiaffinity> term using the topologyKey
<topology.kubernetes.io/zone> in the same number of MDS pods.
In the case that will be more MDS pods than zones, the <antiaffinity> term won't
be added in these extra pods.
Important Note:
The antiaffiniy term only will work with nodes labeled using the NEW label:
<topology.kubernetes.io/zone> present in k8s V.1.17 clusters.
For previous versions of k8s clusters the label used is:
<failure-domain.beta.kubernetes.io/zone>
And therefore, in this kind of clusters this modification won't work.
Resolves#4641
[test ceph]
Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
A new CRD option `continueUpgradeAfterChecksEvenIfNotHealthy` is added.
When upgrading, Rook goes OSD by OSD and then waits for PGs to be clean
before proceeding to the next OSD. Currently, Rook waits for 5 hours but
there might be circumstances where PGs need more time to settle.
Thus setting `continueUpgradeAfterChecksEvenIfNotHealthy` to true will
pursue the upgrade process, even if PGs are not 100% active+clean.
Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1786029
Signed-off-by: Sébastien Han <seb@redhat.com>
Reduce from 5 minutes to less than 50 seconds the time needed to restart a
mgr and osds(PVCs based),rgw,mds,rbd,nfs and toolbox pod that was running
in a k8s "NotReady" Node.
A new Rook Operator setting has been added to allow the user to set the
time used in pod's toleration <node.kubernetes.io/unreachable>:
ROOK_UNREACHABLE_NODE_TOLERATION_SECONDS = <5>
New setting added in release notes
Addressed @travisn,@leseb, and @blaineEXE suggestions
Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
When using the "errors" package, using `%+v` (extended format),
each Frame of the error's StackTrace will be printed in detail.
Let's only print `%v` to print the error.
If the error has a Cause it will be printed recursively.
Basically `%+v` has been replaced with `%v` for all `error` type
interfaces, whether the logger is Info, Warning or Error.
Signed-off-by: Sébastien Han <seb@redhat.com>
The Ceph Mimic release introduced the monitor config store which allows
us to set configuration flag per daemon in a centralized fashion. Let's
use it for Mimic and newer and keep the CLI flag of daemon startup for
older versions.
The main benefit is that the config of a daemon can now be overridden.
Also, this commit refactors some code and add methods that are unit
tested.
Closes: https://github.com/rook/rook/issues/4307
Signed-off-by: Sébastien Han <seb@redhat.com>
This change standardizes the ownerRefs of Ceph keyrings to
k8s Deployment resources across all daemons and refactors a
bit of duplicate code on keyring association for ease of
maintenance.
Fixes: #4089
Signed-off-by: egafford <egafford@redhat.com>
Failure to create a keyring prior to creating a deployment
dependent on that keyring created a race conditition resulting
in an intermittent and unnecessary pod failure. This change
creates a keyring prior to deployment creation for MDS
daemons.
Partially fixes: rook#4089
Signed-off-by: egafford <egafford@redhat.com>
Following the Ceph OSD rework, my test environment regularly failed to
start OSDs, as they would start before the postStart lifecyle hook which
was chown'ing the log and data directories could finish. Pods failed to
start for an arbitrarily long time before starting successfully. This
was a manifestation of a race condition identified in Ceph monitors in
https://github.com/rook/rook/pull/3594#discussion_r312279176
Create a Ceph operator-level method to create an init container that
will chown the necessary directories for all Ceph daemons (any daemon
which runs as the ceph:ceph user-and-group). Also add this container to
expectations for Ceph pod templates in unit tests.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
The resources for rgw, mds, and nfs were being cleaned up explicitly when
the CR was deleted. This is legacy from before the owner references were
set on the CRs. Now the owner references are set to the appropriate CRs so
the resources will be cleaned up upon deletion of the specific CR
rather than the cluster CR.
Signed-off-by: travisn <tnielsen@redhat.com>
On rare occasion, the ceph checks might not be 100% correct and the user
might decide to force an upgrade anyway.
Correctly, if we are in an upgrade, we perform `ok-to-stop` check for
each daemon, if one of them fails, we retry and eventually fail.
They are corner-cases where users would like to force the upgrade anyway.
Enhance the new CR property: "skipUpgradeChecks: true".
Closes: https://github.com/rook/rook/issues/3872
Signed-off-by: Sébastien Han <seb@redhat.com>
From now on, Rook will only perform check before upgrades when there is
an actual upgrade. So if the Ceph image changed and a new version is
desired Rook will go through all the daemons and update them one by one
and perform checks in between.
Closes: https://github.com/rook/rook/issues/3583
Signed-off-by: Sébastien Han <seb@redhat.com>
ceph: Use the new NetworkSpec api
ceph: Change all tests to use the NetworkSpec
ceph: Remove TypeMeta field from NetworkSpec
ceph: Change IsHost network func signature
Signed-off-by: giovanism <giovanism@outlook.co.id>
When configuring an external cluster the orchestration of discover and all the ceph
daemons will be skipped. The operator will still watch for the creation of
crds for this namespace. When a filesystem, object, object user, or
ganesha crd are created, the operator will simply print an error
to the log.
Signed-off-by: travisn <tnielsen@redhat.com>
Signed-off-by: Sébastien Han <seb@redhat.com>
When deleting mds deployment the error is logged, but not propagated
further down (it always returns nil). The two places using the function
checks the error, and acts upon it.
Signed-off-by: Odin Ugedal <odin@ugedal.com>
With the 1.1 release we should not be creating bluestore OSDs
with rook's legacy partitioning scheme. All new bluestore OSDs
should be created with ceph-volume. Therefore, the miminum
version allowed is a version of mimic supporting ceph-volume.
This commit also adds consistent version checking between
the Add and Update methods for the operator handling the
CR events.
Signed-off-by: travisn <tnielsen@redhat.com>
Sébastien Han
ushen
Arun Kumar Mohan
Santosh Pillai
subhamkrai
Alexander Trost
Travis Nielsen
Blaine Gardner
RAJAT SINGH
Madhu Rajanna
n.fraison
Nizamudeen
Juan Miguel Olmo Martínez
d-luu
egafford
giovanism
Odin Ugedal