On rare occasion, the ceph checks might not be 100% correct and the user
might decide to force an upgrade anyway.
Correctly, if we are in an upgrade, we perform `ok-to-stop` check for
each daemon, if one of them fails, we retry and eventually fail.
They are corner-cases where users would like to force the upgrade anyway.
Enhance the new CR property: "skipUpgradeChecks: true".
Closes: https://github.com/rook/rook/issues/3872
Signed-off-by: Sébastien Han <seb@redhat.com>
This adds a list of unusable directories paths to the Ceph Cluster CRD
doc page and the Ceph operator validation code. Reason for adding these
directories is that they are causing errors during Ceph OSD deployment.
Example: When an user wants to use `/rook` as a directory for an OSD,
the cluster deployment fails because the `mountPath`s must be unique:
```
Job.batch "rook-ceph-osd-prepare-NODE" is invalid: spec.template.spec.containers[1].volumeMounts[4].mountPath: Invalid value: "/rook": must be unique
```
Signed-off-by: Alexander Trost <galexrt@googlemail.com>
Add support for MachineDisruptionBudget controller and MachineLabel controller required for fencing
in OpenShift. This will ensure that machines are only fenced and OSDs are only stopped
when Ceph is in a healthy state.
Co-authored-by: Ashish Ranjan <aranjan@redhat.com>
Signed-off-by: travisn <tnielsen@redhat.com>
We can now enable via the cluster CR any manager module with a
setting under the new mgr element
mgr:
modules:
- name: pg_autoscaler
enabled: true
Co-authored-by: Sébastien Han <seb@redhat.com>
Co-authored-by: travisn <tnielsen@redhat.com>
Signed-off-by: travisn <tnielsen@redhat.com>
Some users have reported issues running Ceph OSDs in environments
without beefy systems. After consulting with Ceph experts, we believe
2Gb is still a safe minimum for Ceph for users with more modest systems
that won't result in disfunction in Ceph.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
After further consideration and working with the feature for a few
weeks, the `configOverride` functionality seems to be more complexity
than is worth it for the expected benefit. Users still have the most
preferred option of setting configs in Ceph's config database using
`ceph cofig set` or the dashboard, and the override ConfigMap still
exists for fallback scenarios. The config database can still be used
by Rook for setting necessary configs, but the user has no way of
telling Rook to do so.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
Add deviceClass as a per OSD config option under
storage/nodes/devices/config for setting a custom crush device class
per OSD.
Signed-off-by: Michael Vollman <michael.b.vollman@gmail.com>
Allow users to specify overrides in the CephCluster CRD. The override
ConfigMap still exists for emergency situations and is mounted into
daemon pods directly instead of being merged into a Rook-created config
file.
Rook Ceph no longer uses a config file for managing daemons with the
exception of the OSDs which still generate a config in an init
container.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
The tag pipeline currently only allows a strict versioning tag to be
applied with the major, minor, and patch numbers such as v1.1.0.
This change allows adding a suffix to the tag to support alpha,
beta, or rc releases. For example, the version can now be
v1.1.0-alpha.0 or v1.1.0-beta.1 or v1.1.0-rc.2
Signed-off-by: travisn <tnielsen@redhat.com>
With the 1.1 release we should not be creating bluestore OSDs
with rook's legacy partitioning scheme. All new bluestore OSDs
should be created with ceph-volume. Therefore, the miminum
version allowed is a version of mimic supporting ceph-volume.
This commit also adds consistent version checking between
the Add and Update methods for the operator handling the
CR events.
Signed-off-by: travisn <tnielsen@redhat.com>
- Updated Ceph Cluster CRD docuementation on how to use StorageClassDeviceSets to create OSD on PVC with examples CRD
- Updated the release notes with the new StorageClassDeviceSets functionality.
Co-authored-by: rohan47 <rohgupta@redhat.com>
Co-authored-by: Ashish Ranjan <aranjan@redhat.com>
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
CSI is now the preferred storage driver for Rook.
By default both the CSI and flex drivers will be started
by the operator. In a future release the flex driver
will be deprecated, but for now is still supported.
The drivers can be disabled with an environment
variable in the operator deployment in operator.yaml.
If users are only using one driver or the other there is
no need to enable both drivers.
Signed-off-by: travisn <tnielsen@redhat.com>
this adds the access for rook to create monitoring resources
across namespaces and provides servicemonitor config to
monitor the current service being created
Signed-off-by: Umanga Chapagain <chapagainumanga@gmail.com>
fail when metadataDevice is specified but ceph-volume batch does not
automatically use the metadataDevice as the block.db. (#3092)
Signed-off-by: Michael Vollman <michael.b.vollman@gmail.com>
Since most PRs are focused on a single storage provider,
the tests for other storage providers do not need to be
run. The CI will allow skipping tests for other storage
providers by specifying keywords in the PR description.
Similar to [skip ci], you can now specify [test storage-provider].
For example, [test ceph] will only trigger the test
suites for ceph and none of the other storage providers.
Signed-off-by: travisn <tnielsen@redhat.com>
this patch introduces failure-domain aware scheduling for monitor pods.
previously the only policy was to avoid scheduling monitors on the same
nodes. the new policy tries to spread monitors across zones and nodes.
NOTE: this patch only enforces the well-known k8s "zone"
failure-domain label (i.e. it does not look for region labels). this
decision is made for two reasons. first, eliminating explicit
scheduling is an important goal and will be done as part of upcoming
work. this will enable all supported failure-domain labels to be
handled. second, the "zone" domain is likely more appropriate for a
larger portion of users (as opposed to the geographical distinction
made with region), and supporting multiple failure domain types would
add significant complexity for a temporary resolution to failure
domain scheduling.
at a high-level this patch introduces a data structure `NodeUsage` that
is a pairing of a v1.Node and metadata relevant to scheduling: number of
monitor pods on the node, and its status w.r.t. to being scheduable for
new monitor pods.
each scheduling event computes a unified data structure that organizes
for each node a `NodeUsage` structure into per-failure-zone groups.
subsequent algorithms rely soley on this unified structure to make
scheduling decisions.
there are two entry points to the changes made:
1) during orchestration mon:Cluster:assignMons is invoked to make a
place new monitor pods onto nodes. the core scheduling logic is now
self-contained in mon:scheduleMonitor which implements node and zone
aware scheduling policies.
2) periodic health checks resolve any conflicts that may occur due to
changes to the cluster (e.g. new nodes, configuration changes, etc...).
the two existing checks: mons on invalid nodes, and overloaded nodes are
both handled.
fixes: #2603
Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
Linear raid can be created using `mdadm --create --level=linear`,
By creating a linear raid disk on top of a logical volume,
we can pass the linear device to `ceph-volume lvm batch --prepare`
to provision a new OSD on top of the logical volume since
`ceph-volume` does not take lv as the data device.
Signed-off-by: Ash Wu <hSATAC@gmail.com>
This commit does:
* remove dead code SetCrushTunables()
* stop creating initial crushmap
There is no need to create an initial crushmap, since Ceph natively does
that for us.
We don't delete the existing configmap containing the initial crush map
of already deployed cluster, perhaps people are using for whatever
reason. It does not harm to keep it around.
Closes: https://github.com/rook/rook/issues/3138
Signed-off-by: Sébastien Han <seb@redhat.com>
Previously, Rook Agent and Discovery DaemonSet deployment didn't allow
adding nodeAffinity. This commit adds nodeAffinity spec to daemonSet
deployment, which can be configured through environment variables in
operator deployment yaml.
+ Support multiple LabelKey, each with multiple LabelValue
+ Support multiple LabelKey with no value
Signed-off-by: Umanga Chapagain <chapagainumanga@gmail.com>
OwnerReferences are now created with the fully qualified apiVersion
such that the references will work properly on OpenShift.
Signed-off-by: travisn <tnielsen@redhat.com>
This requirement is long gone and was reported in 2017. Clients should
have been updated by then, so there is no reason to set the CRUSH
tunable to such an old client. Actually, we should let Ceph run its own
tunable. They can always be changed later. Also, this was really
restrictive and applied on every orchestration and thus would override
any other config done by an administrator. Even worse, this will trigger
data movement back and forth...
Closes: https://github.com/rook/rook/issues/3138
Signed-off-by: Sébastien Han <seb@redhat.com>
When the operator first starts, the only operation needed
is to watch for new cephcluster crds to be created and
start the discovery to find available devices. The flexvolume
agent, the csi driver, and the volume provisioning can all be delayed
starting until the first cluster is created.
Signed-off-by: travisn <tnielsen@redhat.com>
This commit does multiple things:
* remove support for AllNodes where we would deploy one rgw per node on
all the nodes.
* a transition path is implemented in the code so that if someone has an
existing deployment, daemonsets will be removed and replaced by an
deployments.
* when using "instances", each rgw deployed has its own key which makes
Ceph reporting the exact number of rgw running, see:
```
[root@rook-ceph-operator-775cf575c5-bh4sr /]# ceph -s
cluster:
id: 611fcf39-0669-4864-9a12-debb35c0397a
health: HEALTH_OK
services:
mon: 3 daemons, quorum a,b,c (age 12h)
mgr: a(active, since 12h)
osd: 3 osds: 3 up (since 12h), 3 in (since 12h)
rgw: 3 daemons active (my.store.a, my.store.b, my.store.c)
data:
pools: 6 pools, 600 pgs
objects: 235 objects, 3.8 KiB
usage: 3.0 GiB used, 84 GiB / 87 GiB avail
pgs: 600 active+clean
```
Closes: https://github.com/rook/rook/issues/2474, https://github.com/rook/rook/issues/2957 and https://github.com/rook/rook/issues/3245
Signed-off-by: Sébastien Han <seb@redhat.com>
Sébastien Han
travisn
Alexander Trost
Blaine Gardner
Maksim Nabokikh
Rohan CJ
Michael Vollman
rohan47
Noah Watkins
Sameer Kulkarni
Umanga Chapagain
Mateusz Gozdek
Ash Wu