With the desire to run the integration tests on the five
most recent versions of K8s, 1.16 is added and 1.11
is removed from the integration test matrix.
Signed-off-by: travisn <tnielsen@redhat.com>
As with the rest of the Ceph daemons, use the new config patters for
dir-based OSDs. Do not generate a config to use in an init container;
instead use CLI flags and the mon config database.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
Log file path on host for OSDs was dataDirHostPath/log/<namespace>
instead of dataDirHostPath/<namespace>/log as with other daemons.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
A new CRD property `PreservePoolsOnDelete` has been added to Filesystem(fs) and
Object Store(os) resources in order to increase protection against data loss.
If it is set to `true`, associated pools won't be deleted when the main
resource(fs/os) is deleted. Creating again the deleted fs/os with the same name
will reuse the preserved pools.
Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
This updates the Ceph mon disaster recovery guide to use the correct
paths and commands for the current Rook Ceph release.
Signed-off-by: Alexander Trost <galexrt@googlemail.com>
A job is started to detect the Ceph version. This job now allows setting
the node affinity and tolerations with the same setting that is
specified for the mon placement. No new placement spec is required, but
we will simply use the same spec that the mons use.
Signed-off-by: travisn <tnielsen@redhat.com>
Previously, "run dir" was used to place a number of file and config on
dataDirHostPath. Now, since a lot of configs and options have moved
either on the OSD's startup CLI line or remove with bluestore we don't
need to change the default.
Also, it was confusing for user and difficult to find the socket.
The is is not breaking any config in dataDirHostPath since all the
elements (keyrings) are hardcoded in the config file.
Closes: https://github.com/rook/rook/issues/3966
Signed-off-by: Sébastien Han <seb@redhat.com>
On rare occasion, the ceph checks might not be 100% correct and the user
might decide to force an upgrade anyway.
Correctly, if we are in an upgrade, we perform `ok-to-stop` check for
each daemon, if one of them fails, we retry and eventually fail.
They are corner-cases where users would like to force the upgrade anyway.
Enhance the new CR property: "skipUpgradeChecks: true".
Closes: https://github.com/rook/rook/issues/3872
Signed-off-by: Sébastien Han <seb@redhat.com>
This adds a list of unusable directories paths to the Ceph Cluster CRD
doc page and the Ceph operator validation code. Reason for adding these
directories is that they are causing errors during Ceph OSD deployment.
Example: When an user wants to use `/rook` as a directory for an OSD,
the cluster deployment fails because the `mountPath`s must be unique:
```
Job.batch "rook-ceph-osd-prepare-NODE" is invalid: spec.template.spec.containers[1].volumeMounts[4].mountPath: Invalid value: "/rook": must be unique
```
Signed-off-by: Alexander Trost <galexrt@googlemail.com>
Add support for MachineDisruptionBudget controller and MachineLabel controller required for fencing
in OpenShift. This will ensure that machines are only fenced and OSDs are only stopped
when Ceph is in a healthy state.
Co-authored-by: Ashish Ranjan <aranjan@redhat.com>
Signed-off-by: travisn <tnielsen@redhat.com>
We can now enable via the cluster CR any manager module with a
setting under the new mgr element
mgr:
modules:
- name: pg_autoscaler
enabled: true
Co-authored-by: Sébastien Han <seb@redhat.com>
Co-authored-by: travisn <tnielsen@redhat.com>
Signed-off-by: travisn <tnielsen@redhat.com>
Some users have reported issues running Ceph OSDs in environments
without beefy systems. After consulting with Ceph experts, we believe
2Gb is still a safe minimum for Ceph for users with more modest systems
that won't result in disfunction in Ceph.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
After further consideration and working with the feature for a few
weeks, the `configOverride` functionality seems to be more complexity
than is worth it for the expected benefit. Users still have the most
preferred option of setting configs in Ceph's config database using
`ceph cofig set` or the dashboard, and the override ConfigMap still
exists for fallback scenarios. The config database can still be used
by Rook for setting necessary configs, but the user has no way of
telling Rook to do so.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
Add deviceClass as a per OSD config option under
storage/nodes/devices/config for setting a custom crush device class
per OSD.
Signed-off-by: Michael Vollman <michael.b.vollman@gmail.com>
Allow users to specify overrides in the CephCluster CRD. The override
ConfigMap still exists for emergency situations and is mounted into
daemon pods directly instead of being merged into a Rook-created config
file.
Rook Ceph no longer uses a config file for managing daemons with the
exception of the OSDs which still generate a config in an init
container.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
The tag pipeline currently only allows a strict versioning tag to be
applied with the major, minor, and patch numbers such as v1.1.0.
This change allows adding a suffix to the tag to support alpha,
beta, or rc releases. For example, the version can now be
v1.1.0-alpha.0 or v1.1.0-beta.1 or v1.1.0-rc.2
Signed-off-by: travisn <tnielsen@redhat.com>
With the 1.1 release we should not be creating bluestore OSDs
with rook's legacy partitioning scheme. All new bluestore OSDs
should be created with ceph-volume. Therefore, the miminum
version allowed is a version of mimic supporting ceph-volume.
This commit also adds consistent version checking between
the Add and Update methods for the operator handling the
CR events.
Signed-off-by: travisn <tnielsen@redhat.com>
- Updated Ceph Cluster CRD docuementation on how to use StorageClassDeviceSets to create OSD on PVC with examples CRD
- Updated the release notes with the new StorageClassDeviceSets functionality.
Co-authored-by: rohan47 <rohgupta@redhat.com>
Co-authored-by: Ashish Ranjan <aranjan@redhat.com>
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
CSI is now the preferred storage driver for Rook.
By default both the CSI and flex drivers will be started
by the operator. In a future release the flex driver
will be deprecated, but for now is still supported.
The drivers can be disabled with an environment
variable in the operator deployment in operator.yaml.
If users are only using one driver or the other there is
no need to enable both drivers.
Signed-off-by: travisn <tnielsen@redhat.com>
this adds the access for rook to create monitoring resources
across namespaces and provides servicemonitor config to
monitor the current service being created
Signed-off-by: Umanga Chapagain <chapagainumanga@gmail.com>
fail when metadataDevice is specified but ceph-volume batch does not
automatically use the metadataDevice as the block.db. (#3092)
Signed-off-by: Michael Vollman <michael.b.vollman@gmail.com>
Since most PRs are focused on a single storage provider,
the tests for other storage providers do not need to be
run. The CI will allow skipping tests for other storage
providers by specifying keywords in the PR description.
Similar to [skip ci], you can now specify [test storage-provider].
For example, [test ceph] will only trigger the test
suites for ceph and none of the other storage providers.
Signed-off-by: travisn <tnielsen@redhat.com>
this patch introduces failure-domain aware scheduling for monitor pods.
previously the only policy was to avoid scheduling monitors on the same
nodes. the new policy tries to spread monitors across zones and nodes.
NOTE: this patch only enforces the well-known k8s "zone"
failure-domain label (i.e. it does not look for region labels). this
decision is made for two reasons. first, eliminating explicit
scheduling is an important goal and will be done as part of upcoming
work. this will enable all supported failure-domain labels to be
handled. second, the "zone" domain is likely more appropriate for a
larger portion of users (as opposed to the geographical distinction
made with region), and supporting multiple failure domain types would
add significant complexity for a temporary resolution to failure
domain scheduling.
at a high-level this patch introduces a data structure `NodeUsage` that
is a pairing of a v1.Node and metadata relevant to scheduling: number of
monitor pods on the node, and its status w.r.t. to being scheduable for
new monitor pods.
each scheduling event computes a unified data structure that organizes
for each node a `NodeUsage` structure into per-failure-zone groups.
subsequent algorithms rely soley on this unified structure to make
scheduling decisions.
there are two entry points to the changes made:
1) during orchestration mon:Cluster:assignMons is invoked to make a
place new monitor pods onto nodes. the core scheduling logic is now
self-contained in mon:scheduleMonitor which implements node and zone
aware scheduling policies.
2) periodic health checks resolve any conflicts that may occur due to
changes to the cluster (e.g. new nodes, configuration changes, etc...).
the two existing checks: mons on invalid nodes, and overloaded nodes are
both handled.
fixes: #2603
Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
travisn
Blaine Gardner
Juan Miguel Olmo Martínez
Sébastien Han
Alexander Trost
Maksim Nabokikh
Rohan CJ
Michael Vollman
rohan47
Noah Watkins
Sameer Kulkarni
Umanga Chapagain
Mateusz Gozdek