RHEL8 introduces some lvm changes. For OSDs on PVs to work
we need to set allow_changes_with_duplicate_pvs = 1.
This is required since we're copying the block image
under another location.
Signed-off-by: travisn <tnielsen@redhat.com>
When running osds on pvc, the lvm is still holding the device when the cluster is deleted. As a result the PVs go into `failed` state
when the cluster is deleted. On deactivating the device, the 'failed' pvs get deleted as well. Solution involves:
- Running `lvchange -an <vg group name>` to deactivate the device.
- This command runs after 'osd prepare' completes.
- This command also runs after the 'osd daemon pod' is deleted or if the ceph-osd process terminates for some reasons.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
2019-09-20 18:08:13 +05:30
Sébastien Hanand: Tuan Do Anh <tuanda@vn.fujitsu.com>
Several commands were not using the cli finalizer function which builds
up the right argument to run a ceph command. This was working on the
operator since it used to have an /etc/ceph/ceph.conf file so we could
ran the command without specifying the ceph conf and key paths.
However, we recently removed that and it silently broke this "already"
broken code :).
Signed-off-by: Sébastien Han <seb@redhat.com>
ceph-volume now sends status and log output to stderr, which breaks
`ceph-volume lvm prepare` if not other things as well. Use the combined
stdout and stderr output from ceph-volume to fix this.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
OSD pods when running on PVC fail randomly due to some race condition:
- OSD daemon pod was activating the volume group using vgchange -an without any volume group name.
- This caused the pods to fail randomly when number of OSDs is more, say greater than 10.
- This fix adds Volume group name while activating/deactivating volume groups to fix this
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
udev is not running in the container so lvm might be stuck requesting lvm
devices. So let's not talk to udev to get lvm devices.
Signed-off-by: Sébastien Han <seb@redhat.com>
Maintenance is only defined when managedDisruptionBudgets are enabled.
Details can be found in the design doc for manged poddisriptionbudgets.
Signed-off-by: Rohan CJ <rohantmp@gmail.com>
Add deviceClass as a per OSD config option under
storage/nodes/devices/config for setting a custom crush device class
per OSD.
Signed-off-by: Michael Vollman <michael.b.vollman@gmail.com>
Adding support for the db-devices flag for ceph versions 14.2.1 and
newer. This flag will allow for explictly setting the db device for
bluestore configurations to the device specified as metadataDevice.
Fixing the databaseSizeMB config parameter to ensure it is used to set
the size of the db-device partition when specified at either the global
or individual device level. (#3652)
Signed-off-by: Michael Vollman <michael.b.vollman@gmail.com>
Allow users to specify overrides in the CephCluster CRD. The override
ConfigMap still exists for emergency situations and is mounted into
daemon pods directly instead of being merged into a Rook-created config
file.
Rook Ceph no longer uses a config file for managing daemons with the
exception of the OSDs which still generate a config in an init
container.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
Ensure that the metadata device and osd devices are passed as a space
separated list to ceph-volume. Currently osds-per-device is placed
between the metadataDevice and the osd devices causing an error.
Signed-off-by: Michael Vollman <michael.b.vollman@gmail.com>
With this change, the admin can specify a single mon and the operator
will add any other mons to both ClusterInfo and the configmap of mon
endpoints.
It will periodically check for changes too and update both ClusterInfo
and the configmap accordingly.
Signed-off-by: Sébastien Han <seb@redhat.com>
GetMonStatus is called by checkHealth in health.go and is already
printing GetMonStatus's response so there is no need to print it twice.
Signed-off-by: Sébastien Han <seb@redhat.com>
External clusters might only listen on msgr2, so we must acknowledge
that and set the correct prefix to 'v2:' instead of 'v1:' on the
operator's ceph config file.
Signed-off-by: Sébastien Han <seb@redhat.com>
Fix variable scoping issue that is ignoring osdsPerDevice config
setting. Ensure that in all scenarios both the global and per OSD
osdsPerDevice setting are honored. When both are set the more specific
setting will be used. (#3599)
Signed-off-by: Michael Vollman <michael.b.vollman@gmail.com>
- OSD backed by PVC required updating the lvm.conf file which was done by executing the 'sed' commands.
- This patch removes the 'sed' command and uses golang utlities to update the lvm.conf file.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
When setting metadataDevice per OSD the check that c-v is properly using
the metadata device is failing because it is only looking at the global
md setting which is nil and not the per OSD setting.
Signed-off-by: Michael Vollman <michael.b.vollman@gmail.com>
The daemon name always needs to have the prefix rook-ceph-mds-
removed from the deployment name in order to translate to the
daemon name. The daemon name is not dependent on the namespace.
Signed-off-by: travisn <tnielsen@redhat.com>
With the 1.1 release we should not be creating bluestore OSDs
with rook's legacy partitioning scheme. All new bluestore OSDs
should be created with ceph-volume. Therefore, the miminum
version allowed is a version of mimic supporting ceph-volume.
This commit also adds consistent version checking between
the Add and Update methods for the operator handling the
CR events.
Signed-off-by: travisn <tnielsen@redhat.com>
This is not changing the behavior of the code but making the
flow of logic in the createReplicationCrushRule() function consistent
by removing an else branch. This was just a reflex when I was reading
the code.
Signed-off-by: Michael Adam <obnox@redhat.com>
- Added code to support StorageClassDeviceSet spec provided in the cluster-on-pvc.yaml
- The code reads the StorageClassDeviceSet spec and creates pvc based on the ‘count’ field for each device set.
- OSD prepare job is started for each PVC which activates the ceph-volume on each PVC
- Finally OSD is started on each of the PVC device.
Co-authored-by: rohan47 <rohgupta@redhat.com>
Co-authored-by: Ashish Ranjan <aranjan@redhat.com>
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
Replace usage of the custom osd copybins command with the new top-level
copy-binaries command introduced with cmd-reporter.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
fail when metadataDevice is specified but ceph-volume batch does not
automatically use the metadataDevice as the block.db. (#3092)
Signed-off-by: Michael Vollman <michael.b.vollman@gmail.com>
Refactor the Ceph NFS (Ganesha) daemon to use the new config pattern.
Use of a custom Rook binary to run NFS has been removed in favor of
instead running the required DBUS process as a sidecar. The effect here
is that the NFS Ganesha container itself may fail and restart a few
times before the DBUS sidecar is running.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
We do not determine the daemon type and name by string parsing anymore
but passing that info directly in the update call, it's more reliable.
Also removing a bunch of unused functions.
Signed-off-by: Sébastien Han <seb@redhat.com>
Do not consider clusterInfo Ceph version since it's the one picked by
the spec image.
The CephVersion provided here needs to be the current running version
of Ceph for when it is later passed to OkToStop and checked with
IsAtLeast. If you are upgrading from a pre 14.2.1 version to a
version 14.2.1 or newer, IsAtLeast will incorrectly return true and
then fail trying to run mon ok-to-stop on a mon pod running a pre 14.2.1 version.
Signed-off-by: Sébastien Han <seb@redhat.com>
When a cluster is updated with a different image version, this triggers
a serialized restart of all the pods. Prior to this commit, no safety
check were performed and rook was hoping for the best outcome.
Now before doing restarting a daemon we check it can be restarted. Once
it's restarted we also check we can pursue with the rest of the
platform. For instance, with monitors we check that they are in quorum,
for OSD we check that PGs are clean and for MDS we make sure they are
all active.
Fixes: https://github.com/rook/rook/issues/2889
Signed-off-by: Sébastien Han <seb@redhat.com>
travisn
Santosh Pillai
Sébastien Hanand: Tuan Do Anh <tuanda@vn.fujitsu.com>
Blaine Gardner
Rohan CJ
Alexander Trost
Ashish Ranjan
Umanga Chapagain
Maksim Nabokikh
Michael Vollman
Michael Adam
rohan47andAshish Ranjan