When running osds on pvc, the lvm is still holding the device when the cluster is deleted. As a result the PVs go into `failed` state
when the cluster is deleted. On deactivating the device, the 'failed' pvs get deleted as well. Solution involves:
- Running `lvchange -an <vg group name>` to deactivate the device.
- This command runs after 'osd prepare' completes.
- This command also runs after the 'osd daemon pod' is deleted or if the ceph-osd process terminates for some reasons.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
The node selector for running OSDs on PVCs was using the node name
rather than the node's hostname. In clusters where the node name
is different from the hostname this would cause the osd daemons
to be in pending state indefinitely. Now the node selector will
use the hostname as expected.
Signed-off-by: travisn <tnielsen@redhat.com>
On rare occasion, the ceph checks might not be 100% correct and the user
might decide to force an upgrade anyway.
Correctly, if we are in an upgrade, we perform `ok-to-stop` check for
each daemon, if one of them fails, we retry and eventually fail.
They are corner-cases where users would like to force the upgrade anyway.
Enhance the new CR property: "skipUpgradeChecks: true".
Closes: https://github.com/rook/rook/issues/3872
Signed-off-by: Sébastien Han <seb@redhat.com>
Several commands were not using the cli finalizer function which builds
up the right argument to run a ceph command. This was working on the
operator since it used to have an /etc/ceph/ceph.conf file so we could
ran the command without specifying the ceph conf and key paths.
However, we recently removed that and it silently broke this "already"
broken code :).
Signed-off-by: Sébastien Han <seb@redhat.com>
If a Ceph container image is specified which has a entrypoint then the
Init container for osd prepare job blkdevmapper will fail. So set the
Command property.
Signed-off-by: rohan47 <rohgupta@redhat.com>
Some users have reported issues running Ceph OSDs in environments
without beefy systems. After consulting with Ceph experts, we believe
2Gb is still a safe minimum for Ceph for users with more modest systems
that won't result in disfunction in Ceph.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
OSD pods when running on PVC fail randomly due to some race condition:
- OSD daemon pod was activating the volume group using vgchange -an without any volume group name.
- This caused the pods to fail randomly when number of OSDs is more, say greater than 10.
- This fix adds Volume group name while activating/deactivating volume groups to fix this
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
The init container for copying binaries was being overwritten by the
init container for initializing pvcs. This change appends the init
containers so all expected init containers are preserved.
Signed-off-by: travisn <tnielsen@redhat.com>
Add deviceClass as a per OSD config option under
storage/nodes/devices/config for setting a custom crush device class
per OSD.
Signed-off-by: Michael Vollman <michael.b.vollman@gmail.com>
Adding support for the db-devices flag for ceph versions 14.2.1 and
newer. This flag will allow for explictly setting the db device for
bluestore configurations to the device specified as metadataDevice.
Fixing the databaseSizeMB config parameter to ensure it is used to set
the size of the db-device partition when specified at either the global
or individual device level. (#3652)
Signed-off-by: Michael Vollman <michael.b.vollman@gmail.com>
Allow users to specify overrides in the CephCluster CRD. The override
ConfigMap still exists for emergency situations and is mounted into
daemon pods directly instead of being merged into a Rook-created config
file.
Rook Ceph no longer uses a config file for managing daemons with the
exception of the OSDs which still generate a config in an init
container.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
OSD prepare job pods don't require node selector when the OSDs are based
on PVC.
The bug was intruduced in https://github.com/rook/rook/pull/3707 which
added node selector to the prepare job pod when portable property of
storageClassDeviceSets is false.
Adding node selector set kubernetes.io/hostname=example-storagedeviceset-xxx, while there is no host example-storagedeviceset-xxx the prepare job failed.
Signed-off-by: rohan47 <rohgupta@redhat.com>
From now on, Rook will only perform check before upgrades when there is
an actual upgrade. So if the Ceph image changed and a new version is
desired Rook will go through all the daemons and update them one by one
and perform checks in between.
Closes: https://github.com/rook/rook/issues/3583
Signed-off-by: Sébastien Han <seb@redhat.com>
ceph: Use the new NetworkSpec api
ceph: Change all tests to use the NetworkSpec
ceph: Remove TypeMeta field from NetworkSpec
ceph: Change IsHost network func signature
Signed-off-by: giovanism <giovanism@outlook.co.id>
Fix variable scoping issue that is ignoring osdsPerDevice config
setting. Ensure that in all scenarios both the global and per OSD
osdsPerDevice setting are honored. When both are set the more specific
setting will be used. (#3599)
Signed-off-by: Michael Vollman <michael.b.vollman@gmail.com>
Previously, the value was set to commonArgs, resulting in dead code,
since common arg isn't used after the append.
Signed-off-by: Odin Ugedal <odin@ugedal.com>
This commit fixes resource limits and placements not getting applied to osd pods provisioned using storageClassDeviceSets
Signed-off-by: Ashish Ranjan <aranjan@redhat.com>
- Added code to support StorageClassDeviceSet spec provided in the cluster-on-pvc.yaml
- The code reads the StorageClassDeviceSet spec and creates pvc based on the ‘count’ field for each device set.
- OSD prepare job is started for each PVC which activates the ceph-volume on each PVC
- Finally OSD is started on each of the PVC device.
Co-authored-by: rohan47 <rohgupta@redhat.com>
Co-authored-by: Ashish Ranjan <aranjan@redhat.com>
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
Replace usage of the custom osd copybins command with the new top-level
copy-binaries command introduced with cmd-reporter.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
At the beginning of every container startup we now change the ownership
of every known ceph directories to ceph:ceph.
This will allow daemons to start properly as well as allowing daemon to
writes their logs on files.
This is needed because ceph daemon now run as 'ceph' instead of 'root'
Signed-off-by: Sébastien Han <seb@redhat.com>
We do not determine the daemon type and name by string parsing anymore
but passing that info directly in the update call, it's more reliable.
Also removing a bunch of unused functions.
Signed-off-by: Sébastien Han <seb@redhat.com>
Do not consider clusterInfo Ceph version since it's the one picked by
the spec image.
The CephVersion provided here needs to be the current running version
of Ceph for when it is later passed to OkToStop and checked with
IsAtLeast. If you are upgrading from a pre 14.2.1 version to a
version 14.2.1 or newer, IsAtLeast will incorrectly return true and
then fail trying to run mon ok-to-stop on a mon pod running a pre 14.2.1 version.
Signed-off-by: Sébastien Han <seb@redhat.com>
These were used on a previous patch as a debug, let's remove them. They
are not intended to be displayed.
Signed-off-by: Sébastien Han <seb@redhat.com>
When a cluster is updated with a different image version, this triggers
a serialized restart of all the pods. Prior to this commit, no safety
check were performed and rook was hoping for the best outcome.
Now before doing restarting a daemon we check it can be restarted. Once
it's restarted we also check we can pursue with the rest of the
platform. For instance, with monitors we check that they are in quorum,
for OSD we check that PGs are clean and for MDS we make sure they are
all active.
Fixes: https://github.com/rook/rook/issues/2889
Signed-off-by: Sébastien Han <seb@redhat.com>
Empty deployment list made the operator to crash as the condition check
was checking if the value at the 0th location is nill wich fails in case
of empty list. Updated it to check length of list not equals 0.
Signed-off-by: rohan47 <rohgupta@redhat.com>
Travis Nielsen
Santosh Pillai
Sébastien Han
rohan47
Rohan CJ
Blaine Gardner
Michael Vollman
rohan47
giovanism
Odin Ugedal
Ashish Ranjan
xieyanker