The OSD needs to shut down quickly during upgrade, or other scenarios
where the OSD is being restarted. To facilitate this fast shutdown,
rook will run kill -9 on the osd process. The Ceph OSD is designed
to be safe even when killed like this. This allows the
ECONNREFUSED to be returned sooner, which will redirect the OSD
traffic to other OSDs and cause less downtime.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
When the OSD is running on a PVC, the LV needs to be released immediately
after the OSD process exits. The cleanup was being skipped when the OSD
process exited and returning an error rather than allowing the
LV cleanup to proceed.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The topology of the cluster should be based on the node labels rather
than a setting in the cluster CR. This allows a much richer and more
dynamic topology to be configured. The location will now be ignored
if specified in the cluster CR.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Some testing on OpenShift has shown that enabling Hostnetworking causes
issues with ceph-volume creating OSD.
This is a pure Red Hat OpenShift downstream issue.
This is meant to be temporary while we are waiting for the Selinux team
to get back to us.
Signed-off-by: Sébastien Han <seb@redhat.com>
OSD on PVC does not upgrade when the user upgrades the ceph version on cluster-on-pvc yaml.
The solution incudes:
- upgrade osd prepare and daemon pods on upgrade
- skip c-v prepare if filesystem is already present on the pvc device.
- skip lvm release in case of upgrade.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
PopulateDeviceInfo() in pkg/clusterd/disk.go returns nil as *sys.LocalDisk
if an error has occurred. This causes a nil pointer exception at
getAvailableDevices() in pkg/daemon/ceph/osd/daemon.go. At least the returned
value should be checked at the caller.
I added an explicit error to the return value of PopulateDeviceInfo(). This
naturally revokes an error check at the caller.
I modified PopulateDeviceUdevInfo() in the same file too.
Signed-off-by: morimoto-cybozu <kenji_morimoto@cybozu.co.jp>
The full path to vgchange should not be necessary. For some
downstream releases the path may even be in a different path.
Therefore we require that vgchange be in the path.
Signed-off-by: travisn <tnielsen@redhat.com>
Previously, we were applying lvm config changes to OSD running on PV. So
now, we do this for any type of cluster deployment (bare metal or
Cloud).
Signed-off-by: Sébastien Han <seb@redhat.com>
Previously, "run dir" was used to place a number of file and config on
dataDirHostPath. Now, since a lot of configs and options have moved
either on the OSD's startup CLI line or remove with bluestore we don't
need to change the default.
Also, it was confusing for user and difficult to find the socket.
The is is not breaking any config in dataDirHostPath since all the
elements (keyrings) are hardcoded in the config file.
Closes: https://github.com/rook/rook/issues/3966
Signed-off-by: Sébastien Han <seb@redhat.com>
When running osds on pvc, the lvm is still holding the device when the cluster is deleted. As a result the PVs go into `failed` state
when the cluster is deleted. On deactivating the device, the 'failed' pvs get deleted as well. Solution involves:
- Running `lvchange -an <vg group name>` to deactivate the device.
- This command runs after 'osd prepare' completes.
- This command also runs after the 'osd daemon pod' is deleted or if the ceph-osd process terminates for some reasons.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
OSD pods when running on PVC fail randomly due to some race condition:
- OSD daemon pod was activating the volume group using vgchange -an without any volume group name.
- This caused the pods to fail randomly when number of OSDs is more, say greater than 10.
- This fix adds Volume group name while activating/deactivating volume groups to fix this
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
Allow users to specify overrides in the CephCluster CRD. The override
ConfigMap still exists for emergency situations and is mounted into
daemon pods directly instead of being merged into a Rook-created config
file.
Rook Ceph no longer uses a config file for managing daemons with the
exception of the OSDs which still generate a config in an init
container.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
- Added code to support StorageClassDeviceSet spec provided in the cluster-on-pvc.yaml
- The code reads the StorageClassDeviceSet spec and creates pvc based on the ‘count’ field for each device set.
- OSD prepare job is started for each PVC which activates the ceph-volume on each PVC
- Finally OSD is started on each of the PVC device.
Co-authored-by: rohan47 <rohgupta@redhat.com>
Co-authored-by: Ashish Ranjan <aranjan@redhat.com>
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
When ceph-volume is called, the default python logging will buffer everything until
the process exits, which means the pod log doesnt show what is happening
until ceph-volume is done. This change will allow logs to be written
immediately as tasks are performed inside ceph-volume
Signed-off-by: travisn <tnielsen@redhat.com>
We must expose the ceph version to the osd pod so that we can generate
the proper ceph configuration file. If we don't do this all the version
check will assume they run the opposite of the desired version and the
formatting of the `ceph.conf` will be wrong.
Signed-off-by: Sébastien Han <seb@redhat.com>
All devices detected by the discovery pod were being passed to the OSD provisioning pod
thus not always honoring the desired device list that should be provisioned.
Now the provisioning pod will be given the desired state from the crd,
then apply that state depending on the actual devices detected.
Also added a helper to ensure OSDsPerDevice is always valid.
Signed-off-by: travisn <tnielsen@redhat.com>
Rook's behavior should not be to create a default OSD in dataDirHostPath
when no devices are present on a node. Any preexisting fallback osd
should be kept as long as no dirs or disks have been specified to keep
the legacy behavior for those clusters running with these osds in place.
The legacy deletion behavior is also kept, removing the osd as soon as
any dir or device is specified other than the dataDirHostPath dir.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
Create a cephconfig module in Ceph's daemon pkg source, and refactor the
config and keyring generation that exists in the mon package into the
new cephconfig package. The config/keyring generation code is used by
most all daemons and not just mon, so a new package is a more
appropriate place for this.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
run "rook discover" on storage nodes and discover devices on each node. The discovered disks are saved in a per node configmap, local-device-nodename.
Device information consits of name and persistent names, uuid, partition, filesystem, rotational, readonly, size, etc.
Signed-off-by: Huamin Chen <hchen@redhat.com>
If watching the osd orchestration status map fails, ensure the loop is broken out of so a new watcher can be created.
Log the osd device/dir mapping structures in a more friendly way
Signed-off-by: Jared Watts <jbw976@gmail.com>
Travis Nielsen
Sébastien Han
Santosh Pillai
morimoto-cybozu
Blaine Gardner
rohan47andAshish Ranjan
Alexander Trost
Huamin Chen
Jared Watts
Travis Nielsen