The OSD needs to shut down quickly during upgrade, or other scenarios
where the OSD is being restarted. To facilitate this fast shutdown,
rook will run kill -9 on the osd process. The Ceph OSD is designed
to be safe even when killed like this. This allows the
ECONNREFUSED to be returned sooner, which will redirect the OSD
traffic to other OSDs and cause less downtime.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
When the OSD is running on a PVC, the LV needs to be released immediately
after the OSD process exits. The cleanup was being skipped when the OSD
process exited and returning an error rather than allowing the
LV cleanup to proceed.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Since the lvm config might change again in the future and because the
filter in the conf is commented as an example, let's append our filter
instead of searching and replacing.
Signed-off-by: Sébastien Han <seb@redhat.com>
**Description of your changes:**
This modification adds the information extracted from 'ceph-volume inventory':
command to the device configmaps generated by the discovery daemon when
"rook discover" starts with the new boolean "--use-ceph-volume" parameter.
Resolves #
https://github.com/rook/rook/issues/2606
Now the <cephVolumeData> field contains all the information returned
from <ceph-volume inventory> command.
Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
The topology of the cluster should be based on the node labels rather
than a setting in the cluster CR. This allows a much richer and more
dynamic topology to be configured. The location will now be ignored
if specified in the cluster CR.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Some testing on OpenShift has shown that enabling Hostnetworking causes
issues with ceph-volume creating OSD.
This is a pure Red Hat OpenShift downstream issue.
This is meant to be temporary while we are waiting for the Selinux team
to get back to us.
Signed-off-by: Sébastien Han <seb@redhat.com>
If a volume is being attached, it should be verified that the volume
is safe to attach. The volume was assumed to be safe to attach if it was
for the same pod. But this was assuming the pod of the same name would be
on the same node. This is true for pods created from deployments, but not
for pods that are part of a stateful set. A stateful set will maintain the
pod name even as the pod is failed over to a new node. Therefore, the fencing
much check if the pod is from the same node before allowing the attach to
continue. Otherwise, we need to wait for the volume to be detached from the
other node.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
In Octopus, this commit
https://github.com/ceph/ceph/pull/30859/commits/c09b82a80a392ccd0da7677c7b424ce5cd3fa5d6
moved the quorum_status command to a socket call. Thus, the `ceph
quorum_status` command does not exist anymore, to use it we need to do
`ceph daemon mon.$ID quorum_status`. However, this command only works
locally so it won't work if we run it from the operator.
The `ceph mon_status` command is a general command that run from
anywhere and returns the mon map, let's use this one on the operator
code.
This is not a breaking change.
Signed-off-by: Sébastien Han <seb@redhat.com>
OSD on PVC does not upgrade when the user upgrades the ceph version on cluster-on-pvc yaml.
The solution incudes:
- upgrade osd prepare and daemon pods on upgrade
- skip c-v prepare if filesystem is already present on the pvc device.
- skip lvm release in case of upgrade.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
The command `fs new` already enable the application pool settings for
the cephfs pools so we don't need to run that command again.
Signed-off-by: Sébastien Han <seb@redhat.com>
Checking for clean PGs is a recurring event when reconciling nodes
for upgrades and fencing. No need to log every request.
Signed-off-by: travisn <tnielsen@redhat.com>
PopulateDeviceInfo() in pkg/clusterd/disk.go returns nil as *sys.LocalDisk
if an error has occurred. This causes a nil pointer exception at
getAvailableDevices() in pkg/daemon/ceph/osd/daemon.go. At least the returned
value should be checked at the caller.
I added an explicit error to the return value of PopulateDeviceInfo(). This
naturally revokes an error check at the caller.
I modified PopulateDeviceUdevInfo() in the same file too.
Signed-off-by: morimoto-cybozu <kenji_morimoto@cybozu.co.jp>
The full path to vgchange should not be necessary. For some
downstream releases the path may even be in a different path.
Therefore we require that vgchange be in the path.
Signed-off-by: travisn <tnielsen@redhat.com>
We now differentiate the cases where:
* we only consume the external cluster
* we consume the external cluster as well as creating stateless
resources in Kubernetes (bootstrap mds,rgw, nfs)
This is mostly controlled via the image property spec. If not defined,
not extra CRs won't be able to be created.
Now the external cluster feature supports Ceph cluster as of Luminous 12.2.
Signed-off-by: Sébastien Han <seb@redhat.com>
A new CRD property `PreservePoolsOnDelete` has been added to Filesystem(fs) and
Object Store(os) resources in order to increase protection against data loss.
If it is set to `true`, associated pools won't be deleted when the main
resource(fs/os) is deleted. Creating again the deleted fs/os with the same name
will reuse the preserved pools.
Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
currently we are not checking the pool is empty or
not before deleting, This PR adds a check to check
if any images/snapshots using the pool, if yes it will
not delete the pool
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
Previously, we were applying lvm config changes to OSD running on PV. So
now, we do this for any type of cluster deployment (bare metal or
Cloud).
Signed-off-by: Sébastien Han <seb@redhat.com>
Previously, "run dir" was used to place a number of file and config on
dataDirHostPath. Now, since a lot of configs and options have moved
either on the OSD's startup CLI line or remove with bluestore we don't
need to change the default.
Also, it was confusing for user and difficult to find the socket.
The is is not breaking any config in dataDirHostPath since all the
elements (keyrings) are hardcoded in the config file.
Closes: https://github.com/rook/rook/issues/3966
Signed-off-by: Sébastien Han <seb@redhat.com>
RHEL8 introduces some lvm changes. For OSDs on PVs to work
we need to set allow_changes_with_duplicate_pvs = 1.
This is required since we're copying the block image
under another location.
Signed-off-by: travisn <tnielsen@redhat.com>
When running osds on pvc, the lvm is still holding the device when the cluster is deleted. As a result the PVs go into `failed` state
when the cluster is deleted. On deactivating the device, the 'failed' pvs get deleted as well. Solution involves:
- Running `lvchange -an <vg group name>` to deactivate the device.
- This command runs after 'osd prepare' completes.
- This command also runs after the 'osd daemon pod' is deleted or if the ceph-osd process terminates for some reasons.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
2019-09-20 18:08:13 +05:30
Sébastien Hanand: Tuan Do Anh <tuanda@vn.fujitsu.com>
Several commands were not using the cli finalizer function which builds
up the right argument to run a ceph command. This was working on the
operator since it used to have an /etc/ceph/ceph.conf file so we could
ran the command without specifying the ceph conf and key paths.
However, we recently removed that and it silently broke this "already"
broken code :).
Signed-off-by: Sébastien Han <seb@redhat.com>
ceph-volume now sends status and log output to stderr, which breaks
`ceph-volume lvm prepare` if not other things as well. Use the combined
stdout and stderr output from ceph-volume to fix this.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
OSD pods when running on PVC fail randomly due to some race condition:
- OSD daemon pod was activating the volume group using vgchange -an without any volume group name.
- This caused the pods to fail randomly when number of OSDs is more, say greater than 10.
- This fix adds Volume group name while activating/deactivating volume groups to fix this
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
udev is not running in the container so lvm might be stuck requesting lvm
devices. So let's not talk to udev to get lvm devices.
Signed-off-by: Sébastien Han <seb@redhat.com>
Maintenance is only defined when managedDisruptionBudgets are enabled.
Details can be found in the design doc for manged poddisriptionbudgets.
Signed-off-by: Rohan CJ <rohantmp@gmail.com>
Travis Nielsen
Sébastien Han
Juan Miguel Olmo Martínez
Santosh Pillai
morimoto-cybozu
Madhu Rajanna
Blaine Gardner
Rohan CJ
Alexander Trost
Ashish Ranjan
Umanga Chapagain
Maksim Nabokikh