Enabling pg_autoscalar module fails most probably becasue `ceph mgr module enable <>` is happening very quickly after ceph mgr starts.
This PR adds a retry (5 times) with a sleep interval of 5 seconds (for each interval) to handle failure while enabling mgr modules.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
When using the "errors" package, using `%+v` (extended format),
each Frame of the error's StackTrace will be printed in detail.
Let's only print `%v` to print the error.
If the error has a Cause it will be printed recursively.
Basically `%+v` has been replaced with `%v` for all `error` type
interfaces, whether the logger is Info, Warning or Error.
Signed-off-by: Sébastien Han <seb@redhat.com>
An owner reference cannot be set with BlockOwnerDeletion: true
if the operator does not have privs to add a finalizer to the parent.
The discover resources don't need to block deletion based on the
parent so we remove that property from the owner reference.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Removing an OSD unintentionally can result in data loss,
which is to be prevented at all costs. Rook's functionality
to remove OSDs automatically when a node was removed from the
cluster CR has led on several occasions to users having OSDs
unintentionally removed from their cluster.
If the admin decides they really need to remove OSDs they should
do it as a one-time action that they intentionally run
rather than the operator attempting to manage the deletion
with desired state. The risk of misinterpreting the desired
state, even with a typo, is too great.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
It's not 100% safe since before Nautilus osd safe-to-destroy command
didn't have any json output. So added a version check.
Signed-off-by: rohan47 <rohgupta@redhat.com>
Add owner reference to the discover config map so that when the
discover pod goes away we also remove its configmap.
Signed-off-by: Sébastien Han <seb@redhat.com>
Added default regex values.
Created an environment variable and using that variable the user can give custom regex values
Updated the PendingReleaseNotes.md
Signed-off-by: Nizamudeen <nia@redhat.com>
Just as with the mon, mgr, mds, rgw, and rbd-mirror daemons, do not
generate a ceph.conf in an init container for filestore OSDs.
Instead use the commandline to supply all the needed arguments for
running these OSDs.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
ceph client CRD refactoring
Add info about client crd to documentation
Add tests for client controller CRD
Signed-off-by: Mateusz Los <los.mateusz@gmail.com>
"OSD on PVC" doesn't work for PV backed by LV. Fixing this problem
by the following changes.
- Rook accepts LVM disk type.
- If a LV-backed device is passed, Rook/Ceph invokes
"ceph-volume lvm prepare" with "--data vg/lv"
instead of "--data /path/to/device".
- If a LV-backed device is passed, Rook/Ceph suppresses
activation/deactivation of VG that owns this LV.
Fixes: https://github.com/rook/rook/issues/4185
Signed-off-by: dulltz <isrgnoe@gmail.com>
Co-authored-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
We shouldn't need to compute the location from the main OSD pod, since
the prepare pod runs on the same node we can populate the crush location
information from that pod and pass it along to the main osd container.
This is an initial commit that aims to eventually remove the need of
using the rook binary when running the main osd container.
Closes: https://github.com/rook/rook/issues/4362
Signed-off-by: Sébastien Han <seb@redhat.com>
`ceph-volume` cannot handle a device correctly if the device has an alias
under `/dev/mapper/`.
This commit fixes it by checking whether the device has that kind of alias
and using the alias as an argument for `ceph-volume`.
Signed-off-by: Kazuhito MATSUDA <kazuto.jinnai@gmail.com>
This reverts commit 7553318235.
The append puts the filter at the end of the file and it's out of any
lvm section, so it does not work anymore.
LVM complains with:
Configuration setting "filter" invalid. It's not part of any section.
Signed-off-by: Sébastien Han <seb@redhat.com>
lvmFilter doesn't contain a newline character. So when we will add more
lines after lvm.conf's filter configuration line, lvm.conf will corrupt.
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
The OSD needs to shut down quickly during upgrade, or other scenarios
where the OSD is being restarted. To facilitate this fast shutdown,
rook will run kill -9 on the osd process. The Ceph OSD is designed
to be safe even when killed like this. This allows the
ECONNREFUSED to be returned sooner, which will redirect the OSD
traffic to other OSDs and cause less downtime.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
When the OSD is running on a PVC, the LV needs to be released immediately
after the OSD process exits. The cleanup was being skipped when the OSD
process exited and returning an error rather than allowing the
LV cleanup to proceed.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Since the lvm config might change again in the future and because the
filter in the conf is commented as an example, let's append our filter
instead of searching and replacing.
Signed-off-by: Sébastien Han <seb@redhat.com>
**Description of your changes:**
This modification adds the information extracted from 'ceph-volume inventory':
command to the device configmaps generated by the discovery daemon when
"rook discover" starts with the new boolean "--use-ceph-volume" parameter.
Resolves #
https://github.com/rook/rook/issues/2606
Now the <cephVolumeData> field contains all the information returned
from <ceph-volume inventory> command.
Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
The topology of the cluster should be based on the node labels rather
than a setting in the cluster CR. This allows a much richer and more
dynamic topology to be configured. The location will now be ignored
if specified in the cluster CR.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Some testing on OpenShift has shown that enabling Hostnetworking causes
issues with ceph-volume creating OSD.
This is a pure Red Hat OpenShift downstream issue.
This is meant to be temporary while we are waiting for the Selinux team
to get back to us.
Signed-off-by: Sébastien Han <seb@redhat.com>
If a volume is being attached, it should be verified that the volume
is safe to attach. The volume was assumed to be safe to attach if it was
for the same pod. But this was assuming the pod of the same name would be
on the same node. This is true for pods created from deployments, but not
for pods that are part of a stateful set. A stateful set will maintain the
pod name even as the pod is failed over to a new node. Therefore, the fencing
much check if the pod is from the same node before allowing the attach to
continue. Otherwise, we need to wait for the volume to be detached from the
other node.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
In Octopus, this commit
https://github.com/ceph/ceph/pull/30859/commits/c09b82a80a392ccd0da7677c7b424ce5cd3fa5d6
moved the quorum_status command to a socket call. Thus, the `ceph
quorum_status` command does not exist anymore, to use it we need to do
`ceph daemon mon.$ID quorum_status`. However, this command only works
locally so it won't work if we run it from the operator.
The `ceph mon_status` command is a general command that run from
anywhere and returns the mon map, let's use this one on the operator
code.
This is not a breaking change.
Signed-off-by: Sébastien Han <seb@redhat.com>
OSD on PVC does not upgrade when the user upgrades the ceph version on cluster-on-pvc yaml.
The solution incudes:
- upgrade osd prepare and daemon pods on upgrade
- skip c-v prepare if filesystem is already present on the pvc device.
- skip lvm release in case of upgrade.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
The command `fs new` already enable the application pool settings for
the cephfs pools so we don't need to run that command again.
Signed-off-by: Sébastien Han <seb@redhat.com>
Checking for clean PGs is a recurring event when reconciling nodes
for upgrades and fencing. No need to log every request.
Signed-off-by: travisn <tnielsen@redhat.com>
PopulateDeviceInfo() in pkg/clusterd/disk.go returns nil as *sys.LocalDisk
if an error has occurred. This causes a nil pointer exception at
getAvailableDevices() in pkg/daemon/ceph/osd/daemon.go. At least the returned
value should be checked at the caller.
I added an explicit error to the return value of PopulateDeviceInfo(). This
naturally revokes an error check at the caller.
I modified PopulateDeviceUdevInfo() in the same file too.
Signed-off-by: morimoto-cybozu <kenji_morimoto@cybozu.co.jp>
The full path to vgchange should not be necessary. For some
downstream releases the path may even be in a different path.
Therefore we require that vgchange be in the path.
Signed-off-by: travisn <tnielsen@redhat.com>
We now differentiate the cases where:
* we only consume the external cluster
* we consume the external cluster as well as creating stateless
resources in Kubernetes (bootstrap mds,rgw, nfs)
This is mostly controlled via the image property spec. If not defined,
not extra CRs won't be able to be created.
Now the external cluster feature supports Ceph cluster as of Luminous 12.2.
Signed-off-by: Sébastien Han <seb@redhat.com>
A new CRD property `PreservePoolsOnDelete` has been added to Filesystem(fs) and
Object Store(os) resources in order to increase protection against data loss.
If it is set to `true`, associated pools won't be deleted when the main
resource(fs/os) is deleted. Creating again the deleted fs/os with the same name
will reuse the preserved pools.
Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
currently we are not checking the pool is empty or
not before deleting, This PR adds a check to check
if any images/snapshots using the pool, if yes it will
not delete the pool
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
Previously, we were applying lvm config changes to OSD running on PV. So
now, we do this for any type of cluster deployment (bare metal or
Cloud).
Signed-off-by: Sébastien Han <seb@redhat.com>
Santosh Pillai
Sébastien Han
Travis Nielsen
Blaine Gardner
rohan47
Nizamudeen
Mateusz Los
Mateusz Gozdek
dulltzandSatoru Takeuchi
Kazuhito MATSUDA
Juan Miguel Olmo Martínez
morimoto-cybozu
Madhu Rajanna