This commit replaces `os.Setenv` with `t.Setenv` in tests. The
environment variable is automatically restored to its original value
when the test and all its subtests complete.
Reference: https://pkg.go.dev/testing#T.Setenv
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
Rook supports raw mode OSD in host-based cluster. So we can also
support OSD on logical volume in this kind of cluster.
Logical volumes aren't picked by filters (i.e. `useAllDevices: true`
and `device{Path,}Filter` to avoid unwanted LV consumption on upgrade.
Closes: https://github.com/rook/rook/issues/2047
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
With octopus coming to end of life, we remove support from
Rook for deploying Ceph Octopus and assume a min version of
Pacific v16. Any checks for octopus or earlier are removed
from the reconciles since they are obsolete.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Let it fail further once we run c-v activate. The error reported by LVM
could potentially ignored. If there is a failure, the c-v command will
fail anyways.
Signed-off-by: Sébastien Han <seb@redhat.com>
Block deletion of CephFilesystems when there are any raw Ceph
subvolumegroups present that have subvolumes in them. Empty
subvolumegroups will not block deletion.
One important subvolume group is "csi" which is the default location
where CSI subvolumes are kept. If this group is empty, it means that
there are no PVCs created based on the CephFilesystem in question. This
also holds true if there are external consumers of the filesystem in
external cluster mode.
Similarly, if there are any subvolumegroups (for example "_nogroup",
which includes subvolumes in the filesystem root) that contain
manually-created subvolumes, Rook will also see this and block deletion.
This comes into play currently with manually-created NFS exports.
A work-in progress aims to create a Ceph-CSI NFS export provisioner
which will likely create subvolumes in the "csi" group as well. This
implementation will catch this case also.
Rook still checks for CephFilesystemSubVolumeGroups explicitly in
addition to the check added here. This is to ensure that even empty
groups will block deletion if they are created via this CR type.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
Return the error instead of hiding it, err could be non-nil for a lot of
reasons, eg: a canceled context or a non-existing object.
Signed-off-by: Sébastien Han <seb@redhat.com>
Ceph added a check recently that prevents creation of the
built-in pools that start with a dot, such as .nfs, .mgr,
and .rgw.root. These pools require a --yes-i-really-mean-it
flag to be passed to force the creation now.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
- disk UUID should only be got for "disk" type device.
- It's not necessary to fail prepare job when failing to get disk UUID
- We should consider that `sgdisk` reports UUID even if there is no GPT.
Closes: https://github.com/rook/rook/issues/9948
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
Once we are done cleaning up the content of the encrypted osd, let's
close the main LUKS device.
This avoids having dm devices on the system after the cleanup.
Closes: https://github.com/rook/rook/issues/10181
Signed-off-by: Sébastien Han <seb@redhat.com>
The CSI package needs to load clusterInfo, today this code is in the mon
package which makes the call of LoadClusterInfo impossible without
having a circular import.
Signed-off-by: Sébastien Han <seb@redhat.com>
There is a corner case where an env variable populated with the prepare
job could influence and trigger some encryption code. We have an env
variable to discover whether the prepare job will encrypt a drive or not
so let's use that instead of running code as a noop and sometimes
trigger a corner case. In this scenario, the prepare job had an "IBM_"
env variable present in the OS image (IBM OS) so the setKEKinEnv()
function would get called for no reason.
Signed-off-by: Sébastien Han <seb@redhat.com>
This introduces a new CRD to add the ability
to create rados namespace for a given
ceph block pool. Typically the name of the pool
is the name of the blockpool created by rook.
Closes: #7035
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
This commits makes Rook usable on NixOS.
NixOS system holds "installed" binaries in `/run/current-system/sw/{bin,sbin}`.
The only binaries residing outside those paths are `/usr/bin/env` and `/bin/sh`.
I got a suggestion on Slack to simply add those NixOS specific paths
Signed-off-by: Krzysztof Nazarewski <3494992+nazarewk@users.noreply.github.com>
msgr2 allows for encryption and/or compression across the wire.
Settings for enabling the encryption and compression are now
available in the cluster CR to that ceph will be automatically
configured with these settings when desired.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Since Nautilus, mgr2 is already enabled by default. No longer
does the operator need to enable msgr2 after the mons are
configured or upgraded.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
When kube authentication is used, let's clarify the error message and
hopefully lead to a better understanding on where to look for.
It could be misleading to print "token auth" when using kube auth,
although internally a token is used (the JWT from the Service
Account).
Signed-off-by: Sébastien Han <seb@redhat.com>
If a specific disk is configured to have more than one OSD per device,
or if it has a metadata device specified, exclude it from raw mode on an
individual basis.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
The osd destroy command leaves the osd id in use instead
of fully purging the osd id. If the osd id is not also
removed, it gives the impression that there is still
something that needs to be cleaned up from the old osd.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The purge job is intended to clean up all the resources
related to an OSD that is to be removed. An OSD with
a metadata, wal, and data PVC was only cleaning up the
data PVC. Now the metadata and wal PVCs will also be
cleaned up.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
This is fixing a potential race where the context is canceled and the
external cluster is trying to fetch the cluster info. If the context is
canceled then the loop will be stuck forever. Now we check for the
canceled context and return appropriately.
Signed-off-by: Sébastien Han <seb@redhat.com>
The OSD prepare job was succeeding even if the OSDs failed to be listed
by ceph-volume. These errors should be returned and fail the job, resulting
in a retry to retrieve the OSDs. Otherwise, the failure is masked and
the admin needs to dig through logs to find why the OSDs were not configured
on that node.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
If the KMS connection details for Vault have a VAULT_NAMESPACE we must
use it when building the vault client to auto-detect the KV internal
version.
Signed-off-by: Sébastien Han <seb@redhat.com>
The reconcile was skipping updating most pool properties for
object stores. The implementation of pools between the file,
object, and pool controllers had some duplicate code, so
this change also factors out the common code for better
reuse in a single place. Anytime a pool is created or updated,
it will now consistently update all the pool properties
that are expected to be modifiable.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
We can eliminate the OSD creation code for Ceph < v15.2.0 because Nautilus(v14.2.z)
is no longer supported. In other words, we can now expect that all newly-created
OSDs on PVC are raw mode.
Please note that the existing lvm mode OSDs on PVCs still be supported after this change.
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
If the cluster is external, users might still want to take advantage of
the ceph-csi configmap configuration for ceph filesystem subvolume
group. This configuration is handled by the controller whenever a
cephFilesystemSubVolumeGroup CR is created or deleted.
If the present cluster CephCluster is external, the controller
(cephFilesystemSubVolumeGroup) won't handle creation nor deletion of sub volume groups.
It just assumes that the given name is correct and the controller goes ahead with the
ceph-csi configuration.
Closes: #9551
Signed-off-by: Sébastien Han <seb@redhat.com>
In volume.go, remove if condition in `callCephVolume`
method where we always pass false.
In exec.go, removing `ExecuteCommandWithFile*` methods
which was not used anywhere.
Signed-off-by: subhamkrai <srai@redhat.com>
It's easier to read the config if there is a new line before instead
having a truncated version like:
```
2022-01-26 06:39:41.421373 D | cephclient: config file @ /etc/ceph/ceph.conf: [global]
fsid = 0650a1d5-d688-4575-97fc-7b3052a8a3dd
mon initial members = a
mon host = [v2:10.104.192.209:3300,v1:10.104.192.209:6789]
[client.admin]
keyring = /var/lib/rook/rook-ceph/client.admin.keyring
```
Now we have:
```
2022-01-26 06:39:41.421373 D | cephclient: config file @ /etc/ceph/ceph.conf:
[global]
fsid = 0650a1d5-d688-4575-97fc-7b3052a8a3dd
mon initial members = a
mon host = [v2:10.104.192.209:3300,v1:10.104.192.209:6789]
[client.admin]
keyring = /var/lib/rook/rook-ceph/client.admin.keyring
```
Signed-off-by: Sébastien Han <seb@redhat.com>
If we fail fetching the ceph version let's also print the stdout since
ceph tends to put the error here too.
Signed-off-by: Sébastien Han <seb@redhat.com>
If the pool application is already set, skip setting it again
to avoid a warning message being logged that it is already
set.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
For better consistency with IBM_KP_SERVICE_API_KEY and use
"ibmkeyprotect" as a provider name for more clarity.
Signed-off-by: Sébastien Han <seb@redhat.com>
The cluster-wide encryption feature now has a new supported key
management system: IBM Key protect. More information about the backend
can be found in Rook documentation under the
Today's implementation stores OSD encryption keys has "Standard" keys.
Signed-off-by: Sébastien Han <seb@redhat.com>
This introduces a new CRD to add the ability to create subvolumegroup
for a given ceph filesystem volume. Typically the name of the volume is
the name of the filesystem created by rook.
Closes: https://github.com/rook/rook/issues/7036
Signed-off-by: Sébastien Han <seb@redhat.com>
This is handling a tricky scenario where the OSD deployment is manually
removed and the OSD never reconvers. This is unlikely to happen, but
still OSD should be able to run after that action. Essentially after a
manual deletion, we need to run the prepare job again to re-hydrate the
OSD information so that the OSD deployment can be deployed.
On encryption, it is a little bit tricky since ceph-volume list again
the main block won't return anything, so we need to target the encrypted
block to list.
There is another case this PR does not handle, which is the removal of
the OSD deployment and then the node is restarted. This means that the
encrypted container is not opened anymore. However, opening it requires
more work like writing the key on the filesystem (if not coming from the
Kubernete secret, eg,. KMS vault) and then run luksOpen. This is an
extreme corner case probably not worth worrying about for now.
Signed-off-by: Sébastien Han <seb@redhat.com>
Travis Nielsen
Eng Zer Jun
Liang Zheng
Sébastien Han
Satoru Takeuchi
Josh Soref
Blaine Gardner
Madhu Rajanna
subhamkrai
Krzysztof Nazarewski
Divyansh Kamboj