Even if we can use raw mode, do NOT use raw mode on disks. Ceph bluestore disks can
sometimes appear as though they have "phantom" Atari (AHDI) partitions created on them
when they don't in reality. This is due to a series of bugs in the Linux kernel when it
is built with Atari support enabled. This behavior does not appear for raw mode OSDs on
partitions, and we need the raw mode to create partition-based OSDs. We cannot merely
skip creating OSDs on "phantom" partitions due to a bug in `ceph-volume raw inventory`
which reports only the phantom partitions (and malformed OSD info) when they exist and
ignores the original (correct) OSDs created on the raw disk.
Resolves https://github.com/rook/rook/issues/7940
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
Currently it is not possible to set resource limits based on device classes
for different OSDs. Now adding an ability to use predefined keys for main
cluster spec Resource section to reflect resource limits for different OSDs
with different device classes.
Closes: https://github.com/rook/rook/issues/8007
Signed-off-by: Denis Egorenko <degorenko@mirantis.com>
When configuring encrypted on-pvc clusters we now set the cluster fsid
in the LUKS header. If needed we can then determine if the encrypted
block is part of our cluster or not. We also attach the pvc_name in case
it might become useful.
Closes: https://github.com/rook/rook/issues/7991
Signed-off-by: Sébastien Han <seb@redhat.com>
The deviceClass was only being set for LVM mode OSDs and was
missed for the raw mode for non-PVC. Now the deviceClass will be set
for all expected scenarios.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
This fixes cases where raw provisioning mode still cannot be used for
Rook v1.6 and Ceph Pacific clusters where json.Unmarshal fails to read
the ceph-volume output after OSD provisioning.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
This mops up the last of the multiple-imports within ceph-related
packages, in pkg/daemon/ceph/agent/flexvolume/manager/ceph and
pkg/daemon/ceph/osd.
Signed-off-by: Lars Lehtonen <lars.lehtonen@gmail.com>
Update OSDs in parallel per the design in
design/ceph/update-osds-in-parallel.md
The max number of OSDs updated in parallel is currently fixed at 20.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
When c-v runs after the end of the prepare sequence, it will look for
lvm and raw-based OSDs. The raw mode detects LVM OSDs so we must trim
the list of OSD, only to report the LVM ones if their ID exists.
Closes: https://github.com/rook/rook/issues/7359
Signed-off-by: Sébastien Han <seb@redhat.com>
For simple OSD scenario that don't require the following:
* an encrypted osd
* an osd with a backing device
* multiple osd on the same block device
The OSD will be prepared using the ceph-volume raw mode which does
not use LVM on non-pvc case.
Closes: https://github.com/rook/rook/issues/4768
Signed-off-by: Sébastien Han <seb@redhat.com>
block if not defined was used with osdInfo.Device so not empty anymore
in the loop, which means it will have the same value for all OSDs in the
loop.
This is only called when running on PVC, where a single device is
present so the loop has a single iteration, so not a big issue.
Signed-off-by: Sébastien Han <seb@redhat.com>
With 5f57b6fd3a, we introduced a
regression in the validation code.
If the OSD PV comes from another cluster, GetCephVolumeRawOSDs must
return an error.
The prepare job will crash loop but at least with a useful and
comprehensive message.
Signed-off-by: Sébastien Han <seb@redhat.com>
14.2.15 lvm batch command prepare report changes
output format. This commit skips md check if ceph version
is greater than 14.2.13.
Signed-off-by: shenjiatong <yshxxsjt715@gmail.com>
The deviceClass property was being ignored when creating the
non-pvc OSDs. Now the deviceClass will be specified as a property
for individual devices, all devices on a node, or all OSDs in the
cluster, depending on the level where the config is applied in the
cluster CR.
Co-authored-by: shenjiatong <yshxxsjt715@gmail.com>
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The code was assuming that devices were passed by the user as
"/dev/sda", this is bad! We all know people should be using paths like
/dev/disk/by-id so we must support them.
Closes: https://github.com/rook/rook/issues/6685
Signed-off-by: Sébastien Han <seb@redhat.com>
When the Ceph cluster runs on PVC and the OSDs are encrypted we can
store LUKS's Key Encryption Key inside a Key Management System. Today,
Rook only supports HashiCorp Vault: https://www.vaultproject.io/
The CephCluster has now a new "security" field which will plug onto the
KMS. Here is an example:
security:
kms:
tokenSecretName: <name of the secret containing a Vault token, used
to authenticate>
connectionDetails: < a map of strings containing connection
information>
Refer to the ceph-cluster-crd documentation to lear more.
Closes: https://github.com/rook/rook/issues/6105
Signed-off-by: Sébastien Han <seb@redhat.com>
When deploying on non-PVC, we still need to gather c-v logs on failures.
Now if the bootstrap fails, the c-v log will be printed.
Signed-off-by: Sébastien Han <seb@redhat.com>
callCephVolume() was using our exec interface with CombinedOutput which
means return both stdout and stderr. This is fine when preparing OSDs
since we read back the stdout and also want any errors if any (stderr).
However when listing OSD we only want the output since ceph-volume tends
to mix stdout and stderr in the output and does not always silence the
warning types/harmless errors. If the listing has a problem we report it
by printing the ceph-volume log file.
Signed-off-by: Sébastien Han <seb@redhat.com>
Now the metadata pvc will also be encrypted if the storageClassDeviceSet
has the "encrypted" flag turned on.
Signed-off-by: Sébastien Han <seb@redhat.com>
Based on the feature of metadata PVC for OSD: #4818, add the wal device.
For this, you need to create a new volumeClaimTemplates, its name must be "wal", if "wal" is
set, "metadata" device will refers specially for block.db device, and "wal" refers for block.wal
device in BlueStore.
A template will look like this:
```
volumeClaimTemplates:
- metadata:
name: data
spec:
resources:
requests:
storage: 10Gi
# IMPORTANT: Change the storage class depending on your environment (e.g. local-storage, gp2)
storageClassName: gp2
volumeMode: Block
accessModes:
- ReadWriteOnce
- metadata:
name: metadata
spec:
resources:
requests:
storage: 6Gi
# IMPORTANT: Change the storage class depending on your environment (e.g. local-storage, io1)
storageClassName: io1
volumeMode: Block
accessModes:
- ReadWriteOnce
- metadata:
name: wal
spec:
resources:
requests:
storage: 6Gi
# IMPORTANT: Change the storage class depending on your environment (e.g. local-storage, io1)
storageClassName: io1
volumeMode: Block
accessModes:
- ReadWriteOnce
```
We now map block, block.db and block.wal directly inside the container instead of
running ceph-volume activate.
Signed-off-by: Chen, Tingjie <tingjie.chen@intel.com>
We can now encrypted OSD device that were provisioned via a storage
class using the PV interface.
The encryption works at the storageClassDeviceSets level, which means we
can have encrypted and non-encrypted sets.
Using the new key `encrypted` we can turn it on.
Signed-off-by: Sébastien Han <seb@redhat.com>
File permissions need to be reduced to the minimum to avoid config
files from being accessed. While a connection to the pod would be
required to access the files, min privileges are set just as
another layer of security. This fixes gosec error G306.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
This commit fixes where there are areas in which files and directories are
written and created with statically defined permissions. Many of these
permissions are rather open, potentially allowing other system tenants to
view and interact with their contents, which may be sensitive.
Fixed G303: Poor file permissions used for directory by using the most narrow permission
Resoves: https://github.com/rook/rook/issues/4580
Signed-off-by: Nizamudeen <nia@redhat.com>
this commit suppress the gosec errors for
g204: Audit use of command execution.
g304: File path provided as taint input.
g101: Look for hard coded credentials.
Signed-off-by: subhamkrai <subhamkumarrai03@gmail.com>
To ensure a file handle is closed, we defer the close command
so it is guaranteed to run when the method returns. The closing
of the file handle is not going to fail in our usage since we aren't
using the SetDeadline on the files that would cancel a request
and return an error.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Now, we can not only cleanup monitor data, logs and crashes but the
disks too. As part of the cleanupPolicy CR spec, we have a new setting
called sanitizeDisk which holds more details:
* method: indicates if the entire disk should be sanitized or simply ceph's metadata.
Possible choices are 'complete' or 'quick' (default)
* dataSource: indicate where to get random bytes from to write on the disk.
Possible choices are 'zero' (default) or 'random'
Using random sources will consume entropy from the system and will take much more time then the zero source
* iteration: overwrite N times instead of the default (1). Takes an integer value
See the documentation for more details.
Signed-off-by: Sébastien Han <seb@redhat.com>
To provide more context for executing commands in a ceph cluster,
the full clusterInfo is now passed to the ceph execution commands.
All information about the cluster will now be available throughout
all the areas of the operator. The namespace, ceph credentials,
mon endpoints, and other info is a core part of that cluster info.
Arguments passed through the controllers are also simplified for
mons, mgr, osds, and other daemons where the parameters had
become too complex.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Add the ability to provision Ceph OSDs with Drive Groups.
This adds Drive Groups to the CephCluster CRD, and it sets code
in place for propagating this config to the OSD provisioning pod.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
Updated initializeBlockPVC method to add
crushDeviceClass along with
immediateExecuteArgs instead of
metadataArg. Also added unit tests for this method.
Signed-off-by: subhamkrai <subhamkumarrai03@gmail.com>
Now, we can not only cleanup monitor data, logs and crashes but the
disks too. As part of the cleanupPolicy CR spec, we have a new setting
called sanitizeDisks which holds more details:
* confirmation: the confirmation message to sanitize disks,
use "yes-really-sanitize-disks" to confirm
This will **only** wipe the metadata, so it's a fast cleanup allowing
you to re-install later but won't remove all the data from the drive.
Signed-off-by: Sébastien Han <seb@redhat.com>
subhamkrai
Jonas Zeiger
Blaine Gardner
Travis Nielsen
Denis Egorenko
Sébastien Han
Lars Lehtonen
shenjiatong
Daichi Sakaue
Chen, Tingjie
Satoru Takeuchi
Nizamudeen
subhamkrai
Blaine Gardner