This implements the "Ceph Config via Ceph Cluster CRD" design document
as a `cephConfig:` structure on the CRD.
This also fixes the `yq` commands used to manipulate the
`cluster-test.yaml` that caused CI issues for this PR and potentially
unknowingly others.
Signed-off-by: Alexander Trost <galexrt@googlemail.com>
Originally we create it using this cmd
ceph fs subvolume create <vol_name> <subvol_name>
So we can have 2 variables filesystem and subvolume name,
Currently the CR doesn't allow us to make subvolume-name
as constant as needed to "csi" because of k8s limitations
Signed-off-by: parth-gr <paarora@redhat.com>
This commit adds new CSIDriverOptions section in
cephCluster CR. This section contains settings
for read affinity and kernel+fuse Mount options
These settings will be injected directly into
rook-ceph-csi-config cm to be applicable per
ceph cluster.
Signed-off-by: Rakshith R <rar@redhat.com>
This patch adds `pgHealthyRegex` field to DisruptionManagementSpec.
`pgHealthyRegex` is a regular expression that is used to determine which
PG states should be considered healthy. The default value of
`pgHealthyRegex` is:
^(active\+clean|active\+clean\+scrubbing|active\+clean\+scrubbing\+deep)$
which is effectively the same as before.
Signed-off-by: Ryotaro Banno <ryotaro.banno@gmail.com>
The activeStandby property of the filesystem CR was not
taking effect when changed from true to false. The standby
was only being enabled when true, but never applied when
changed to false.
Signed-off-by: travisn <tnielsen@redhat.com>
The json tag in struct MonMap had a spelling error where "json" was incorrectly splled as "josn". This commit corrects the spelling to "json".
Signed-off-by: go-bai <599500688@qq.com>
Dashboard ac-user-create cmd was taking more time
then the usual ceph command to run,
So increased the timeout to run the cmd, and
now dashboard admin user is sucessfully created
Closes: https://github.com/rook/rook/issues/12113
Signed-off-by: parth-gr <paarora@redhat.com>
few functions got change as they were deprecated
for ex: ioutil.Readfile change to os.Readfile
ioutil.TempFile change to os.CreateTemp
And fixed golang-ci-lint-issues
Signed-off-by: parth-gr <paarora@redhat.com>
Mons that are configured on msgr1 will be using the v1 port
in the csi driver since only one port is allowed in the configmap.
To allow mons to be started on v2 while keeping quorum with the
v1 mons, the v1 mons must continue running on the v1 port.
So even if msgr2 is required, allow existing v1 mons to continue
running on that port.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The ceph.conf was being written with 0x444 permissions, which means
the file could not be updated if it already exists. Specifically, when
an osd is purged, if the operation is retried multiple times, only
the first operation was succeeding. The subsequent requests to purge
an osd were failing due to the config already existing. If we make
the config writeable with 0x644 then the operator can always write the file.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The mons that are out of quorum may cause ceph commands to timeout
or fail unnecessarily trying to connect to a mon that is no longer
online. Now the mon health check will update the mon endpoints configmap
when a mon is detected out of quorum. This also means that if the
operator is restarted during a mon failover, the failed mon will no
longer remain in the ceph.conf, thus allowing the quorum to be more
likely to respond to the mons that are still in quorum.
The update for mons out of quorum only applies if other mons are in
quorum. If quorum is down, the configmap will not keep track of the offline
mons since too many are offline.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Run the ganesha-rados-grace command in a remote pod when multus
networking is enabled.
Signed-off-by: parth-gr <paarora@redhat.com>
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
The tool gofmt in go 1.19 requires certain formatting
in the comments section for better rendering.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
With octopus coming to end of life, we remove support from
Rook for deploying Ceph Octopus and assume a min version of
Pacific v16. Any checks for octopus or earlier are removed
from the reconciles since they are obsolete.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Block deletion of CephFilesystems when there are any raw Ceph
subvolumegroups present that have subvolumes in them. Empty
subvolumegroups will not block deletion.
One important subvolume group is "csi" which is the default location
where CSI subvolumes are kept. If this group is empty, it means that
there are no PVCs created based on the CephFilesystem in question. This
also holds true if there are external consumers of the filesystem in
external cluster mode.
Similarly, if there are any subvolumegroups (for example "_nogroup",
which includes subvolumes in the filesystem root) that contain
manually-created subvolumes, Rook will also see this and block deletion.
This comes into play currently with manually-created NFS exports.
A work-in progress aims to create a Ceph-CSI NFS export provisioner
which will likely create subvolumes in the "csi" group as well. This
implementation will catch this case also.
Rook still checks for CephFilesystemSubVolumeGroups explicitly in
addition to the check added here. This is to ensure that even empty
groups will block deletion if they are created via this CR type.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
Ceph added a check recently that prevents creation of the
built-in pools that start with a dot, such as .nfs, .mgr,
and .rgw.root. These pools require a --yes-i-really-mean-it
flag to be passed to force the creation now.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
This introduces a new CRD to add the ability
to create rados namespace for a given
ceph block pool. Typically the name of the pool
is the name of the blockpool created by rook.
Closes: #7035
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
msgr2 allows for encryption and/or compression across the wire.
Settings for enabling the encryption and compression are now
available in the cluster CR to that ceph will be automatically
configured with these settings when desired.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Since Nautilus, mgr2 is already enabled by default. No longer
does the operator need to enable msgr2 after the mons are
configured or upgraded.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
This is fixing a potential race where the context is canceled and the
external cluster is trying to fetch the cluster info. If the context is
canceled then the loop will be stuck forever. Now we check for the
canceled context and return appropriately.
Signed-off-by: Sébastien Han <seb@redhat.com>
The reconcile was skipping updating most pool properties for
object stores. The implementation of pools between the file,
object, and pool controllers had some duplicate code, so
this change also factors out the common code for better
reuse in a single place. Anytime a pool is created or updated,
it will now consistently update all the pool properties
that are expected to be modifiable.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
If the cluster is external, users might still want to take advantage of
the ceph-csi configmap configuration for ceph filesystem subvolume
group. This configuration is handled by the controller whenever a
cephFilesystemSubVolumeGroup CR is created or deleted.
If the present cluster CephCluster is external, the controller
(cephFilesystemSubVolumeGroup) won't handle creation nor deletion of sub volume groups.
It just assumes that the given name is correct and the controller goes ahead with the
ceph-csi configuration.
Closes: #9551
Signed-off-by: Sébastien Han <seb@redhat.com>
It's easier to read the config if there is a new line before instead
having a truncated version like:
```
2022-01-26 06:39:41.421373 D | cephclient: config file @ /etc/ceph/ceph.conf: [global]
fsid = 0650a1d5-d688-4575-97fc-7b3052a8a3dd
mon initial members = a
mon host = [v2:10.104.192.209:3300,v1:10.104.192.209:6789]
[client.admin]
keyring = /var/lib/rook/rook-ceph/client.admin.keyring
```
Now we have:
```
2022-01-26 06:39:41.421373 D | cephclient: config file @ /etc/ceph/ceph.conf:
[global]
fsid = 0650a1d5-d688-4575-97fc-7b3052a8a3dd
mon initial members = a
mon host = [v2:10.104.192.209:3300,v1:10.104.192.209:6789]
[client.admin]
keyring = /var/lib/rook/rook-ceph/client.admin.keyring
```
Signed-off-by: Sébastien Han <seb@redhat.com>
If we fail fetching the ceph version let's also print the stdout since
ceph tends to put the error here too.
Signed-off-by: Sébastien Han <seb@redhat.com>
If the pool application is already set, skip setting it again
to avoid a warning message being logged that it is already
set.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
parth-gr
Alexander Trost
Rakshith R
Ryotaro Banno
travisn
subhamkrai
sp98
go-bai
Madhu Rajanna
Miles Simpson
Blaine Gardner
zhucan
Liang Zheng
Sébastien Han
Josh Soref
Divyansh Kamboj
Eng Zer Jun