With octopus coming to end of life, we remove support from
Rook for deploying Ceph Octopus and assume a min version of
Pacific v16. Any checks for octopus or earlier are removed
from the reconciles since they are obsolete.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Block deletion of CephFilesystems when there are any raw Ceph
subvolumegroups present that have subvolumes in them. Empty
subvolumegroups will not block deletion.
One important subvolume group is "csi" which is the default location
where CSI subvolumes are kept. If this group is empty, it means that
there are no PVCs created based on the CephFilesystem in question. This
also holds true if there are external consumers of the filesystem in
external cluster mode.
Similarly, if there are any subvolumegroups (for example "_nogroup",
which includes subvolumes in the filesystem root) that contain
manually-created subvolumes, Rook will also see this and block deletion.
This comes into play currently with manually-created NFS exports.
A work-in progress aims to create a Ceph-CSI NFS export provisioner
which will likely create subvolumes in the "csi" group as well. This
implementation will catch this case also.
Rook still checks for CephFilesystemSubVolumeGroups explicitly in
addition to the check added here. This is to ensure that even empty
groups will block deletion if they are created via this CR type.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
If the filesystem is in a bad state such as never being
created successfully in the first place, the filesystem
removal will continuously fail and prevent the CR from
being removed. Thus, we treat the filesystem removal
as a best effort and log the issue instead of failing
the uninstall reconcile.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The reconcile was skipping updating most pool properties for
object stores. The implementation of pools between the file,
object, and pool controllers had some duplicate code, so
this change also factors out the common code for better
reuse in a single place. Anytime a pool is created or updated,
it will now consistently update all the pool properties
that are expected to be modifiable.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The built-in pools device_health_metrics and .nfs created by ceph
need to be configured for replicas, failure domain, etc.
To support this, we allow the pool to be created as a CR.
Since K8s does not support underscores in the resource names
the operator must translate this special pool name into
the name expected by ceph.
This also sets the basis for allowing filesystem data
pools to specify the desired pool name instead of requiring
a generated name.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The MDS core team suggested with deploy the MDS daemon first and then do
the filesystem creation and configuration. Reversing the sequence lets
us avoid spurious FS_DOWN warnings when creating the filesystem.
Closes: #8745
Signed-off-by: Sébastien Han <seb@redhat.com>
Multiple filesystems are supported as of the Ceph Pacific release. So we
remove the flag from the operator configuration and just have a ceph
version check instead.
The Operator configuration option `ROOK_ALLOW_MULTIPLE_FILESYSTEMS`
has been removed in favor of simply verifying the Ceph version is
at least Pacific.
Multiple filesystems are stable since Ceph Pacific.
So users who had `ROOK_ALLOW_MULTIPLE_FILESYSTEMS` enabled will
need to update their Ceph version to Pacific.
Closes: https://github.com/rook/rook/issues/7183
Signed-off-by: Sébastien Han <seb@redhat.com>
we should not use --force when creating filesystem(fs)
as it will recreate fs which will leads to data loss
when `preserveFilesystemOnDelete` is set to 'true'.
this commit removes --force argument when create fs.
Signed-off-by: subhamkrai <srai@redhat.com>
This fixes libraries that were being imported multiple times in
pkg/operator/ceph/file and its subpackages.
Signed-off-by: Lars Lehtonen <lars.lehtonen@gmail.com>
It's better to validate ownerReferences when setting them. In addition, we should use
controllerrutil.Set{Controller,Owner}Reference, that have such validation, as possible.
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
The upcoming Rook release 1.6 will support Ceph Pacific which introduces
stable support for multiple Ceph Filesystems in the same cluster. So we
now allow the creation of multiple Filesystem if the release is Pacific.
Also, we remove the Operator config flag which was not necessary.
Here I have 2 filesystems:
```
[root@rook-ceph-tools-6b4889fdfd-vncck /]# ceph fs ls
name: myfs, metadata pool: myfs-metadata, data pools: [myfs-data0 ]
name: myfs2, metadata pool: myfs2-metadata, data pools: [myfs2-data0 ]
```
```
cluster:
id: cc747950-589a-4862-bad2-ff7de1217b1a
health: HEALTH_WARN
mons a,b,c are low on available space
6 pool(s) have no replicas configured
services:
mon: 3 daemons, quorum a,b,c (age 2d)
mgr: a(active, since 2d)
mds: myfs:1 myfs2:1 {myfs2:0=myfs2-b=up:active,myfs:0=myfs-b=up:active} 2 up:standby-replay
osd: 3 osds: 3 up (since 2m), 3 in (since 6d)
data:
pools: 6 pools, 717 pgs
objects: 44 objects, 4.2 KiB
usage: 46 MiB used, 90 GiB / 90 GiB avail
pgs: 717 active+clean
io:
client: 1.8 KiB/s rd, 4 op/s rd, 0 op/s wr
progress:
Global Recovery Event (45s)
[===========================.]
```
Signed-off-by: Sébastien Han <seb@redhat.com>
When new pools are added to an already existing filesystem, we would
create them but not actually associate them to the filesystem. Ensure
that we do this.
Also, while we're here, we refactor the CreateFilesystem test: before,
we were not properly testing creating the filesystem from scratch
because we would return valid JSON for *every* ceph command. Change this
by making the first run return an error so that we get some better
coverage.
After this, we also add a check for the correctness of adding new pools
to the CreateFilesystem unittest.
Fixes#5876
Signed-off-by: Lalit Maganti <lalitm@google.com>
Found by running the following command:
codespell -S .git,*.png,*.jpg -L \
aks,keyserver,atleast,dne,ser,ist,files\',ba,dum,iam,te -f -H
Signed-off-by: Mateusz Gozdek <mgozdekof@gmail.com>
Pools in stretch clusters will all use the same crush rule that
is generated by the operator when configuring the cluster for
stretch mode, with replica 4 and two replicas per failure domain.
Pools cannot create new crush rules, so we require that replica be
4 when the pools is created, EC is not allowed, and no new
crush rule will be created.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Due to #6492, preservePoolsOnDelete is not useful at all for CephFS;
after the filesystem is deleted, the leftover pools cannot be
reassocaited with a newly created filesystem without wiping all
metadata. The only way we can actually preserve data is keeping around
the entire filesystem.
This commit implements a `preserveFilesystemOnDelete` option which work
similar to the existing pool preservation option but instead keeps the
whole CephFS while taking it down and removing all MDSes.
This commit also changes all documentation to refer to this new option
with the intent of essentially deprecating `preservePoolsOnDelete`. IMO,
keeping around `preservePoolsOnDelete` is actively harmful because it
lulls users into thinking their data will be safe but, in reality,
recovering from this situation is highly complex and has large potential
for data loss.
Signed-off-by: Lalit Maganti <lalitm@google.com>
The CRUSH map does not allow duplicate values (nor keys) for the
labels. That means that if the currently hardcoded label value
`default` occurs anywhere else in the tree (for example, because
some of the topology labels provided by the cloud provider use it),
the cluster cannot sucessfully spawn.
By giving the user control over the label value used for the root
CRUSH map label, it is possible to adapt the cluster to this
situation.
To be able to actually use the `default` value, the root=default
node which is created by default by ceph itself has to be removed;
since that node is accompanied by a replicated crush rule, we have
to remove that rule, too.
The integration tests are modified to sometimes use the custom
crushRoot (based on an arbitrary criterium) to get coverage across
the various suites. Separate tests could be added, but were not
deemed necessary at this point.
Fixes#4993.
Signed-off-by: Jonas Schäfer <jonas.schaefer@cloudandheat.com>
this commit handles all the gosec g601
error code (i.e Implicit memory aliasing
of items from a range statement).
Signed-off-by: subhamkrai <subhamkumarrai03@gmail.com>
ceph-file-controller gets stuck in loops, this happens when it tires to delete the filesystem which doesn't exist.Filesystem should only be deleted when it exist.
Signed-off-by: RAJAT SINGH <rajasing@redhat.com>
To provide more context for executing commands in a ceph cluster,
the full clusterInfo is now passed to the ceph execution commands.
All information about the cluster will now be available throughout
all the areas of the operator. The namespace, ceph credentials,
mon endpoints, and other info is a core part of that cluster info.
Arguments passed through the controllers are also simplified for
mons, mgr, osds, and other daemons where the parameters had
become too complex.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The ceph config and client packages are conceptually the same.
To avoid circular dependencies in some cases, we simplify by combining
the packages into the client package.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
ceph mds deactivate is not valid command for Nautilus, changing the mechnaism of upgrading so that it can be done in the way Nautilus does it.
Closes: https://github.com/rook/rook/issues/5278
Signed-off-by: RAJAT SINGH <rajasing@redhat.com>
Remove specific mds config added to the mon configuration database when deleting a CephObjectstore
or when scaling down number of mds
Also remove mds CephX key
Signed-off-by: n.fraison <n.fraison@criteo.com>
CSI CephFS provisioner fails to provision a Persistent Volume after upgrading from Rook Ceph Operator 1.1.2 to 1.1.7.
This issue can be fixed by removing givePoolAppTag function whenever there is an appName cephFs.
Signed-off-by: RAJAT SINGH <rajasing@redhat.com>
The old Filesystem's pool is not getting updated when changed with the new File system.
So, we added two helper functions namely generateDataPoolNames and generateMetaDataPoolName that will generate the names of the pools and then change those pool's values using the SetPoolSize method.
Signed-off-by: RAJAT SINGH <rajasing@redhat.com>
The pools had some legacy structs that translated between the
ceph.v1 types used by the CRDs and the internal implementation
of the pools. This simplifies the pool implementation by removing
the intermediate model and leaving us only with the ceph v1
pool types and no unnecessary translation.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The CRD watcher has been replaced by the new controller-runtime
framework.
This brings robustness in our operator, meaning that any resources that
are modified will be reconciled into the desired state.
Closes: https://github.com/rook/rook/issues/4940
Signed-off-by: Sébastien Han <seb@redhat.com>
We now always perform ok-to-stop change when the deployment is going to
be updated. This could be for various reasons like change in the spec or
a new Ceph image.
In any case, if Ceph resources are going to be restarted this must be
done carefully.
Also trimmed the isUpgrade variable, the upcoming commit will remove the
need of that variable.
Keeping isUpgrade from the controller code since we need it to detect
whether an orchestration upgrade should be triggered or not (based on
the ceph cluster status).
Also, we keep it on the child controllers so we don't go into an
orchestration phase if the cluster image hasn't changed. The child does
not need to be updated/go into its orchestration phase because the
cluster CR changed.
Closes: https://github.com/rook/rook/issues/4379
Signed-off-by: Sébastien Han <seb@redhat.com>
When using the "errors" package, using `%+v` (extended format),
each Frame of the error's StackTrace will be printed in detail.
Let's only print `%v` to print the error.
If the error has a Cause it will be printed recursively.
Basically `%+v` has been replaced with `%v` for all `error` type
interfaces, whether the logger is Info, Warning or Error.
Signed-off-by: Sébastien Han <seb@redhat.com>
Failure to create a keyring prior to creating a deployment
dependent on that keyring created a race conditition resulting
in an intermittent and unnecessary pod failure. This change
creates a keyring prior to deployment creation for MDS
daemons.
Partially fixes: rook#4089
Signed-off-by: egafford <egafford@redhat.com>
The command `fs new` already enable the application pool settings for
the cephfs pools so we don't need to run that command again.
Signed-off-by: Sébastien Han <seb@redhat.com>
The resources for rgw, mds, and nfs were being cleaned up explicitly when
the CR was deleted. This is legacy from before the owner references were
set on the CRs. Now the owner references are set to the appropriate CRs so
the resources will be cleaned up upon deletion of the specific CR
rather than the cluster CR.
Signed-off-by: travisn <tnielsen@redhat.com>
A new CRD property `PreservePoolsOnDelete` has been added to Filesystem(fs) and
Object Store(os) resources in order to increase protection against data loss.
If it is set to `true`, associated pools won't be deleted when the main
resource(fs/os) is deleted. Creating again the deleted fs/os with the same name
will reuse the preserved pools.
Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
From now on, Rook will only perform check before upgrades when there is
an actual upgrade. So if the Ceph image changed and a new version is
desired Rook will go through all the daemons and update them one by one
and perform checks in between.
Closes: https://github.com/rook/rook/issues/3583
Signed-off-by: Sébastien Han <seb@redhat.com>
When configuring an external cluster the orchestration of discover and all the ceph
daemons will be skipped. The operator will still watch for the creation of
crds for this namespace. When a filesystem, object, object user, or
ganesha crd are created, the operator will simply print an error
to the log.
Signed-off-by: travisn <tnielsen@redhat.com>
Signed-off-by: Sébastien Han <seb@redhat.com>
It's quite convinient to expose /var/log/ceph so that we can decide to
activate logs locally on the machine and see what's going on.
This is only a placeholder when a daemon is stuck crashlooping and we
want to allow administrator to gather log files.
We still do not log on file but this can be activated via a config
option passed to the centralized config option store.
For some daemons, which typically do not store any data (rgw, rbd-mirror
and mds) we had to propagate dataDirHostPath from the cluster spec to
each creation call so that the bindmount can happen.
Fixes: https://github.com/rook/rook/issues/2881
Signed-off-by: Sébastien Han <seb@redhat.com>
A ceph filesystem was brought down by marking the filesystem
as down and individually failing the MDSes. From nautilus onwards,
this could be more simply and efficiently done by using
`fs fail` command. See,
https://github.com/ceph/ceph/commit/4c49f165ec1Resolves: #2841
Signed-off-by: Ramana Raja <rraja@redhat.com>
The basic idea here is to replace the release name in the ClusterInfo
object with the fine grained version information queried at runtime.
This patch also removes the Name field from the cluster spec, which was
also runtime determined and was redundant with ClusterInfo relase name.
Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
From ceph mimic (13.x) onwards, `mds deactivate` command is
deprecated. To scale down the number of active MDSes of a filesystem,
it's sufficient to set the desired number of active MDSes using
'max_mds' setting, and is no longer required to individually
deactivate the unwanted MDSes.
Resolves: #2725
Signed-off-by: Ramana Raja <rraja@redhat.com>
The MDS mds_standby_for_*, mon_force_standby_active, and mds_standby_replay
configuration options have been obsoleted. Instead, the operator may now set
the new "allow_standby_replay" flag on the CephFS file system. This setting
causes standbys to become standby-replay for any available rank in the file
system.
Closes: https://github.com/rook/rook/issues/2768
Signed-off-by: Sébastien Han <seb@redhat.com>
Josh Soref
Travis Nielsen
Blaine Gardner
FengJiankui
Denis Egorenko
Sébastien Han
subhamkrai
Lars Lehtonen
Satoru Takeuchi
Lalit Maganti
Mateusz Gozdek
Jonas Schäfer
subhamkrai
RAJAT SINGH
Madhu Rajanna
n.fraison
egafford
Juan Miguel Olmo Martínez
Ramana Raja
Noah Watkins