The MDS core team suggested with deploy the MDS daemon first and then do
the filesystem creation and configuration. Reversing the sequence lets
us avoid spurious FS_DOWN warnings when creating the filesystem.
Closes: #8745
Signed-off-by: Sébastien Han <seb@redhat.com>
Multiple filesystems are supported as of the Ceph Pacific release. So we
remove the flag from the operator configuration and just have a ceph
version check instead.
The Operator configuration option `ROOK_ALLOW_MULTIPLE_FILESYSTEMS`
has been removed in favor of simply verifying the Ceph version is
at least Pacific.
Multiple filesystems are stable since Ceph Pacific.
So users who had `ROOK_ALLOW_MULTIPLE_FILESYSTEMS` enabled will
need to update their Ceph version to Pacific.
Closes: https://github.com/rook/rook/issues/7183
Signed-off-by: Sébastien Han <seb@redhat.com>
we should not use --force when creating filesystem(fs)
as it will recreate fs which will leads to data loss
when `preserveFilesystemOnDelete` is set to 'true'.
this commit removes --force argument when create fs.
Signed-off-by: subhamkrai <srai@redhat.com>
This fixes libraries that were being imported multiple times in
pkg/operator/ceph/file and its subpackages.
Signed-off-by: Lars Lehtonen <lars.lehtonen@gmail.com>
It's better to validate ownerReferences when setting them. In addition, we should use
controllerrutil.Set{Controller,Owner}Reference, that have such validation, as possible.
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
The upcoming Rook release 1.6 will support Ceph Pacific which introduces
stable support for multiple Ceph Filesystems in the same cluster. So we
now allow the creation of multiple Filesystem if the release is Pacific.
Also, we remove the Operator config flag which was not necessary.
Here I have 2 filesystems:
```
[root@rook-ceph-tools-6b4889fdfd-vncck /]# ceph fs ls
name: myfs, metadata pool: myfs-metadata, data pools: [myfs-data0 ]
name: myfs2, metadata pool: myfs2-metadata, data pools: [myfs2-data0 ]
```
```
cluster:
id: cc747950-589a-4862-bad2-ff7de1217b1a
health: HEALTH_WARN
mons a,b,c are low on available space
6 pool(s) have no replicas configured
services:
mon: 3 daemons, quorum a,b,c (age 2d)
mgr: a(active, since 2d)
mds: myfs:1 myfs2:1 {myfs2:0=myfs2-b=up:active,myfs:0=myfs-b=up:active} 2 up:standby-replay
osd: 3 osds: 3 up (since 2m), 3 in (since 6d)
data:
pools: 6 pools, 717 pgs
objects: 44 objects, 4.2 KiB
usage: 46 MiB used, 90 GiB / 90 GiB avail
pgs: 717 active+clean
io:
client: 1.8 KiB/s rd, 4 op/s rd, 0 op/s wr
progress:
Global Recovery Event (45s)
[===========================.]
```
Signed-off-by: Sébastien Han <seb@redhat.com>
When new pools are added to an already existing filesystem, we would
create them but not actually associate them to the filesystem. Ensure
that we do this.
Also, while we're here, we refactor the CreateFilesystem test: before,
we were not properly testing creating the filesystem from scratch
because we would return valid JSON for *every* ceph command. Change this
by making the first run return an error so that we get some better
coverage.
After this, we also add a check for the correctness of adding new pools
to the CreateFilesystem unittest.
Fixes#5876
Signed-off-by: Lalit Maganti <lalitm@google.com>
Found by running the following command:
codespell -S .git,*.png,*.jpg -L \
aks,keyserver,atleast,dne,ser,ist,files\',ba,dum,iam,te -f -H
Signed-off-by: Mateusz Gozdek <mgozdekof@gmail.com>
Pools in stretch clusters will all use the same crush rule that
is generated by the operator when configuring the cluster for
stretch mode, with replica 4 and two replicas per failure domain.
Pools cannot create new crush rules, so we require that replica be
4 when the pools is created, EC is not allowed, and no new
crush rule will be created.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Due to #6492, preservePoolsOnDelete is not useful at all for CephFS;
after the filesystem is deleted, the leftover pools cannot be
reassocaited with a newly created filesystem without wiping all
metadata. The only way we can actually preserve data is keeping around
the entire filesystem.
This commit implements a `preserveFilesystemOnDelete` option which work
similar to the existing pool preservation option but instead keeps the
whole CephFS while taking it down and removing all MDSes.
This commit also changes all documentation to refer to this new option
with the intent of essentially deprecating `preservePoolsOnDelete`. IMO,
keeping around `preservePoolsOnDelete` is actively harmful because it
lulls users into thinking their data will be safe but, in reality,
recovering from this situation is highly complex and has large potential
for data loss.
Signed-off-by: Lalit Maganti <lalitm@google.com>
The CRUSH map does not allow duplicate values (nor keys) for the
labels. That means that if the currently hardcoded label value
`default` occurs anywhere else in the tree (for example, because
some of the topology labels provided by the cloud provider use it),
the cluster cannot sucessfully spawn.
By giving the user control over the label value used for the root
CRUSH map label, it is possible to adapt the cluster to this
situation.
To be able to actually use the `default` value, the root=default
node which is created by default by ceph itself has to be removed;
since that node is accompanied by a replicated crush rule, we have
to remove that rule, too.
The integration tests are modified to sometimes use the custom
crushRoot (based on an arbitrary criterium) to get coverage across
the various suites. Separate tests could be added, but were not
deemed necessary at this point.
Fixes#4993.
Signed-off-by: Jonas Schäfer <jonas.schaefer@cloudandheat.com>
this commit handles all the gosec g601
error code (i.e Implicit memory aliasing
of items from a range statement).
Signed-off-by: subhamkrai <subhamkumarrai03@gmail.com>
ceph-file-controller gets stuck in loops, this happens when it tires to delete the filesystem which doesn't exist.Filesystem should only be deleted when it exist.
Signed-off-by: RAJAT SINGH <rajasing@redhat.com>
To provide more context for executing commands in a ceph cluster,
the full clusterInfo is now passed to the ceph execution commands.
All information about the cluster will now be available throughout
all the areas of the operator. The namespace, ceph credentials,
mon endpoints, and other info is a core part of that cluster info.
Arguments passed through the controllers are also simplified for
mons, mgr, osds, and other daemons where the parameters had
become too complex.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The ceph config and client packages are conceptually the same.
To avoid circular dependencies in some cases, we simplify by combining
the packages into the client package.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
ceph mds deactivate is not valid command for Nautilus, changing the mechnaism of upgrading so that it can be done in the way Nautilus does it.
Closes: https://github.com/rook/rook/issues/5278
Signed-off-by: RAJAT SINGH <rajasing@redhat.com>
Remove specific mds config added to the mon configuration database when deleting a CephObjectstore
or when scaling down number of mds
Also remove mds CephX key
Signed-off-by: n.fraison <n.fraison@criteo.com>
CSI CephFS provisioner fails to provision a Persistent Volume after upgrading from Rook Ceph Operator 1.1.2 to 1.1.7.
This issue can be fixed by removing givePoolAppTag function whenever there is an appName cephFs.
Signed-off-by: RAJAT SINGH <rajasing@redhat.com>
The old Filesystem's pool is not getting updated when changed with the new File system.
So, we added two helper functions namely generateDataPoolNames and generateMetaDataPoolName that will generate the names of the pools and then change those pool's values using the SetPoolSize method.
Signed-off-by: RAJAT SINGH <rajasing@redhat.com>
The pools had some legacy structs that translated between the
ceph.v1 types used by the CRDs and the internal implementation
of the pools. This simplifies the pool implementation by removing
the intermediate model and leaving us only with the ceph v1
pool types and no unnecessary translation.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The CRD watcher has been replaced by the new controller-runtime
framework.
This brings robustness in our operator, meaning that any resources that
are modified will be reconciled into the desired state.
Closes: https://github.com/rook/rook/issues/4940
Signed-off-by: Sébastien Han <seb@redhat.com>
We now always perform ok-to-stop change when the deployment is going to
be updated. This could be for various reasons like change in the spec or
a new Ceph image.
In any case, if Ceph resources are going to be restarted this must be
done carefully.
Also trimmed the isUpgrade variable, the upcoming commit will remove the
need of that variable.
Keeping isUpgrade from the controller code since we need it to detect
whether an orchestration upgrade should be triggered or not (based on
the ceph cluster status).
Also, we keep it on the child controllers so we don't go into an
orchestration phase if the cluster image hasn't changed. The child does
not need to be updated/go into its orchestration phase because the
cluster CR changed.
Closes: https://github.com/rook/rook/issues/4379
Signed-off-by: Sébastien Han <seb@redhat.com>
When using the "errors" package, using `%+v` (extended format),
each Frame of the error's StackTrace will be printed in detail.
Let's only print `%v` to print the error.
If the error has a Cause it will be printed recursively.
Basically `%+v` has been replaced with `%v` for all `error` type
interfaces, whether the logger is Info, Warning or Error.
Signed-off-by: Sébastien Han <seb@redhat.com>
Failure to create a keyring prior to creating a deployment
dependent on that keyring created a race conditition resulting
in an intermittent and unnecessary pod failure. This change
creates a keyring prior to deployment creation for MDS
daemons.
Partially fixes: rook#4089
Signed-off-by: egafford <egafford@redhat.com>
The command `fs new` already enable the application pool settings for
the cephfs pools so we don't need to run that command again.
Signed-off-by: Sébastien Han <seb@redhat.com>
The resources for rgw, mds, and nfs were being cleaned up explicitly when
the CR was deleted. This is legacy from before the owner references were
set on the CRs. Now the owner references are set to the appropriate CRs so
the resources will be cleaned up upon deletion of the specific CR
rather than the cluster CR.
Signed-off-by: travisn <tnielsen@redhat.com>
A new CRD property `PreservePoolsOnDelete` has been added to Filesystem(fs) and
Object Store(os) resources in order to increase protection against data loss.
If it is set to `true`, associated pools won't be deleted when the main
resource(fs/os) is deleted. Creating again the deleted fs/os with the same name
will reuse the preserved pools.
Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
From now on, Rook will only perform check before upgrades when there is
an actual upgrade. So if the Ceph image changed and a new version is
desired Rook will go through all the daemons and update them one by one
and perform checks in between.
Closes: https://github.com/rook/rook/issues/3583
Signed-off-by: Sébastien Han <seb@redhat.com>
When configuring an external cluster the orchestration of discover and all the ceph
daemons will be skipped. The operator will still watch for the creation of
crds for this namespace. When a filesystem, object, object user, or
ganesha crd are created, the operator will simply print an error
to the log.
Signed-off-by: travisn <tnielsen@redhat.com>
Signed-off-by: Sébastien Han <seb@redhat.com>
It's quite convinient to expose /var/log/ceph so that we can decide to
activate logs locally on the machine and see what's going on.
This is only a placeholder when a daemon is stuck crashlooping and we
want to allow administrator to gather log files.
We still do not log on file but this can be activated via a config
option passed to the centralized config option store.
For some daemons, which typically do not store any data (rgw, rbd-mirror
and mds) we had to propagate dataDirHostPath from the cluster spec to
each creation call so that the bindmount can happen.
Fixes: https://github.com/rook/rook/issues/2881
Signed-off-by: Sébastien Han <seb@redhat.com>
A ceph filesystem was brought down by marking the filesystem
as down and individually failing the MDSes. From nautilus onwards,
this could be more simply and efficiently done by using
`fs fail` command. See,
https://github.com/ceph/ceph/commit/4c49f165ec1Resolves: #2841
Signed-off-by: Ramana Raja <rraja@redhat.com>
The basic idea here is to replace the release name in the ClusterInfo
object with the fine grained version information queried at runtime.
This patch also removes the Name field from the cluster spec, which was
also runtime determined and was redundant with ClusterInfo relase name.
Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
From ceph mimic (13.x) onwards, `mds deactivate` command is
deprecated. To scale down the number of active MDSes of a filesystem,
it's sufficient to set the desired number of active MDSes using
'max_mds' setting, and is no longer required to individually
deactivate the unwanted MDSes.
Resolves: #2725
Signed-off-by: Ramana Raja <rraja@redhat.com>
The MDS mds_standby_for_*, mon_force_standby_active, and mds_standby_replay
configuration options have been obsoleted. Instead, the operator may now set
the new "allow_standby_replay" flag on the CephFS file system. This setting
causes standbys to become standby-replay for any available rank in the file
system.
Closes: https://github.com/rook/rook/issues/2768
Signed-off-by: Sébastien Han <seb@redhat.com>
Configure the Ceph mds daemon completely from the operator a la the
recent changes to the Ceph mon and mgr operators.
Create the mds deployments first and then
create the keyring secrets for them with their owner reference as the
corresponding deployment. This will mean that the secrets do not need to
be micromanaged. When the deployment is deleted, the secret is also
deleted. This has not been necessary for the mons or the manager since
the mons share a keyring with a lifespan of the cluster, as does the
mgr, which currently has single-mgr support only.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
design/external-management.md says that we should allow specifying
Filesystems that have no pools. When this is done, the CRD should
interepret that as a request to just manage the MDS daemons without
creating a new filesystem.
Don't fail the verification if there are no data pools in the spec,
and just skip creating a new filesystem if that's the case.
Also, add a testcase for this usage.
Signed-off-by: Jeff Layton <jlayton@redhat.com>
This seems a bit misplaced, since CreateFilesystem is all about
doing the initial setup of the filesystem, and the daemons are
managed in a different block of code.
In a later patch, we're going to want to allow the CRD to spin
up MDS' for filesystems that already exist. Lift the code that
sets the MDS count into the caller.
Signed-off-by: Jeff Layton <jlayton@redhat.com>
fix#2193
Fix regression introduced by PR allowing MDS config creation in init
container. This allows the new deployment-based mds clusters to scale
down where currently they can only remain constant or scale up.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
Sébastien Han
subhamkrai
Lars Lehtonen
Satoru Takeuchi
Lalit Maganti
Mateusz Gozdek
Travis Nielsen
Jonas Schäfer
subhamkrai
RAJAT SINGH
Madhu Rajanna
n.fraison
egafford
Juan Miguel Olmo Martínez
Ramana Raja
Noah Watkins
Blaine Gardner
Jeff Layton