The check for whether to set the OwnerReferences is no longer
necessary. When passing the correct value to the apiVersion,
the OwnerReferences can be successfully set for OpenShift.
Signed-off-by: travisn <tnielsen@redhat.com>
This commit does multiple things:
* remove support for AllNodes where we would deploy one rgw per node on
all the nodes.
* a transition path is implemented in the code so that if someone has an
existing deployment, daemonsets will be removed and replaced by an
deployments.
* when using "instances", each rgw deployed has its own key which makes
Ceph reporting the exact number of rgw running, see:
```
[root@rook-ceph-operator-775cf575c5-bh4sr /]# ceph -s
cluster:
id: 611fcf39-0669-4864-9a12-debb35c0397a
health: HEALTH_OK
services:
mon: 3 daemons, quorum a,b,c (age 12h)
mgr: a(active, since 12h)
osd: 3 osds: 3 up (since 12h), 3 in (since 12h)
rgw: 3 daemons active (my.store.a, my.store.b, my.store.c)
data:
pools: 6 pools, 600 pgs
objects: 235 objects, 3.8 KiB
usage: 3.0 GiB used, 84 GiB / 87 GiB avail
pgs: 600 active+clean
```
Closes: https://github.com/rook/rook/issues/2474, https://github.com/rook/rook/issues/2957 and https://github.com/rook/rook/issues/3245
Signed-off-by: Sébastien Han <seb@redhat.com>
The CephCluster CR contains settings that are needed by other
CRs to configure the Ceph daemons. When the CephCluster CR
is updated, the updates will now be passed on to each of the
CR controllers to ensure the daemons are updated properly
without requiring an operator restart.
When calling the controllers from another controller,
we ensure that only a single goroutine is handling
CRs at any given time to prevent contention across
multiple CRs of the same type.
Signed-off-by: travisn <tnielsen@redhat.com>
Add a 'ceph-version' label to application controllers in the same manner
as the prior 'rook-version' label to help with upgrades.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
Allow the user to set annotations in the CRDs of the following operators which will then be set on
the resulting Pods (Deployments, StatefulSets and so on).
Operators:
* Cassandra
* Ceph
* CockroachDB
* EdgeFS
* Minio
* NFS
The Annotations related structures have been added to the
rook.io/v1alpha2 pkg.
Signed-off-by: Alexander Trost <galexrt@googlemail.com>
Because the deployment update-and-wait function waits for the deployment
to be ready, and that cannot happen due to the keyring not
having been updated yet. The keyring has its
owner set to the deployment so that no special keyring handling must
be done, but it must be created after the deployment UID is known.
Change the order of operations so that the existing deployment's UID can
be retrieved if it exists and the keyring created/updated before the
update-and-wait function is called.
This order-of-operations issue currently only applies to the mds. Other
daemons whose keyrings are owned by the replication controller do not
use the update-and-wait function.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
It's quite convinient to expose /var/log/ceph so that we can decide to
activate logs locally on the machine and see what's going on.
This is only a placeholder when a daemon is stuck crashlooping and we
want to allow administrator to gather log files.
We still do not log on file but this can be activated via a config
option passed to the centralized config option store.
For some daemons, which typically do not store any data (rgw, rbd-mirror
and mds) we had to propagate dataDirHostPath from the cluster spec to
each creation call so that the bindmount can happen.
Fixes: https://github.com/rook/rook/issues/2881
Signed-off-by: Sébastien Han <seb@redhat.com>
Add a `rook-version` label to all controller resources used by Ceph:
deployments, daemonsets, and jobs. Labels are added to controller
resources only and not to the pod templates within because changing pod
templates causes updates due to the label change. The controller
resources themselves are not updated with the label addition and
therefore don't cause an unnecessary upgrade.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
A ceph filesystem was brought down by marking the filesystem
as down and individually failing the MDSes. From nautilus onwards,
this could be more simply and efficiently done by using
`fs fail` command. See,
https://github.com/ceph/ceph/commit/4c49f165ec1Resolves: #2841
Signed-off-by: Ramana Raja <rraja@redhat.com>
All usages of k8s go client are now also using the versioned `AppsV1() `
call for the client.
Updated MySQL and Wordpress, and Kube Registy examples to use apps/v1
Deployments.
Signed-off-by: Alexander Trost <galexrt@googlemail.com>
The basic idea here is to replace the release name in the ClusterInfo
object with the fine grained version information queried at runtime.
This patch also removes the Name field from the cluster spec, which was
also runtime determined and was redundant with ClusterInfo relase name.
Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
If someone sets limits to pod, we want to ensure the possible
experience, so we want to make sure that people do not configure
inapropriate values for certain daemons.
We decide to fail if the memory.limit is too low.
Signed-off-by: Sébastien Han <seb@redhat.com>
If the spec of the MDS daemon contains resource limits, especially
memory.limit we will use that value for setting the mds cache memory
limit value so the MDS knows the amount of memory available.
Signed-off-by: Sébastien Han <seb@redhat.com>
We now expose resource as environment variable in the pod.
They will be used by Ceph to report as a daemon metadata.
Signed-off-by: Sébastien Han <seb@redhat.com>
In the form of mon, mgr, mds, and rgw, convert the rbd-mirror to be
configured completely from the operator. This is a straightforward
conversion with one functional addition: the rbd-mirror daemon stores
*no* data and has no default data dir, so the concept of a `Dataless`
daemon is here introduced.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
From ceph mimic (13.x) onwards, `mds deactivate` command is
deprecated. To scale down the number of active MDSes of a filesystem,
it's sufficient to set the desired number of active MDSes using
'max_mds' setting, and is no longer required to individually
deactivate the unwanted MDSes.
Resolves: #2725
Signed-off-by: Ramana Raja <rraja@redhat.com>
The MDS mds_standby_for_*, mon_force_standby_active, and mds_standby_replay
configuration options have been obsoleted. Instead, the operator may now set
the new "allow_standby_replay" flag on the CephFS file system. This setting
causes standbys to become standby-replay for any available rank in the file
system.
Closes: https://github.com/rook/rook/issues/2768
Signed-off-by: Sébastien Han <seb@redhat.com>
The mgr orchestrator modules need the container image to run
the ceph image for blinking the lights when a disk is down
Signed-off-by: travisn <tnielsen@redhat.com>
Configure the Ceph mds daemon completely from the operator a la the
recent changes to the Ceph mon and mgr operators.
Create the mds deployments first and then
create the keyring secrets for them with their owner reference as the
corresponding deployment. This will mean that the secrets do not need to
be micromanaged. When the deployment is deleted, the secret is also
deleted. This has not been necessary for the mons or the manager since
the mons share a keyring with a lifespan of the cluster, as does the
mgr, which currently has single-mgr support only.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
This commit introduces the necessary changes to support the new
messenger feature coming with Ceph Nautilus (currently in development).
What changes? Now the monitor listens on two port:
* old 6789 for messenger v1, which will help us support older client
(e,g: krbd)
* new 3300 for messengers v2, which brings new improvement in the
messaging layer. This new transport layer brings numerous advantages
such as encryption improvement, speed improvement, pluggable nature to
support different network stack than TCP and many more.
We still have one Service IP, however it has 2 ports, see:
```
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
rook-ceph-mon-a ClusterIP 10.106.217.160 <none> 3300/TCP,6789/TCP 4h
rook-ceph-mon-b ClusterIP 10.99.36.175 <none> 3300/TCP,6789/TCP 4h
rook-ceph-mon-c ClusterIP 10.108.220.74 <none> 3300/TCP,6789/TCP 4h
````
The `ceph.conf` has changed and we don't force the port when using an IP
address (public addr etc). Ceph, depending on its version will naturally
start the monitors on their right port, 6789.
A new --ceph-version-name CLI argument has been added to the Rook binary
so that when the pod starts it passes the ceph version name and the
configuration of the ceph.conf, as well as daemon startup flags, happen
properly.
Given that the Rook Operator remembers the port of all the monitors it
deployed (through Pod definition), this change is not an issue and will
maintain backward compatibility.
Note that to test this you must build rook with dev container image,
which contains the dev Nautilus version. So you should do something
like:
`make -j4 BASEIMAGE='ceph/daemon-base:latest-master' IMAGES='ceph' build`
Resolves: #2525
Signed-off-by: Sébastien Han <seb@redhat.com>
Introduce mount security mode for basic multi tenancy
Fixes#2164.
This adds three new parameters/options to StorageClass/flexvolume entry:
* `mountUser`
* `mountSecret`
* `mountSecretNamespace`
Signed-off-by: Alexander Trost <galexrt@googlemail.com>
Verify that the expected deployments are updated in the Ceph mgr,
and mds unit tests.
This also allows those unit tests to pass at all since
UpdateDeploymentAndWait was blocking the unit tests from finishing due
to unexpected behavior of generated `Deployment.Update` unit test mock.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
Use the `UpdateDeploymentAndWait` method used by the osd operator to
update the deployments for mgrs, mdses, and rgws when the Rook
orchestrator image is updated.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
design/external-management.md says that we should allow specifying
Filesystems that have no pools. When this is done, the CRD should
interepret that as a request to just manage the MDS daemons without
creating a new filesystem.
Don't fail the verification if there are no data pools in the spec,
and just skip creating a new filesystem if that's the case.
Also, add a testcase for this usage.
Signed-off-by: Jeff Layton <jlayton@redhat.com>
This seems a bit misplaced, since CreateFilesystem is all about
doing the initial setup of the filesystem, and the daemons are
managed in a different block of code.
In a later patch, we're going to want to allow the CRD to spin
up MDS' for filesystems that already exist. Lift the code that
sets the MDS count into the caller.
Signed-off-by: Jeff Layton <jlayton@redhat.com>
fix#2193
Fix regression introduced by PR allowing MDS config creation in init
container. This allows the new deployment-based mds clusters to scale
down where currently they can only remain constant or scale up.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
Progress toward issue #2003.
Includes design from design doc PR #1578
Use init containers to create configuration for Ceph mgrs. There is only
one init container in this design. The init container calls the Rook
binary to create Ceph config files which are then shared with the mds
daemon main container.
Once this init is run, the main mds daemon is run. Leaving room to use
the Ceph-versioned image in the future, call `ceph-mds --foreground ...`
to run the Ceph mds.
The refactor to using an init container also necessitated refactoring
the mdses replicaset implementation to a deployment-per-pod
implementation due to a chicken-egg problem. With a single container (in
the before times) the Rook binary was able to call the ceph-mds daemon
with an id generated from the pod name. Since the pod name is not known
before runtime, and the id is one of the few params that must be
specified to Ceph daemons on run, it is necessary to know the id
beforehand; thus the move to a deployment architecture following the
likes of the mon and mgr daemons.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
Ceph expects to be able to reason about physical hardware / nodes.
Before this patch the NODE_NAME that was available referred to the pod
name. This patch exposes the underlying node name, pod, and pod
namespace for the ceph daemons to consume.
fixes: #2078
Signed-off-by: Noah Watkins <nwatkins@redhat.com>
To help with code readability, include daemons from `pkg/daemon/ceph`
as `<dmn>daemon "github.com/rook/rook/pkg/daemon/ceph/<dmn>`. This will
make it easier to understand code lines without needing to scroll up to
check the includes, as there are mon/mds/mgr/osd/rgw packages for
daemons as well as operator and cmd.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
travisn
Sébastien Han
Blaine Gardner
Alexander Trost
Ramana Raja
Noah Watkins
Huamin Chen
Jeff Layton
Noah Watkins
Ganesh Maharaj Mahalingam
Jared Watts