If someone sets limits to pod, we want to ensure the possible
experience, so we want to make sure that people do not configure
inapropriate values for certain daemons.
We decide to fail if the memory.limit is too low.
Signed-off-by: Sébastien Han <seb@redhat.com>
If the spec of the OSD daemon contains resource limits, especially
memory.limit we will use that value for setting the osd memory target
value so the bluestore OSD knows the amount of memory available for its
cache.
Signed-off-by: Sébastien Han <seb@redhat.com>
If the spec of the MDS daemon contains resource limits, especially
memory.limit we will use that value for setting the mds cache memory
limit value so the MDS knows the amount of memory available.
Signed-off-by: Sébastien Han <seb@redhat.com>
We now expose resource as environment variable in the pod.
They will be used by Ceph to report as a daemon metadata.
Signed-off-by: Sébastien Han <seb@redhat.com>
Only the args necessary have traditionally been passed to the mon package.
This has led to many properties in the mon struct, whereas we will
now simply pass the cluster crd spec to simplify.
Signed-off-by: travisn <tnielsen@redhat.com>
The orchestrations need to be triggered on any update to the crd.
Users today restart the operator for various scenarios when the
operator could take care of it automatically. If unsupported options
are updated, they should just be ignored.
There are two goroutines that are working with the mons.
The first is an orchestration that is triggered by the operator
at startup, or when the cluster crd is updated. The second
is the health check that triggers periodically by default every
45 seconds. These goroutines must not try to make updates at the
same time. A mutex is added so one will block if the other is
still active. The mutex is only active for the duration of working
with mons, and not an entire orchestration of mgr, osd, etc
Signed-off-by: travisn <tnielsen@redhat.com>
In the form of mon, mgr, mds, and rgw, convert the rbd-mirror to be
configured completely from the operator. This is a straightforward
conversion with one functional addition: the rbd-mirror daemon stores
*no* data and has no default data dir, so the concept of a `Dataless`
daemon is here introduced.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
From ceph mimic (13.x) onwards, `mds deactivate` command is
deprecated. To scale down the number of active MDSes of a filesystem,
it's sufficient to set the desired number of active MDSes using
'max_mds' setting, and is no longer required to individually
deactivate the unwanted MDSes.
Resolves: #2725
Signed-off-by: Ramana Raja <rraja@redhat.com>
Configure the Ceph rgw daemon completely from the operator a la the
recent changes to the Ceph mon, mgr, and mds operators.
Create the rgw deployment or daemonset first, and then create the
keyring secret for the object store with its owner reference as the
corresponding deployment or daemonset. When the replication controller
is deleted, the secret is also deleted.
The RGW's mime.types file is now stored in a configmap with a different
file created for each object store. This is primarily just a means to
get the mime.types file into the rgw pod, but the added benefit is that
the administrator can modify the configmap, which could reduce
susceptibility to file type execution vulnerabilities (worst case).
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
The MDS mds_standby_for_*, mon_force_standby_active, and mds_standby_replay
configuration options have been obsoleted. Instead, the operator may now set
the new "allow_standby_replay" flag on the CephFS file system. This setting
causes standbys to become standby-replay for any available rank in the file
system.
Closes: https://github.com/rook/rook/issues/2768
Signed-off-by: Sébastien Han <seb@redhat.com>
The mgr orchestrator modules need the container image to run
the ceph image for blinking the lights when a disk is down
Signed-off-by: travisn <tnielsen@redhat.com>
this is needed as cryptsetup in the pod and udev on the host coordinate
via a semaphore that must be visible in both contexts.
note that additionally the host needs to have dmcrypt compiled with udev
synchronization, which at least one minikube version does not have.
fixes: #923
Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
Configure the Ceph mds daemon completely from the operator a la the
recent changes to the Ceph mon and mgr operators.
Create the mds deployments first and then
create the keyring secrets for them with their owner reference as the
corresponding deployment. This will mean that the secrets do not need to
be micromanaged. When the deployment is deleted, the secret is also
deleted. This has not been necessary for the mons or the manager since
the mons share a keyring with a lifespan of the cluster, as does the
mgr, which currently has single-mgr support only.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
We should probably not hardcode this value here, and instead use
whatever the syscall module thinks this is.
Signed-off-by: Jeff Layton <jlayton@redhat.com>
All devices detected by the discovery pod were being passed to the OSD provisioning pod
thus not always honoring the desired device list that should be provisioned.
Now the provisioning pod will be given the desired state from the crd,
then apply that state depending on the actual devices detected.
Also added a helper to ensure OSDsPerDevice is always valid.
Signed-off-by: travisn <tnielsen@redhat.com>
This commit introduces the necessary changes to support the new
messenger feature coming with Ceph Nautilus (currently in development).
What changes? Now the monitor listens on two port:
* old 6789 for messenger v1, which will help us support older client
(e,g: krbd)
* new 3300 for messengers v2, which brings new improvement in the
messaging layer. This new transport layer brings numerous advantages
such as encryption improvement, speed improvement, pluggable nature to
support different network stack than TCP and many more.
We still have one Service IP, however it has 2 ports, see:
```
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
rook-ceph-mon-a ClusterIP 10.106.217.160 <none> 3300/TCP,6789/TCP 4h
rook-ceph-mon-b ClusterIP 10.99.36.175 <none> 3300/TCP,6789/TCP 4h
rook-ceph-mon-c ClusterIP 10.108.220.74 <none> 3300/TCP,6789/TCP 4h
````
The `ceph.conf` has changed and we don't force the port when using an IP
address (public addr etc). Ceph, depending on its version will naturally
start the monitors on their right port, 6789.
A new --ceph-version-name CLI argument has been added to the Rook binary
so that when the pod starts it passes the ceph version name and the
configuration of the ceph.conf, as well as daemon startup flags, happen
properly.
Given that the Rook Operator remembers the port of all the monitors it
deployed (through Pod definition), this change is not an issue and will
maintain backward compatibility.
Note that to test this you must build rook with dev container image,
which contains the dev Nautilus version. So you should do something
like:
`make -j4 BASEIMAGE='ceph/daemon-base:latest-master' IMAGES='ceph' build`
Resolves: #2525
Signed-off-by: Sébastien Han <seb@redhat.com>
Travis Nielsen
Alexander Trostandnull
Anton Skriptsov
Sébastien Han
Noah Watkins
Stefan Haas
Blaine Gardner
Jose A. Rivera
Ramana Raja
zhoulin xie
Jeff Layton
Bogdan Luca