The mgr orchestrator modules need the container image to run
the ceph image for blinking the lights when a disk is down
Signed-off-by: travisn <tnielsen@redhat.com>
this is needed as cryptsetup in the pod and udev on the host coordinate
via a semaphore that must be visible in both contexts.
note that additionally the host needs to have dmcrypt compiled with udev
synchronization, which at least one minikube version does not have.
fixes: #923
Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
Configure the Ceph mds daemon completely from the operator a la the
recent changes to the Ceph mon and mgr operators.
Create the mds deployments first and then
create the keyring secrets for them with their owner reference as the
corresponding deployment. This will mean that the secrets do not need to
be micromanaged. When the deployment is deleted, the secret is also
deleted. This has not been necessary for the mons or the manager since
the mons share a keyring with a lifespan of the cluster, as does the
mgr, which currently has single-mgr support only.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
We should probably not hardcode this value here, and instead use
whatever the syscall module thinks this is.
Signed-off-by: Jeff Layton <jlayton@redhat.com>
All devices detected by the discovery pod were being passed to the OSD provisioning pod
thus not always honoring the desired device list that should be provisioned.
Now the provisioning pod will be given the desired state from the crd,
then apply that state depending on the actual devices detected.
Also added a helper to ensure OSDsPerDevice is always valid.
Signed-off-by: travisn <tnielsen@redhat.com>
This commit introduces the necessary changes to support the new
messenger feature coming with Ceph Nautilus (currently in development).
What changes? Now the monitor listens on two port:
* old 6789 for messenger v1, which will help us support older client
(e,g: krbd)
* new 3300 for messengers v2, which brings new improvement in the
messaging layer. This new transport layer brings numerous advantages
such as encryption improvement, speed improvement, pluggable nature to
support different network stack than TCP and many more.
We still have one Service IP, however it has 2 ports, see:
```
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
rook-ceph-mon-a ClusterIP 10.106.217.160 <none> 3300/TCP,6789/TCP 4h
rook-ceph-mon-b ClusterIP 10.99.36.175 <none> 3300/TCP,6789/TCP 4h
rook-ceph-mon-c ClusterIP 10.108.220.74 <none> 3300/TCP,6789/TCP 4h
````
The `ceph.conf` has changed and we don't force the port when using an IP
address (public addr etc). Ceph, depending on its version will naturally
start the monitors on their right port, 6789.
A new --ceph-version-name CLI argument has been added to the Rook binary
so that when the pod starts it passes the ceph version name and the
configuration of the ceph.conf, as well as daemon startup flags, happen
properly.
Given that the Rook Operator remembers the port of all the monitors it
deployed (through Pod definition), this change is not an issue and will
maintain backward compatibility.
Note that to test this you must build rook with dev container image,
which contains the dev Nautilus version. So you should do something
like:
`make -j4 BASEIMAGE='ceph/daemon-base:latest-master' IMAGES='ceph' build`
Resolves: #2525
Signed-off-by: Sébastien Han <seb@redhat.com>
Rename createCephNFS to upCephNFS and have it take a parameter with the
old active count. Rename deleteGanesha to downCephNFS and have it take a
parameter with the new desired active count. Both the existing callers
pass in "0" for the new count parameters.
Fix up the onUpdate handler for CephNFS objects, such that changes to
the active count are reflected in the number of active servers. Have it
pass in the appropriate values when scaling servers up or down.
Finally, fix downCephNFS to shut down the highest-index servers first
and count down, instead of doing them in the order they were started.
This should eventually allow us more predictable behavior when scaling
down the number of server nodes.
Resolves: https://github.com/rook/rook/issues/2644
Signed-off-by: Jeff Layton <jlayton@redhat.com>
Configure the Ceph mgr daemon completely from the operator a la the
recent changes to the Ceph mon operator.
The pod spec for the mgr changed quite a bit, and instead of updating
the unit tests of questionable use, some additional unit test tools
applicable to any Ceph daemon have been added and used with the mgr. The
mgrs unit tests should now be more useful and get in the way of devs
less, and they can be used by other daemons later.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
Make the Rook config-init unnecessary for mons, and remove that init
container. Perform all mon configuration steps in the operator, and set
up the mon pods and k8s environment such that only Ceph containers are
needed for running mons.
This should help streamline changes to the mons, as there will be no
need to change the `daemon/mon` code or `cmd/rook/ceph` code with mon
changes in the future.
This work starts to lay the groundwork for supporting the
`design/ceph-config-updates.md` design.
Notable new bits:
Create a keyring secret store helper for storing dameon keyrings, and
use it to store the mon keyring. Mon pods mount the keyring into a
k8s secret-backed volume.
Create a configmap store for the Ceph config file which can be mounted
into pods/containers directly to /etc/ceph/ceph.conf. Also store
individual mon_host and mon_initial_members values which can be mapped
into pods as environment variables and used in Ceph commandline flags,
enabling the mon pods to have the most up-to-date information about the
mon cluster when restarting and without need for operator intervention.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
The mon operator code is fairly substantial. Separate large files into
more smaller ones with a good separation of names. This will be
important for trying to move all the mon code into the operator.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
Set each Ceph mon's `public bind addr` from the commandline instead of
by inserting it into the config file. This eliminates an extra step from
initializing mon configs.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
Ganesha recently grew the ability to watch a RADOS object for
notifications, which will cue it to reload its configuration. Add the
directive to the RADOS_URLS config block.
Signed-off-by: Jeff Layton <jlayton@redhat.com>
We now refuse to start more than one monitor on the same machine if
hostnetworking is enabled.
Supporting this creates a lot more complexity from the rook side. Also
having 3 monitors running on the same machine is not a production setup.
This means multi-cluster support on the same machine
is not possible anymore when hostnetworking is enabled.
Resolves: https://github.com/rook/rook/issues/2604
Signed-off-by: Sébastien Han <seb@redhat.com>
Rook's behavior should not be to create a default OSD in dataDirHostPath
when no devices are present on a node. Any preexisting fallback osd
should be kept as long as no dirs or disks have been specified to keep
the legacy behavior for those clusters running with these osds in place.
The legacy deletion behavior is also kept, removing the osd as soon as
any dir or device is specified other than the dataDirHostPath dir.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
Given that we have mon_host and mon_initial_members sections in our
ceph.conf, building a monitor map is not needed. During the monitor mkfs
a monmap will get generated on the fly based on the section mentioned
above.
One benefit from that commit is a potential speed improvement as we have
removed the container that was generating the map from the monitor
initialization sequence.
Resolves: https://github.com/rook/rook/issues/2576
Signed-off-by: Sébastien Han <seb@redhat.com>
The Endpoint has a format such as ip:port and the comment was implying
something like hostname:port. Thus fixing the comment with the right
value.
Signed-off-by: Sébastien Han <seb@redhat.com>
Blaine Gardner
travisn
Noah Watkins
Jeff Layton
Alexander Trost
Sébastien Han
Dmitry Yusupov
Bogdan Luca
Huamin Chen
Yannis Zarkadas
Anton Skriptsov
Madhu Rajanna