Configure the Ceph mgr daemon completely from the operator a la the
recent changes to the Ceph mon operator.
The pod spec for the mgr changed quite a bit, and instead of updating
the unit tests of questionable use, some additional unit test tools
applicable to any Ceph daemon have been added and used with the mgr. The
mgrs unit tests should now be more useful and get in the way of devs
less, and they can be used by other daemons later.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
extra_binds has to be on same level as extra_args. Currently it will interpreted as another argument resuling in kubelet being called with the non-existent argument "--extraBinds=/usr/libexec/kubernetes/kubelet-plugins/volume/exec:/usr/libexec/kubernetes/kubelet-plugins/volume/exec" leading to a crash/reboot-loop of the kubelet.
Signed-off-by: Maximilian Heinrich <maximilian.heinrich@esono.de>
The namespace referred to by the prose text in the "Create a Rook Cluster" when creating the cluster was incorrect and did not match the example below, which was correct.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
Make the Rook config-init unnecessary for mons, and remove that init
container. Perform all mon configuration steps in the operator, and set
up the mon pods and k8s environment such that only Ceph containers are
needed for running mons.
This should help streamline changes to the mons, as there will be no
need to change the `daemon/mon` code or `cmd/rook/ceph` code with mon
changes in the future.
This work starts to lay the groundwork for supporting the
`design/ceph-config-updates.md` design.
Notable new bits:
Create a keyring secret store helper for storing dameon keyrings, and
use it to store the mon keyring. Mon pods mount the keyring into a
k8s secret-backed volume.
Create a configmap store for the Ceph config file which can be mounted
into pods/containers directly to /etc/ceph/ceph.conf. Also store
individual mon_host and mon_initial_members values which can be mapped
into pods as environment variables and used in Ceph commandline flags,
enabling the mon pods to have the most up-to-date information about the
mon cluster when restarting and without need for operator intervention.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
The mon operator code is fairly substantial. Separate large files into
more smaller ones with a good separation of names. This will be
important for trying to move all the mon code into the operator.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
Set each Ceph mon's `public bind addr` from the commandline instead of
by inserting it into the config file. This eliminates an extra step from
initializing mon configs.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
Ganesha recently grew the ability to watch a RADOS object for
notifications, which will cue it to reload its configuration. Add the
directive to the RADOS_URLS config block.
Signed-off-by: Jeff Layton <jlayton@redhat.com>
We now refuse to start more than one monitor on the same machine if
hostnetworking is enabled.
Supporting this creates a lot more complexity from the rook side. Also
having 3 monitors running on the same machine is not a production setup.
This means multi-cluster support on the same machine
is not possible anymore when hostnetworking is enabled.
Resolves: https://github.com/rook/rook/issues/2604
Signed-off-by: Sébastien Han <seb@redhat.com>
Rook's behavior should not be to create a default OSD in dataDirHostPath
when no devices are present on a node. Any preexisting fallback osd
should be kept as long as no dirs or disks have been specified to keep
the legacy behavior for those clusters running with these osds in place.
The legacy deletion behavior is also kept, removing the osd as soon as
any dir or device is specified other than the dataDirHostPath dir.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
The Ceph CLI has the flag option '--connect-timeout' which controls the
timeout of the command when reaching out to a monitor. If the connection
cannot be established within 15 seconds, the command will fail. The
default value in Ceph is relatively long (300sec IIRC). This is a
problem when monitors have trouble reaching out quorum, since this means
the command will only fail after 300 sec.
Now we fail faster.
Resolves: https://github.com/rook/rook/issues/2574
Signed-off-by: Sébastien Han <seb@redhat.com>
Given that we have mon_host and mon_initial_members sections in our
ceph.conf, building a monitor map is not needed. During the monitor mkfs
a monmap will get generated on the fly based on the section mentioned
above.
One benefit from that commit is a potential speed improvement as we have
removed the container that was generating the map from the monitor
initialization sequence.
Resolves: https://github.com/rook/rook/issues/2576
Signed-off-by: Sébastien Han <seb@redhat.com>
The Endpoint has a format such as ip:port and the comment was implying
something like hostname:port. Thus fixing the comment with the right
value.
Signed-off-by: Sébastien Han <seb@redhat.com>
Add steps to clean the disks on a node after Rook has been removed so
that they are in a usable state again.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
The io.MultiReader(r1, r2) reads r1 until EOF before moving on to r2.
When used for stdout/stderr stderr will not be written to the log until
stdout reaches EOF. This patch reads stderr in a go routine so that the
two streams can be interleaved properly.
Fixes: #2479
Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
Propose a new design to update Rook's Ceph configuration strategy to
align with Ceph's recent changes to the config process.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
Blaine Gardner
Travis Nielsen
hchiramm
Alexander Trost
Maximilian Heinrich
Christian Hüning
Jared Watts
John Mulligan
Jeff Layton
Jason Murray
Dmitry Yusupov
Sébastien Han
Andri Steiner
Dmitry Yusupov
Anton Skriptsov
Noah Watkins