Commit Graph
29 Commits
Author SHA1 Message Date
travisn cb82f36eba ceph: no need for --public-bind-addr with host networking
The mons require the --public-bind-addr when host networking is not in use
since the pod ip will be different from the service ip that will be
advertised to clients. When host networking is enabled, there is no
need for this argument since the endpoints will be the same to bind
inside the pod as what is advertised to clients.

Signed-off-by: travisn <tnielsen@redhat.com>
2019-05-10 15:07:46 -06:00
travisn dbe3da97a1 ceph: preserve the non-default port for hostnetworking
Clusters using host networking will not work if the non-default port is
being used by the mons. Previous to rook 1.0 the mons were all
using the non-default port 6790 and now they are using the default port
of 6789 from 1.0. Now host networking will preserve the non-default port
to allow upgrades to continue working.

Signed-off-by: travisn <tnielsen@redhat.com>
2019-05-10 12:24:57 -06:00
Blaine Gardner f4e2be50a6 ceph spec: add ceph-version label to controllers
Add a 'ceph-version' label to application controllers in the same manner
as the prior 'rook-version' label to help with upgrades.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2019-04-18 15:09:33 -06:00
travisn bc87b440fb update to the k8s 1.14 client libraries
Signed-off-by: travisn <tnielsen@redhat.com>
2019-04-18 07:51:41 -06:00
Alexander Trost 0c6eb7aa3d crds: Set annotations pods, deployments and so on
Allow the user to set annotations in the CRDs of the following operators which will then be set on
the resulting Pods (Deployments, StatefulSets and so on).
Operators:
* Cassandra
* Ceph
* CockroachDB
* EdgeFS
* Minio
* NFS

The Annotations related structures have been added to the
rook.io/v1alpha2 pkg.

Signed-off-by: Alexander Trost <galexrt@googlemail.com>
2019-04-16 20:47:05 +02:00
Blaine Gardner 685a8df08d ceph: add 'rook-version' label to controllers
Add a `rook-version` label to all controller resources used by Ceph:
deployments, daemonsets, and jobs. Labels are added to controller
resources only and not to the pod templates within because changing pod
templates causes updates due to the label change. The controller
resources themselves are not updated with the label addition and
therefore don't cause an unnecessary upgrade.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2019-04-12 13:56:31 -06:00
Blaine Gardner feff587992 ceph: mon service forwards default bind port 6789
The Ceph mon flag `--public-bind-addr` does not keep the same port as
the previous daemon run when the port is left off the flag's value. A
port is undesired since that could interfere with Nautilus' msgr2
protocol, so instead configure the mon's services to forward the default
bind port on pods (6789) to the mon endpoint's port. The endpoint port
will be 6789 for new mons, but it may be 6790 for legacy clusters.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2019-04-03 15:53:56 -06:00
Noah Watkins 379cb98689 ceph: replace release name with fine-grained version
The basic idea here is to replace the release name in the ClusterInfo
object with the fine grained version information queried at runtime.
This patch also removes the Name field from the cluster spec, which was
also runtime determined and was redundant with ClusterInfo relase name.

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
2019-03-18 12:57:29 -07:00
travisn c1628141aa minimize args to mon creation in the operator
Only the args necessary have traditionally been passed to the mon package.
This has led to many properties in the mon struct, whereas we will
now simply pass the cluster crd spec to simplify.

Signed-off-by: travisn <tnielsen@redhat.com>
2019-03-14 13:10:52 -06:00
Blaine Gardner c12aa6c7c5 ceph rbd-mirror: configure entirely in operator
In the form of mon, mgr, mds, and rgw, convert the rbd-mirror to be
configured completely from the operator. This is a straightforward
conversion with one functional addition: the rbd-mirror daemon stores
*no* data and has no default data dir, so the concept of a `Dataless`
daemon is here introduced.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2019-03-13 09:54:09 -06:00
travisn 31848e23e7 add the CONTAINER_IMAGE env var to the ceph daemon pods
The mgr orchestrator modules need the container image to run
the ceph image for blinking the lights when a disk is down

Signed-off-by: travisn <tnielsen@redhat.com>
2019-03-04 13:33:41 -07:00
Blaine Gardner 4eb4fb7aa6 ceph mds: configure completely from operator
Configure the Ceph mds daemon completely from the operator a la the
recent changes to the Ceph mon and mgr operators.

Create the mds deployments first and then
create the keyring secrets for them with their owner reference as the
corresponding deployment. This will mean that the secrets do not need to
be micromanaged. When the deployment is deleted, the secret is also
deleted. This has not been necessary for the mons or the manager since
the mons share a keyring with a lifespan of the cluster, as does the
mgr, which currently has single-mgr support only.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2019-02-28 12:31:43 -07:00
Sébastien Han eb3c424d57 Ceph messengers 2 support
This commit introduces the necessary changes to support the new
messenger feature coming with Ceph Nautilus (currently in development).

What changes? Now the monitor listens on two port:

* old 6789 for messenger v1, which will help us support older client
(e,g: krbd)
* new 3300 for messengers v2, which brings new improvement in the
messaging layer. This new transport layer brings numerous advantages
such as encryption improvement, speed improvement, pluggable nature to
support different network stack than TCP and many more.

We still have one Service IP, however it has 2 ports, see:

```
NAME                      TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
rook-ceph-mon-a           ClusterIP   10.106.217.160   <none>        3300/TCP,6789/TCP   4h
rook-ceph-mon-b           ClusterIP   10.99.36.175     <none>        3300/TCP,6789/TCP   4h
rook-ceph-mon-c           ClusterIP   10.108.220.74    <none>        3300/TCP,6789/TCP   4h
````

The `ceph.conf` has changed and we don't force the port when using an IP
address (public addr etc). Ceph, depending on its version will naturally
start the monitors on their right port, 6789.

A new --ceph-version-name CLI argument has been added to the Rook binary
so that when the pod starts it passes the ceph version name and the
configuration of the ceph.conf, as well as daemon startup flags, happen
properly.

Given that the Rook Operator remembers the port of all the monitors it
deployed (through Pod definition), this change is not an issue and will
maintain backward compatibility.

Note that to test this you must build rook with dev container image,
which contains the dev Nautilus version. So you should do something
like:

`make -j4 BASEIMAGE='ceph/daemon-base:latest-master' IMAGES='ceph' build`

Resolves: #2525
Signed-off-by: Sébastien Han <seb@redhat.com>
2019-02-22 00:09:37 +01:00
Huamin Chen 1b21e42a90 convert extensions to apps
Signed-off-by: Huamin Chen <hchen@redhat.com>
2019-02-14 16:06:05 -05:00
Blaine Gardner 03ea5f0a7c ceph mgr: configure completely from operator
Configure the Ceph mgr daemon completely from the operator a la the
recent changes to the Ceph mon operator.

The pod spec for the mgr changed quite a bit, and instead of updating
the unit tests of questionable use, some additional unit test tools
applicable to any Ceph daemon have been added and used with the mgr. The
mgrs unit tests should now be more useful and get in the way of devs
less, and they can be used by other daemons later.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2019-02-11 14:35:01 -07:00
Blaine Gardner ec416cdd92 ceph mons: set up entirely in operator
Make the Rook config-init unnecessary for mons, and remove that init
container. Perform all mon configuration steps in the operator, and set
up the mon pods and k8s environment such that only Ceph containers are
needed for running mons.

This should help streamline changes to the mons, as there will be no
need to change the `daemon/mon` code or `cmd/rook/ceph` code with mon
changes in the future.

This work starts to lay the groundwork for supporting the
`design/ceph-config-updates.md` design.

Notable new bits:

Create a keyring secret store helper for storing dameon keyrings, and
use it to store the mon keyring. Mon pods mount the keyring into a
k8s secret-backed volume.

Create a configmap store for the Ceph config file which can be mounted
into pods/containers directly to /etc/ceph/ceph.conf. Also store
individual mon_host and mon_initial_members values which can be mapped
into pods as environment variables and used in Ceph commandline flags,
enabling the mon pods to have the most up-to-date information about the
mon cluster when restarting and without need for operator intervention.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2019-02-05 06:19:59 -07:00
Blaine Gardner e13956c0bf ceph mon: set public-bind-addr from commandline
Set each Ceph mon's `public bind addr` from the commandline instead of
by inserting it into the config file. This eliminates an extra step from
initializing mon configs.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2019-02-05 06:06:35 -07:00
Sébastien Han ff2bc6509f mon: do not generate nor inject a monmap
Given that we have mon_host and mon_initial_members sections in our
ceph.conf, building a monitor map is not needed. During the monitor mkfs
a monmap will get generated on the fly based on the section mentioned
above.

One benefit from that commit is a potential speed improvement as we have
removed the container that was generating the map from the monitor
initialization sequence.

Resolves: https://github.com/rook/rook/issues/2576
Signed-off-by: Sébastien Han <seb@redhat.com>
2019-01-28 16:55:31 +01:00
travisn 94a7a0d4bc run daemons with the ceph image instead of rook
Signed-off-by: travisn <tnielsen@redhat.com>
2018-11-01 11:49:35 -06:00
Blaine Gardner ce803fbf9d Ceph mds/file: Set up config in init container
Progress toward issue #2003.
Includes design from design doc PR #1578

Use init containers to create configuration for Ceph mgrs. There is only
one init container in this design. The init container calls the Rook
binary to create Ceph config files which are then shared with the mds
daemon main container.

Once this init is run, the main mds daemon is run. Leaving room to use
the Ceph-versioned image in the future, call `ceph-mds --foreground ...`
to run the Ceph mds.

The refactor to using an init container also necessitated refactoring
the mdses replicaset implementation to a deployment-per-pod
implementation due to a chicken-egg problem. With a single container (in
the before times) the Rook binary was able to call the ceph-mds daemon
with an id generated from the pod name. Since the pod name is not known
before runtime, and the id is one of the few params that must be
specified to Ceph daemons on run, it is necessary to know the id
beforehand; thus the move to a deployment architecture following the
likes of the mon and mgr daemons.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2018-10-05 19:01:12 -06:00
Noah Watkins e015a21cbd ceph: expose node,pod,ns names to ceph daemons
Ceph expects to be able to reason about physical hardware / nodes.
Before this patch the NODE_NAME that was available referred to the pod
name. This patch exposes the underlying node name, pod, and pod
namespace for the ceph daemons to consume.

fixes: #2078

Signed-off-by: Noah Watkins <nwatkins@redhat.com>
2018-09-24 13:26:24 -07:00
Blaine Gardner ca71b7a311 Ceph mgr: Set up config in init container
Progress toward issue #2003.
Includes design from design doc PR #1578

Use init containers to create configuration for Ceph mgrs. There is only
1 init container in this design:
 1. Using the Rook image, call the Rook binary to create Ceph config
files shared with the mgr daemgr container.

Once this init is run, the main mgr daemgr is run. Leaving room to use
the Ceph-versioned image in the future, call `ceph-mgr --foreground ...`
to run the Ceph mgr.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2018-09-21 13:11:08 -06:00
Blaine Gardner 7a5fe8498c Ceph op: Add test package, container tests
Add a 'test' package to the Ceph operator and define a test to verify
that containers produced by Rook-Ceph match what is expected. Because
this is for 'containers' and not strictly for 'ceph containers', this
could be moved a level up to the operator test package; however, if
other backends wish to use the container tests, they will likely need to
make modifications, and there is concern that this might make the Ceph
tests brittle.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2018-09-21 09:58:32 -06:00
Blaine Gardner e3d24106d9 Ceph op: Add spec package, shared vol/mnt defns
Add a package called 'spec' to the Ceph operator to store resource
definition specs that are shared by all Ceph daemons.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2018-09-21 09:58:32 -06:00
Blaine Gardner 655692e845 Ceph mon: Set up config in init containers
Progress toward issue #2003.
Includes design from design doc PR #1578

Use init containers to create configuration for Ceph mons. There are 3
init containers in this design:
 1. Using the Rook image, call the Rook binary to create Ceph config
files ultimately stored in a k8s EmptyDir which is shared between all
containers in a mon pod.
 2. Leaving room to use the Ceph-versioned image in the future, call
`monmaptool` to generate the monmap, stored in a shared config dir.
 3. Leaving room to use the Ceph-versioned image in the future, call
`ceph-mon --mkfs ...` to set up the mon filesystem on the host's disk.

Once this init is run, the main mon daemon is run. Leaving room to use
the Ceph-versioned image in the future, call `ceph-mon --foreground ...`
to run the Ceph mon.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2018-09-12 17:56:08 -06:00
travisn f840e3574e mon: start a deployment instead of replicaset
Signed-off-by: travisn <tnielsen@redhat.com>
2018-09-01 00:12:42 -06:00
travisn 5f01178e83 mon: set names based on chars instead of integers
Signed-off-by: travisn <tnielsen@redhat.com>
2018-08-30 22:18:53 -06:00
travisn 89cfccd5a2 enable rook to run on openshift
Signed-off-by: travisn <tnielsen@redhat.com>
2018-07-05 11:20:43 -06:00
Jared Watts f3df68e573 operator, daemon, and cmd updates for supporting multiple storage types
Signed-off-by: Jared Watts <jbw976@gmail.com>
2018-05-18 14:32:20 -07:00