Commit Graph
33 Commits
Author SHA1 Message Date
d-luu 9c755fac66 ceph: added priority classes to components
Adds priority class support to Ceph components
  to influence scheduler's pod preemption

Signed-off-by: d-luu <david@davidluu.info>
2019-11-15 16:28:41 -06:00
Sébastien Han bed14cfd71 ceph: improve upgrade on image change
From now on, Rook will only perform check before upgrades when there is
an actual upgrade. So if the Ceph image changed and a new version is
desired Rook will go through all the daemons and update them one by one
and perform checks in between.

Closes: https://github.com/rook/rook/issues/3583
Signed-off-by: Sébastien Han <seb@redhat.com>
2019-08-29 10:18:19 +02:00
Noah Watkins 94bd70a14c ceph/mon: support monitor migration without rebuils
this patch adds two new related features. first, it removes the
requirement that all monitor deployments be pinned to nodes using node
selector. this is allowed when not using host networking and when PVC
storage is used. the second feature introduced in this patch is the
elimination of the Rook scheduler for placing monitors that continue to
use the node selector by using canary pods to that are scheduled by k8s,
after which Rook uses the scheduling results to pin the monitors to
nodes.

fixes: #3667

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
2019-08-27 17:43:35 -07:00
giovanism cc62702d39 ceph: Use rook's NetworkSpec with type embedding
ceph: Use the new NetworkSpec api

ceph: Change all tests to use the NetworkSpec

ceph: Remove TypeMeta field from NetworkSpec

ceph: Change IsHost network func signature

Signed-off-by: giovanism <giovanism@outlook.co.id>
2019-08-26 16:17:03 +07:00
Sébastien Han 25158d3d2b ceph: purge external cluster metadata
Add code to remove configmap and secret injected and used by Rook to
connect to the external cluster.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-08-22 17:19:33 +02:00
John Mulligan 17fad8e958 ceph: plumb a mutex for csi config map updates through the application
Create a mutex at the rook operator level that will be used when
updating the shared-between-clusters config map that Ceph CSI consumes.
Pass the mutex down to the function that will update the CM.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2019-08-12 16:41:26 -04:00
Noah Watkins 4173c5544e ceph: base monitor storage of pvc
Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
2019-08-02 10:30:23 -07:00
Sébastien Han 871054e887 ceph: implement pod restriction on memory
If someone sets limits to pod, we want to ensure the possible
experience, so we want to make sure that people do not configure
inapropriate values for certain daemons.
We decide to fail if the memory.limit is too low.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-03-15 17:35:44 +01:00
travisn c1628141aa minimize args to mon creation in the operator
Only the args necessary have traditionally been passed to the mon package.
This has led to many properties in the mon struct, whereas we will
now simply pass the cluster crd spec to simplify.

Signed-off-by: travisn <tnielsen@redhat.com>
2019-03-14 13:10:52 -06:00
travisn 2fd1f7345d Trigger an orchestration on any crd update and only allow a single orchestration of mons at a time
The orchestrations need to be triggered on any update to the crd.
Users today restart the operator for various scenarios when the
operator could take care of it automatically. If unsupported options
are updated, they should just be ignored.

There are two goroutines that are working with the mons.
The first is an orchestration that is triggered by the operator
at startup, or when the cluster crd is updated. The second
is the health check that triggers periodically by default every
45 seconds. These goroutines must not try to make updates at the
same time. A mutex is added so one will block if the other is
still active. The mutex is only active for the duration of working
with mons, and not an entire orchestration of mgr, osd, etc

Signed-off-by: travisn <tnielsen@redhat.com>
2019-03-14 13:10:12 -06:00
Blaine Gardner 7827ea8ff0 ceph mon: use new spec test framework
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2019-03-07 08:13:07 -07:00
travisn 31848e23e7 add the CONTAINER_IMAGE env var to the ceph daemon pods
The mgr orchestrator modules need the container image to run
the ceph image for blinking the lights when a disk is down

Signed-off-by: travisn <tnielsen@redhat.com>
2019-03-04 13:33:41 -07:00
Blaine Gardner 4eb4fb7aa6 ceph mds: configure completely from operator
Configure the Ceph mds daemon completely from the operator a la the
recent changes to the Ceph mon and mgr operators.

Create the mds deployments first and then
create the keyring secrets for them with their owner reference as the
corresponding deployment. This will mean that the secrets do not need to
be micromanaged. When the deployment is deleted, the secret is also
deleted. This has not been necessary for the mons or the manager since
the mons share a keyring with a lifespan of the cluster, as does the
mgr, which currently has single-mgr support only.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2019-02-28 12:31:43 -07:00
Sébastien Han eb3c424d57 Ceph messengers 2 support
This commit introduces the necessary changes to support the new
messenger feature coming with Ceph Nautilus (currently in development).

What changes? Now the monitor listens on two port:

* old 6789 for messenger v1, which will help us support older client
(e,g: krbd)
* new 3300 for messengers v2, which brings new improvement in the
messaging layer. This new transport layer brings numerous advantages
such as encryption improvement, speed improvement, pluggable nature to
support different network stack than TCP and many more.

We still have one Service IP, however it has 2 ports, see:

```
NAME                      TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
rook-ceph-mon-a           ClusterIP   10.106.217.160   <none>        3300/TCP,6789/TCP   4h
rook-ceph-mon-b           ClusterIP   10.99.36.175     <none>        3300/TCP,6789/TCP   4h
rook-ceph-mon-c           ClusterIP   10.108.220.74    <none>        3300/TCP,6789/TCP   4h
````

The `ceph.conf` has changed and we don't force the port when using an IP
address (public addr etc). Ceph, depending on its version will naturally
start the monitors on their right port, 6789.

A new --ceph-version-name CLI argument has been added to the Rook binary
so that when the pod starts it passes the ceph version name and the
configuration of the ceph.conf, as well as daemon startup flags, happen
properly.

Given that the Rook Operator remembers the port of all the monitors it
deployed (through Pod definition), this change is not an issue and will
maintain backward compatibility.

Note that to test this you must build rook with dev container image,
which contains the dev Nautilus version. So you should do something
like:

`make -j4 BASEIMAGE='ceph/daemon-base:latest-master' IMAGES='ceph' build`

Resolves: #2525
Signed-off-by: Sébastien Han <seb@redhat.com>
2019-02-22 00:09:37 +01:00
Blaine Gardner ec416cdd92 ceph mons: set up entirely in operator
Make the Rook config-init unnecessary for mons, and remove that init
container. Perform all mon configuration steps in the operator, and set
up the mon pods and k8s environment such that only Ceph containers are
needed for running mons.

This should help streamline changes to the mons, as there will be no
need to change the `daemon/mon` code or `cmd/rook/ceph` code with mon
changes in the future.

This work starts to lay the groundwork for supporting the
`design/ceph-config-updates.md` design.

Notable new bits:

Create a keyring secret store helper for storing dameon keyrings, and
use it to store the mon keyring. Mon pods mount the keyring into a
k8s secret-backed volume.

Create a configmap store for the Ceph config file which can be mounted
into pods/containers directly to /etc/ceph/ceph.conf. Also store
individual mon_host and mon_initial_members values which can be mapped
into pods as environment variables and used in Ceph commandline flags,
enabling the mon pods to have the most up-to-date information about the
mon cluster when restarting and without need for operator intervention.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2019-02-05 06:19:59 -07:00
Blaine Gardner e13956c0bf ceph mon: set public-bind-addr from commandline
Set each Ceph mon's `public bind addr` from the commandline instead of
by inserting it into the config file. This eliminates an extra step from
initializing mon configs.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2019-02-05 06:06:35 -07:00
Sébastien Han ff2bc6509f mon: do not generate nor inject a monmap
Given that we have mon_host and mon_initial_members sections in our
ceph.conf, building a monitor map is not needed. During the monitor mkfs
a monmap will get generated on the fly based on the section mentioned
above.

One benefit from that commit is a potential speed improvement as we have
removed the container that was generating the map from the monitor
initialization sequence.

Resolves: https://github.com/rook/rook/issues/2576
Signed-off-by: Sébastien Han <seb@redhat.com>
2019-01-28 16:55:31 +01:00
travisn 62f7e5d6ea ceph: update docs, code, and tests to use v1 crd types
Signed-off-by: travisn <tnielsen@redhat.com>
2018-12-05 14:33:11 -07:00
travisn 94a7a0d4bc run daemons with the ceph image instead of rook
Signed-off-by: travisn <tnielsen@redhat.com>
2018-11-01 11:49:35 -06:00
Blaine Gardner ce803fbf9d Ceph mds/file: Set up config in init container
Progress toward issue #2003.
Includes design from design doc PR #1578

Use init containers to create configuration for Ceph mgrs. There is only
one init container in this design. The init container calls the Rook
binary to create Ceph config files which are then shared with the mds
daemon main container.

Once this init is run, the main mds daemon is run. Leaving room to use
the Ceph-versioned image in the future, call `ceph-mds --foreground ...`
to run the Ceph mds.

The refactor to using an init container also necessitated refactoring
the mdses replicaset implementation to a deployment-per-pod
implementation due to a chicken-egg problem. With a single container (in
the before times) the Rook binary was able to call the ceph-mds daemon
with an id generated from the pod name. Since the pod name is not known
before runtime, and the id is one of the few params that must be
specified to Ceph daemons on run, it is necessary to know the id
beforehand; thus the move to a deployment architecture following the
likes of the mon and mgr daemons.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2018-10-05 19:01:12 -06:00
Noah Watkins e015a21cbd ceph: expose node,pod,ns names to ceph daemons
Ceph expects to be able to reason about physical hardware / nodes.
Before this patch the NODE_NAME that was available referred to the pod
name. This patch exposes the underlying node name, pod, and pod
namespace for the ceph daemons to consume.

fixes: #2078

Signed-off-by: Noah Watkins <nwatkins@redhat.com>
2018-09-24 13:26:24 -07:00
Blaine Gardner ca71b7a311 Ceph mgr: Set up config in init container
Progress toward issue #2003.
Includes design from design doc PR #1578

Use init containers to create configuration for Ceph mgrs. There is only
1 init container in this design:
 1. Using the Rook image, call the Rook binary to create Ceph config
files shared with the mgr daemgr container.

Once this init is run, the main mgr daemgr is run. Leaving room to use
the Ceph-versioned image in the future, call `ceph-mgr --foreground ...`
to run the Ceph mgr.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2018-09-21 13:11:08 -06:00
Blaine Gardner 7a5fe8498c Ceph op: Add test package, container tests
Add a 'test' package to the Ceph operator and define a test to verify
that containers produced by Rook-Ceph match what is expected. Because
this is for 'containers' and not strictly for 'ceph containers', this
could be moved a level up to the operator test package; however, if
other backends wish to use the container tests, they will likely need to
make modifications, and there is concern that this might make the Ceph
tests brittle.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2018-09-21 09:58:32 -06:00
Blaine Gardner 78c2cec1ad Add k8s volume/mount tests to operator test pkg
Add functions for helping test Kubernetes volumes and volume mounts.

- Create functions for testing the existence of volumes/mounts by
name in a list of vols/mounts without needing to know the index of
the vol/mount in the list.

- Create functions for printing vols/mounts in a human-readable format
so that tests may output more useful errors.

- Create a test definition for ensuring that all the volumes a pod
provides and all the mounts from each pod container match. For each
volume, there must be at least one container which mounts the volume,
and for each mount, there must be a source volume.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2018-09-21 09:58:31 -06:00
Blaine Gardner 655692e845 Ceph mon: Set up config in init containers
Progress toward issue #2003.
Includes design from design doc PR #1578

Use init containers to create configuration for Ceph mons. There are 3
init containers in this design:
 1. Using the Rook image, call the Rook binary to create Ceph config
files ultimately stored in a k8s EmptyDir which is shared between all
containers in a mon pod.
 2. Leaving room to use the Ceph-versioned image in the future, call
`monmaptool` to generate the monmap, stored in a shared config dir.
 3. Leaving room to use the Ceph-versioned image in the future, call
`ceph-mon --mkfs ...` to set up the mon filesystem on the host's disk.

Once this init is run, the main mon daemon is run. Leaving room to use
the Ceph-versioned image in the future, call `ceph-mon --foreground ...`
to run the Ceph mon.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2018-09-12 17:56:08 -06:00
Blaine Gardner 3731f06176 Ceph: Refactor mon config/keyring to cephconfig
Create a cephconfig module in Ceph's daemon pkg source, and refactor the
config and keyring generation that exists in the mon package into the
new cephconfig package. The config/keyring generation code is used by
most all daemons and not just mon, so a new package is a more
appropriate place for this.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2018-09-05 10:17:11 -06:00
travisn f840e3574e mon: start a deployment instead of replicaset
Signed-off-by: travisn <tnielsen@redhat.com>
2018-09-01 00:12:42 -06:00
travisn 5f01178e83 mon: set names based on chars instead of integers
Signed-off-by: travisn <tnielsen@redhat.com>
2018-08-30 22:18:53 -06:00
Jared Watts 189e3eb611 ceph: migration logic for ceph.rook.io/v1alpha1 to ceph.rook.io/v1beta1, update all type references
Signed-off-by: Jared Watts <jbw976@gmail.com>
2018-07-11 09:23:46 -07:00
travisn 89cfccd5a2 enable rook to run on openshift
Signed-off-by: travisn <tnielsen@redhat.com>
2018-07-05 11:20:43 -06:00
Alexander Trost 86568cafd3 operator/ceph/mon: Cleaned up tests
To make it more applicable to what is actually running when using the
operator the used test mon names have been replaced by the actual format
used in normal operations: `rook-ceph-mon[0-9]` instead of `mon[0-9]`.

This also removes some duplicated test code and used the already
available function for it.

Signed-off-by: Alexander Trost <galexrt@googlemail.com>
2018-06-04 23:01:29 +02:00
Alexander Trost edfa135560 ceph mon: Only run one mon on a node
This introduces a MonSpec to the Ceph Cluster CRD which contains settings for
mons, current options are the count and if more than one mon is allowed
to run per node.

Fixes #1386

Signed-off-by: Alexander Trost <galexrt@googlemail.com>
2018-06-02 13:36:30 +02:00
Jared Watts f3df68e573 operator, daemon, and cmd updates for supporting multiple storage types
Signed-off-by: Jared Watts <jbw976@gmail.com>
2018-05-18 14:32:20 -07:00