Commit Graph
60 Commits
Author SHA1 Message Date
Sébastien Han 93b2448619 ceph: refactor rgw bootstrap
This commit does multiple things:

* remove support for AllNodes where we would deploy one rgw per node on
all the nodes.
* a transition path is implemented in the code so that if someone has an
existing deployment, daemonsets will be removed and replaced by an
deployments.
* when using "instances", each rgw deployed has its own key which makes
Ceph reporting the exact number of rgw running, see:

```
[root@rook-ceph-operator-775cf575c5-bh4sr /]# ceph -s
  cluster:
    id:     611fcf39-0669-4864-9a12-debb35c0397a
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum a,b,c (age 12h)
    mgr: a(active, since 12h)
    osd: 3 osds: 3 up (since 12h), 3 in (since 12h)
    rgw: 3 daemons active (my.store.a, my.store.b, my.store.c)

  data:
    pools:   6 pools, 600 pgs
    objects: 235 objects, 3.8 KiB
    usage:   3.0 GiB used, 84 GiB / 87 GiB avail
    pgs:     600 active+clean
```

Closes: https://github.com/rook/rook/issues/2474, https://github.com/rook/rook/issues/2957 and https://github.com/rook/rook/issues/3245
Signed-off-by: Sébastien Han <seb@redhat.com>
2019-06-14 17:55:15 +02:00
Christian Hüning f0699c40bf Added Liveness Probe to Ceph Mgr
This will configure a liveness probe for every ceph mgr deployment

Signed-off-by: Christian Hüning <christian.huening@figo.io>
2019-05-17 07:24:28 +02:00
Noah Watkins e33eb98047 ceph: run self-signed cert creation with timeout
fixes: #2784

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
2019-05-06 10:10:05 -07:00
Noah Watkins c2dbb93096 ceph: simplify the ceph command execution interface
this adds a structure to hold various settings related to executing ceph
cli commands, and introduces a common interface for configuring and
running such commands.

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
2019-05-06 10:10:05 -07:00
Noah Watkins a661b2db02 ceph: rewind http bind fix on upgrade
when upgrading to a newer version of ceph that doesn't need the http
binding fix, this patch unwinds the configuration fix which may still be
present when upgrading from a cluster without the fix. if the
configuration fix remains then the dashboard and prometheus servers will
not be able to properly configure themselves.

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
2019-05-01 22:30:12 -07:00
Noah Watkins fc7baef6aa ceph: add tests for dashboard http bind fix
Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
2019-04-30 10:57:37 -07:00
Noah Watkins 1efb8af87b ceph: selectively apply http bind addr fix
fixes: #2492

the http bind address fix in rook address an upstream issue in ceph in
which cherrypy break when binding to `::` when the host doesn't have an
ipv6 address. it fixes the issue by always binding prometheus and the
dashboard webservers to the pod address. by doing this, it prevents
using the kubectl port-forward technique, which some users would like.

a patch to cherrypy is upstream in ceph. this PR checks the ceph version
and selectively disables the fix so that both port-forward and the
normal service method works. this is the verison matrix:

          FIX     VER
          ---     ---
luminous merged (>= 12.2.12)
mimic    merged (>= 13.2.6)
nautilus merged (>= 14.1.1)

when all the containers for these versions exists and sufficient time
passes the unreachable code guarded by the version checks in this patch
can be removed.

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
2019-04-30 10:06:39 -07:00
wangxiao86 a5a8d22478 Set the module's server_addr path with the mgr's name.
To enable multiple mgrs to run properly, we need to set the respective server_addr path with the mgr's name for the modules of each mgr.

When multiple mgrs are running, the modules( e.g. dashboard and Prometheus ) of each mgr need to bind its own server_addr when it is started. The way to get server_addr when ceph mgr starts the module is as follows:  first,  get the server_addr value by "mgr/<module-name>/<mgr-daemonid>/server_addr" path;  and if this path does not exist, then get the server_addr value by" mgr/<module-name>/server_addr" path.
According to the original rook setting for server_addr,  the modules of all mgrs take the same server_addr on ceph luminous and mimic, this produces a wrong result that only one mgr's modules can run properly ( the modules of other mgrs failed to start with the wrong server_addr).
This change is made to be compatible with ceph luminous, mimic, and nautilus.  and Modify according to: the "HOST NAME AND PORT" description of http://docs.ceph.com/docs/master/mgr/dashboard/ .

Signed-off-by: wangxiao86 <wangxiao1@sensetime.com>
2019-04-30 17:14:07 +08:00
Maximilian Mack bdbd67fd7f ceph: fix mgr - dashboard settings
This fix passes on the mgrName to the configureDashboard function.

Signed-off-by: Maximilian Mack <max@mack.io>
2019-04-24 08:26:22 +02:00
Blaine Gardner f4e2be50a6 ceph spec: add ceph-version label to controllers
Add a 'ceph-version' label to application controllers in the same manner
as the prior 'rook-version' label to help with upgrades.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2019-04-18 15:09:33 -06:00
Alexander Trost 0c6eb7aa3d crds: Set annotations pods, deployments and so on
Allow the user to set annotations in the CRDs of the following operators which will then be set on
the resulting Pods (Deployments, StatefulSets and so on).
Operators:
* Cassandra
* Ceph
* CockroachDB
* EdgeFS
* Minio
* NFS

The Annotations related structures have been added to the
rook.io/v1alpha2 pkg.

Signed-off-by: Alexander Trost <galexrt@googlemail.com>
2019-04-16 20:47:05 +02:00
Sébastien Han d4c48bb293 ceph: expose /var/log/ceph to the host
It's quite convinient to expose /var/log/ceph so that we can decide to
activate logs locally on the machine and see what's going on.
This is only a placeholder when a daemon is stuck crashlooping and we
want to allow administrator to gather log files.
We still do not log on file but this can be activated via a config
option passed to the centralized config option store.

For some daemons, which typically do not store any data (rgw, rbd-mirror
and mds) we had to propagate dataDirHostPath from the cluster spec to
each creation call so that the bindmount can happen.

Fixes: https://github.com/rook/rook/issues/2881
Signed-off-by: Sébastien Han <seb@redhat.com>
2019-04-15 10:47:09 +02:00
Blaine Gardner 685a8df08d ceph: add 'rook-version' label to controllers
Add a `rook-version` label to all controller resources used by Ceph:
deployments, daemonsets, and jobs. Labels are added to controller
resources only and not to the pod templates within because changing pod
templates causes updates due to the label change. The controller
resources themselves are not updated with the label addition and
therefore don't cause an unnecessary upgrade.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2019-04-12 13:56:31 -06:00
Alexander Trost 8bc26d9096 k8sclient: Update all operators to use apps/v1
All usages of k8s go client are now also using the versioned `AppsV1() `
call for the client.

Updated MySQL and Wordpress, and Kube Registy examples to use apps/v1
Deployments.

Signed-off-by: Alexander Trost <galexrt@googlemail.com>
2019-04-12 09:25:42 +02:00
rohan47 0ecce1015c ceph: update env vars for ceph-mgr containers
pass env vars with rook-ceph-system namespace and API version to
ceph-mgr containers to remove hardcoded values from rook orchestrator in
ceph-mgr and updated the unit tests for the same

Signed-off-by: rohan47 <rohgupta@redhat.com>
2019-04-04 02:11:44 +05:30
Blaine Gardner 7f26f4aa13 ceph spec tests: allow adding required env vars
Allow adding additional required environment variables on a situational
basis to Ceph's spec unit tests.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
(cherry picked from commit 94bf2a6e14b88a8e965b4ad5ce1289b606e5c32d)
2019-04-04 02:11:44 +05:30
Noah Watkins 379cb98689 ceph: replace release name with fine-grained version
The basic idea here is to replace the release name in the ClusterInfo
object with the fine grained version information queried at runtime.
This patch also removes the Name field from the cluster spec, which was
also runtime determined and was redundant with ClusterInfo relase name.

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
2019-03-18 12:57:29 -07:00
Travis Nielsen 511cc6c365 Merge pull request #2764 from leseb/ceph-process-tuning
Ceph memory process tuning
2019-03-15 12:18:33 -06:00
Sébastien Han 871054e887 ceph: implement pod restriction on memory
If someone sets limits to pod, we want to ensure the possible
experience, so we want to make sure that people do not configure
inapropriate values for certain daemons.
We decide to fail if the memory.limit is too low.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-03-15 17:35:44 +01:00
Sébastien Han fa61c0441b ceph: pod: expose resource limits and requests
We now expose resource as environment variable in the pod.
They will be used by Ceph to report as a daemon metadata.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-03-15 11:45:29 +01:00
Noah Watkins 1ee7cdfa8b ceph: refactor dashboard retry logic to use helper
Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
2019-03-14 14:52:44 -07:00
Noah Watkins 8ebf9ac722 ceph: retry dashboard credentials setup
may need retry if manager hasn't fully loaded its modules.

fixes: #2809

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
2019-03-14 14:51:50 -07:00
Blaine Gardner c12aa6c7c5 ceph rbd-mirror: configure entirely in operator
In the form of mon, mgr, mds, and rgw, convert the rbd-mirror to be
configured completely from the operator. This is a straightforward
conversion with one functional addition: the rbd-mirror daemon stores
*no* data and has no default data dir, so the concept of a `Dataless`
daemon is here introduced.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2019-03-13 09:54:09 -06:00
travisn 31848e23e7 add the CONTAINER_IMAGE env var to the ceph daemon pods
The mgr orchestrator modules need the container image to run
the ceph image for blinking the lights when a disk is down

Signed-off-by: travisn <tnielsen@redhat.com>
2019-03-04 13:33:41 -07:00
Travis Nielsen eaf6b97b59 Merge pull request #2702 from SUSE/mds-all-in-operator
ceph mds: configure completely from operator
2019-03-01 06:59:15 -07:00
Blaine Gardner 4eb4fb7aa6 ceph mds: configure completely from operator
Configure the Ceph mds daemon completely from the operator a la the
recent changes to the Ceph mon and mgr operators.

Create the mds deployments first and then
create the keyring secrets for them with their owner reference as the
corresponding deployment. This will mean that the secrets do not need to
be micromanaged. When the deployment is deleted, the secret is also
deleted. This has not been necessary for the mons or the manager since
the mons share a keyring with a lifespan of the cluster, as does the
mgr, which currently has single-mgr support only.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2019-02-28 12:31:43 -07:00
Jeff Layton c0910dab2c ceph: retry "ceph orchestrator set backend rook" if command returns EINVAL
Use similar method as that used by the dashboard initialization code.

Fixes: https://github.com/rook/rook/issues/2617
Signed-off-by: Jeff Layton <jlayton@redhat.com>
2019-02-27 10:55:24 -05:00
Jeff Layton 56abc00b78 ceph: fix invalidArgErrorCode definition
We should probably not hardcode this value here, and instead use
whatever the syscall module thinks this is.

Signed-off-by: Jeff Layton <jlayton@redhat.com>
2019-02-27 10:53:10 -05:00
Sébastien Han eb3c424d57 Ceph messengers 2 support
This commit introduces the necessary changes to support the new
messenger feature coming with Ceph Nautilus (currently in development).

What changes? Now the monitor listens on two port:

* old 6789 for messenger v1, which will help us support older client
(e,g: krbd)
* new 3300 for messengers v2, which brings new improvement in the
messaging layer. This new transport layer brings numerous advantages
such as encryption improvement, speed improvement, pluggable nature to
support different network stack than TCP and many more.

We still have one Service IP, however it has 2 ports, see:

```
NAME                      TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
rook-ceph-mon-a           ClusterIP   10.106.217.160   <none>        3300/TCP,6789/TCP   4h
rook-ceph-mon-b           ClusterIP   10.99.36.175     <none>        3300/TCP,6789/TCP   4h
rook-ceph-mon-c           ClusterIP   10.108.220.74    <none>        3300/TCP,6789/TCP   4h
````

The `ceph.conf` has changed and we don't force the port when using an IP
address (public addr etc). Ceph, depending on its version will naturally
start the monitors on their right port, 6789.

A new --ceph-version-name CLI argument has been added to the Rook binary
so that when the pod starts it passes the ceph version name and the
configuration of the ceph.conf, as well as daemon startup flags, happen
properly.

Given that the Rook Operator remembers the port of all the monitors it
deployed (through Pod definition), this change is not an issue and will
maintain backward compatibility.

Note that to test this you must build rook with dev container image,
which contains the dev Nautilus version. So you should do something
like:

`make -j4 BASEIMAGE='ceph/daemon-base:latest-master' IMAGES='ceph' build`

Resolves: #2525
Signed-off-by: Sébastien Han <seb@redhat.com>
2019-02-22 00:09:37 +01:00
Huamin Chen 1b21e42a90 convert extensions to apps
Signed-off-by: Huamin Chen <hchen@redhat.com>
2019-02-14 16:06:05 -05:00
Blaine Gardner 03ea5f0a7c ceph mgr: configure completely from operator
Configure the Ceph mgr daemon completely from the operator a la the
recent changes to the Ceph mon operator.

The pod spec for the mgr changed quite a bit, and instead of updating
the unit tests of questionable use, some additional unit test tools
applicable to any Ceph daemon have been added and used with the mgr. The
mgrs unit tests should now be more useful and get in the way of devs
less, and they can be used by other daemons later.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2019-02-11 14:35:01 -07:00
travisn 9afbe36ae6 set mgr privs for access to the rbd mirroring settings
Signed-off-by: travisn <tnielsen@redhat.com>
2019-01-19 21:29:19 -07:00
Noah Watkins e9a9509313 ceph: support dashboard ssl on/off
fixes: #2433

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
2019-01-09 12:08:51 -08:00
Noah Watkins e66ad7463b ceph: support custom dashboard port
Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
2019-01-09 12:08:49 -08:00
Travis Nielsen 2da22001d6 Merge pull request #2414 from travisn/retry-dashboard-cert
Retry creating self signed cert if module is not ready
2018-12-21 07:43:42 -07:00
travisn b770b80a88 mgr: retry creating self signed cert if module is not ready
Signed-off-by: travisn <tnielsen@redhat.com>
2018-12-20 16:07:00 -07:00
travisn 261aae3f46 mgr: set the server_addr for the dashboard and prometheus modules
Signed-off-by: travisn <tnielsen@redhat.com>
2018-12-20 12:08:37 -07:00
Alexander Trost 51a6de0151 mgr: use Recreate update strategy
Signed-off-by: Alexander Trost <galexrt@googlemail.com>
2018-12-07 19:38:45 -07:00
travisn 62f7e5d6ea ceph: update docs, code, and tests to use v1 crd types
Signed-off-by: travisn <tnielsen@redhat.com>
2018-12-05 14:33:11 -07:00
Blaine Gardner 290b496069 ceph mon/mgr/mds units: verify updated deployments
Verify that the expected deployments are updated in the Ceph mgr,
and mds unit tests.

This also allows those unit tests to pass at all since
UpdateDeploymentAndWait was blocking the unit tests from finishing due
to unexpected behavior of generated `Deployment.Update` unit test mock.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2018-12-04 11:35:06 -07:00
Blaine Gardner 3d89c78508 ceph: Update deployments on rook img update
Use the `UpdateDeploymentAndWait` method used by the osd operator to
update the deployments for mgrs, mdses, and rgws when the Rook
orchestrator image is updated.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2018-12-04 11:33:40 -07:00
travisn 16c2a56591 add ceph mgr roles to the tests and helm chart
Signed-off-by: travisn <tnielsen@redhat.com>
2018-11-30 13:39:58 -07:00
John Spray cff953696c Set service account on ceph-mgr pod
This is to enable the `rook` ceph-mgr module to drive
the Rook and k8s APIs, exposing functionality via
its own CLI and GUI.

The service account gets access to two different roles:
one for the items that are within the cluster's namespace,
and one for reading configmaps in the rook-ceph-system namespace,
which is needed for reporting available devices.

Signed-off-by: John Spray <john.spray@redhat.com>
2018-11-30 10:50:20 -07:00
travisn fdd64bb95a enable rbd mirror daemon in the cluster crd
Signed-off-by: travisn <tnielsen@redhat.com>
2018-11-28 08:29:57 -07:00
Andrey Laguta c247e8c0f7 Add urlPrefix setting for ceph dashboard
Signed-off-by: Andrey Laguta <13280256+laevilgenius@users.noreply.github.com>
2018-11-06 02:58:47 +02:00
travisn 6cfd2ee11a port 7000 for the luminous dashboard and 8443 for mimic and above
Signed-off-by: travisn <tnielsen@redhat.com>
2018-11-01 11:49:37 -06:00
travisn 906add05ce consume the ceph images from the ceph/ceph repo
Signed-off-by: travisn <tnielsen@redhat.com>
2018-11-01 11:49:37 -06:00
travisn f5b09a5ca5 enable the orchestrator modules on nautilus
Signed-off-by: travisn <tnielsen@redhat.com>
2018-11-01 11:49:37 -06:00
travisn 39117a7a0b mgr: configure a self-signed cert and admin user for the dashboard
Signed-off-by: travisn <tnielsen@redhat.com>
2018-11-01 11:49:36 -06:00
travisn 94a7a0d4bc run daemons with the ceph image instead of rook
Signed-off-by: travisn <tnielsen@redhat.com>
2018-11-01 11:49:35 -06:00