Commit Graph
31 Commits
Author SHA1 Message Date
Rohan CJ fb4d23345e Revert "Merge pull request #2986 from rohantmp/mainMon"
This reverts commit b7a564e929, reversing
changes made to 6996571b5c.

Reverting: Linking ceph noout to a node taint.
The taint is applied for reasons other than maintenance as well.
This also needs a way to respect user-set noout.
Need to revisit this with a better design.

Signed-off-by: Rohan CJ <rohantmp@gmail.com>
2019-04-24 22:38:38 +05:30
Blaine Gardner f4e2be50a6 ceph spec: add ceph-version label to controllers
Add a 'ceph-version' label to application controllers in the same manner
as the prior 'rook-version' label to help with upgrades.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2019-04-18 15:09:33 -06:00
travisn bc87b440fb update to the k8s 1.14 client libraries
Signed-off-by: travisn <tnielsen@redhat.com>
2019-04-18 07:51:41 -06:00
Travis Nielsen b7a564e929 Merge pull request #2986 from rohantmp/mainMon
Ceph: Set noout on cluster when storage nodes are in maintenance
2019-04-17 09:56:50 -06:00
Travis Nielsen 6996571b5c Merge pull request #3005 from leseb/osd-mem-tune
ceph: osd, do not auto tune osd-memory-target
2019-04-17 09:56:03 -06:00
Rohan CJ 894f4bffbd - Set noout on cluster when storage nodes are in maintenance (unschedulable)
- Add noout functions to osd client
- export the functions DiscoverStorageNodes,GetAllStorageNodes
  from pkg/operator/ceph/cluster/osd
- Update PendingReleaseNotes.md

Signed-off-by: Rohan CJ <rohantmp@gmail.com>
2019-04-17 19:31:55 +05:30
Alexander Trost ea03048ef1 Merge pull request #2831 from galexrt/feature_2308
crds: Set annotations pods, deployments and so on
2019-04-17 10:27:14 +02:00
Sébastien Han f0b0f340f8 ceph: osd, do not auto tune osd-memory-target
As per https://github.com/ceph/ceph/pull/26856 Ceph tunes the value of
osd-memory-target when running the OSD process so there is no need to
enforce it.

We also set a safety factor to that value which has been calculated by
the Ceph eng/perf team. Without this factor, the OSD will get OOM killed
since it is consuming memory for other purposes too.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-04-17 00:14:04 +02:00
Alexander Trost 0c6eb7aa3d crds: Set annotations pods, deployments and so on
Allow the user to set annotations in the CRDs of the following operators which will then be set on
the resulting Pods (Deployments, StatefulSets and so on).
Operators:
* Cassandra
* Ceph
* CockroachDB
* EdgeFS
* Minio
* NFS

The Annotations related structures have been added to the
rook.io/v1alpha2 pkg.

Signed-off-by: Alexander Trost <galexrt@googlemail.com>
2019-04-16 20:47:05 +02:00
Noah Watkins c25e262c75 ceph/osd: mount /run/udev for lvs speedup
When Ceph OSD spins up using ceph-volume, `ceph-volume lvm activate
--no-systemd --bluestore...` runs `lvs` which can cause significant
delays when /run/udev is not mounted from the host.

Before:

[root@rook-ceph-osd-0-858f49b5-md4t9 /]# time lvs
  WARNING: Failed to connect to lvmetad. Falling back to device scanning.
  WARNING: Device /dev/sda not initialized in udev database even after waiting 10000000 microseconds.
...
  LV                                            VG                                        Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  osd-data-729c1fb6-440f-434c-be49-f199fa59f1c0 ceph-3bf7823b-6db3-42b6-bf8f-c36301e85238 -wi-ao---- <10.00g

real    1m30.545s

After:

[root@rook-ceph-osd-0-d8c4698b8-5vqnk /]# time lvs
  WARNING: Failed to connect to lvmetad. Falling back to device scanning.
  LV                                            VG                                        Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  osd-data-6c7249cc-cbd1-4f3b-9027-03725886bb30 ceph-9808319f-8567-4930-803a-41f70e537c1f -wi-ao---- <10.00g

real    0m0.043s

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
2019-04-16 11:09:13 -07:00
Sébastien Han 4438672e47 ceph: expose ceph version to osd pod
We must expose the ceph version to the osd pod so that we can generate
the proper ceph configuration file. If we don't do this all the version
check will assume they run the opposite of the desired version and the
formatting of the `ceph.conf` will be wrong.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-04-15 15:57:23 +02:00
Sébastien Han b8874d7f3f ceph: allow logging control
We now run all the daemon with a new option from Nautilus 14.2.1 which
allows us to tell to a daemon to not log on file. However, we can decide
to activate logging by editing the configuration flag 'log_to_file' via
the centralized config store like this for a particular daemon:

ceph config set mon.a log_to_file true

This is useful when a daemon keeps crashing and we want to collect log
files on the system.

Fixes: https://github.com/rook/rook/issues/2881
Signed-off-by: Sébastien Han <seb@redhat.com>
2019-04-15 10:52:03 +02:00
Sébastien Han d4c48bb293 ceph: expose /var/log/ceph to the host
It's quite convinient to expose /var/log/ceph so that we can decide to
activate logs locally on the machine and see what's going on.
This is only a placeholder when a daemon is stuck crashlooping and we
want to allow administrator to gather log files.
We still do not log on file but this can be activated via a config
option passed to the centralized config option store.

For some daemons, which typically do not store any data (rgw, rbd-mirror
and mds) we had to propagate dataDirHostPath from the cluster spec to
each creation call so that the bindmount can happen.

Fixes: https://github.com/rook/rook/issues/2881
Signed-off-by: Sébastien Han <seb@redhat.com>
2019-04-15 10:47:09 +02:00
Blaine Gardner 685a8df08d ceph: add 'rook-version' label to controllers
Add a `rook-version` label to all controller resources used by Ceph:
deployments, daemonsets, and jobs. Labels are added to controller
resources only and not to the pod templates within because changing pod
templates causes updates due to the label change. The controller
resources themselves are not updated with the label addition and
therefore don't cause an unnecessary upgrade.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2019-04-12 13:56:31 -06:00
Sébastien Han b7fd64074c ceph: osd spec: tune osd memory target
If the spec of the OSD daemon contains resource limits, especially
memory.limit we will use that value for setting the osd memory target
value so the bluestore OSD knows the amount of memory available for its
cache.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-03-15 11:45:29 +01:00
travisn 31848e23e7 add the CONTAINER_IMAGE env var to the ceph daemon pods
The mgr orchestrator modules need the container image to run
the ceph image for blinking the lights when a disk is down

Signed-off-by: travisn <tnielsen@redhat.com>
2019-03-04 13:33:41 -07:00
Travis Nielsen a145f4dcb7 Merge pull request #2703 from travisn/osds-per-device-with-filter
Clean up provisioning of OSDs on devices
2019-03-04 07:58:56 -07:00
Noah Watkins 6597531ed0 ceph: share host ipc namespace with provision pod
this is needed as cryptsetup in the pod and udev on the host coordinate
via a semaphore that must be visible in both contexts.

note that additionally the host needs to have dmcrypt compiled with udev
synchronization, which at least one minikube version does not have.

fixes: #923

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
2019-03-01 10:50:19 -08:00
travisn bdc3cf8146 osd: fix the device filter and improve device provisioning reliability
All devices detected by the discovery pod were being passed to the OSD provisioning pod
thus not always honoring the desired device list that should be provisioned.
Now the provisioning pod will be given the desired state from the crd,
then apply that state depending on the actual devices detected.
Also added a helper to ensure OSDsPerDevice is always valid.

Signed-off-by: travisn <tnielsen@redhat.com>
2019-02-26 16:11:25 -07:00
Sébastien Han eb3c424d57 Ceph messengers 2 support
This commit introduces the necessary changes to support the new
messenger feature coming with Ceph Nautilus (currently in development).

What changes? Now the monitor listens on two port:

* old 6789 for messenger v1, which will help us support older client
(e,g: krbd)
* new 3300 for messengers v2, which brings new improvement in the
messaging layer. This new transport layer brings numerous advantages
such as encryption improvement, speed improvement, pluggable nature to
support different network stack than TCP and many more.

We still have one Service IP, however it has 2 ports, see:

```
NAME                      TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
rook-ceph-mon-a           ClusterIP   10.106.217.160   <none>        3300/TCP,6789/TCP   4h
rook-ceph-mon-b           ClusterIP   10.99.36.175     <none>        3300/TCP,6789/TCP   4h
rook-ceph-mon-c           ClusterIP   10.108.220.74    <none>        3300/TCP,6789/TCP   4h
````

The `ceph.conf` has changed and we don't force the port when using an IP
address (public addr etc). Ceph, depending on its version will naturally
start the monitors on their right port, 6789.

A new --ceph-version-name CLI argument has been added to the Rook binary
so that when the pod starts it passes the ceph version name and the
configuration of the ceph.conf, as well as daemon startup flags, happen
properly.

Given that the Rook Operator remembers the port of all the monitors it
deployed (through Pod definition), this change is not an issue and will
maintain backward compatibility.

Note that to test this you must build rook with dev container image,
which contains the dev Nautilus version. So you should do something
like:

`make -j4 BASEIMAGE='ceph/daemon-base:latest-master' IMAGES='ceph' build`

Resolves: #2525
Signed-off-by: Sébastien Han <seb@redhat.com>
2019-02-22 00:09:37 +01:00
Huamin Chen 1b21e42a90 convert extensions to apps
Signed-off-by: Huamin Chen <hchen@redhat.com>
2019-02-14 16:06:05 -05:00
travisn 3a64288356 avoid a duplicate volume for a directory and dataDirHostPath
Signed-off-by: travisn <tnielsen@redhat.com>
2019-02-05 22:26:50 -07:00
travisnandjtlayton 37a005b71e nfs: configure and start daemons when the nfs crd is created
Co-authored-by: jtlayton <jlayton@redhat.com>
Signed-off-by: travisn <tnielsen@redhat.com>
2019-01-15 13:48:58 -07:00
travisn 1389cc56db ceph: enable multiple osds per device
Signed-off-by: travisn <tnielsen@redhat.com>
2018-12-07 09:29:15 -07:00
travisn e6572e41e6 enable ceph-volume to provision and start filestore osds
Signed-off-by: travisn <tnielsen@redhat.com>
2018-12-06 21:28:58 -07:00
travisn 45c53dd5b2 detect if ceph-volume is supported and allow legacy osds to be started
Signed-off-by: travisn <tnielsen@redhat.com>
2018-12-06 21:28:58 -07:00
travisn 78b8011226 osds: prototype provisioning with ceph-volume
Signed-off-by: travisn <tnielsen@redhat.com>
2018-12-06 21:28:58 -07:00
travisn ea03f58048 rbac: rename service account rook-ceph-cluster to rook-ceph-osd
Signed-off-by: travisn <tnielsen@redhat.com>
2018-11-30 13:39:58 -07:00
travisn 94a7a0d4bc run daemons with the ceph image instead of rook
Signed-off-by: travisn <tnielsen@redhat.com>
2018-11-01 11:49:35 -06:00
travisn 7d97b6ad76 osd: use the node hostname labels instead of node names
Signed-off-by: travisn <tnielsen@redhat.com>
2018-10-01 08:09:06 -06:00
travisn 15f6ee5b14 osd: enable filestore on device scenario to run with ceph image
update osd specs to be consistent with other daemons

Signed-off-by: travisn <tnielsen@redhat.com>
2018-09-26 10:36:16 -06:00