Commit Graph
566 Commits
Author SHA1 Message Date
Blaine Gardner a80082988d ceph: remove osds only when certain
Make the Ceph operator more cautious about when it decides to remove
nodes from the Rook-Ceph cluster which are acting as osd hosts.

When `useAllNodes` is set to `true` we assume that the user wants to
have the most hands-off experience. Node removals are allowed when a
node is delted from Kubernetes and when a node has its taints/affinities
modified by the user (but not by automatic k8s modification as much as
possible).

When `useAllnodes` is set to `false` the only time a node is removed is
if it is removed from the Ceph cluster definition.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2019-04-11 09:41:54 -06:00
Blaine Gardner f126887b52 tests: add T/F/Unk. status to node ready condition
Rook previously relied on the presence of `Ready` to determine if a node
is ready, but `Ready` can have different statuses. Correct this.

https://kubernetes.io/docs/concepts/architecture/nodes/#condition

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2019-04-09 15:40:37 -06:00
Noah Watkins 379cb98689 ceph: replace release name with fine-grained version
The basic idea here is to replace the release name in the ClusterInfo
object with the fine grained version information queried at runtime.
This patch also removes the Name field from the cluster spec, which was
also runtime determined and was redundant with ClusterInfo relase name.

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
2019-03-18 12:57:29 -07:00
Sébastien Han 871054e887 ceph: implement pod restriction on memory
If someone sets limits to pod, we want to ensure the possible
experience, so we want to make sure that people do not configure
inapropriate values for certain daemons.
We decide to fail if the memory.limit is too low.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-03-15 17:35:44 +01:00
Sébastien Han b7fd64074c ceph: osd spec: tune osd memory target
If the spec of the OSD daemon contains resource limits, especially
memory.limit we will use that value for setting the osd memory target
value so the bluestore OSD knows the amount of memory available for its
cache.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-03-15 11:45:29 +01:00
travisn 31848e23e7 add the CONTAINER_IMAGE env var to the ceph daemon pods
The mgr orchestrator modules need the container image to run
the ceph image for blinking the lights when a disk is down

Signed-off-by: travisn <tnielsen@redhat.com>
2019-03-04 13:33:41 -07:00
Travis Nielsen a145f4dcb7 Merge pull request #2703 from travisn/osds-per-device-with-filter
Clean up provisioning of OSDs on devices
2019-03-04 07:58:56 -07:00
Travis Nielsen 816d698655 Merge pull request #2742 from noahdesu/dmcrypt
ceph: share host ipc namespace with provision pod
2019-03-01 13:21:05 -07:00
Noah Watkins 6597531ed0 ceph: share host ipc namespace with provision pod
this is needed as cryptsetup in the pod and udev on the host coordinate
via a semaphore that must be visible in both contexts.

note that additionally the host needs to have dmcrypt compiled with udev
synchronization, which at least one minikube version does not have.

fixes: #923

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
2019-03-01 10:50:19 -08:00
Blaine Gardner 4eb4fb7aa6 ceph mds: configure completely from operator
Configure the Ceph mds daemon completely from the operator a la the
recent changes to the Ceph mon and mgr operators.

Create the mds deployments first and then
create the keyring secrets for them with their owner reference as the
corresponding deployment. This will mean that the secrets do not need to
be micromanaged. When the deployment is deleted, the secret is also
deleted. This has not been necessary for the mons or the manager since
the mons share a keyring with a lifespan of the cluster, as does the
mgr, which currently has single-mgr support only.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2019-02-28 12:31:43 -07:00
travisn bdc3cf8146 osd: fix the device filter and improve device provisioning reliability
All devices detected by the discovery pod were being passed to the OSD provisioning pod
thus not always honoring the desired device list that should be provisioned.
Now the provisioning pod will be given the desired state from the crd,
then apply that state depending on the actual devices detected.
Also added a helper to ensure OSDsPerDevice is always valid.

Signed-off-by: travisn <tnielsen@redhat.com>
2019-02-26 16:11:25 -07:00
Sébastien Han eb3c424d57 Ceph messengers 2 support
This commit introduces the necessary changes to support the new
messenger feature coming with Ceph Nautilus (currently in development).

What changes? Now the monitor listens on two port:

* old 6789 for messenger v1, which will help us support older client
(e,g: krbd)
* new 3300 for messengers v2, which brings new improvement in the
messaging layer. This new transport layer brings numerous advantages
such as encryption improvement, speed improvement, pluggable nature to
support different network stack than TCP and many more.

We still have one Service IP, however it has 2 ports, see:

```
NAME                      TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
rook-ceph-mon-a           ClusterIP   10.106.217.160   <none>        3300/TCP,6789/TCP   4h
rook-ceph-mon-b           ClusterIP   10.99.36.175     <none>        3300/TCP,6789/TCP   4h
rook-ceph-mon-c           ClusterIP   10.108.220.74    <none>        3300/TCP,6789/TCP   4h
````

The `ceph.conf` has changed and we don't force the port when using an IP
address (public addr etc). Ceph, depending on its version will naturally
start the monitors on their right port, 6789.

A new --ceph-version-name CLI argument has been added to the Rook binary
so that when the pod starts it passes the ceph version name and the
configuration of the ceph.conf, as well as daemon startup flags, happen
properly.

Given that the Rook Operator remembers the port of all the monitors it
deployed (through Pod definition), this change is not an issue and will
maintain backward compatibility.

Note that to test this you must build rook with dev container image,
which contains the dev Nautilus version. So you should do something
like:

`make -j4 BASEIMAGE='ceph/daemon-base:latest-master' IMAGES='ceph' build`

Resolves: #2525
Signed-off-by: Sébastien Han <seb@redhat.com>
2019-02-22 00:09:37 +01:00
Huamin Chen 1b21e42a90 convert extensions to apps
Signed-off-by: Huamin Chen <hchen@redhat.com>
2019-02-14 16:06:05 -05:00
travisn 3a64288356 avoid a duplicate volume for a directory and dataDirHostPath
Signed-off-by: travisn <tnielsen@redhat.com>
2019-02-05 22:26:50 -07:00
Blaine Gardner 78d4719125 ceph: no longer create fallback osd
Rook's behavior should not be to create a default OSD in dataDirHostPath
when no devices are present on a node. Any preexisting fallback osd
should be kept as long as no dirs or disks have been specified to keep
the legacy behavior for those clusters running with these osds in place.
The legacy deletion behavior is also kept, removing the osd as soon as
any dir or device is specified other than the dataDirHostPath dir.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2019-02-01 09:41:32 -07:00
travisnandjtlayton 37a005b71e nfs: configure and start daemons when the nfs crd is created
Co-authored-by: jtlayton <jlayton@redhat.com>
Signed-off-by: travisn <tnielsen@redhat.com>
2019-01-15 13:48:58 -07:00
travisn 1389cc56db ceph: enable multiple osds per device
Signed-off-by: travisn <tnielsen@redhat.com>
2018-12-07 09:29:15 -07:00
travisn e6572e41e6 enable ceph-volume to provision and start filestore osds
Signed-off-by: travisn <tnielsen@redhat.com>
2018-12-06 21:28:58 -07:00
travisn 45c53dd5b2 detect if ceph-volume is supported and allow legacy osds to be started
Signed-off-by: travisn <tnielsen@redhat.com>
2018-12-06 21:28:58 -07:00
travisn 78b8011226 osds: prototype provisioning with ceph-volume
Signed-off-by: travisn <tnielsen@redhat.com>
2018-12-06 21:28:58 -07:00
travisn 62f7e5d6ea ceph: update docs, code, and tests to use v1 crd types
Signed-off-by: travisn <tnielsen@redhat.com>
2018-12-05 14:33:11 -07:00
travisn ea03f58048 rbac: rename service account rook-ceph-cluster to rook-ceph-osd
Signed-off-by: travisn <tnielsen@redhat.com>
2018-11-30 13:39:58 -07:00
Alexander Trost b12d1e82cf Use truncated node name "everywhere"
Signed-off-by: Alexander Trost <galexrt@googlemail.com>
2018-11-23 10:02:35 +01:00
travisn b2a0dc77fa automatically detect the major version of the Ceph image
Signed-off-by: travisn <tnielsen@redhat.com>
2018-11-01 11:49:36 -06:00
travisn 94a7a0d4bc run daemons with the ceph image instead of rook
Signed-off-by: travisn <tnielsen@redhat.com>
2018-11-01 11:49:35 -06:00
travisn 7d97b6ad76 osd: use the node hostname labels instead of node names
Signed-off-by: travisn <tnielsen@redhat.com>
2018-10-01 08:09:06 -06:00
travisn 15f6ee5b14 osd: enable filestore on device scenario to run with ceph image
update osd specs to be consistent with other daemons

Signed-off-by: travisn <tnielsen@redhat.com>
2018-09-26 10:36:16 -06:00
Noah Watkins e015a21cbd ceph: expose node,pod,ns names to ceph daemons
Ceph expects to be able to reason about physical hardware / nodes.
Before this patch the NODE_NAME that was available referred to the pod
name. This patch exposes the underlying node name, pod, and pod
namespace for the ceph daemons to consume.

fixes: #2078

Signed-off-by: Noah Watkins <nwatkins@redhat.com>
2018-09-24 13:26:24 -07:00
Huamin Chen 1c225185b6 adjust cluster and osd provision timeout
Signed-off-by: Huamin Chen <hchen@redhat.com>
2018-09-13 11:21:11 -04:00
Huamin Chen 9ec37d5c00 reset resource version if watch channel closed
Signed-off-by: Huamin Chen <hchen@redhat.com>
2018-09-11 13:27:26 -04:00
travisn c0af417b00 osd: delete legacy deployment with -id in the name
Signed-off-by: travisn <tnielsen@redhat.com>
2018-09-04 14:15:18 -05:00
Sage Weil 8e773612ca Remove unnecessary "-id" from OSD pod names
The -id portion of the name is unnecessary and inconsistent with the other
pod names.

Signed-off-by: Sage Weil <sage@redhat.com>
2018-09-04 14:15:18 -05:00
Blaine Gardner 406067788d ceph osd oper: Allow explicit runAsNonRoot=false
Rook no longer supports Kubernetes < 1.7
Remove a workaround for k8s < 1.6.4 where `runAsNonRoot` could not be
explicitly set to false.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2018-08-24 15:11:45 -06:00
Dennis Chen ec84e6543c Return immediately when no valid node to run osd
We don't need to handle other osd(s) related tasks if there's
no valid nodes available in the cluster to run osd(s), log a
warning message to help user identify the cause of the issue.

Signed-off-by: Dennis Chen <dennis.chen@arm.com>
2018-08-07 10:09:14 +08:00
travisn 1d928ecd36 osd: init container to initialize config at startup
Signed-off-by: travisn <tnielsen@redhat.com>
2018-07-31 14:39:53 -06:00
Travis Nielsen 5e4fba2aec Merge pull request #1952 from SUSE/simple-linting
Fix spellcheck and trailing space/newline issues
2018-07-27 14:02:00 -06:00
Travis Nielsen add4bd2fd2 Merge pull request #1926 from galexrt/fix_1920
operator/osd: Use UpdateStrategy Recreate
2018-07-26 14:45:56 -06:00
Blaine Gardner fe5fb394c7 Fix spellcheck and trailing space/newline issues
Fix some basic spellcheck errors. Also remove trailing spaces and make
sure files have a newline (my editor does automatically).

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2018-07-26 11:36:09 -06:00
Huamin Chen c6f463d0e7 validate storage nodes before preparing osds
Signed-off-by: Huamin Chen <hchen@redhat.com>
2018-07-26 17:18:47 +00:00
Alexander Trost 56544c51d4 operator/osd: Use UpdateStrategy Recreate
This fixes issues with a new OSD Pod in the same Deployment coming up and
directly entering CrashLoopBackOff because the old OSD Pod hasn't been fully
terminated yet (released it's locks).

Signed-off-by: Alexander Trost <galexrt@googlemail.com>
2018-07-21 12:10:09 +02:00
Alexander Trost faddfb0184 Truncate too long name for ObjectMeta.Name
Resolves #1907

Signed-off-by: Alexander Trost <galexrt@googlemail.com>
2018-07-20 10:35:20 +02:00
Travis Nielsen 363e5ee58a Merge pull request #1892 from jcsp/wip-osd-remove-idempotent
Fix retry of OSD deletion
2018-07-17 15:47:00 -06:00
John Spray bed1d01e9c Fix retry of OSD deletion
Previously, if this failed mid-way it would then throw
an exception at the first operation when retrying (because it's
trying to reweight an OSD that might not be in the map any more)

Signed-off-by: John Spray <john.spray@redhat.com>
2018-07-17 14:08:23 +01:00
Alexander Trost cdb708f6a1 Apply placement to OSD Pod Spec
Resolves #1895

Signed-off-by: Alexander Trost <galexrt@googlemail.com>
2018-07-17 14:01:08 +02:00
Jared Watts a7ce7a361b Merge pull request #1766 from galexrt/1708
osd: skip wait loop when initial PGs is 0
2018-07-11 09:38:22 -07:00
Alexander Trost 979360e85b osd: skip wait loop when initial PGs or used KBs is 0 for an OSD
Resolves #1708

Signed-off-by: Alexander Trost <galexrt@googlemail.com>
2018-07-11 07:59:43 +02:00
travisn 962b4f22da osd: available devices should include previously configured devices
Signed-off-by: travisn <tnielsen@redhat.com>
2018-07-10 12:41:43 -06:00
travisn 86f6b3ccf0 avoid setting ownerref when not available for crds
Signed-off-by: travisn <tnielsen@redhat.com>
2018-07-06 16:35:31 -06:00
travisn 55e0ee26c0 monitor the OSDs for the lifetime of the cluster
Signed-off-by: travisn <tnielsen@redhat.com>
2018-07-06 15:08:59 -06:00
travisn c9406597e1 osd: remove storage nodes centrally in the operator
Signed-off-by: travisn <tnielsen@redhat.com>
2018-07-05 16:09:22 -06:00