Commit Graph
90 Commits
Author SHA1 Message Date
Sébastien Han cba9a359a0 ceph: upgrade apply osd nautilus flag
When OSDs are running on Nautilus we always disable old osd features and
aplpy the onces for Nautilus as described in the upgrade doc.
During an upgrade or the next time an orchestration will be called the
command will be applied. The command is idempotent so we can run it each
time.
This can be backported for 1.0.3

Closes: https://github.com/rook/rook/issues/2960
Signed-off-by: Sébastien Han <seb@redhat.com>
2019-06-13 22:15:15 +02:00
rohan47 a380ca7dc1 fixed ceph pg dump pgs_brief json deserialization
Signed-off-by: rohan47 <rohgupta@redhat.com>
2019-06-06 21:54:37 +05:30
Noah Watkins 71c88ae9e9 ceph: log osd recovery at info level
for the osd recovery case (DOWN->UP) log at a matching log level as the
message indicating the OSD went down.

fixes: #2904

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
2019-06-04 13:57:49 -07:00
Jose A. Rivera f3a1512983 Read node labels for provider topology
Signed-off-by: Jose A. Rivera <jarrpa@redhat.com>
2019-05-14 13:18:00 -05:00
Travis Nielsen 221560bf24 Merge pull request #3090 from kshlm/delete-active-job-if-requested
ceph: Delete and recreate rook-ceph-detect-version job
2019-05-01 10:16:18 -06:00
travisn 77c7eb52f9 ceph: query the latest list of nodes for osd orchestration
The list of nodes to be queried for configuring OSDs
should include all of the K8s nodes before the placement
or other filters are applied. The operator was sometimes
missing configuring OSDs on nodes if the discover pod
hadn't finished running on a node. Now the osd prepare
job will be triggered on a node even if the discover pod
hasn't yet run there.

Signed-off-by: travisn <tnielsen@redhat.com>
2019-04-30 17:07:12 -06:00
Kaushal M 5e8375e921 ceph: Delete and recreate rook-ceph-detect-version job
The ceph controller will delete and recreate the detec-version job, when
an already existing job is found. This helps avoid cluster creation from
getting hung up on waiting for an already existing/stale job to finish.

Fixes #3081.

Signed-off-by: Kaushal M <kshlmster@gmail.com>
2019-04-30 18:19:56 +05:30
travisn b6568b07b4 ceph: stop disabling scrubbing during osd orchestration
Signed-off-by: travisn <tnielsen@redhat.com>
2019-04-26 09:35:50 -06:00
Rohan CJ fb4d23345e Revert "Merge pull request #2986 from rohantmp/mainMon"
This reverts commit b7a564e929, reversing
changes made to 6996571b5c.

Reverting: Linking ceph noout to a node taint.
The taint is applied for reasons other than maintenance as well.
This also needs a way to respect user-set noout.
Need to revisit this with a better design.

Signed-off-by: Rohan CJ <rohantmp@gmail.com>
2019-04-24 22:38:38 +05:30
Blaine Gardner f4e2be50a6 ceph spec: add ceph-version label to controllers
Add a 'ceph-version' label to application controllers in the same manner
as the prior 'rook-version' label to help with upgrades.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2019-04-18 15:09:33 -06:00
travisn bc87b440fb update to the k8s 1.14 client libraries
Signed-off-by: travisn <tnielsen@redhat.com>
2019-04-18 07:51:41 -06:00
Travis Nielsen b7a564e929 Merge pull request #2986 from rohantmp/mainMon
Ceph: Set noout on cluster when storage nodes are in maintenance
2019-04-17 09:56:50 -06:00
Travis Nielsen 6996571b5c Merge pull request #3005 from leseb/osd-mem-tune
ceph: osd, do not auto tune osd-memory-target
2019-04-17 09:56:03 -06:00
Rohan CJ 894f4bffbd - Set noout on cluster when storage nodes are in maintenance (unschedulable)
- Add noout functions to osd client
- export the functions DiscoverStorageNodes,GetAllStorageNodes
  from pkg/operator/ceph/cluster/osd
- Update PendingReleaseNotes.md

Signed-off-by: Rohan CJ <rohantmp@gmail.com>
2019-04-17 19:31:55 +05:30
Alexander Trost ea03048ef1 Merge pull request #2831 from galexrt/feature_2308
crds: Set annotations pods, deployments and so on
2019-04-17 10:27:14 +02:00
Sébastien Han f0b0f340f8 ceph: osd, do not auto tune osd-memory-target
As per https://github.com/ceph/ceph/pull/26856 Ceph tunes the value of
osd-memory-target when running the OSD process so there is no need to
enforce it.

We also set a safety factor to that value which has been calculated by
the Ceph eng/perf team. Without this factor, the OSD will get OOM killed
since it is consuming memory for other purposes too.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-04-17 00:14:04 +02:00
Alexander Trost 0c6eb7aa3d crds: Set annotations pods, deployments and so on
Allow the user to set annotations in the CRDs of the following operators which will then be set on
the resulting Pods (Deployments, StatefulSets and so on).
Operators:
* Cassandra
* Ceph
* CockroachDB
* EdgeFS
* Minio
* NFS

The Annotations related structures have been added to the
rook.io/v1alpha2 pkg.

Signed-off-by: Alexander Trost <galexrt@googlemail.com>
2019-04-16 20:47:05 +02:00
Noah Watkins c25e262c75 ceph/osd: mount /run/udev for lvs speedup
When Ceph OSD spins up using ceph-volume, `ceph-volume lvm activate
--no-systemd --bluestore...` runs `lvs` which can cause significant
delays when /run/udev is not mounted from the host.

Before:

[root@rook-ceph-osd-0-858f49b5-md4t9 /]# time lvs
  WARNING: Failed to connect to lvmetad. Falling back to device scanning.
  WARNING: Device /dev/sda not initialized in udev database even after waiting 10000000 microseconds.
...
  LV                                            VG                                        Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  osd-data-729c1fb6-440f-434c-be49-f199fa59f1c0 ceph-3bf7823b-6db3-42b6-bf8f-c36301e85238 -wi-ao---- <10.00g

real    1m30.545s

After:

[root@rook-ceph-osd-0-d8c4698b8-5vqnk /]# time lvs
  WARNING: Failed to connect to lvmetad. Falling back to device scanning.
  LV                                            VG                                        Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  osd-data-6c7249cc-cbd1-4f3b-9027-03725886bb30 ceph-9808319f-8567-4930-803a-41f70e537c1f -wi-ao---- <10.00g

real    0m0.043s

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
2019-04-16 11:09:13 -07:00
Sébastien Han 4438672e47 ceph: expose ceph version to osd pod
We must expose the ceph version to the osd pod so that we can generate
the proper ceph configuration file. If we don't do this all the version
check will assume they run the opposite of the desired version and the
formatting of the `ceph.conf` will be wrong.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-04-15 15:57:23 +02:00
Sébastien Han b8874d7f3f ceph: allow logging control
We now run all the daemon with a new option from Nautilus 14.2.1 which
allows us to tell to a daemon to not log on file. However, we can decide
to activate logging by editing the configuration flag 'log_to_file' via
the centralized config store like this for a particular daemon:

ceph config set mon.a log_to_file true

This is useful when a daemon keeps crashing and we want to collect log
files on the system.

Fixes: https://github.com/rook/rook/issues/2881
Signed-off-by: Sébastien Han <seb@redhat.com>
2019-04-15 10:52:03 +02:00
Sébastien Han d4c48bb293 ceph: expose /var/log/ceph to the host
It's quite convinient to expose /var/log/ceph so that we can decide to
activate logs locally on the machine and see what's going on.
This is only a placeholder when a daemon is stuck crashlooping and we
want to allow administrator to gather log files.
We still do not log on file but this can be activated via a config
option passed to the centralized config option store.

For some daemons, which typically do not store any data (rgw, rbd-mirror
and mds) we had to propagate dataDirHostPath from the cluster spec to
each creation call so that the bindmount can happen.

Fixes: https://github.com/rook/rook/issues/2881
Signed-off-by: Sébastien Han <seb@redhat.com>
2019-04-15 10:47:09 +02:00
Sébastien Han 4627b66636 ceph: expose clusterInfo to the osd code
When instanciating an osd, we now pass clusterInfo which basically
represents a collection of information about a particular Ceph cluster. Rook uses information
// about the cluster to configure daemons to connect to the desired cluster.

From clusterInfo, we use the CephVersion which helps us putting
conditions for our code execution.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-04-15 10:47:09 +02:00
Blaine Gardner 685a8df08d ceph: add 'rook-version' label to controllers
Add a `rook-version` label to all controller resources used by Ceph:
deployments, daemonsets, and jobs. Labels are added to controller
resources only and not to the pod templates within because changing pod
templates causes updates due to the label change. The controller
resources themselves are not updated with the label addition and
therefore don't cause an unnecessary upgrade.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2019-04-12 13:56:31 -06:00
Alexander Trost 8bc26d9096 k8sclient: Update all operators to use apps/v1
All usages of k8s go client are now also using the versioned `AppsV1() `
call for the client.

Updated MySQL and Wordpress, and Kube Registy examples to use apps/v1
Deployments.

Signed-off-by: Alexander Trost <galexrt@googlemail.com>
2019-04-12 09:25:42 +02:00
Blaine Gardner a80082988d ceph: remove osds only when certain
Make the Ceph operator more cautious about when it decides to remove
nodes from the Rook-Ceph cluster which are acting as osd hosts.

When `useAllNodes` is set to `true` we assume that the user wants to
have the most hands-off experience. Node removals are allowed when a
node is delted from Kubernetes and when a node has its taints/affinities
modified by the user (but not by automatic k8s modification as much as
possible).

When `useAllnodes` is set to `false` the only time a node is removed is
if it is removed from the Ceph cluster definition.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2019-04-11 09:41:54 -06:00
Blaine Gardner f126887b52 tests: add T/F/Unk. status to node ready condition
Rook previously relied on the presence of `Ready` to determine if a node
is ready, but `Ready` can have different statuses. Correct this.

https://kubernetes.io/docs/concepts/architecture/nodes/#condition

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2019-04-09 15:40:37 -06:00
Noah Watkins 379cb98689 ceph: replace release name with fine-grained version
The basic idea here is to replace the release name in the ClusterInfo
object with the fine grained version information queried at runtime.
This patch also removes the Name field from the cluster spec, which was
also runtime determined and was redundant with ClusterInfo relase name.

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
2019-03-18 12:57:29 -07:00
Sébastien Han 871054e887 ceph: implement pod restriction on memory
If someone sets limits to pod, we want to ensure the possible
experience, so we want to make sure that people do not configure
inapropriate values for certain daemons.
We decide to fail if the memory.limit is too low.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-03-15 17:35:44 +01:00
Sébastien Han b7fd64074c ceph: osd spec: tune osd memory target
If the spec of the OSD daemon contains resource limits, especially
memory.limit we will use that value for setting the osd memory target
value so the bluestore OSD knows the amount of memory available for its
cache.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-03-15 11:45:29 +01:00
travisn 31848e23e7 add the CONTAINER_IMAGE env var to the ceph daemon pods
The mgr orchestrator modules need the container image to run
the ceph image for blinking the lights when a disk is down

Signed-off-by: travisn <tnielsen@redhat.com>
2019-03-04 13:33:41 -07:00
Travis Nielsen a145f4dcb7 Merge pull request #2703 from travisn/osds-per-device-with-filter
Clean up provisioning of OSDs on devices
2019-03-04 07:58:56 -07:00
Travis Nielsen 816d698655 Merge pull request #2742 from noahdesu/dmcrypt
ceph: share host ipc namespace with provision pod
2019-03-01 13:21:05 -07:00
Noah Watkins 6597531ed0 ceph: share host ipc namespace with provision pod
this is needed as cryptsetup in the pod and udev on the host coordinate
via a semaphore that must be visible in both contexts.

note that additionally the host needs to have dmcrypt compiled with udev
synchronization, which at least one minikube version does not have.

fixes: #923

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
2019-03-01 10:50:19 -08:00
Blaine Gardner 4eb4fb7aa6 ceph mds: configure completely from operator
Configure the Ceph mds daemon completely from the operator a la the
recent changes to the Ceph mon and mgr operators.

Create the mds deployments first and then
create the keyring secrets for them with their owner reference as the
corresponding deployment. This will mean that the secrets do not need to
be micromanaged. When the deployment is deleted, the secret is also
deleted. This has not been necessary for the mons or the manager since
the mons share a keyring with a lifespan of the cluster, as does the
mgr, which currently has single-mgr support only.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2019-02-28 12:31:43 -07:00
travisn bdc3cf8146 osd: fix the device filter and improve device provisioning reliability
All devices detected by the discovery pod were being passed to the OSD provisioning pod
thus not always honoring the desired device list that should be provisioned.
Now the provisioning pod will be given the desired state from the crd,
then apply that state depending on the actual devices detected.
Also added a helper to ensure OSDsPerDevice is always valid.

Signed-off-by: travisn <tnielsen@redhat.com>
2019-02-26 16:11:25 -07:00
Sébastien Han eb3c424d57 Ceph messengers 2 support
This commit introduces the necessary changes to support the new
messenger feature coming with Ceph Nautilus (currently in development).

What changes? Now the monitor listens on two port:

* old 6789 for messenger v1, which will help us support older client
(e,g: krbd)
* new 3300 for messengers v2, which brings new improvement in the
messaging layer. This new transport layer brings numerous advantages
such as encryption improvement, speed improvement, pluggable nature to
support different network stack than TCP and many more.

We still have one Service IP, however it has 2 ports, see:

```
NAME                      TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
rook-ceph-mon-a           ClusterIP   10.106.217.160   <none>        3300/TCP,6789/TCP   4h
rook-ceph-mon-b           ClusterIP   10.99.36.175     <none>        3300/TCP,6789/TCP   4h
rook-ceph-mon-c           ClusterIP   10.108.220.74    <none>        3300/TCP,6789/TCP   4h
````

The `ceph.conf` has changed and we don't force the port when using an IP
address (public addr etc). Ceph, depending on its version will naturally
start the monitors on their right port, 6789.

A new --ceph-version-name CLI argument has been added to the Rook binary
so that when the pod starts it passes the ceph version name and the
configuration of the ceph.conf, as well as daemon startup flags, happen
properly.

Given that the Rook Operator remembers the port of all the monitors it
deployed (through Pod definition), this change is not an issue and will
maintain backward compatibility.

Note that to test this you must build rook with dev container image,
which contains the dev Nautilus version. So you should do something
like:

`make -j4 BASEIMAGE='ceph/daemon-base:latest-master' IMAGES='ceph' build`

Resolves: #2525
Signed-off-by: Sébastien Han <seb@redhat.com>
2019-02-22 00:09:37 +01:00
Huamin Chen 1b21e42a90 convert extensions to apps
Signed-off-by: Huamin Chen <hchen@redhat.com>
2019-02-14 16:06:05 -05:00
travisn 3a64288356 avoid a duplicate volume for a directory and dataDirHostPath
Signed-off-by: travisn <tnielsen@redhat.com>
2019-02-05 22:26:50 -07:00
Blaine Gardner 78d4719125 ceph: no longer create fallback osd
Rook's behavior should not be to create a default OSD in dataDirHostPath
when no devices are present on a node. Any preexisting fallback osd
should be kept as long as no dirs or disks have been specified to keep
the legacy behavior for those clusters running with these osds in place.
The legacy deletion behavior is also kept, removing the osd as soon as
any dir or device is specified other than the dataDirHostPath dir.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2019-02-01 09:41:32 -07:00
travisnandjtlayton 37a005b71e nfs: configure and start daemons when the nfs crd is created
Co-authored-by: jtlayton <jlayton@redhat.com>
Signed-off-by: travisn <tnielsen@redhat.com>
2019-01-15 13:48:58 -07:00
travisn 1389cc56db ceph: enable multiple osds per device
Signed-off-by: travisn <tnielsen@redhat.com>
2018-12-07 09:29:15 -07:00
travisn e6572e41e6 enable ceph-volume to provision and start filestore osds
Signed-off-by: travisn <tnielsen@redhat.com>
2018-12-06 21:28:58 -07:00
travisn 45c53dd5b2 detect if ceph-volume is supported and allow legacy osds to be started
Signed-off-by: travisn <tnielsen@redhat.com>
2018-12-06 21:28:58 -07:00
travisn 78b8011226 osds: prototype provisioning with ceph-volume
Signed-off-by: travisn <tnielsen@redhat.com>
2018-12-06 21:28:58 -07:00
travisn 62f7e5d6ea ceph: update docs, code, and tests to use v1 crd types
Signed-off-by: travisn <tnielsen@redhat.com>
2018-12-05 14:33:11 -07:00
travisn ea03f58048 rbac: rename service account rook-ceph-cluster to rook-ceph-osd
Signed-off-by: travisn <tnielsen@redhat.com>
2018-11-30 13:39:58 -07:00
Alexander Trost b12d1e82cf Use truncated node name "everywhere"
Signed-off-by: Alexander Trost <galexrt@googlemail.com>
2018-11-23 10:02:35 +01:00
travisn b2a0dc77fa automatically detect the major version of the Ceph image
Signed-off-by: travisn <tnielsen@redhat.com>
2018-11-01 11:49:36 -06:00
travisn 94a7a0d4bc run daemons with the ceph image instead of rook
Signed-off-by: travisn <tnielsen@redhat.com>
2018-11-01 11:49:35 -06:00
travisn 7d97b6ad76 osd: use the node hostname labels instead of node names
Signed-off-by: travisn <tnielsen@redhat.com>
2018-10-01 08:09:06 -06:00