Commit Graph
87 Commits
Author SHA1 Message Date
Blaine Gardner f32cecb701 Merge pull request #3206 from d-luu/ceph_priority_classes
ceph: added priority classes to components
2019-11-18 11:27:57 -07:00
d-luu 9c755fac66 ceph: added priority classes to components
Adds priority class support to Ceph components
  to influence scheduler's pod preemption

Signed-off-by: d-luu <david@davidluu.info>
2019-11-15 16:28:41 -06:00
Sébastien Han 3e82a0f4a6 ceph: osd add env var to skip uded on lvm
With DM_DISABLE_UDEV, LVM will avoid interaction with udev.
LVM will manage the relevant nodes in /dev directly.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-11-15 22:54:03 +01:00
Sébastien Han d9679e5cf4 Merge pull request #3316 from leseb/ceph-crash
ceph: add ceph-crash controller
2019-11-12 23:16:02 +01:00
Sébastien Han a0fdf1bb5f ceph: add CEPH_VOLUME_SKIP_RESTORECON env var
This will tell ceph-volume not to run any restorecon commands.

This relies on https://github.com/ceph/ceph/pull/31421

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-11-12 16:27:20 +01:00
Sébastien HanandRohan CJ 90cd8224f4 ceph: add ceph-crash controller
A new controller to bootstrap ceph-crash pod on Ceph nodes running Ceph
pods only.

This implementation is unfortunately a mix of 'hacks' to make the daemon
working correclty. The ceph-crash script faced different issues:

* it's not a daemon
* it does not have any key in cephx
* it runs ceph commands under the hood which need an admin key and a
ceph.conf

On upstream Ceph, ceph-crash needs to grow better, once that happens we
will improve our implementation.

This also enhances osd provisionConfig struct with DataPathMap

Pod volumes and volume mount needs DataPathMap to perform the right
actions on log and crash dir. Exposing DataPathMap makes that possible.

Obviously, this is exposing ceph crash reports on the host as
well as pushing them into the mgr.
Basically, if a daemon fails, it'll put its core dump into
/var/lib/ceph/crash, bindmounting this dir on the host, ensures that the
crashes don't get lost when the pod dies.

Example:

```
[leseb@tarox~/go/src/github.com/rook/rook][rgw-liveprobe !] kubectl -n rook-ceph exec -ti rook-ceph-crashcollector-minikube-574858b99c-zvg4z bash
bash: warning: setlocale: LC_CTYPE: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_COLLATE: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_MESSAGES: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_NUMERIC: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_TIME: cannot change locale (en_US.UTF-8): No such file or directory
[root@rook-ceph-crashcollector-minikube-574858b99c-zvg4z /]# ceph crash ls

[root@rook-ceph-osd-2-7644f99695-cljzh /]# pidof ceph-osd
13258 5415 5397
[root@rook-ceph-osd-2-7644f99695-cljzh /]# kill -SIGABRT 13258
[root@rook-ceph-osd-2-7644f99695-cljzh /]# ls /var/lib/ceph/crash/
2019-11-12_12:56:51.404109Z_39f060e1-776d-4605-8f28-85c97e53de96     posted

... wait maximum 10 min (ceph-crash scraps every 10 minutes)
... the container will exit

[root@rook-ceph-osd-2-7644f99695-cljzh /]# exit
[leseb@tarox~/go/src/github.com/rook/rook][rgw-liveprobe !]
[leseb@tarox~/go/src/github.com/rook/rook][rgw-liveprobe !] kubectl -n rook-ceph exec -ti rook-ceph-crashcollector-minikube-574858b99c-zvg4z bash
bash: warning: setlocale: LC_CTYPE: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_COLLATE: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_MESSAGES: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_NUMERIC: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_TIME: cannot change locale (en_US.UTF-8): No such file or directory
[root@rook-ceph-crashcollector-minikube-574858b99c-zvg4z /]#
[root@rook-ceph-crashcollector-minikube-574858b99c-zvg4z /]#
[root@rook-ceph-crashcollector-minikube-574858b99c-zvg4z /]# ceph crash ls
2019-11-12_12:56:51.404109Z_39f060e1-776d-4605-8f28-85c97e53de96 osd.1
[root@rook-ceph-crashcollector-minikube-574858b99c-zvg4z /]# ls /var/lib/ceph/crash/
posted
[root@rook-ceph-crashcollector-minikube-574858b99c-zvg4z /]# ls /var/lib/ceph/crash/posted/
2019-11-12_12:56:51.404109Z_39f060e1-776d-4605-8f28-85c97e53de96
[root@rook-ceph-crashcollector-minikube-574858b99c-zvg4z /]# tail /var/lib/ceph/crash/posted/2019-11-12_12\:56\:51.404109Z_39f060e1-776d-4605-8f28-85c97e53de96/log
   1/ 5 mgr
   1/ 5 mgrc
   1/ 5 dpdk
   1/ 5 eventtrace
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent     10000
  max_new         1000
  log_file /var/lib/ceph/crash/2019-11-12_12:56:51.404109Z_39f060e1-776d-4605-8f28-85c97e53de96/log
--- end dump of recent events ---
```

Signed-off-by: Sébastien Han <seb@redhat.com>
Co-authored-by: Rohan CJ <rohantmp@gmail.com>
2019-11-12 16:06:07 +01:00
Sébastien Han 76c33e0f85 Revert "ceph: osd hide 'restorecon' when necessary"
This reverts commit d224fbdef1.
Since https://github.com/ceph/ceph/pull/31421, the monkey patching of
'restorecon' is not needed anymore.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-11-12 15:34:12 +01:00
Travis Nielsen 47764715fe ceph: remove the location from the cluster CR
The topology of the cluster should be based on the node labels rather
than a setting in the cluster CR. This allows a much richer and more
dynamic topology to be configured. The location will now be ignored
if specified in the cluster CR.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2019-11-04 23:07:13 -07:00
Sébastien Han ef6815ebfa ceph: remove topologyAware CRD option
We removed the topologyAware CRD option since it was redundant with the
use of an OSD being backed by a PVC.
So now, if an OSD is backed by a PVC we assume the topology aware
decision and will discover zone and region labels on that host.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-11-04 09:53:14 -07:00
Sébastien Han d224fbdef1 ceph: osd hide 'restorecon' when necessary
Some testing on OpenShift has shown that enabling Hostnetworking causes
issues with ceph-volume creating OSD.
This is a pure Red Hat OpenShift downstream issue.

This is meant to be temporary while we are waiting for the Selinux team
to get back to us.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-10-30 11:32:15 +01:00
Sébastien Han 9f053ca4e1 ceph: osd remove --conf flag
Even legacy osds were still running with this flag. In my previous patch
I was trying to minimize the impact of the change and was afraid the
config file was holding important information but that appears not to be
true.

The only option in the config could have the keyring path, but it's
passed in the CLI startup line.

Closes: https://github.com/rook/rook/issues/4063
Signed-off-by: Sébastien Han <seb@redhat.com>
2019-10-23 11:43:28 -06:00
Sébastien Han 1765ad6508 ceph: expose prepare pod resource limits
The osd-prepare jobs should not inherit osd resources since this might
cause scheduling issues.
Instead, we can now use the new CRD property `prepareosd` in the resource section.

Refer to the documentation for more details.

Closes: https://github.com/rook/rook/issues/2502
Signed-off-by: Sébastien Han <seb@redhat.com>
2019-10-18 16:48:03 +02:00
Mateusz Los cb52aea9a3 ceph: add OSDs to proper buckets in crush hierarchy while deployment
add missing topologyAware and location env variables to OSD pods

Signed-off-by: Mateusz Los <los.mateusz@gmail.com>
2019-10-15 17:15:01 -06:00
Blaine Gardner db7a6da09e ceph: do ceph:ceph chown in init containers
Following the Ceph OSD rework, my test environment regularly failed to
start OSDs, as they would start before the postStart lifecyle hook which
was chown'ing the log and data directories could finish. Pods failed to
start for an arbitrarily long time before starting successfully. This
was a manifestation of a race condition identified in Ceph monitors in
https://github.com/rook/rook/pull/3594#discussion_r312279176

Create a Ceph operator-level method to create an init container that
will chown the necessary directories for all Ceph daemons (any daemon
which runs as the ceph:ceph user-and-group). Also add this container to
expectations for Ceph pod templates in unit tests.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2019-10-15 12:12:46 -06:00
Blaine Gardner 70810dd55a ceph: osd: do not init ceph.conf for dir OSDs
Just as with the mon, mgr, mds, rgw, and rbd-mirror daemons, do not
generate a ceph.conf in an init container for directory-based OSDs.
Instead use the mon config database and the commandline to supply all
the needed arguments for running these OSDs.

Also generate a keyring secret to mount to OSD pods. Use this
secret for directory-based OSDs for now with the intention to
use this for other OSDs in the future.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2019-10-15 12:12:46 -06:00
Blaine Gardner e2155afa0f ceph: osd: use mon DB config for dir-based OSDs
As with the rest of the Ceph daemons, use the new config patters for
dir-based OSDs. Do not generate a config to use in an init container;
instead use CLI flags and the mon config database.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2019-10-15 10:10:08 -06:00
Blaine Gardner 9ab2b885d4 ceph: osd: use DataPathMap for dir-based OSDs
This is just one small unit of work that still results in usable
dir-based OSDs. The goal is to build on this to run dir-based OSDs using
the new config patterns used by the other Ceph daemons. This also
removes an unnecessary mount of the dataDirHostPath in directory-based
OSD containers.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2019-10-15 10:10:07 -06:00
Erik KnudtsonandSébastien Han ec4d0c8925 ceph: remove decimal places for osdMemoryTargetValue
Update pkg/operator/ceph/cluster/osd/spec.go

Rook v1.1.2 uses a different command to start an OSD, and seems to
have problems when --osd-memory-target contains a number with a
decimal point.

Co-Authored-By: Sébastien Han <seb@redhat.com>
Signed-off-by: Erik Knudtson <eknudtson@discogsinc.com>
2019-10-07 09:32:49 -07:00
Sébastien Han ae8e45585f ceph: osd add fsid to daemon flag
Now that the ceph.conf is gone, it's safer to run the OSD with the
cluster fsid flag.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-09-26 15:29:24 +02:00
Sébastien Han a4a180b8fb ceph: osd remove config file
We don't need to force any Ceph config file for ceph-volume based OSDs
since the 'ceph-volume activate' command mount everything at start time
in the right place.
So now, OSD will keep running normally and they will also read back
normally the default location of the ceph configuration:
/etc/ceph/ceph.conf which will allow OSDs to read any configuration
overrides.
We cannot remove the generation of that file in /var/lib/rook/osd* since
we cannot tell at the stage that triggers the generation if we run
ceph-volume or not. So let's keep it around for some time.
If you are wondering how the OSD connects to the mons this is done via
the CEPH_ARGS env variable which was introduced in
https://github.com/rook/rook/pull/3894.

Closes: https://github.com/rook/rook/issues/3926
Signed-off-by: Sébastien Han <seb@redhat.com>
2019-09-26 15:07:53 +02:00
Sébastien Han a42cc44d11 Merge pull request #3894 from mvollman/osd_ceph_conf
ceph: fix encrypted osd startup
2019-09-25 09:00:06 +02:00
Sébastien Han 9b2afbca14 ceph: osd add debug logs for ceph-volume on prepare
Previously the debug flag was turned on for the 'activate' call only,
now 'prepare' will also take advantage of it.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-09-24 17:17:29 +02:00
Michael Vollman e12bbd6f7c ceph: fix encrypted osd startup
Encrypted OSDs are failing because ceph-volume is unable to determine
how to contact the monitors.  Setting the CEPH_CONF env variable for OSD
pods fixes this issue. (#3846)

Signed-off-by: Michael Vollman <michael.b.vollman@gmail.com>
2019-09-20 16:36:52 -04:00
Santosh Pillai 9db863377a release the device held onto by lvm
When running osds on pvc, the lvm is still holding the device when the cluster is deleted. As a result the PVs go into `failed` state
when the cluster is deleted. On deactivating the device, the 'failed' pvs get deleted as well. Solution involves:
- Running `lvchange -an <vg group name>` to deactivate the device.
- This command runs after 'osd prepare' completes.
- This command also runs after the 'osd daemon pod' is deleted or if the ceph-osd process terminates for some reasons.

Signed-off-by: Santosh Pillai <sapillai@redhat.com>
2019-09-20 18:08:13 +05:30
Sébastien Han c96fe8b5f8 ceph: enable ceph-volume debug logs
It's fine to enable debug logs from c-v, it's not so verbose and really
useful when debugging.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-09-18 13:55:18 +02:00
rohan47 fcbce4898c Set the Command property for the OSD prepare init container blkdevmapper
If a Ceph container image is specified which has a entrypoint then the
Init container for osd prepare job blkdevmapper will fail. So set the
Command property.

Signed-off-by: rohan47 <rohgupta@redhat.com>
2019-09-10 22:05:50 +05:30
Rohan CJ f8b7537830 Fix topologyAware on PVC-based OSDs
Signed-off-by: Rohan CJ <rohantmp@gmail.com>
2019-09-06 23:17:01 +05:30
Santosh Pillai ead0653068 fix random OSD daemon pod failures when running OSD on PVC
OSD pods when running on PVC fail randomly due to some race condition:
 - OSD daemon pod was activating the volume group using vgchange -an without any volume group name.
 - This caused the pods to fail randomly when number of OSDs is more, say greater than 10.
 - This fix adds Volume group name while activating/deactivating volume groups to fix this

Signed-off-by: Santosh Pillai <sapillai@redhat.com>
2019-09-05 20:34:03 +05:30
Sébastien Han 87ca208640 Merge pull request #3596 from rohantmp/managedPDB
Managed PDBs
2019-08-30 14:34:31 +02:00
Sébastien Han e2c8a9935c Merge pull request #3696 from mvollman/db-devices-n-size
ceph: add db-devices flag and fix DatabaseSizeMB
2019-08-30 09:49:42 +02:00
travisn 49115749ab ceph: copy binaries init container for osds on pvcs provisioning
The init container for copying binaries was being overwritten by the
init container for initializing pvcs. This change appends the init
containers so all expected init containers are preserved.

Signed-off-by: travisn <tnielsen@redhat.com>
2019-08-29 22:23:04 -06:00
Rohan CJ d6b23dd549 Add nodedrain controller for deploying canary deployments for every node.
Canary pods can be watched to determine when a node is draining.

Signed-off-by: Rohan CJ <rohantmp@gmail.com>
2019-08-30 09:01:11 +05:30
Travis Nielsen 5eab7ca658 Merge pull request #3573 from SUSE/mon-kv-store
Ceph: override configs in CRD
2019-08-29 19:04:16 -06:00
Michael Vollman ed9519953a ceph: add per OSD custom deviceClass support
Add deviceClass as a per OSD config option under
storage/nodes/devices/config for setting a custom crush device class
per OSD.

Signed-off-by: Michael Vollman <michael.b.vollman@gmail.com>
2019-08-29 20:39:41 -04:00
Michael Vollman 496c638306 ceph: add db-devices flag and fix DatabaseSizeMB
Adding support for the db-devices flag for ceph versions 14.2.1 and
newer.  This flag will allow for explictly setting the db device for
bluestore configurations to the device specified as metadataDevice.

Fixing the databaseSizeMB config parameter to ensure it is used to set
the size of the db-device partition when specified at either the global
or individual device level. (#3652)

Signed-off-by: Michael Vollman <michael.b.vollman@gmail.com>
2019-08-29 20:39:41 -04:00
Blaine Gardner 1be43290be Ceph: override configs in CRD
Allow users to specify overrides in the CephCluster CRD. The override
ConfigMap still exists for emergency situations and is mounted into
daemon pods directly instead of being merged into a Rook-created config
file.

Rook Ceph no longer uses a config file for managing daemons with the
exception of the OSDs which still generate a config in an init
container.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2019-08-29 15:38:27 -06:00
rohan47 a098cdd1a6 OSD: fixed label selector for osd prepare job pod
OSD prepare job pods don't require node selector when the OSDs are based
on PVC.
The bug was intruduced in https://github.com/rook/rook/pull/3707 which
added node selector to the prepare job pod when portable property of
storageClassDeviceSets is false.
Adding node selector set kubernetes.io/hostname=example-storagedeviceset-xxx, while there is no host example-storagedeviceset-xxx the prepare job failed.

Signed-off-by: rohan47 <rohgupta@redhat.com>
2019-08-29 19:30:02 +05:30
rohan47 21deea9d64 OSD: CRUSH map can be based on node or PVC in case of PVC backed OSD
Signed-off-by: rohan47 <rohgupta@redhat.com>
2019-08-29 04:01:41 +05:30
Sébastien Han 0bdc98f333 Merge pull request #3576 from giovanism/multus
Multi-homing networking
2019-08-26 15:17:24 +02:00
giovanism cc62702d39 ceph: Use rook's NetworkSpec with type embedding
ceph: Use the new NetworkSpec api

ceph: Change all tests to use the NetworkSpec

ceph: Remove TypeMeta field from NetworkSpec

ceph: Change IsHost network func signature

Signed-off-by: giovanism <giovanism@outlook.co.id>
2019-08-26 16:17:03 +07:00
rohan47 7307575637 Added portable and failure-domain label to the OSD deployment
Signed-off-by: rohan47 <rohgupta@redhat.com>
2019-08-23 01:49:14 +05:30
Sébastien Han b3f58863e9 Merge pull request #3600 from mvollman/bugfix-osdsPerDevice
ceph: fix osdsPerDevice config
2019-08-22 20:24:01 +02:00
Travis Nielsen 1d28489c4a Merge pull request #3633 from ashishranjan738/fixsds
fix(storageClassDeviceSet): fixes resource limits and placements not getting applied to osds
2019-08-20 17:38:47 -06:00
Michael Vollman 33711b7d36 ceph: fix osdsPerDevice config
Fix variable scoping issue that is ignoring osdsPerDevice config
setting.  Ensure that in all scenarios both the global and per OSD
osdsPerDevice setting are honored.  When both are set the more specific
setting will be used. (#3599)

Signed-off-by: Michael Vollman <michael.b.vollman@gmail.com>
2019-08-20 07:09:12 -04:00
Odin Ugedal 7bd1ff1049 ceph: remove duplicated code
Signed-off-by: Odin Ugedal <odin@ugedal.com>
2019-08-18 11:10:28 +02:00
Odin Ugedal 0e73bbe2f3 ceph: set osd mem target correctly
Previously, the value was set to commonArgs, resulting in dead code,
since common arg isn't used after the append.

Signed-off-by: Odin Ugedal <odin@ugedal.com>
2019-08-16 17:29:38 +02:00
Ashish Ranjan a8827f3652 fix(storageClassDeviceSet): fixes resource limits and placements for osds
This commit fixes resource limits and placements not getting applied to osd pods provisioned using storageClassDeviceSets

Signed-off-by: Ashish Ranjan <aranjan@redhat.com>
2019-08-16 02:44:37 +05:30
rohan47andAshish Ranjan d2f52aebe5 Adds support for storageClassDeviceSet in rook-ceph operator
- Added code to support StorageClassDeviceSet spec provided in the cluster-on-pvc.yaml
- The code reads the StorageClassDeviceSet spec and creates pvc based on the ‘count’ field for each device set.
- OSD prepare job is started for each PVC which activates the ceph-volume on each PVC
- Finally OSD is started on each of the PVC device.

Co-authored-by: rohan47 <rohgupta@redhat.com>
Co-authored-by: Ashish Ranjan <aranjan@redhat.com>
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
2019-08-12 09:24:13 -06:00
Michael Vollman f3ef680298 ceph: Add support for per OSD metdataDevices
Add ceph-volume support for per OSD metdataDevices

Signed-off-by: Michael Vollman <michael.b.vollman@gmail.com>
2019-08-02 07:41:45 -04:00
Blaine Gardner 55a726fda9 ceph: osd use copy-binaries top level cmd
Replace usage of the custom osd copybins command with the new top-level
copy-binaries command introduced with cmd-reporter.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2019-07-29 09:23:45 -06:00