Commit Graph
61 Commits
Author SHA1 Message Date
Nizamudeen 78ef4abeec ceph: Fixing the potential file inclusion error
This commit is also part of gosec error handling which handles the following issue:

Fixed G304: Potential File inclusion by cleaning up the file path

Signed-off-by: Nizamudeen <nia@redhat.com>
2020-02-14 12:22:49 +05:30
Sébastien HanandSean Micklethwaite b04e35dab5 ceph: add support for disk selection by full path
Rook-Ceph now supports specify device by their fullpath instead of links
created by the kernel like /dev/sdb.

Closes: https://github.com/rook/rook/issues/1228
Co-authored-by: Sean Micklethwaite <sean@wayve.ai>
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-02-13 17:31:38 +01:00
Sébastien Han c08c3ced05 ceph: add support for metadata PVC for OSD on PVC
We now support the addition of the PVC that acts as a metadata device
for a given OSD.
For this, you need to create a new `volumeClaimTemplates`, its name must
be "metadata" otherwise, Rook won't pick it up.

A template will look like this:

```
volumeClaimTemplates:
- metadata:
    name: data
  spec:
    resources:
      requests:
        storage: 10Gi
    # IMPORTANT: Change the storage class depending on your environment (e.g. local-storage, gp2)
    storageClassName: gp2
    volumeMode: Block
    accessModes:
      - ReadWriteOnce
- metadata:
    name: metadata
  spec:
    resources:
      requests:
        storage: 6Gi
    # IMPORTANT: Change the storage class depending on your environment (e.g. local-storage, gp2)
    storageClassName: gp2
    volumeMode: Block
    accessModes:
      - ReadWriteOnce
```

We now map block and block.db directly inside the container instead of
running ceph-volume activate. This is much cleaner.

Closes: https://github.com/rook/rook/issues/3852
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-02-11 17:30:02 +01:00
Sébastien Han 1c3d449eb7 ceph: fail if no devices found
This will likely happen when re-running an orchestration and
bootstrapping a new cluster with unclean drives.
The prepare pod won't prepare nor list existing disks, thus the code
will later fail on reading disks properties.

Closes: https://github.com/rook/rook/issues/4810
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-02-05 22:16:26 +01:00
Sébastien Han df3a8564b6 ceph: skip partitions depending on the ceph version
If we detect a partition we should not use it if the ceph version is not
at least 14.2.8 since it has the necessary changes to support partitions
with ceph-volume.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-01-30 18:49:34 +01:00
Sébastien Han 227d2d527a ceph: osd store refactor
Multiple things:

1. We removed all the function/methods/tests that were used to
create and manage rook legacy OSDS as well as bringing support to
Bluestore OSD only.
It also fixes various go-lint issues in the respectives files.

2. use c-v inventory to detect available devices:
Now we rely on the 'ceph-volume inventory' command to tell us if a
device is available or not.

3. implement raw mode for osd on pvc
When an OSD will be bootstrap on a PVC, the new c-v raw mode will be
used. It consists of putting block, db and wal under the same device.
Here LVM is out of the picture and the raw device is used as is. The
implementation is backward compatible so existing OSD on PVC will LVM
will continue to operate.

Closes: https://github.com/rook/rook/issues/4363
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-01-23 19:13:09 +01:00
Dustin Guerrero 4ff18d148a osd: fix misleading log messages when matching device filters
This removes a misleading log message that implies device filters
match when they do not.

Signed-off-by: Dustin Guerrero <dguerrero@twilio.com>
2020-01-22 09:58:09 -08:00
Sébastien Han dd659de46f ceph: add partition support
We now support partitions via 2 ways:

* if `useAllDevice: true`: partitions will be taken into account and
presented as OSD candidate
* if specified in the cluster CR: it'll picked up as well

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-01-16 09:34:46 +01:00
Sébastien Han ad95c7296f ceph: do not print extended format for loggers.
When using the "errors" package, using `%+v` (extended format),
each Frame of the error's StackTrace will be printed in detail.
Let's only print `%v` to print the error.
If the error has a Cause it will be printed recursively.

Basically `%+v` has been replaced with `%v` for all `error` type
interfaces, whether the logger is Info, Warning or Error.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-12-16 18:35:54 +01:00
Travis Nielsen fbf3837f13 ceph: stop removing osds for any cephcluster CR update
Removing an OSD unintentionally can result in data loss,
which is to be prevented at all costs. Rook's functionality
to remove OSDs automatically when a node was removed from the
cluster CR has led on several occasions to users having OSDs
unintentionally removed from their cluster.

If the admin decides they really need to remove OSDs they should
do it as a one-time action that they intentionally run
rather than the operator attempting to manage the deletion
with desired state. The risk of misinterpreting the desired
state, even with a typo, is too great.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2019-12-13 08:26:23 -07:00
Blaine Gardner caba125b84 Merge pull request #4370 from SUSE/osd-refactor-legacy
Ceph: OSD: refactor config for filestore OSDs
2019-12-09 16:20:17 -07:00
Blaine Gardner 64bad24712 ceph: osd: do not init ceph.conf for filestore osd
Just as with the mon, mgr, mds, rgw, and rbd-mirror daemons, do not
generate a ceph.conf in an init container for filestore OSDs.
Instead use the commandline to supply all the needed arguments for
running these OSDs.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2019-12-09 14:15:12 -07:00
Sébastien Han 5ce2ed220e ceph: use "github.com/pkg/errors"
We now use the error package.
Kubernetes errors have been renamed kerrors since they are lower than
'errors'.

Closes: https://github.com/rook/rook/issues/4054
Signed-off-by: Sébastien Han <seb@redhat.com>
2019-12-09 16:58:32 +01:00
dulltzandSatoru Takeuchi dfe45ac6b0 ceph: support OSD on PVC backed by LV
"OSD on PVC" doesn't work for PV backed by LV. Fixing this problem
by the following changes.

- Rook accepts LVM disk type.
- If a LV-backed device is passed, Rook/Ceph invokes
  "ceph-volume lvm prepare" with "--data vg/lv"
  instead of "--data /path/to/device".
- If a LV-backed device is passed, Rook/Ceph suppresses
  activation/deactivation of VG that owns this LV.

Fixes: https://github.com/rook/rook/issues/4185
Signed-off-by: dulltz <isrgnoe@gmail.com>
Co-authored-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
2019-12-05 17:26:27 +00:00
Sébastien Han e108f87b06 Merge pull request #4386 from leseb/location-prepare
ceph: do not restart the osds when upgrading the operator
2019-12-05 16:59:33 +01:00
Kazuhito MATSUDA 40d868ebd7 ceph: support device selection for OSDs by device path name patterns
Add devicePathFilter to Storage Selection Settings which enables
device selection with a regular expression for device path names.

Fixes: https://github.com/rook/rook/issues/4275
Signed-off-by: Kazuhito MATSUDA <kazuto.jinnai@gmail.com>
2019-12-05 07:02:44 +00:00
Sébastien Han 4a7258eb0d ceph: osd pass crush location from prepare pod
We shouldn't need to compute the location from the main OSD pod, since
the prepare pod runs on the same node we can populate the crush location
information from that pod and pass it along to the main osd container.

This is an initial commit that aims to eventually remove the need of
using the rook binary when running the main osd container.

Closes: https://github.com/rook/rook/issues/4362
Signed-off-by: Sébastien Han <seb@redhat.com>
2019-12-04 23:03:02 +01:00
Kazuhito MATSUDA 05ac9489b3 ceph: use /dev/mapper/<name> for ceph-volume instead of /dev/md-x
`ceph-volume` cannot handle a device correctly if the device has an alias
under `/dev/mapper/`.
This commit fixes it by checking whether the device has that kind of alias
and using the alias as an argument for `ceph-volume`.

Signed-off-by: Kazuhito MATSUDA <kazuto.jinnai@gmail.com>
2019-12-04 12:53:50 +00:00
Travis Nielsen ba6fcffd8f ceph: stop osd process more quickly during pod shutdown
The OSD needs to shut down quickly during upgrade, or other scenarios
where the OSD is being restarted. To facilitate this fast shutdown,
rook will run kill -9 on the osd process. The Ceph OSD is designed
to be safe even when killed like this. This allows the
ECONNREFUSED to be returned sooner, which will redirect the OSD
traffic to other OSDs and cause less downtime.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2019-11-18 19:01:47 -08:00
Travis Nielsen 9eea60f9f4 ceph: release the LV immediately after the osd process exits
When the OSD is running on a PVC, the LV needs to be released immediately
after the OSD process exits. The cleanup was being skipped when the OSD
process exited and returning an error rather than allowing the
LV cleanup to proceed.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2019-11-18 19:01:02 -08:00
Sébastien Han 76c33e0f85 Revert "ceph: osd hide 'restorecon' when necessary"
This reverts commit d224fbdef1.
Since https://github.com/ceph/ceph/pull/31421, the monkey patching of
'restorecon' is not needed anymore.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-11-12 15:34:12 +01:00
Travis Nielsen 47764715fe ceph: remove the location from the cluster CR
The topology of the cluster should be based on the node labels rather
than a setting in the cluster CR. This allows a much richer and more
dynamic topology to be configured. The location will now be ignored
if specified in the cluster CR.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2019-11-04 23:07:13 -07:00
Sébastien Han d224fbdef1 ceph: osd hide 'restorecon' when necessary
Some testing on OpenShift has shown that enabling Hostnetworking causes
issues with ceph-volume creating OSD.
This is a pure Red Hat OpenShift downstream issue.

This is meant to be temporary while we are waiting for the Selinux team
to get back to us.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-10-30 11:32:15 +01:00
Sébastien Han 6dff2afe8b ceph: osd update error message
Add a more meaninful message.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-10-29 16:58:16 +01:00
Santosh Pillai 8ea693a740 ceph: fix operator reconcile to restart osds daemons
OSD on PVC does not upgrade when the user upgrades the ceph version on cluster-on-pvc yaml.
The solution incudes:
   - upgrade osd prepare and daemon pods on upgrade
   - skip c-v prepare if filesystem is already present on the pvc device.
   - skip lvm release in case of upgrade.

Signed-off-by: Santosh Pillai <sapillai@redhat.com>
2019-10-23 16:56:15 +05:30
Travis Nielsen fdd92ec883 Merge pull request #4100 from morimoto-cybozu/clarify-errors-in-clusterd
Return explicit errors
2019-10-15 08:57:06 -06:00
morimoto-cybozu 815927d943 clusterd: return explicit errors
PopulateDeviceInfo() in pkg/clusterd/disk.go returns nil as *sys.LocalDisk
if an error has occurred.  This causes a nil pointer exception at
getAvailableDevices() in pkg/daemon/ceph/osd/daemon.go.  At least the returned
value should be checked at the caller.
I added an explicit error to the return value of PopulateDeviceInfo().  This
naturally revokes an error check at the caller.
I modified PopulateDeviceUdevInfo() in the same file too.

Signed-off-by: morimoto-cybozu <kenji_morimoto@cybozu.co.jp>
2019-10-15 08:30:45 +00:00
travisn 22391377a8 ceph: remove full path from vgchange call
The full path to vgchange should not be necessary. For some
downstream releases the path may even be in a different path.
Therefore we require that vgchange be in the path.

Signed-off-by: travisn <tnielsen@redhat.com>
2019-10-14 09:29:45 -06:00
Sébastien Han 24fc94cc57 Merge pull request #3955 from leseb/cv-debug-prepare
ceph: various osd fixes and improvements with lvm and ceph-volume
2019-09-25 08:46:34 +02:00
Sébastien Han 42cebf6311 ceph: osd run lvm config all the time
Previously, we were applying lvm config changes to OSD running on PV. So
now, we do this for any type of cluster deployment (bare metal or
Cloud).

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-09-24 22:49:39 +02:00
Sébastien Han ad514606bf ceph: osd reset 'run dir' to default location
Previously, "run dir" was used to place a number of file and config on
dataDirHostPath. Now, since a lot of configs and options have moved
either on the OSD's startup CLI line or remove with bluestore we don't
need to change the default.

Also, it was confusing for user and difficult to find the socket.
The is is not breaking any config in dataDirHostPath since all the
elements (keyrings) are hardcoded in the config file.

Closes: https://github.com/rook/rook/issues/3966
Signed-off-by: Sébastien Han <seb@redhat.com>
2019-09-24 10:27:59 +02:00
Sébastien Han d8813513eb ceph: fix various spelling issues
Correct spelling.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-09-24 10:26:22 +02:00
Santosh Pillai 9db863377a release the device held onto by lvm
When running osds on pvc, the lvm is still holding the device when the cluster is deleted. As a result the PVs go into `failed` state
when the cluster is deleted. On deactivating the device, the 'failed' pvs get deleted as well. Solution involves:
- Running `lvchange -an <vg group name>` to deactivate the device.
- This command runs after 'osd prepare' completes.
- This command also runs after the 'osd daemon pod' is deleted or if the ceph-osd process terminates for some reasons.

Signed-off-by: Santosh Pillai <sapillai@redhat.com>
2019-09-20 18:08:13 +05:30
Santosh Pillai ead0653068 fix random OSD daemon pod failures when running OSD on PVC
OSD pods when running on PVC fail randomly due to some race condition:
 - OSD daemon pod was activating the volume group using vgchange -an without any volume group name.
 - This caused the pods to fail randomly when number of OSDs is more, say greater than 10.
 - This fix adds Volume group name while activating/deactivating volume groups to fix this

Signed-off-by: Santosh Pillai <sapillai@redhat.com>
2019-09-05 20:34:03 +05:30
Blaine Gardner 1be43290be Ceph: override configs in CRD
Allow users to specify overrides in the CephCluster CRD. The override
ConfigMap still exists for emergency situations and is mounted into
daemon pods directly instead of being merged into a Rook-created config
file.

Rook Ceph no longer uses a config file for managing daemons with the
exception of the OSDs which still generate a config in an init
container.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2019-08-29 15:38:27 -06:00
rohan47andAshish Ranjan d2f52aebe5 Adds support for storageClassDeviceSet in rook-ceph operator
- Added code to support StorageClassDeviceSet spec provided in the cluster-on-pvc.yaml
- The code reads the StorageClassDeviceSet spec and creates pvc based on the ‘count’ field for each device set.
- OSD prepare job is started for each PVC which activates the ceph-volume on each PVC
- Finally OSD is started on each of the PVC device.

Co-authored-by: rohan47 <rohgupta@redhat.com>
Co-authored-by: Ashish Ranjan <aranjan@redhat.com>
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
2019-08-12 09:24:13 -06:00
travisn aae4d7c7f6 osds: write ceph-volume log entries as they occur
When ceph-volume is called, the default python logging will buffer everything until
the process exits, which means the pod log doesnt show what is happening
until ceph-volume is done. This change will allow logs to be written
immediately as tasks are performed inside ceph-volume

Signed-off-by: travisn <tnielsen@redhat.com>
2019-04-16 15:47:05 -06:00
Sébastien Han 4438672e47 ceph: expose ceph version to osd pod
We must expose the ceph version to the osd pod so that we can generate
the proper ceph configuration file. If we don't do this all the version
check will assume they run the opposite of the desired version and the
formatting of the `ceph.conf` will be wrong.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-04-15 15:57:23 +02:00
Blaine Gardner 51cc7c2e85 Ceph integration tests: add more verbose logging
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2019-03-05 14:14:09 -07:00
travisn bdc3cf8146 osd: fix the device filter and improve device provisioning reliability
All devices detected by the discovery pod were being passed to the OSD provisioning pod
thus not always honoring the desired device list that should be provisioned.
Now the provisioning pod will be given the desired state from the crd,
then apply that state depending on the actual devices detected.
Also added a helper to ensure OSDsPerDevice is always valid.

Signed-off-by: travisn <tnielsen@redhat.com>
2019-02-26 16:11:25 -07:00
Blaine Gardner 78d4719125 ceph: no longer create fallback osd
Rook's behavior should not be to create a default OSD in dataDirHostPath
when no devices are present on a node. Any preexisting fallback osd
should be kept as long as no dirs or disks have been specified to keep
the legacy behavior for those clusters running with these osds in place.
The legacy deletion behavior is also kept, removing the osd as soon as
any dir or device is specified other than the dataDirHostPath dir.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2019-02-01 09:41:32 -07:00
travisn fe4a645844 osd: detect existing osds not to be passed to ceph-volume
Signed-off-by: travisn <tnielsen@redhat.com>
2018-12-07 10:50:37 -07:00
travisn 1389cc56db ceph: enable multiple osds per device
Signed-off-by: travisn <tnielsen@redhat.com>
2018-12-07 09:29:15 -07:00
travisn e6572e41e6 enable ceph-volume to provision and start filestore osds
Signed-off-by: travisn <tnielsen@redhat.com>
2018-12-06 21:28:58 -07:00
travisn 45c53dd5b2 detect if ceph-volume is supported and allow legacy osds to be started
Signed-off-by: travisn <tnielsen@redhat.com>
2018-12-06 21:28:58 -07:00
travisn 78b8011226 osds: prototype provisioning with ceph-volume
Signed-off-by: travisn <tnielsen@redhat.com>
2018-12-06 21:28:58 -07:00
Alexander Trost b12d1e82cf Use truncated node name "everywhere"
Signed-off-by: Alexander Trost <galexrt@googlemail.com>
2018-11-23 10:02:35 +01:00
Blaine Gardner 3731f06176 Ceph: Refactor mon config/keyring to cephconfig
Create a cephconfig module in Ceph's daemon pkg source, and refactor the
config and keyring generation that exists in the mon package into the
new cephconfig package. The config/keyring generation code is used by
most all daemons and not just mon, so a new package is a more
appropriate place for this.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2018-09-05 10:17:11 -06:00
travisn 1d928ecd36 osd: init container to initialize config at startup
Signed-off-by: travisn <tnielsen@redhat.com>
2018-07-31 14:39:53 -06:00
travisn 55e0ee26c0 monitor the OSDs for the lifetime of the cluster
Signed-off-by: travisn <tnielsen@redhat.com>
2018-07-06 15:08:59 -06:00