Commit Graph
35 Commits
Author SHA1 Message Date
Sébastien Han 7553318235 ceph: osd append instead of replace
Since the lvm config might change again in the future and because the
filter in the conf is commented as an example, let's append our filter
instead of searching and replacing.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-11-15 22:54:10 +01:00
Sébastien Han 8c8b8e7fef ceph: osd add more details when changing lvm conf
Let's be more verbose and informative when we are performing actions.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-11-15 22:54:09 +01:00
Sébastien Han 8c3db31e35 ceph: update lvm filter
The following commit from LVM
https://github.com/lvmteam/lvm2/commit/08396b4bce45fb8311979250623f04ec0ddb628c#diff-13c602a6258e57ce666a240e67c44f38
updated the lvm.conf and thus changed our filter. So now it does not
match anymore.
Updating the filter to match fixes that issue.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-11-13 17:46:47 +01:00
Sébastien Han 76c33e0f85 Revert "ceph: osd hide 'restorecon' when necessary"
This reverts commit d224fbdef1.
Since https://github.com/ceph/ceph/pull/31421, the monkey patching of
'restorecon' is not needed anymore.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-11-12 15:34:12 +01:00
Sébastien Han d224fbdef1 ceph: osd hide 'restorecon' when necessary
Some testing on OpenShift has shown that enabling Hostnetworking causes
issues with ceph-volume creating OSD.
This is a pure Red Hat OpenShift downstream issue.

This is meant to be temporary while we are waiting for the Selinux team
to get back to us.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-10-30 11:32:15 +01:00
Santosh Pillai 8ea693a740 ceph: fix operator reconcile to restart osds daemons
OSD on PVC does not upgrade when the user upgrades the ceph version on cluster-on-pvc yaml.
The solution incudes:
   - upgrade osd prepare and daemon pods on upgrade
   - skip c-v prepare if filesystem is already present on the pvc device.
   - skip lvm release in case of upgrade.

Signed-off-by: Santosh Pillai <sapillai@redhat.com>
2019-10-23 16:56:15 +05:30
Sébastien Han 42cebf6311 ceph: osd run lvm config all the time
Previously, we were applying lvm config changes to OSD running on PV. So
now, we do this for any type of cluster deployment (bare metal or
Cloud).

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-09-24 22:49:39 +02:00
Sébastien Han 3404460d3c ceph: osd, fix lvm filter regexp
The lvm documentation uses commas to separate the regular expressions.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-09-24 17:17:29 +02:00
travisn f3ce214822 ceph: configure additional lvm settings for rhel8
RHEL8 introduces some lvm changes. For OSDs on PVs to work
we need to set allow_changes_with_duplicate_pvs = 1.
This is required since we're copying the block image
under another location.

Signed-off-by: travisn <tnielsen@redhat.com>
2019-09-20 11:51:07 -06:00
Santosh Pillai 9db863377a release the device held onto by lvm
When running osds on pvc, the lvm is still holding the device when the cluster is deleted. As a result the PVs go into `failed` state
when the cluster is deleted. On deactivating the device, the 'failed' pvs get deleted as well. Solution involves:
- Running `lvchange -an <vg group name>` to deactivate the device.
- This command runs after 'osd prepare' completes.
- This command also runs after the 'osd daemon pod' is deleted or if the ceph-osd process terminates for some reasons.

Signed-off-by: Santosh Pillai <sapillai@redhat.com>
2019-09-20 18:08:13 +05:30
Blaine Gardner 704786f273 Ceph: use combined output from ceph-volume
ceph-volume now sends status and log output to stderr, which breaks
`ceph-volume lvm prepare` if not other things as well. Use the combined
stdout and stderr output from ceph-volume to fix this.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2019-09-09 10:29:51 -06:00
Alexander Trost 10f17bc1e0 Merge pull request #3791 from ashishranjan738/osdpanic
fix(osd prepare): osd prepare job getting into panic state
2019-09-06 17:38:13 +02:00
Ashish Ranjan f16c8dc060 fix(osd prepare): osd prepare job getting into panic state
Osd pods getting into panic state due to index out of range error.

logs of osd prepare job:

```
[aranjan@localhost:~]$ oc logs -f rook-ceph-osd-prepare-set1-0-data-6f2nh-zh4zl
2019-09-06 07:48:11.448085 I | cephcmd: desired devices to configure osds: [{Name:/mnt/set1-0-data-6f2nh OSDsPerDevice:1 MetadataDevice: DatabaseSizeMB:0 DeviceClass: IsFilter:false}]
2019-09-06 07:48:11.449841 I | rookcmd: starting Rook v1.0.0-623.gc18d575c.dirty with arguments '/rook/rook ceph osd provision'
2019-09-06 07:48:11.449866 I | rookcmd: flag values: --cluster-id=308f232a-d07a-11e9-96ba-0adfe2bb9bfc, --data-device-filter=, --data-devices=/mnt/set1-0-data-6f2nh, --data-directories=, --encrypted-device=false, --force-format=false, --
help=false, --location=, --log-flush-frequency=5s, --log-level=INFO, --metadata-device=, --node-name=set1-0-data-6f2nh, --operator-image=, --osd-database-size=0, --osd-journal-size=5120, --osd-store=, --osd-wal-size=576, --osds-per-devic
e=1, --pvc-backed-osd=true, --service-account=
2019-09-06 07:48:11.449875 I | op-mon: parsing mon endpoints: a=172.30.150.154:6789
2019-09-06 07:48:11.470058 I | cephconfig: writing config file /var/lib/rook/rook-ceph/rook-ceph.config
2019-09-06 07:48:11.470797 I | cephconfig: generated admin config in /var/lib/rook/rook-ceph
2019-09-06 07:48:11.470989 I | cephosd: discovering hardware
2019-09-06 07:48:11.471030 I | exec: Running command: lsblk /mnt/set1-0-data-6f2nh --bytes --nodeps --pairs --output SIZE,ROTA,RO,TYPE,PKNAME
2019-09-06 07:48:11.473887 I | exec: Running command: sgdisk --print /mnt/set1-0-data-6f2nh
2019-09-06 07:48:11.476048 I | cephosd: creating and starting the osds
2019-09-06 07:48:11.476070 I | exec: Running command: lsblk /mnt/set1-0-data-6f2nh --bytes --pairs --output NAME,SIZE,TYPE,PKNAME
2019-09-06 07:48:11.478030 I | sys: Output: NAME="xvdbg" SIZE="10737418240" TYPE="disk" PKNAME=""
2019-09-06 07:48:11.478056 I | cephosd: /mnt/set1-0-data-6f2nh found in the desired devices
2019-09-06 07:48:11.488231 I | cephosd: configuring osd devices: {"Entries":{"/mnt/set1-0-data-6f2nh":{"Data":-1,"Metadata":null,"Config":{"Name":"/mnt/set1-0-data-6f2nh","OSDsPerDevice":1,"MetadataDevice":"","DatabaseSizeMB":0,"DeviceCl
ass":"","IsFilter":false},"LegacyPartitionsFound":false}}}
2019-09-06 07:48:11.492191 I | cephosd: device /mnt/set1-0-data-6f2nh to be configured by ceph-volume
2019-09-06 07:48:11.492213 I | cephosd: 0/0 pre-ceph-volume osd devices succeeded on this node
2019-09-06 07:48:11.492399 I | exec: Running command: ceph auth get-or-create-key client.bootstrap-osd mon allow profile bootstrap-osd --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/va
r/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/973674470
2019-09-06 07:48:12.020858 I | cephosd: Successfully updated lvm config file
2019-09-06 07:48:12.020894 I | cephosd: configuring new device /mnt/set1-0-data-6f2nh
2019-09-06 07:48:12.020908 I | exec: Running command: stdbuf -oL ceph-volume lvm prepare --data /mnt/set1-0-data-6f2nh
2019-09-06 07:48:55.172366 I | cephosd:
panic: runtime error: index out of range

goroutine 1 [running]:
github.com/rook/rook/pkg/daemon/ceph/osd.getLVPath(0x0, 0x0, 0x1e3c60d, 0x2)
        /home/aranjan/Project/go/src/github.com/rook/rook/pkg/daemon/ceph/osd/volume.go:128 +0x2fd
github.com/rook/rook/pkg/daemon/ceph/osd.(*OsdAgent).initializeBlockPVC(0xc000829c30, 0xc00072e1e0, 0xc000736e70, 0x1e4927d, 0xd, 0x0, 0x0)
        /home/aranjan/Project/go/src/github.com/rook/rook/pkg/daemon/ceph/osd/volume.go:108 +0x50b
github.com/rook/rook/pkg/daemon/ceph/osd.(*OsdAgent).configureCVDevices(0xc000829c30, 0xc00072e1e0, 0xc000736e70, 0x38, 0xc000829310, 0x2, 0x2, 0x0)
        /home/aranjan/Project/go/src/github.com/rook/rook/pkg/daemon/ceph/osd/volume.go:70 +0x33a
github.com/rook/rook/pkg/daemon/ceph/osd.(*OsdAgent).configureDevices(0xc000829c30, 0xc00072e1e0, 0xc0000bc370, 0x1c, 0xc0008296e8, 0x1, 0x1, 0xd)
        /home/aranjan/Project/go/src/github.com/rook/rook/pkg/daemon/ceph/osd/agent.go:213 +0x97d
github.com/rook/rook/pkg/daemon/ceph/osd.Provision(0xc00072e1e0, 0xc000829c30, 0xc0000560d0, 0x24)
        /home/aranjan/Project/go/src/github.com/rook/rook/pkg/daemon/ceph/osd/daemon.go:166 +0xeb2
github.com/rook/rook/cmd/rook/ceph.prepareOSD(0x359be60, 0x35cd220, 0x0, 0x0, 0x0, 0x0)
        /home/aranjan/Project/go/src/github.com/rook/rook/cmd/rook/ceph/osd.go:252 +0x838
github.com/rook/rook/vendor/github.com/spf13/cobra.(*Command).execute(0x359be60, 0x35cd220, 0x0, 0x0, 0x359be60, 0x35cd220)
        /home/aranjan/Project/go/src/github.com/rook/rook/vendor/github.com/spf13/cobra/command.go:762 +0x465
github.com/rook/rook/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x359e200, 0xc0006eff30, 0xb, 0xb)
        /home/aranjan/Project/go/src/github.com/rook/rook/vendor/github.com/spf13/cobra/command.go:852 +0x2ec
github.com/rook/rook/vendor/github.com/spf13/cobra.(*Command).Execute(...)
        /home/aranjan/Project/go/src/github.com/rook/rook/vendor/github.com/spf13/cobra/command.go:800
main.main()
        /home/aranjan/Project/go/src/github.com/rook/rook/cmd/rook/main.go:35 +0x12e
```

Signed-off-by: Ashish Ranjan <aranjan@redhat.com>
2019-09-06 13:40:22 +05:30
Santosh Pillai ead0653068 fix random OSD daemon pod failures when running OSD on PVC
OSD pods when running on PVC fail randomly due to some race condition:
 - OSD daemon pod was activating the volume group using vgchange -an without any volume group name.
 - This caused the pods to fail randomly when number of OSDs is more, say greater than 10.
 - This fix adds Volume group name while activating/deactivating volume groups to fix this

Signed-off-by: Santosh Pillai <sapillai@redhat.com>
2019-09-05 20:34:03 +05:30
Sébastien Han 1393b1af3c ceph: osd lvm flag
udev is not running in the container so lvm might be stuck requesting lvm
devices. So let's not talk to udev to get lvm devices.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-08-30 17:39:49 +02:00
Michael Vollman ed9519953a ceph: add per OSD custom deviceClass support
Add deviceClass as a per OSD config option under
storage/nodes/devices/config for setting a custom crush device class
per OSD.

Signed-off-by: Michael Vollman <michael.b.vollman@gmail.com>
2019-08-29 20:39:41 -04:00
Michael Vollman 496c638306 ceph: add db-devices flag and fix DatabaseSizeMB
Adding support for the db-devices flag for ceph versions 14.2.1 and
newer.  This flag will allow for explictly setting the db device for
bluestore configurations to the device specified as metadataDevice.

Fixing the databaseSizeMB config parameter to ensure it is used to set
the size of the db-device partition when specified at either the global
or individual device level. (#3652)

Signed-off-by: Michael Vollman <michael.b.vollman@gmail.com>
2019-08-29 20:39:41 -04:00
Michael Vollman 9eccffd794 ceph: fix md and dev ordering for c-v batch
Ensure that the metadata device and osd devices are passed as a space
separated list to ceph-volume.  Currently osds-per-device is placed
between the metadataDevice and the osd devices causing an error.

Signed-off-by: Michael Vollman <michael.b.vollman@gmail.com>
2019-08-26 19:17:52 -04:00
Michael Vollman 33711b7d36 ceph: fix osdsPerDevice config
Fix variable scoping issue that is ignoring osdsPerDevice config
setting.  Ensure that in all scenarios both the global and per OSD
osdsPerDevice setting are honored.  When both are set the more specific
setting will be used. (#3599)

Signed-off-by: Michael Vollman <michael.b.vollman@gmail.com>
2019-08-20 07:09:12 -04:00
Travis Nielsen 27cad9d008 Merge pull request #3606 from sp98/sed_to_golang
Replace "sed" commands with native golang file handling
2019-08-19 12:03:43 -06:00
Santosh Pillai b1c9ad97ae replace "sed" commands with native golang file handling
- OSD backed by PVC required updating the lvm.conf file which was done by executing the 'sed' commands.
- This patch removes the 'sed' command and uses golang utlities to update the lvm.conf file.

Signed-off-by: Santosh Pillai <sapillai@redhat.com>
2019-08-19 12:38:56 +05:30
Michael Vollman 46f8012268 ceph: fix per osd metadata support
When setting metadataDevice per OSD the check that c-v is properly using
the metadata device is failing because it is only looking at the global
md setting which is nil and not the per OSD setting.

Signed-off-by: Michael Vollman <michael.b.vollman@gmail.com>
2019-08-15 07:22:00 -04:00
rohan47andAshish Ranjan d2f52aebe5 Adds support for storageClassDeviceSet in rook-ceph operator
- Added code to support StorageClassDeviceSet spec provided in the cluster-on-pvc.yaml
- The code reads the StorageClassDeviceSet spec and creates pvc based on the ‘count’ field for each device set.
- OSD prepare job is started for each PVC which activates the ceph-volume on each PVC
- Finally OSD is started on each of the PVC device.

Co-authored-by: rohan47 <rohgupta@redhat.com>
Co-authored-by: Ashish Ranjan <aranjan@redhat.com>
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
2019-08-12 09:24:13 -06:00
Michael Vollman f3ef680298 ceph: Add support for per OSD metdataDevices
Add ceph-volume support for per OSD metdataDevices

Signed-off-by: Michael Vollman <michael.b.vollman@gmail.com>
2019-08-02 07:41:45 -04:00
Michael Vollman bd58df9d3b ceph: fail when metadataDevice not used by cephvol
fail when metadataDevice is specified but ceph-volume batch does not
automatically use the metadataDevice as the block.db. (#3092)

Signed-off-by: Michael Vollman <michael.b.vollman@gmail.com>
2019-07-23 07:23:51 -04:00
Santosh Pillai ec2e6ea888 use ceph fsid to filter ceph volume OSDs
- Updated getCephVolumeOSDs method to use fsid filter while retriving devices.
- This solves "failed to fetch mon config (--no-mon-config to skip)" error when multiple ceph clusters are running on same Node

Signed-off-by: Santosh Pillai <sapillai@redhat.com>
2019-06-05 12:05:56 +05:30
Michael Vollman a839527421 ceph: provision OSDs with metadataDevice config
provision OSDs with ceph-volume when metadtaDevice is set (#3108)
always print ceph-volume report before executing ceph-volume.

Signed-off-by: Michael Vollman <michael.b.vollman@gmail.com>
2019-05-10 11:47:15 -04:00
Ash WuandChia-liang Kao 118a1383e6 Support metadataDevice & databaseSizeMB for ceph-volume based osd
1. Add support of `metadataDevice` & `databaseSizeMB` options to
   OSDs provisioned by `ceph-volume`.

2. Add `ceph-volume --report` output before the actual run to
   provide more info.

Signed-off-by: Ash Wu <hSATAC@gmail.com>
Co-authored-by: Ash Wu <hSATAC@gmail.com>
Co-authored-by: Chia-liang Kao <clkao@clkao.org>
2019-04-23 16:04:21 +08:00
travisn aae4d7c7f6 osds: write ceph-volume log entries as they occur
When ceph-volume is called, the default python logging will buffer everything until
the process exits, which means the pod log doesnt show what is happening
until ceph-volume is done. This change will allow logs to be written
immediately as tasks are performed inside ceph-volume

Signed-off-by: travisn <tnielsen@redhat.com>
2019-04-16 15:47:05 -06:00
travisn bdc3cf8146 osd: fix the device filter and improve device provisioning reliability
All devices detected by the discovery pod were being passed to the OSD provisioning pod
thus not always honoring the desired device list that should be provisioned.
Now the provisioning pod will be given the desired state from the crd,
then apply that state depending on the actual devices detected.
Also added a helper to ensure OSDsPerDevice is always valid.

Signed-off-by: travisn <tnielsen@redhat.com>
2019-02-26 16:11:25 -07:00
travisn fe4a645844 osd: detect existing osds not to be passed to ceph-volume
Signed-off-by: travisn <tnielsen@redhat.com>
2018-12-07 10:50:37 -07:00
travisn 1389cc56db ceph: enable multiple osds per device
Signed-off-by: travisn <tnielsen@redhat.com>
2018-12-07 09:29:15 -07:00
travisn e6572e41e6 enable ceph-volume to provision and start filestore osds
Signed-off-by: travisn <tnielsen@redhat.com>
2018-12-06 21:28:58 -07:00
travisn 45c53dd5b2 detect if ceph-volume is supported and allow legacy osds to be started
Signed-off-by: travisn <tnielsen@redhat.com>
2018-12-06 21:28:58 -07:00
travisn 78b8011226 osds: prototype provisioning with ceph-volume
Signed-off-by: travisn <tnielsen@redhat.com>
2018-12-06 21:28:58 -07:00