Commit Graph
199 Commits
Author SHA1 Message Date
Satoru Takeuchi 26c8fd9bd1 ceph: improve owner reference management
It's better to validate ownerReferences when setting them. In addition, we should use
controllerrutil.Set{Controller,Owner}Reference, that have such validation, as possible.

Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
2021-03-16 10:29:33 +00:00
Travis Nielsen c0eacc89d5 Merge pull request #7235 from travisn/mgr-sidecar
ceph: Allow two mgr daemons and actively reconcile the mgr services with a sidecar
2021-03-10 10:35:40 -07:00
Travis Nielsen 388ff3e78b ceph: start sidecar to monitor active mgr
The mgr daemon may be failed over by ceph if the active mgr is not
responding and the standby mgr is available. If the active mgr changes
the services for the dashboard and metrics will be updated with a
label selector for the new active mgr. The services cannot direct
traffic to the standby mgr or else they will be incorrectly redirected
to the active mgr.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-03-10 08:37:17 -07:00
Sébastien Han a51d8c5f2e ceph: use raw mode for pacific on non-pvc
We will now use raw mode on non-pvc for pacific and onward.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-03-10 12:18:31 +01:00
Sébastien Han 504e9aa806 ceph: do not duplicate osd info
When c-v runs after the end of the prepare sequence, it will look for
lvm and raw-based OSDs. The raw mode detects LVM OSDs so we must trim
the list of OSD, only to report the LVM ones if their ID exists.

Closes: https://github.com/rook/rook/issues/7359
Signed-off-by: Sébastien Han <seb@redhat.com>
2021-03-09 15:17:32 +01:00
Sébastien Han 3dde2609fa ceph: add raw mode for non-pvc osd
For simple OSD scenario that don't require the following:

* an encrypted osd
* an osd with a backing device
* multiple osd on the same block device

The OSD will be prepared using the ceph-volume raw mode which does
not use LVM on non-pvc case.

Closes: https://github.com/rook/rook/issues/4768
Signed-off-by: Sébastien Han <seb@redhat.com>
2021-03-03 14:11:26 +01:00
Sébastien Han 06810b9700 ceph: fix iteration
block if not defined was used with osdInfo.Device so not empty anymore
in the loop, which means it will have the same value for all OSDs in the
loop.
This is only called when running on PVC, where a single device is
present so the loop has a single iteration, so not a big issue.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-03-03 14:06:19 +01:00
Sébastien Han a134d1df6e Merge pull request #7336 from leseb/revert-vault-backend
ceph: revert kms VAULT_KV_VERS to VAULT_BACKEND
2021-03-02 10:44:43 +01:00
Sébastien Han e4c3a4cef7 ceph: revert kms VAULT_KV_VERS to VAULT_BACKEND
The vault env variable VAULT_BACKEND is read by the secret library so we
must not change it. This is reverting
ee6a03dfb8 with a few extra fixes for the
rgw code.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-03-02 10:08:39 +01:00
Blaine Gardner efa7d66d45 ceph: remove driveGroup support
Remove features and design that supports adding OSDs to Ceph clusters
via `spec:driveGroups`. Update the Ceph upgrade doc that informs users
who currently use Drive Groups (we believe there are none of these
users) how to migrate to using the `spec:storage` config.

Resolves https://github.com/rook/rook/issues/7275

Revert "ceph: fix drive group deployment failure"
This reverts commit 76f1d9944e.

Revert "ceph: osd: add drive groups spec to cluster CR"
This reverts commit 7117fc12b7.

Revert "design: ceph orchestrator module add/remove OSDs"
This reverts commit 178187d035.

Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
2021-02-25 17:08:47 -07:00
Travis Nielsen 291bf9e061 ceph: enforce portable osds in same topology as osd prepare job
The osd prepare job detects the failure domain for the OSD by querying
the node topology labels. This topology is then assigned to the OSD
daemon in the CRUSH map. Previously, the affinity was required to be
set in the cluster CR, but it was very difficult to get right. Now the
operator will enforce the correct topology label on the OSD daemon
nodeAffinity by setting the label of the lowest topology in the hierarcy.

For example, if there are region, zone, and rack labels, the rack label
would be used to set the node affinity for the OSD daemon. If an
OSD prepare job is run in rack1, the corresponding OSD daemon will
have node affinity to rack1 to ensure the same topology. Previously,
the OSD could have ended up in rack2 unless the storageClassDeviceSet
placement was very carefully crafted.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-02-18 12:49:37 -07:00
Sébastien Han f8ce9df5e3 Merge pull request #7001 from crombus/master
ceph: Add ceph archive cmd in osd removal
2021-02-18 17:09:52 +01:00
crombus 76ed166e86 ceph: add ceph archive cmd in osd removal
Add ceph archive cmd in osd removal for
Silencing the warning when osd is removed.

Signed-off-by: crombus <pkundra@redhat.com>
2021-02-18 21:08:16 +05:30
Travis Nielsen dfc24ddd24 Merge pull request #7240 from leseb/fix-vault-key-override
ceph: do not override existing keys
2021-02-16 16:09:34 -07:00
Sébastien Han 9a604640ac ceph: do not override existing keys
If the cluster was expanded we would iterate over the entire list of
OSDs again, create a new key and override in Vault... The Secret store
is not impacted since we have a handler for "already exist".

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-02-16 19:20:20 +01:00
Blaine Gardner a95734160e ceph: create new osds on pvc before update existing
Add a new OSD provisioner status (reported by provisioner configmap)
that denotes that an OSD is preexisting and that OSD prepare does not
need to be run for the OSD. This only applies to OSDs on PVC currently
where existence of a deployment for the OSD indicates no further
provisioning needs to occur. New status is "preexisting".

Allow creating new OSDs on PVCs before updating existing ones by
allowing deferring OSDs during processing of OSD provisioning status
ConfigMaps. Deferral is identified by new "preexisting" status.

Add a unit test to ensure OSD on PVC provisioning occurs as expected
including deferring already-created OSDs to be updated after new PVCs
are created.

Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
2021-02-12 17:38:35 -07:00
Sébastien Han e6e9c8c901 ceph: do not print pointer memory address
When printing pointers of struct (regardless if it is also a slice or
not) we must implement a stringer interface so they can be displayed
properly, this is especially useful when looking at the prepare job
logs.

Before this patch:

```
2021-02-10 06:39:06.783508 D | inventory: discovered disks are [0xc00029c240 0xc00024a000 0xc00086c480 0xc00024a480]
```

After this patch:

```
2021-02-11 13:05:15.730500 D | inventory: discovered disks are:
2021-02-11 13:05:15.730636 D | inventory: &{Name:dm-0 Parent: HasChildren:false DevLinks:/dev/disk/by-id/dm-uuid-LVM-aIFRq5P3hwE2YhukbOmsEZoLY61HxvX62A1xxb6ajTymVE0N8S4Ga0KWlHSgpvNw /dev/disk/by-id/dm-name-ceph--e29bf82e--86f1--4c8e--8412--c468def5a844-osd--block--2044a1ea--6e5e--4423--88cf--8947bf6957d1 Size:32208060416 UUID:91a852c2-8727-4277-9d92-879ee1996503 Serial: Type:lvm Rotational:true Readonly:false Partitions:[] Filesystem:ceph_bluestore Vendor: Model: WWN: WWNVendorExtension: Empty:false CephVolumeData: RealPath:/dev/mapper/ceph--e29bf82e--86f1--4c8e--8412--c468def5a844-osd--block--2044a1ea--6e5e--4423--88cf--8947bf6957d1 KernelName:dm-0 Encrypted:false}
2021-02-11 13:05:15.730657 D | inventory: &{Name:dm-1 Parent: HasChildren:false DevLinks:/dev/disk/by-id/dm-uuid-LVM-GyLVW5UxDRcws8PwsbQuY2CjrQwdn1VxgPQjsKZ2pGVaS1r2Ct0OGZ4P80yEMtqW /dev/disk/by-id/dm-name-ceph--334c640f--7a7f--4fdb--9db4--2f9459e19f9a-osd--block--66ce11c8--20f0--4600--ae50--e8849c9a0af6 Size:32208060416 UUID:cdbcb218-1b03-4d8c-b7d2-25cb88e33b6a Serial: Type:lvm Rotational:true Readonly:false Partitions:[] Filesystem:ceph_bluestore Vendor: Model: WWN: WWNVendorExtension: Empty:false CephVolumeData: RealPath:/dev/mapper/ceph--334c640f--7a7f--4fdb--9db4--2f9459e19f9a-osd--block--66ce11c8--20f0--4600--ae50--e8849c9a0af6 KernelName:dm-1 Encrypted:false}
2021-02-11 13:05:15.730686 D | inventory: &{Name:dm-2 Parent: HasChildren:false DevLinks:/dev/disk/by-id/dm-name-ceph--a011b5c2--cc3f--497a--b19a--6d284461cbc4-osd--block--4de5491c--ecf8--455d--9858--547001fd0321 /dev/disk/by-id/dm-uuid-LVM-QGzgO4E228fvT3NsighNABGdvAKxQEhFOh9rKiDLnSOTk7jLZWu2r1RIMXe5GNqW Size:32208060416 UUID:2238ce24-79e4-41a0-b8ce-e381ac41d5b0 Serial: Type:lvm Rotational:true Readonly:false Partitions:[] Filesystem:ceph_bluestore Vendor: Model: WWN: WWNVendorExtension: Empty:false CephVolumeData: RealPath:/dev/mapper/ceph--a011b5c2--cc3f--497a--b19a--6d284461cbc4-osd--block--4de5491c--ecf8--455d--9858--547001fd0321 KernelName:dm-2 Encrypted:false}
2021-02-11 13:05:15.730713 D | inventory: &{Name:vda1 Parent:vda HasChildren:false DevLinks:/dev/disk/by-path/pci-0000:00:05.0-part1 /dev/disk/by-path/virtio-pci-0000:00:05.0-part1 /dev/disk/by-uuid/0fdf5e49-6b3c-42f2-a1c5-bf77e490dfa4 /dev/disk/by-partuuid/f9dd32c9-a4a5-654f-aee4-651276fdc754 /dev/disk/by-label/boot2docker-data Size:19998934528 UUID: Serial: Type:part Rotational:true Readonly:false Partitions:[] Filesystem:ext4 Vendor: Model: WWN: WWNVendorExtension: Empty:false CephVolumeData: RealPath:/dev/vda1 KernelName:vda1 Encrypted:false}
```

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-02-11 14:06:42 +01:00
Sébastien Han 7c928f4196 ceph: return an error if PVC is not clean
With 5f57b6fd3a, we introduced a
regression in the validation code.
If the OSD PV comes from another cluster, GetCephVolumeRawOSDs must
return an error.
The prepare job will crash loop but at least with a useful and
comprehensive message.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-02-09 11:28:08 +01:00
Jiffin Tony Thottan da61c9a83e ceph: vault kms configuration for ceph object store
The first patch to configure vault for ceph object store. If the `security.kms` configured in
`clusterSpec` CRD, RGW will be configured with vault kms settings to handle SSE request from s3 clients.

Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
2021-01-22 10:44:03 +05:30
shenjiatong 8d3d460e70 ceph: refactor and remove duplicate codes
Signed-off-by: shenjiatong <yshxxsjt715@gmail.com>
2020-12-18 18:01:30 +08:00
shenjiatong 04aabb3896 ceph: analyze result for new typed return from lvm batch
Signed-off-by: shenjiatong <yshxxsjt715@gmail.com>
2020-12-18 11:11:03 +08:00
ushen 373b952988 ceph: allow ceph 14.2.15 for lvm batch
14.2.15 lvm batch command prepare report changes
output format. This commit skips md check if ceph version
is greater than 14.2.13.

Signed-off-by: shenjiatong <yshxxsjt715@gmail.com>
2020-12-18 11:11:02 +08:00
Travis Nielsenandshenjiatong a0a27bc426 ceph: apply deviceClass properly to new osds
The deviceClass property was being ignored when creating the
non-pvc OSDs. Now the deviceClass will be specified as a property
for individual devices, all devices on a node, or all OSDs in the
cluster, depending on the level where the config is applied in the
cluster CR.

Co-authored-by: shenjiatong <yshxxsjt715@gmail.com>
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-12-11 10:16:00 -07:00
Sébastien Han 5d9612c2ae ceph: fix metadata device passed by-id
The code was assuming that devices were passed by the user as
"/dev/sda", this is bad! We all know people should be using paths like
/dev/disk/by-id so we must support them.

Closes: https://github.com/rook/rook/issues/6685
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-11-25 15:00:47 +01:00
Arun Kumar Mohan 65d16bfc94 ceph: manual changes needed for kubernetes api updates
Fetched latest lib-bucket-provisioner changes as well.

Signed-off-by: Arun Kumar Mohan <amohan@redhat.com>
2020-11-18 21:14:01 +05:30
Mateusz Gozdek 8ba3762fa4 docs: fix bunch of typos
Found by running the following command:

codespell -S .git,*.png,*.jpg -L \
aks,keyserver,atleast,dne,ser,ist,files\',ba,dum,iam,te -f -H

Signed-off-by: Mateusz Gozdek <mgozdekof@gmail.com>
2020-11-06 10:01:04 +01:00
Sébastien Han 8631eef1f7 ceph: correctly populate variable
The wrong variable was referenced in the print.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-11-05 18:06:47 +01:00
Sébastien Han bc5c8d32f1 ceph: fix crush-class osd on non-PVC
We need to use storeConfig instead device config as the later never
seems to be populated.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-11-05 18:06:47 +01:00
Sébastien Han 6b211eacba ceph: add unit test for osd batch mode
We now have much better coverage on the c-v on non-pvc scenario.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-11-05 18:06:11 +01:00
Travis Nielsen 37a56f39a6 ceph: remove the osd pvc when purging the osd
The osd purge job was not removing the osd prepare job or the
osd pvc due to an invalid label query for the pvc. Now the
prepare job and pvc will be removed as expected from the job.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-11-03 08:32:37 -07:00
Sébastien Han b7e0e206d3 ceph: rework unit test
Golang maps order is not always guaranteed to be identical, which makes
the table driven test to fail. Because of the nature of the maps, it is
hard to predict the exact position of an element in the map. This
makes the table driven test flappy.
Reworked the unit test using a contains statement which achieves the
same validation.

Closes: https://github.com/rook/rook/issues/6522
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-11-02 12:14:09 +01:00
Sébastien Han ea1d71cbfb ceph: add vault kms support for osd encryption
When the Ceph cluster runs on PVC and the OSDs are encrypted we can
store LUKS's Key Encryption Key inside a Key Management System. Today,
Rook only supports HashiCorp Vault: https://www.vaultproject.io/

The CephCluster has now a new "security" field which will plug onto the
KMS. Here is an example:

security:
  kms:
    tokenSecretName: <name of the secret containing a Vault token, used
    to authenticate>
    connectionDetails: < a map of strings containing connection
    information>

Refer to the ceph-cluster-crd documentation to lear more.

Closes: https://github.com/rook/rook/issues/6105
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-10-30 16:16:33 +01:00
Sébastien Han f0b2c6eacf Merge pull request #6184 from cybozu-go/ceph-raw-mode-on-lv-backed-pvc
ceph: raw mode OSD on LV-backed PVC
2020-10-15 09:46:56 +02:00
Daichi Sakaue b9f29349ed ceph: use raw mode instead of lvm mode even if lv-backed PVC
It can simplify OSD management in many ways and can avoid many problems
that come from LVM.

Closes: https://github.com/rook/rook/issues/5939
Signed-off-by: Yuji Ito <llamerada.jp@gmail.com>
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
2020-10-14 01:46:25 +00:00
Sébastien Han b623133fb6 ceph: add ceph-volume log to non-pvc scenario
When deploying on non-PVC, we still need to gather c-v logs on failures.
Now if the bootstrap fails, the c-v log will be printed.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-10-12 14:10:17 +02:00
subhamkrai 0fddfcf307 ceph: handle golangci-lint linter errcheck error
this commit handle golangci-lint linter errcheck.

`errcheck` - Errcheck is a program for checking for
unchecked errors in go programs. These unchecked errors
can be critical bugs in some cases

To see only staticcheck linter output
`golangci-lint run --disable-all -E errcheck`

Signed-off-by: subhamkrai <srai@redhat.com>
2020-09-30 22:24:34 +05:30
subhamkrai 829778f251 ceph: handle golangci-lint linter ineffassign
this commit will enable one more linter ineffassign
in golangci-lint.

This linter throws an error when variable is assigned and never used.

`golangci-lint run --disable-all -E ineffassign` is used detects ineffassign
errors only.

Signed-off-by: subhamkrai <subhamkumarrai03@gmail.com>
2020-09-21 14:31:18 +05:30
subhamkrai 1e9bb8e6e2 ceph: handle golangci-lint linter deadcode
this commit will enable one more linter `deadcode`
in golangci-lint .

Signed-off-by: subhamkrai <subhamkumarrai03@gmail.com>
2020-09-18 16:53:49 +05:30
subhamkrai f9fafe62d4 ceph: handle golangci-lint linter unused
this commit will enable one more linter
in golangci-lint.

Signed-off-by: subhamkrai <subhamkumarrai03@gmail.com>
2020-09-17 22:53:24 +05:30
subhamkrai 25c116a4bd ceph: handle golangci-lint linter gosimple
this commit handles all the errors  check for
golangci-lint linter gosimple.

Signed-off-by: subhamkrai <subhamkumarrai03@gmail.com>
2020-09-17 15:10:27 +05:30
Servesha Dudhgaonkar 2cf0518545 ceph: update remove.go
Call logger.Infof() instead of Debugf() in remove.go file.

Signed-off-by: Servesha Dudhgaonkar <sdudhgao@redhat.com>
2020-09-09 20:06:27 +05:30
Servesha Dudhgaonkar 0c323439a6 ceph: purge a down osd with a job
If an OSD is down and needs to be removed from the cluster,
a job can be run that will remove the OSD deployment
and purge the OSD from ceph. If the OSD is still up,
the purge will be rejected.

To purge multiple OSDs at the same time, the OSD IDs can be
specified as a comma-separated list.

Signed-off-by: Servesha Dudhgaonkar <sdudhgao@redhat.com>
2020-09-01 09:36:20 +02:00
Sébastien Han f2313d6db0 ceph: do not exec on rootfs just lookup
Trying to execute a binary from the host within the container can result
in mismatched libraries. The host version is compiled with paths of
libraries on the host where the libs inside the container can differ.
Because the path is known and we only need to make sure whether the bin
exists or not then simply doing a lookup when we fallback on /rootfs is
sufficient.

Closes: https://github.com/rook/rook/issues/6078
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-08-27 15:16:46 +02:00
Sébastien Han aa5362d48d Merge pull request #6120 from leseb/fix-5996
ceph: do not use realPath for osd on pvc
2020-08-19 17:33:46 +02:00
Sébastien Han 944d8cb209 ceph: do not use realPath for osd on pvc
It is fine to use /mnt/set1-data-0-k47mc, it looks like I thought there
was an issue with udev but there is none...

Closes: https://github.com/rook/rook/issues/5996
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-08-19 14:39:55 +02:00
Travis Nielsen b4db31033b Merge pull request #6107 from leseb/fix-6078
ceph: check rootfs for lvm too
2020-08-18 11:52:36 -06:00
Sébastien Han 713d3ecace ceph: check rootfs for lvm too
On rancher OS, the lvm binary is in a different location and cannot be
accessed via the root namespace. So we now also check directly from
/rootfs. When using nsenter, itbreaks out of container's mount
namespace and executes an lvm command.

Closes: https://github.com/rook/rook/issues/6078
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-08-18 17:04:56 +02:00
Sébastien Han adab0e07e5 Merge pull request #6059 from leseb/lvm-list
ceph: improve callCephVolume() for list and prepare
2020-08-18 13:30:49 +02:00
Travis Nielsen 98db380237 Merge pull request #6062 from leseb/wal-encryption
ceph: add support for wal encrypted device on pvc
2020-08-17 14:07:40 -06:00
Sébastien Han dbac1c11d0 ceph: improve callCephVolume() for list and prepare
callCephVolume() was using our exec interface with CombinedOutput which
means return both stdout and stderr. This is fine when preparing OSDs
since we read back the stdout and also want any errors if any (stderr).
However when listing OSD we only want the output since ceph-volume tends
to mix stdout and stderr in the output and does not always silence the
warning types/harmless errors. If the listing has a problem we report it
by printing the ceph-volume log file.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-08-17 12:43:43 +02:00