Commit Graph
566 Commits
Author SHA1 Message Date
Travis Nielsen c73022060a ceph: detect osd pvcs in a batch
The OSD PVCs were being detected one by one with a label selector.
This proves to be slow in large clusters. Instead, we now make this
more efficient by querying for all existing PVCs and looking up their
existing in memory with a map.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-09-28 12:52:17 -06:00
subhamkrai de8dbbcdcc ceph: handle golangci-lint linter staticcheck error
this commit handle golangci-lint linter staticcheck error.

`staticcheck` - Staticcheck is a go vet on steroids,
applying a ton of static analysis checks.

To see only `staticcheck` linter output

`golangci-lint run --disable-all -E staticcheck`

Signed-off-by: subhamkrai <srai@redhat.com>
2020-09-24 15:04:55 +05:30
Travis Nielsen 2b9e7a5335 ceph: allow osd pvc template name set to any value
The pvc template name has either been require to be empty, or one of the
expected values such as data and metadata. Those who upgrade from 1.1
may find that if they changed it to something else, their upgrade will
fail when not one of these values. If there is only a single template,
always treat it as the data template.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-09-22 15:20:54 -06:00
subhamkrai 1e9bb8e6e2 ceph: handle golangci-lint linter deadcode
this commit will enable one more linter `deadcode`
in golangci-lint .

Signed-off-by: subhamkrai <subhamkumarrai03@gmail.com>
2020-09-18 16:53:49 +05:30
Sébastien Han d118bd38a6 Merge pull request #6278 from subhamkrai/golanci-lint-unused
ceph: handle golangci-lint linter unused
2020-09-18 12:44:37 +02:00
Travis Nielsen 524362f9e4 ceph: avoid duplicating env vars in the osd deployments
The OSD deployments have some env vars duplicated in the pod spec for
the daemon container, which may cause the detection for patch changes
to fail. Now we ensure the env vars are not duplicated in the OSD pod
spec.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-09-17 13:43:05 -06:00
subhamkrai f9fafe62d4 ceph: handle golangci-lint linter unused
this commit will enable one more linter
in golangci-lint.

Signed-off-by: subhamkrai <subhamkumarrai03@gmail.com>
2020-09-17 22:53:24 +05:30
subhamkrai 25c116a4bd ceph: handle golangci-lint linter gosimple
this commit handles all the errors  check for
golangci-lint linter gosimple.

Signed-off-by: subhamkrai <subhamkumarrai03@gmail.com>
2020-09-17 15:10:27 +05:30
Travis Nielsen 00e6398fe8 Merge pull request #6235 from fanjunwei/user_truncated_node_name
ceph: Use truncated node name
2020-09-15 13:19:49 -06:00
fanjunwei f013add62e ceph: use truncated node name
Signed-off-by: fanjunwei <fanjunwei@t2cloud.net>
2020-09-11 19:48:38 +08:00
subhamkrai 6d08d94948 ceph: add storageClassDeviceSet label to osd pods
this commit will add a new flag to the osd pods
which will be named according to the name of the
storageClassDeviceSet. This will help in differentiate
pods immediately.

Signed-off-by: subhamkrai <subhamkumarrai03@gmail.com>
2020-09-11 07:26:40 +05:30
Alexander Trost a87c00ddee ceph: allow user to add labels to pods
This allows users to specify additional labels to be added to the components
created by the Rook Ceph operator.
Additionally the rbd mirrors are now getting their annotations and
labels set.

Signed-off-by: Alexander Trost <galexrt@googlemail.com>
2020-08-25 19:38:51 +02:00
Sébastien Han ee893eda5e Merge pull request #6126 from leseb/encryption-expansion-pvc
ceph: add expansion support for encrypted osd on pvc
2020-08-19 21:40:46 +02:00
Travis Nielsen ce92249725 ceph: continue with memory limits below min settings
The operator will now allow the resource limits to be applied below the
recommended minimums. In small clusters, even the recommended minimums
may not be necessary. A warning is still printed to the operator log,
but we allow the configuration to continue.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-08-19 11:07:16 -06:00
Sébastien Han 321e8e0b5a ceph: add more pvc dep unit tests
Added more unit tests scenario for osd on pvc such as:

* block/db/wal
* block encryption
* block/db encryption
* block/db/wal encryption

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-08-19 18:45:31 +02:00
Sébastien Han 803462ceca ceph: add encrypted block status container
This init container displays the status of the encrypted pvc. This can
be really useful for debugging purpose.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-08-19 17:13:46 +02:00
Sébastien Han cb81ee7d1d ceph: add pvc expansion support for encrypted osds
We now support the expansion of encrypted osds. We added a new init
container to run "cryptsetup resize".

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-08-19 17:13:46 +02:00
Sébastien Han 3f42db5ce0 ceph: support osd deployment update on encrypted pvc
If the deployment spec was updated or restarted, the "encryption-open"
init container would fail to open the device since it was already
opened on the node.
Because they are no reliable post stop hook on Kubernetes, the trick is
to use a small bash init container.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-08-19 17:13:25 +02:00
Sébastien Han d5b1e89ce2 Merge pull request #6118 from travisn/osd-prepare-resources
ceph: Set the OSD prepare resources
2020-08-19 15:38:06 +02:00
Travis Nielsen 18d39d909e ceph: set the osd prepare resources
The osd prepare resources were only being set on the main container
while the init containers either did not have resources set, or were
inheriting the resource settings from the device set resources for osds.
Now the osd prepare resources will be consistently set based on the
prepareosd resources in the main section of the cluster CR.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-08-18 22:58:26 -06:00
Travis Nielsen 26f59c0382 ceph: remove redundant check for encrypted osds
This is simply a code simplification to remove an extra check for
whether the OSDs are encrypted. There is no change in functionality.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-08-18 22:50:32 -06:00
Travis Nielsen 0ffa0348b5 ceph: separate placement for osd prepare jobs from osd daemons
The osd prepare jobs and osd daemons may have different placement
requirements. In particular, the topology spread constraints
or pod anti-affinity should only work on either the prepare job or
osd daemon, but does not work effectively on both types at the same
time. Therefore, we split out the osd prepare placement to be separate
from the osd daemon placement. If the osd prepare placement is not set
the operator will fall back to the osd daemon placement for
backward compatibility.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-08-18 15:24:04 -06:00
Sébastien Han 4f9435d226 ceph: add support for wal encrypted device on pvc
Similar to https://github.com/rook/rook/pull/5977 but for wal devices.
So if a cluster is deployed on PVC and wal devices, those will be
encrypted as well.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-08-13 15:42:40 +02:00
Sébastien Han a91c96cb47 ceph: add support for encrypted metadata pvc
Now the metadata pvc will also be encrypted if the storageClassDeviceSet
has the "encrypted" flag turned on.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-08-12 22:24:11 +02:00
Sébastien Han 7d6974b9b7 ceph: add octopus validation for pvc encryption
The c-v encryption is part of 15.2.5 on Octopus.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-08-12 12:06:35 +02:00
Chen, Tingjie 0e87348093 ceph: add support for wal device for OSD on PVC
Based on the feature of metadata PVC for OSD: #4818, add the wal device.

For this, you need to create a new volumeClaimTemplates, its name must be "wal", if "wal" is
set, "metadata" device will refers specially for block.db device, and "wal" refers for block.wal
device in BlueStore.

A template will look like this:

```
volumeClaimTemplates:
- metadata:
    name: data
  spec:
    resources:
      requests:
        storage: 10Gi
    # IMPORTANT: Change the storage class depending on your environment (e.g. local-storage, gp2)
    storageClassName: gp2
    volumeMode: Block
    accessModes:
      - ReadWriteOnce
- metadata:
    name: metadata
  spec:
    resources:
      requests:
        storage: 6Gi
    # IMPORTANT: Change the storage class depending on your environment (e.g. local-storage, io1)
    storageClassName: io1
    volumeMode: Block
    accessModes:
      - ReadWriteOnce
- metadata:
    name: wal
  spec:
    resources:
      requests:
        storage: 6Gi
    # IMPORTANT: Change the storage class depending on your environment (e.g. local-storage, io1)
    storageClassName: io1
    volumeMode: Block
    accessModes:
      - ReadWriteOnce
```

We now map block, block.db and block.wal directly inside the container instead of
running ceph-volume activate.

Signed-off-by: Chen, Tingjie <tingjie.chen@intel.com>
2020-08-10 08:48:20 +08:00
Sébastien Han 89b3225441 ceph: add encryption support for osd pvc
We can now encrypted OSD device that were provisioned via a storage
class using the PV interface.
The encryption works at the storageClassDeviceSets level, which means we
can have encrypted and non-encrypted sets.
Using the new key `encrypted` we can turn it on.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-08-05 10:06:11 +02:00
Travis Nielsen 06c21954e1 ceph: labels are immutable for match selectors
The labels cannot be updated on a deployment match selector.
In v1.4 a new label was added for ceph_daemon_type that is intended
to be on the pod labels, but cannot be applied to the match selectors.
Therefore, we suppress any new labels that are added to the daemons.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-08-04 16:47:00 -06:00
Blaine Gardner dcf8ae2526 ceph: rename pod labels function for more clarity
Rename PodLabels function to CephDaemonAppLabels for more clarity about
what the function's purpose is.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2020-07-30 11:16:30 -06:00
Blaine Gardner 62cbaaec02 ceph: add new "ceph_daemon_type" label to osd pods
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2020-07-30 11:16:26 -06:00
Travis Nielsen 9e6429b4c8 Merge pull request #5893 from subhamkrai/gosec-error-G601
ceph: handling gosec error code g601
2020-07-28 08:25:00 -06:00
subhamkrai 465a0f0aec ceph: handling gosec error code g601
this commit handles all the gosec g601
error code (i.e Implicit memory aliasing
of items from a range statement).

Signed-off-by: subhamkrai <subhamkumarrai03@gmail.com>
2020-07-28 11:34:49 +05:30
Sébastien Han 71dff530d3 ceph: split osd code in files
Makes the code more readable and decoupled into files.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-07-27 23:21:14 +02:00
subhamkrai acc4ed5df9 ceph: handling gosec errors that are not checked
this commit handles all the gosec g104
i.e audit errors not checked inside pkg.

Signed-off-by: subhamkrai <subhamkumarrai03@gmail.com>
2020-07-24 20:02:06 +05:30
Travis Nielsen 77e787b00e Merge pull request #5805 from kesavanvt/add_device_class
ceph: Adding deviceClasses info to CephCluster CR
2020-07-22 16:00:18 -06:00
kesavan 453200045e ceph: adding deviceclasses info to cephcluster cr
The admin needs to start the toolbox and run a command such as ceph osd tree to see the device classes for each OSD. This PR simplifies it by storing the deviceclass info in the CephCluster CR status

closes: https://github.com/rook/rook/issues/5754
Signed-off-by: kesavan <kvellalo@redhat.com>
2020-07-22 17:46:51 +05:30
Sébastien Han f3c4975e35 ceph: fail prepare pod if lvm2 pkg is missing
A long-standing issue that was only solved via a documentation note. Now,
during the prepare pod instantiation, we check for the presence of the
lvm binary on the host. Success will indicate that we can bootstrap
OSDs where failure will refuse to prepare the OSD.

Closes: https://github.com/rook/rook/issues/5627
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-07-20 10:05:05 +02:00
Blaine Gardner 825f1e3eb2 ceph: support device names with special chars
Pass desired devices to the OSD provisioning container by
JSON-marshalling/-unmarshalling a disk ID with StorageConfig settings.
This will allow disks to be specified that contain special characters.
Notably, this will support /dev/disk/by-path/pci-HHHH:HH:HH.H devices
that were unsupported previously due to the format which separated
devices from params with colons (:).

Fixes #5056
Fixes #5535

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2020-07-17 09:44:26 -06:00
Travis Nielsen e74c7eaef8 ceph: refactor context and clusterInfo passed to the ceph commands
To provide more context for executing commands in a ceph cluster,
the full clusterInfo is now passed to the ceph execution commands.
All information about the cluster will now be available throughout
all the areas of the operator. The namespace, ceph credentials,
mon endpoints, and other info is a core part of that cluster info.

Arguments passed through the controllers are also simplified for
mons, mgr, osds, and other daemons where the parameters had
become too complex.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-07-16 15:54:54 -06:00
Travis Nielsen 4681f9e73d ceph: consolidate ceph config and client packages
The ceph config and client packages are conceptually the same.
To avoid circular dependencies in some cases, we simplify by combining
the packages into the client package.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-07-16 15:54:43 -06:00
Travis Nielsen 631b13b906 ceph: refactor creds used by operator
The operator should only connect to ceph with a single set of creds.
In a converged cluster this will be the admin creds and in an external
cluster it will be lower-privileged creds. Independent clusters were
implemented with a separate set of creds. To simplify the code these
are now merged to a single set.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-07-16 15:54:42 -06:00
Blaine Gardner 7117fc12b7 ceph: osd: add drive groups spec to cluster CR
Add the ability to provision Ceph OSDs with Drive Groups.
This adds Drive Groups to the CephCluster CRD, and it sets code
in place for propagating this config to the OSD provisioning pod.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2020-07-14 09:43:03 -06:00
Sébastien Han ccb52b84e6 ceph: configurable status checks and livenessprobe
This commit allows us to configure status check for each daemon:

* "mon": health check on the ceph monitors (quorum)
* "osd": health check on the ceph osds
* "status": ceph health status check

Each check is controlled by the following settings:

* disabled: whether to disable the check (default: false)
* internal: interval to run the check
* timeout: only valid for mons, is the timeout for unresponsive mon
before failling over.

Example to disable the status health check:

```yaml
healthCheck:
  daemonHealth:
    status:
      disabled: true
```

As part of that, pod's livenessprobe can now be configured via the
following settings:

* disabled: whether to enable or not
* probe: override the current probe in place by a new one

```yaml
healthCheck:
  livenessProbe:
    mon:
      disabled: true
```

Closes: https://github.com/rook/rook/issues/5772
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-07-10 19:21:59 +02:00
Travis Nielsen 8f9df75f7b ceph: capture logging from the osd daemon
The OSD daemon has been missing critical flags for logging to stderr
where k8s can capture the logs. Without the --log-to-stderr=true,
all the OSD logging was essentially lost until now.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-07-06 14:38:37 -06:00
Travis Nielsen dbae36b2bb ceph: update to the latest octopus release
Update the base operator image and cluster examples to use the latest
octopus release v15.2.4. Also cleanup some old examples and unit tests
that were still based on mimic.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-07-06 12:11:47 -06:00
UMEZAWA Takeshi 6085ff3223 ceph: add schedulerName property to storageClassDeviceSet
Support schdulerName to use another scheduler for OSD pod placement.

Closes: https://github.com/rook/rook/issues/5536
Signed-off-by: UMEZAWA Takeshi <takeshi-umezawa@cybozu.co.jp>
Signed-off-by: Masayuki Ishii <masa213f@gmail.com>
2020-06-12 11:53:28 +09:00
Travis Nielsen 5a02de5621 ceph: metadata pvcs use a map instead of slice index
The volume sources for an OSD on PVC were relying on the ordering
of the volume sources slice. While this was working, it was causing
headaches and potential future developer error when being
interpreted incorrectly. The addition of the wal pvc device
would make the issue even more pronounced. Now each PVC used
for the OSD will be identified in a map by name such as data
 and metadata.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-06-11 08:58:53 -06:00
Travis Nielsen 282a2a3741 ceph: increase osd size when pvc template increases
When the PVC template increases in size for the storageClassDeviceSet
in the cluster CR, we need to increase the size of the PVC and restart
the OSD. The OSD is restarted by changing a variable on the pod spec
that indicates the desired size of the OSD.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-06-08 14:34:43 -06:00
Travis Nielsen daeb9b56b0 ceph: remove dead osd code
The completeProvisionSkipOSDStart function is no longer used.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-06-08 14:34:43 -06:00
Travis Nielsen ff6a3f80cd ceph: improve osd update logging
The operator should only print helpful info messages
when an OSD is going to be updated. If the OSD hasn't
changed it is just a debug message.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-05-28 10:05:51 -06:00