Commit Graph
566 Commits
Author SHA1 Message Date
Travis Nielsen 07f095c772 ceph: set osd health check interval to 60s
The osd health check every 5 min is not frequent enough to respond
quickly enough to osds going down. Reducing the time interval
to one minute for faster response time.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-05-28 10:05:51 -06:00
Travis Nielsen 171910707b ceph: refactor pod deletion to k8sutil package
The pod deletion is needed by other daemons besides the osds,
so we move the helper method into the k8sutil package.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-05-28 10:05:50 -06:00
Sébastien Han 062e58e053 ceph: do not enforce osd-memory-target
As of Nautilus Ceph auto-tunes its osd_memory_target on the fly so we don't need to force it.
ROok does not support Mimic anymore so this can be removed.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-05-22 11:09:02 +02:00
Travis Nielsen 8bb4f603e6 ceph: default the device set template name to data
In 1.3 we added support for the metadata pvcs, thus requiring the name
to be set on the templates to differentiate between data and metadata.
For backward compatibility we need to assume the template is for data
if it is not set.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-05-21 11:33:16 -06:00
Travis Nielsen 81dae96929 ceph: remove obsolete version from osd health check
The Ceph version information is not needed to be passed
to the osd health checker since mimic support was dropped.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-05-18 12:56:28 -06:00
Travis Nielsen 52e15b6e36 ceph: only run the cluster monitoring goroutines once
The monitoring goroutines that check for mon and osd health,
the ceph cluster status, and the bucket provisioner should
only be started once for each cluster. Previous to this change
and since the controller runtime conversion the goroutines were
being started again every time a reconcile was run. This means
you could potentially have many goroutines monitoring the same
thing.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-05-18 12:55:58 -06:00
Travis Nielsen 65377cbe63 ceph: osd on pvc node assignment is node selector
An OSD on a PVC when portable=false is assigned to a node
with a node selector. The same node assignment is expected
for the lifetime of the cluster. On subsequent reconciles,
the operator was looking up the node assignment from the
nodeName on the pod spec, which is not set if the pod is down.
The operator needs to retrieve the assignment from the
deployment spec node selector.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-05-07 10:51:46 -06:00
Sébastien Han 228041902b Merge pull request #5406 from cybozu-go/ceph-cleanup-osd-on-pvc
ceph: judge whether an OSD is on PVC or not by function
2020-05-07 09:13:39 +02:00
Travis Nielsen 3c6bae56b7 ceph: delete terminating osd pods even if not out
When an OSD pod is stuck in terminating state, there is no need
to check if the OSD is marked as out before force deleting.
As long as the osd is down we will force delete the pod.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-05-06 15:43:04 -06:00
Satoru Takeuchi 417ea131bc ceph: judge whether an OSD is on PVC or not by function
There are many duplicated code to judge whether an OSD is on PVC
or not. Instead, let's introduce a function for this purpose.
It can also be applied to the logic of judging whether an OSD
on PVC has metadata device or not.

Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
2020-05-07 00:49:17 +09:00
Travis Nielsen c713fd30f9 ceph: restart portable osd pod if stuck
If the OSD pod is portable and is on a failed node, the node
will be stuck indefinitely in the terminating state. The pod
can be force deleted in this scenario in order to allow the
underlying volume to be detached and then mounted on another
node.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-04-30 09:52:54 -06:00
Sébastien Han f27fd207ce ceph: convert the CephCluster controller to the controller-runtime
This is the final conversion to controller-runtime conversion. This time the
CephCluster CRD has been converted to use the controller-runtime
library.
The controller incorporates all the previous watchers too, so the Node
and hot-plug configmap are been watched too.
Only the operator setting configmap is not being watcher since it's not
related to the CephCluster CRD.
Not only the patch converts to controller-runtime but also tries to
re-organize the tree of the repo to actually make the code more
readable and have better functions/methods/tests separations.

Closes: https://github.com/rook/rook/issues/4939
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-04-28 09:40:35 +02:00
Satoru Takeuchi f78efc50aa cleanup: use the constants of environment variables properly
There are some code where constants of environment variables
are defined but not used properly. Let's avoid to use string
literals as possible.

Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
2020-04-24 17:31:00 +00:00
Travis Nielsen 668bca27b1 ceph: reduce verbose logging for ok-to-stop failures
When a daemon is not safe-to-stop the operator will print the full
stack trace rather than a simple error message that the daemon
cannot be stopped.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-04-14 14:53:56 -06:00
morimoto-cybozu 9edf8c45b5 ceph: fix resync failure of OSD pod using LV-backed-PV
This fixes the resync of the Deployment of the OSD pod that uses
an LV-backed PV.
`getOSDInfo()` reads the configuration of the OSD pod from the existing
Deployment, but it overlooked this case.

Closes: https://github.com/rook/rook/issues/5139
Signed-off-by: morimoto-cybozu <kenji_morimoto@cybozu.co.jp>
2020-04-07 09:53:00 +00:00
Umanga Chapagain 654efb6d0a Ceph: fixes unhandled type assertions
unhandled type assertions can lead to runtime panic. using
the boolean 'ok' flag to handle failure to assert type.

Issue: #4566
Signed-off-by: Umanga Chapagain <chapagainumanga@gmail.com>
2020-04-04 14:00:19 +05:30
Sébastien Han 20d1543507 ceph: add multus support
You can now use Rook along with Multus. Multus must be up and running
and the right ressources must exist such as NetworkAttachmentDefinition
CR.
The Cluster CR spec has new fields to work with multus:

network:
  provider: multus (or 'host' for hostNetworking)
  selectors:
    public: NetworkAttachmentDefinition name
    cluster: NetworkAttachmentDefinition name

If only a single NetworkAttachmentDefinition is provided Rook will use
both anyway for the Ceph traffic.

Please refer to the doc to learn more.

Closes: https://github.com/rook/rook/issues/4716
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-04-03 18:00:15 +02:00
Sébastien Han 58f1971172 ceph: do not use host PID anymore
Thanks to https://github.com/ceph/ceph/pull/33633, when running on
Octopus, we don't need to use `--pid=host`. This means we are not using
the host PID but use the PID namespace of the pod.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-04-01 15:42:53 -06:00
Sébastien Han 040193bb5a ceph: remove DaemonType type
This type was a string already and was just making us doing string()
calls all the time to it's not worth it.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-04-01 09:08:18 +02:00
Sébastien Han 7788901a88 ceph: add liveness probe to mon, mds and osd daemons
Now Kubernetes will perform liveness checks on mon, mds and osd daemons.
The command will:

* call the socket (check for existence)
* execute a command and check the return code (success if 0)

This handles the case where the daemon is stuck locally and
unresponsive. It's unlikely but not impossible.
These checks bring more robustness to the implementation.

rbd-mirror and nfs have been leftover for the following reason. The
rbd-mirror socket name is different from other daemons (could be fixed
though): /run/ceph/ceph-client.rbd-mirror.a.1.94362516231272.asok also,
the command to call would need to be changed from "status" to "rbd
mirror status" so we can keep this for a later.
The nfs ganesha has no socket only a PID file which doesn't mean much.
No PID means the process does not run so Kubernetes will already handle
this and the pod will crash loop.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-03-31 15:20:42 +02:00
Santosh Pillai 2bfd7c42bc ceph: cleanup cluster.Spec.DataDirHostPath on cluster deletion
In order to ensure proper clean up of all the rook-ceph data when the cluster is deleted, we need to clean up the dataDirHostPath (var/lib/rook)

Signed-off-by: Santosh Pillai <sapillai@redhat.com>
2020-03-30 19:36:01 +05:30
Travis Nielsen 4a2ca1c1ef ceph: enable debug logging in the osd prepare job
All processes created with the exec package will now only have debug logging
for the commands and their output. The osd prepare job relies heavily
on many of these commands so we can enable debug logging for the entire
job.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-03-19 07:49:54 -06:00
Travis Nielsen e8f9cfcb71 exec: always write commands to debug log
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-03-19 07:49:54 -06:00
Travis Nielsen f2ecaa2bda exec: remove the unused actionName param
The helpers for executing a process have long required an actionName
param which is not being used. Now we remove the old param
while also cleaning up various other usages of the exec
package.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-03-19 07:49:53 -06:00
Sébastien Han f44583b2b9 ceph: remove dead code
As part of the transition to support Ceph release **as of** Nautilus, we
left over that portion of code.
We don't need it anymore.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-03-16 14:01:24 +01:00
Sébastien Han 723671f58c ceph: run osd initContainer in priv mode
To access a special block device the container must run privileged.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-03-11 19:33:14 +01:00
Sébastien Han 619f301eb7 ceph: osd re-add udev bindmount
The OSD needs to talk to the udev socket to get information about a device.
like serial, vendor etc. So that it can populate the output of:
`ceph osd metadata $n` correctly.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-03-11 18:35:47 +01:00
Travis Nielsen ba07a63033 ceph: properly lookup the K8s topology labels for OSDs
The OSDs pick up on several topology labels for CRUSH hierarchy.
The GA label topology.kubernetes.io was partially implemented, but
not picked up by the OSDs. Now the OSDs will pick up both the topology
labels from pre-1.17 such as failure-domain.beta.kubernetes.io/zone
and topology.kubernetes.io/zone.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-03-10 12:06:34 -06:00
Sébastien Han 025131c115 Merge pull request #4972 from travisn/osd-status-update
OSD prepare pod was not signaling completion when zero OSDs configured
2020-03-06 13:42:44 +01:00
Travis Nielsen 69c9ed4206 ceph: storage warn iare specified but ignored
In the cluster CR the nodes element will be ignored under storage when
useAllNodes is true. The nodes will only be applied when useAllNodes is
false. It is a common usability problem that users can't figure out
why their nodes are not configured. At least this log entry can give
them a clue.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-03-05 18:26:32 -07:00
Travis Nielsen 0eee2b4dee ceph: always mount /dev in the osd prepare pod
The osd prepare job is now always working with devices so
we always need to mount /dev and run privileged. There is
no scenario for not mounting /dev since filestore support
was removed.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-03-05 18:15:58 -07:00
Travis Nielsen a52948e7f7 ceph: continue orchestration even if osd status update fails
The communication between the operator and OSD prepare job
is done through updates to the configmap. If an update fails
to the configmap, we are anyway going to need to check the logs
for failure. So let's just log the error and make a best effort
to continue the orchestration. If the failure was temporary,
the next update to the status would succeed and likely continue
working as expected.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-03-05 14:08:58 -07:00
Sébastien Han a3068dee0b ceph: separate controller for CephBlockPool CRD
Now, the CephBlockPool CRD is managed with the controller-runtime.
So the watcher is outside of the main controller reconciliation loop of
CephCluster which brings numerous benefit such as:

* having its own reconciliation loop
* won't block anything from the main CephCluster controller loop
* fast than waiting for CephCluster loop to completion

Partially close: https://github.com/rook/rook/issues/1981
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-03-02 17:31:03 +01:00
Travis Nielsen bcb99d86ce crds: pick up the rook types in the v1 package
The rook types used across the storage providers moved from the v1alpha2
package to the v1 package. This commit points the packages at their new
location. Implementation is expected to remain unchanged.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-02-26 11:18:17 -07:00
Sébastien Han 5ca4880ed0 Merge pull request #4888 from leseb/pvc-hybrid-dev-class
ceph: set crush device class via annotations on PVC
2020-02-25 14:07:22 +01:00
Nizamudeen 53883f68cf ceph: Handling Unhandled errors
This commit is to handle all those unhandled errors which raises the gosec warning.

Fixed G104: Unhandled Errors are handled now

Signed-off-by: Nizamudeen <nia@redhat.com>
2020-02-21 22:48:24 +05:30
Sébastien Han a702f463cc ceph: set crush device class via annotations on PVC
When the OSD on PVC is backed by a metadata block PVC, the Ceph CRUSH
device class should be set to something else rather than the rotational
property of the drive.

Closes: https://github.com/rook/rook/issues/4881
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-02-20 18:53:31 +01:00
Sébastien HanandSean Micklethwaite b04e35dab5 ceph: add support for disk selection by full path
Rook-Ceph now supports specify device by their fullpath instead of links
created by the kernel like /dev/sdb.

Closes: https://github.com/rook/rook/issues/1228
Co-authored-by: Sean Micklethwaite <sean@wayve.ai>
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-02-13 17:31:38 +01:00
Sébastien Han 45118185cf ceph: remove mimic support, default to nautilus
As of 1.3, Rook will only support Ceph Nautilus 14.2.5.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-02-13 09:17:34 +01:00
Sébastien Han e1dc677714 ceph: implement osd block expansion on PVC
When running on PVC, if the backend resizes the block, Rook will expand
it and the size of the OSD will grow.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-02-11 18:33:50 +01:00
Sébastien Han c08c3ced05 ceph: add support for metadata PVC for OSD on PVC
We now support the addition of the PVC that acts as a metadata device
for a given OSD.
For this, you need to create a new `volumeClaimTemplates`, its name must
be "metadata" otherwise, Rook won't pick it up.

A template will look like this:

```
volumeClaimTemplates:
- metadata:
    name: data
  spec:
    resources:
      requests:
        storage: 10Gi
    # IMPORTANT: Change the storage class depending on your environment (e.g. local-storage, gp2)
    storageClassName: gp2
    volumeMode: Block
    accessModes:
      - ReadWriteOnce
- metadata:
    name: metadata
  spec:
    resources:
      requests:
        storage: 6Gi
    # IMPORTANT: Change the storage class depending on your environment (e.g. local-storage, gp2)
    storageClassName: gp2
    volumeMode: Block
    accessModes:
      - ReadWriteOnce
```

We now map block and block.db directly inside the container instead of
running ceph-volume activate. This is much cleaner.

Closes: https://github.com/rook/rook/issues/3852
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-02-11 17:30:02 +01:00
Sébastien Han 7390940acd ceph: osd add missing annotation
Setting annotation on pvc deployments too so that we don't
restart/update deployment is not necessary.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-02-11 17:25:51 +01:00
Travis Nielsen 12ff5ab54a Merge pull request #4830 from leseb/octopus-support-osd
ceph: allow osd full upgrade to octopus
2020-02-06 16:25:54 -07:00
Travis Nielsen c46aeffc4f Merge pull request #4584 from jmolmo/issue_4565
ceph: Remove ineffectual assignments
2020-02-06 11:26:34 -07:00
Sébastien Han 25d6d27a8a ceph: allow osd full upgrade to octopus
Once osds will run on octopus we will activate the necessary features
for these OSDs.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-02-06 18:40:04 +01:00
Sébastien Han 6a9f2aec82 Merge pull request #4777 from leseb/tune-slow-dev-fix
ceph: fix TuneSlowDeviceClass spec
2020-01-30 13:58:20 +01:00
Sébastien Han a60a18c477 ceph: fix TuneSlowDeviceClass spec
TuneSlowDeviceClass wasn't part of the StorageClassDeviceSet types but
only in VolumeSource which is not exposed in the CR spec so setting the
option would never work.
The only issue is that this apply to all volumeClaimTemplates
where each can point to a particular storage class...

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-01-30 11:59:18 +01:00
Nizamudeen 4b55a74ac4 ceph: Implementing Conditions on rook-ceph
Fixed conditions getting resetted after the operator restart.
Did the changes which required to implement conditions on the rook ceph cluster
Conditions will eliminate the current status.State and incorporates a type which
provides much more description to the current status of the cluster.

Signed-off-by: Nizamudeen <nia@redhat.com>
2020-01-30 10:21:11 +05:30
Sébastien Han caa8382e25 ceph: do not update deployment if no changes
We know use k8s-objectmatcher to verify whether a deployment is going to
change or not.
This avoids calling Update() from k8s utils which takes time to run.

Closes: https://github.com/rook/rook/issues/4519 and https://github.com/rook/rook/issues/4642
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-01-24 17:02:40 +01:00
Sébastien Han 92c1696bfc ceph: cleanup/trim isUpgrade variable a bit
We now always perform ok-to-stop change when the deployment is going to
be updated. This could be for various reasons like change in the spec or
a new Ceph image.
In any case, if Ceph resources are going to be restarted this must be
done carefully.

Also trimmed the isUpgrade variable, the upcoming commit will remove the
need of that variable.

Keeping isUpgrade from the controller code since we need it to detect
whether an orchestration upgrade should be triggered or not (based on
the ceph cluster status).

Also, we keep it on the child controllers so we don't go into an
orchestration phase if the cluster image hasn't changed. The child does
not need to be updated/go into its orchestration phase because the
cluster CR changed.

Closes: https://github.com/rook/rook/issues/4379
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-01-24 17:02:37 +01:00