Commit Graph
225 Commits
Author SHA1 Message Date
subhamkraiandTravis Nielsen 319e4a41a4 ceph: placement in case of both PVC and non-PVC's
In the case of PVC,
We are giving lower priority to all placement.
We want deviceSet placement to applied and
override in case of overlapping settings and
we are merging nodeAffinity if applied in both
all placement and deviceSet.

In case of non-PVC,
we apply spec.placement

Signed-off-by: subhamkrai <srai@redhat.com>
Co-authored-by: Travis Nielsen <tnielsen@redhat.com>
2021-03-18 10:32:57 +05:30
Satoru Takeuchi 26c8fd9bd1 ceph: improve owner reference management
It's better to validate ownerReferences when setting them. In addition, we should use
controllerrutil.Set{Controller,Owner}Reference, that have such validation, as possible.

Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
2021-03-16 10:29:33 +00:00
Travis Nielsen e7331b0191 ceph: multiple mgrs have antiaffinity across hosts or zones
If there are multiple mgrs running, the operator will automatically
add pod antiaffinity across hosts by default. For stretch clusters,
antiaffinity across the stretch failure domain (e.g. zones)
will be added. Required antiaffinity will be created unless the
test setting allowMultiplePerNode is set to true.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-03-10 08:37:17 -07:00
Travis Nielsen 142ba63b6a ceph: enforce anti-affinity to multiple mgrs
If there are two mgr daemons, they should be running on different
hosts or zones, depending on the failure domain.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-03-10 08:37:17 -07:00
Travis Nielsen 388ff3e78b ceph: start sidecar to monitor active mgr
The mgr daemon may be failed over by ceph if the active mgr is not
responding and the standby mgr is available. If the active mgr changes
the services for the dashboard and metrics will be updated with a
label selector for the new active mgr. The services cannot direct
traffic to the standby mgr or else they will be incorrectly redirected
to the active mgr.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-03-10 08:37:17 -07:00
rohan47 79ff3218bf ceph: removed code for force delete mon and osd pods when stuck
removed the code for force delete pods when stuck in terminating state
for mons and osds as there is common code to delete pods for stuck components

Signed-off-by: rohan47 <rohgupta@redhat.com>
2021-02-23 20:49:23 +05:30
Travis Nielsen 4f7250298c Merge pull request #7084 from travisn/maxmonid-commit
ceph: Only commit the maxmonid after starting a mon daemon
2021-01-28 11:30:10 -07:00
Sébastien Han d7610b7e6f ceph: use errors package instead of fmt
Small fix since all the ceph codebase is using the errors package.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-01-28 08:51:48 +01:00
Travis Nielsen 4fa3c04e44 ceph: only commit the maxmonid after starting a mon daemon
The maxMonId should only track the mons that have been committed to starting.
Currently the maxMonId was tracking all mon scheduling. If the reconcile
was aborted or failed, or the operator restarted in a reconcile, the
maxMonId was being reset by analyzing the mon deployments that existed.
This does not work for scenarios where mon quroum is lost and being
reset to a single mon. Now the maxMonId will only be incremented
when a mon daemon is started, and will never be decremented after
that to ensure that new mons will be incremented to the next id.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-01-27 16:33:36 -07:00
Travis Nielsen 5cf3338bcd ceph: prevent operator crash with host networking and mon not found
When the node is not found for mon assignment with host networking
enabled, the operator may crash. This change ensures the node is
not nil and returns an error instead of crashing.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-01-27 15:34:16 -07:00
Sébastien Han cb7d600841 ceph: set process working dir to /var/log/ceph
By setting the working directory of a Ceph daemon to the log direcor
(which is bindmounted to the host), we can ensure that the coredumps
will be available.

On CentOS 7 **only**, the kernel is configured with:

```
cat /proc/sys/kernel/core_pattern
core
```

which means that the coredumps will end up in the process working
directory and will be named "core".

On CentOS 8, everything is different and this won't work but still
CentOS will be able to consume this and changing the working directory
is harmless.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-01-15 16:06:42 +01:00
Travis Nielsen d0ba899341 ceph: check stretch cluster is ready to configure arbiter
The arbiter can only be configured with the stretch cluster if the
CRUSH map is balanced and there are two zones in the CRUSH map.
After the OSDs are configured, we wait for all the OSD pods to be
running and that the CRUSH map is balanced. If it takes more than
two minutes, we fail the reconcile and try again. This is only done
the first time the stretch cluster is configured. In future reconciles
we first check if the stretch cluster is already enabled before
enabling it again.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-12-09 16:37:24 -07:00
Travis Nielsen 6fff48fbf5 ceph: remove obsolete mon pod and retry variables
The mon timeouts and pod retries were removed back in v1.1,
this just removes some obsolete variables from the legacy
mon scheduling.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-12-09 11:26:18 -07:00
Travis Nielsen bfbae0e4fe ceph: reset maxMonId if mons were not started after canaries
The operator schedules all the mons at the same time, then
serially starts each of the mons. If the operator was restarted
or otherwise failed before all the mons were started, the next
reconcile will schedule new mons again and ignore the scheduling
that was already computed. Since the last mon running is always
expected to be in quorum, we can reset the maxMonId to be
the highest mon currently created.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-12-09 11:00:44 -07:00
Travis Nielsen eebb367e42 ceph: allow arbiter to have separate placement from other mons
The arbiter may have unique requirements for placement settings. Thus, if the arbiter
placement is specified in the cluster CR, that placement will be applied to the arbiter
mon and the other mons will retain their placement from the other mon or all
sections of the placement.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-12-07 10:58:20 -07:00
Travis Nielsen 6a00b4d296 ceph: remove obsolete NodeUsage type
The node usage was used for the obsolete mon selection algorithm, which
was replaced long ago with placement via canary pods.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-12-04 16:02:25 -07:00
Sébastien Han c6a87203ca ceph: add log collector
We can now collect logs directly into a side-car container.
A new CRD spec has been added:

spec:
  logCollector:
    enabled: true
    periodicity: 24h

Every 24h we will rotate log files for each Ceph daemon.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-12-01 16:30:17 +01:00
Sébastien Han ad24990473 ceph: ability to abort orchestration
We can now prioritize orchestrations on certain event. Today only two
events will cancel on-going orchestrations (if any):

* request for cluster deletion
* request for cluster upgrade

If one of the two are caught by the watcher we will cancel the on-going
orchestration.
For that we implemented a simple approach based on check points, where
we will check for a cancellation request in certain part of the
orchestration. Mainly before each mons/mgr/osds orchestration loops.

This solution is not perfect, but we are waiting for the
controller-runtime to release its 0.7 version which will embed context
support. With that we will be able to cancel reconciles more precisely
and rapidly.

Operator log example:

```
2020-11-24 13:54:59.499719 I | op-mon: parsing mon endpoints: a=10.109.126.120:6789
2020-11-24 13:54:59.499719 I | op-mon: parsing mon endpoints: a=10.109.126.120:6789
2020-11-25 12:59:12.986264 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph"
2020-11-25 13:07:33.776947 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff=  v1.ClusterSpec{
        CephVersion: v1.CephVersionSpec{
                Image:            "ceph/ceph:v15.2.5",
-               AllowUnsupported: true,
+               AllowUnsupported: false,
        },
        DriveGroups: nil,
        Storage:     {UseAllNodes: true, Selection: {UseAllDevices: &true}},
        ... // 20 identical fields
  }
2020-11-25 13:07:33.777039 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph"
2020-11-25 13:07:33.785088 I | op-mon: parsing mon endpoints: a=10.107.242.49:6789,b=10.109.71.30:6789,c=10.98.93.224:6789
2020-11-25 13:07:33.788626 I | ceph-cluster-controller: detecting the ceph image version for image ceph/ceph:v15.2.5...
2020-11-25 13:07:35.280789 I | ceph-cluster-controller: detected ceph image version: "15.2.5-0 octopus"
2020-11-25 13:07:35.280806 I | ceph-cluster-controller: validating ceph version from provided image
2020-11-25 13:07:35.285888 I | op-mon: parsing mon endpoints: a=10.107.242.49:6789,b=10.109.71.30:6789,c=10.98.93.224:6789
2020-11-25 13:07:35.287828 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config
2020-11-25 13:07:35.288082 I | cephclient: generated admin config in /var/lib/rook/rook-ceph
2020-11-25 13:07:35.621625 I | ceph-cluster-controller: cluster "rook-ceph": version "15.2.5-0 octopus" detected for image "ceph/ceph:v15.2.5"
2020-11-25 13:07:35.642688 I | op-mon: start running mons
2020-11-25 13:07:35.646323 I | op-mon: parsing mon endpoints: a=10.107.242.49:6789,b=10.109.71.30:6789,c=10.98.93.224:6789
2020-11-25 13:07:35.654070 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.107.242.49:6789","10.109.71.30:6789","10.98.93.224:6789"]}] data:a=10.107.242.49:6789,b=10.109.71.30:6789,c=10.98.93.224:6789 mapping:{"node":{"a":{"Name":"minikube","Hostname":"minikube","Address":"192.168.39.3"},"b":{"Name":"minikube","Hostname":"minikube","Address":"192.168.39.3"},"c":{"Name":"minikube","Hostname":"minikube","Address":"192.168.39.3"}}} maxMonId:2]
2020-11-25 13:07:35.868253 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config
2020-11-25 13:07:35.868573 I | cephclient: generated admin config in /var/lib/rook/rook-ceph
2020-11-25 13:07:37.074353 I | op-mon: targeting the mon count 3
2020-11-25 13:07:38.153435 I | op-mon: checking for basic quorum with existing mons
2020-11-25 13:07:38.178029 I | op-mon: mon "a" endpoint is [v2:10.107.242.49:3300,v1:10.107.242.49:6789]
2020-11-25 13:07:38.670191 I | op-mon: mon "b" endpoint is [v2:10.109.71.30:3300,v1:10.109.71.30:6789]
2020-11-25 13:07:39.477820 I | op-mon: mon "c" endpoint is [v2:10.98.93.224:3300,v1:10.98.93.224:6789]
2020-11-25 13:07:39.874094 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.107.242.49:6789","10.109.71.30:6789","10.98.93.224:6789"]}] data:a=10.107.242.49:6789,b=10.109.71.30:6789,c=10.98.93.224:6789 mapping:{"node":{"a":{"Name":"minikube","Hostname":"minikube","Address":"192.168.39.3"},"b":{"Name":"minikube","Hostname":"minikube","Address":"192.168.39.3"},"c":{"Name":"minikube","Hostname":"minikube","Address":"192.168.39.3"}}} maxMonId:2]
2020-11-25 13:07:40.467999 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config
2020-11-25 13:07:40.469733 I | cephclient: generated admin config in /var/lib/rook/rook-ceph
2020-11-25 13:07:41.071710 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config
2020-11-25 13:07:41.078903 I | cephclient: generated admin config in /var/lib/rook/rook-ceph
2020-11-25 13:07:41.125233 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed
2020-11-25 13:07:41.327778 I | op-k8sutil: updating deployment "rook-ceph-mon-a" after verifying it is safe to stop
2020-11-25 13:07:41.327895 I | op-mon: checking if we can stop the deployment rook-ceph-mon-a
2020-11-25 13:07:44.045644 I | op-k8sutil: finished waiting for updated deployment "rook-ceph-mon-a"
2020-11-25 13:07:44.045706 I | op-mon: checking if we can continue the deployment rook-ceph-mon-a
2020-11-25 13:07:44.045740 I | op-mon: waiting for mon quorum with [a b c]
2020-11-25 13:07:44.109159 I | op-mon: mons running: [a b c]
2020-11-25 13:07:44.474596 I | op-mon: Monitors in quorum: [a b c]
2020-11-25 13:07:44.478565 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed
2020-11-25 13:07:44.493374 I | op-k8sutil: updating deployment "rook-ceph-mon-b" after verifying it is safe to stop
2020-11-25 13:07:44.493403 I | op-mon: checking if we can stop the deployment rook-ceph-mon-b
2020-11-25 13:07:47.135524 I | op-k8sutil: finished waiting for updated deployment "rook-ceph-mon-b"
2020-11-25 13:07:47.135542 I | op-mon: checking if we can continue the deployment rook-ceph-mon-b
2020-11-25 13:07:47.135551 I | op-mon: waiting for mon quorum with [a b c]
2020-11-25 13:07:47.148820 I | op-mon: mons running: [a b c]
2020-11-25 13:07:47.445946 I | op-mon: Monitors in quorum: [a b c]
2020-11-25 13:07:47.448991 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed
2020-11-25 13:07:47.462041 I | op-k8sutil: updating deployment "rook-ceph-mon-c" after verifying it is safe to stop
2020-11-25 13:07:47.462060 I | op-mon: checking if we can stop the deployment rook-ceph-mon-c
2020-11-25 13:07:48.853118 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff=  v1.ClusterSpec{
        CephVersion: v1.CephVersionSpec{
-               Image:            "ceph/ceph:v15.2.5",
+               Image:            "ceph/ceph:v15.2.6",
                AllowUnsupported: false,
        },
        DriveGroups: nil,
        Storage:     {UseAllNodes: true, Selection: {UseAllDevices: &true}},
        ... // 20 identical fields
  }
2020-11-25 13:07:48.853140 I | ceph-cluster-controller: upgrade requested, cancelling any ongoing orchestration
2020-11-25 13:07:50.119584 I | op-k8sutil: finished waiting for updated deployment "rook-ceph-mon-c"
2020-11-25 13:07:50.119606 I | op-mon: checking if we can continue the deployment rook-ceph-mon-c
2020-11-25 13:07:50.119619 I | op-mon: waiting for mon quorum with [a b c]
2020-11-25 13:07:50.130860 I | op-mon: mons running: [a b c]
2020-11-25 13:07:50.431341 I | op-mon: Monitors in quorum: [a b c]
2020-11-25 13:07:50.431361 I | op-mon: mons created: 3
2020-11-25 13:07:50.734156 I | op-mon: waiting for mon quorum with [a b c]
2020-11-25 13:07:50.745763 I | op-mon: mons running: [a b c]
2020-11-25 13:07:51.045108 I | op-mon: Monitors in quorum: [a b c]
2020-11-25 13:07:51.054497 E | ceph-cluster-controller: failed to reconcile. failed to reconcile cluster "rook-ceph": failed to configure local ceph cluster: failed to create cluster: CANCELLING CURRENT ORCHESTATION
2020-11-25 13:07:52.055208 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph"
2020-11-25 13:07:52.070690 I | op-mon: parsing mon endpoints: a=10.107.242.49:6789,b=10.109.71.30:6789,c=10.98.93.224:6789
2020-11-25 13:07:52.088979 I | ceph-cluster-controller: detecting the ceph image version for image ceph/ceph:v15.2.6...
2020-11-25 13:07:53.904811 I | ceph-cluster-controller: detected ceph image version: "15.2.6-0 octopus"
2020-11-25 13:07:53.904862 I | ceph-cluster-controller: validating ceph version from provided image
```

Closes: https://github.com/rook/rook/issues/6587
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-11-25 15:32:04 +01:00
Arun Kumar Mohan 65d16bfc94 ceph: manual changes needed for kubernetes api updates
Fetched latest lib-bucket-provisioner changes as well.

Signed-off-by: Arun Kumar Mohan <amohan@redhat.com>
2020-11-18 21:14:01 +05:30
Travis Nielsen 66de535a8a ceph: restore mon clusterIP if missing
In a disaster recovery scenario, the mon service may have been
accidentally deleted, while the expected mon endpoint is still
found in the mon endpoints configmap. In this case, we create
the mon service with the same endpoint as previously.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-11-17 22:54:19 -07:00
Sébastien Han e3d032dbc1 Merge pull request #6489 from travisn/stretch-mons
ceph: Stretch cluster configuration
2020-11-06 15:16:39 +01:00
Mateusz Gozdek 8ba3762fa4 docs: fix bunch of typos
Found by running the following command:

codespell -S .git,*.png,*.jpg -L \
aks,keyserver,atleast,dne,ser,ist,files\',ba,dum,iam,te -f -H

Signed-off-by: Mateusz Gozdek <mgozdekof@gmail.com>
2020-11-06 10:01:04 +01:00
Travis Nielsen 7c3cdbce50 ceph: use constants for k8s topology labels
Instead of directly defining our own constants, we should be using the k8s
constants for the well-known topology labels.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-11-05 09:23:58 -07:00
Travis Nielsen e72b69f8e0 ceph: validate pools for stretch clusters
Pools in stretch clusters will all use the same crush rule that
is generated by the operator when configuring the cluster for
stretch mode, with replica 4 and two replicas per failure domain.
Pools cannot create new crush rules, so we require that replica be
4 when the pools is created, EC is not allowed, and no new
crush rule will be created.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-11-05 09:23:57 -07:00
Travis Nielsen b91f4211c9 ceph: configure a stretched cluster
In clusters where only two datacenters (or similar failure domains)
are available, a different mon and osd approach is needed to deal
with the network partitions or some other reason for one of the failure
domains going down. The Ceph stretched cluster makes the mons aware
of the failure domains by configuring one as the arbiter in a third
zone, while keeping two replicas of the data in each of the data
zoens.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-11-05 09:17:52 -07:00
Travis Nielsen 772865fa01 ceph: create and remove mon canaries in parallel
The mon canaries are now being added and removed at the same time
instead of starting or stopping them one-by-one. This will simplify
scheduling execution time to much more quickly detect where the mons
can be scheduled.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-11-05 09:17:50 -07:00
Yusei Nishiyama beaddb9c22 ceph: propagte labels to mon pods
To align it with how the other components are implemented in #6084

Signed-off-by: Yusei Nishiyama <yusei.nishiyama@soundcloud.com>
2020-10-30 13:43:18 +01:00
Travis Nielsen efb1cc9f72 ceph: check for orphaned mon resources with every reconcile
The mon resources may be orphaned if there is a mon failover
that was cancelled, for example a mon pvc may remain if the
original mon comes up and the operator was restarted before
the mon failover completed. The cleaning up of mon resources
was only happening was each successful mon failover, now we
perform the check with every reconcile since the resources
were not being cleaned up predictably enough.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-10-26 14:06:17 -06:00
subhamkrai cb0ca66a6b ci: enable gosec linter in golangci-lint
golangci-lint linter gosec showing more errors
than gosec gh. This commit resolve
new errors. And, removing
nosec comments from autogenerated files.

Signed-off-by: subhamkrai <srai@redhat.com>
2020-10-19 11:50:06 +05:30
Travis Nielsen 3eddcf5477 Merge pull request #6283 from sp98/rook-IPv6-support
ceph: support IPv6 single-stack
2020-10-06 07:46:25 -06:00
Santosh Pillai 70c5f70701 ceph: support IPv6 single-stack for ceph
Add --ms-bind-ipv6 true args to mom, osd, mgr, object, mds, rbd daemons.

Signed-off-by: Santosh Pillai <sapillai@redhat.com>
2020-10-02 01:03:03 +05:30
subhamkrai 0fddfcf307 ceph: handle golangci-lint linter errcheck error
this commit handle golangci-lint linter errcheck.

`errcheck` - Errcheck is a program for checking for
unchecked errors in go programs. These unchecked errors
can be critical bugs in some cases

To see only staticcheck linter output
`golangci-lint run --disable-all -E errcheck`

Signed-off-by: subhamkrai <srai@redhat.com>
2020-09-30 22:24:34 +05:30
subhamkrai de8dbbcdcc ceph: handle golangci-lint linter staticcheck error
this commit handle golangci-lint linter staticcheck error.

`staticcheck` - Staticcheck is a go vet on steroids,
applying a ton of static analysis checks.

To see only `staticcheck` linter output

`golangci-lint run --disable-all -E staticcheck`

Signed-off-by: subhamkrai <srai@redhat.com>
2020-09-24 15:04:55 +05:30
subhamkrai 829778f251 ceph: handle golangci-lint linter ineffassign
this commit will enable one more linter ineffassign
in golangci-lint.

This linter throws an error when variable is assigned and never used.

`golangci-lint run --disable-all -E ineffassign` is used detects ineffassign
errors only.

Signed-off-by: subhamkrai <subhamkumarrai03@gmail.com>
2020-09-21 14:31:18 +05:30
subhamkrai 1e9bb8e6e2 ceph: handle golangci-lint linter deadcode
this commit will enable one more linter `deadcode`
in golangci-lint .

Signed-off-by: subhamkrai <subhamkumarrai03@gmail.com>
2020-09-18 16:53:49 +05:30
subhamkrai f9fafe62d4 ceph: handle golangci-lint linter unused
this commit will enable one more linter
in golangci-lint.

Signed-off-by: subhamkrai <subhamkumarrai03@gmail.com>
2020-09-17 22:53:24 +05:30
subhamkrai 25c116a4bd ceph: handle golangci-lint linter gosimple
this commit handles all the errors  check for
golangci-lint linter gosimple.

Signed-off-by: subhamkrai <subhamkumarrai03@gmail.com>
2020-09-17 15:10:27 +05:30
Travis Nielsen da863e4156 ceph: do not change selector labels during upgrade
The selector labels used for mons cannot change during upgrade
or else the upgrade will hang when the cluster is checking
for mon status immediately after the upgrade begins.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-09-14 21:43:25 -06:00
Travis Nielsen 7a6620f2c0 ceph: remove orphaned mon pvc after failed failover
If the mon failover fails, if the mon is on a pvc, the pvc could remain
orphaned. This is likely to happen if there is not a node available
for scheduling the mon. Therefore, the operator will ensure after a failover
that any orphaned mon pvc is cleaned up.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-09-10 10:21:33 -06:00
Travis Nielsen 2e36913dad ceph: set the pvc name label on mon deployments
When the mons are based on pvcs, the pvc_name label was not being set
as expected. Now the label will always be set on both the canary and
the daemon pods.

The mon pvcs were also never being deleted during canary cleanup. This
code is now removed since we must be very careful when the pvcs are
removed.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-09-09 16:08:07 -06:00
Alexander Trost a87c00ddee ceph: allow user to add labels to pods
This allows users to specify additional labels to be added to the components
created by the Rook Ceph operator.
Additionally the rbd mirrors are now getting their annotations and
labels set.

Signed-off-by: Alexander Trost <galexrt@googlemail.com>
2020-08-25 19:38:51 +02:00
Travis Nielsen ce92249725 ceph: continue with memory limits below min settings
The operator will now allow the resource limits to be applied below the
recommended minimums. In small clusters, even the recommended minimums
may not be necessary. A warning is still printed to the operator log,
but we allow the configuration to continue.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-08-19 11:07:16 -06:00
Travis Nielsen d7a83107b6 ceph: mon failover should scale down failed mon first
When a mon is failed over, the new mon is started before the old mon
is removed from quorum and purged from the cluster. If the mon has
anti-affinity, in small clusters or specific toplogies where sufficient
nodes are not available, the new mon may be prevented from starting simply
because the dead mon pod still exists. To ensure a clean failover, the
dead mon pod must first be stopped before the new mon is started.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-08-11 10:30:28 -06:00
Sébastien Han f66473a8f8 ceph: fix healtcheck for external cluster
For external cluster, it is not an issue to have a mon count set to 0
since Rook is not deploying/managing the monitors.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-08-10 11:46:35 +02:00
Travis Nielsen 15fae3c4cd ceph: during upgrade service selector cannot be changed
The service selector labels cannot be updated during the upgrade or else the
mon commands will not respond until the mon deployment is also updated.
Since there is a mon command to check for the ceph versions before the
deployment is updated, the operator would remain stuck during the upgrade.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-08-05 11:32:42 -06:00
Travis Nielsen 06c21954e1 ceph: labels are immutable for match selectors
The labels cannot be updated on a deployment match selector.
In v1.4 a new label was added for ceph_daemon_type that is intended
to be on the pod labels, but cannot be applied to the match selectors.
Therefore, we suppress any new labels that are added to the daemons.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-08-04 16:47:00 -06:00
Blaine Gardner dcf8ae2526 ceph: rename pod labels function for more clarity
Rename PodLabels function to CephDaemonAppLabels for more clarity about
what the function's purpose is.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2020-07-30 11:16:30 -06:00
subhamkrai acc4ed5df9 ceph: handling gosec errors that are not checked
this commit handles all the gosec g104
i.e audit errors not checked inside pkg.

Signed-off-by: subhamkrai <subhamkumarrai03@gmail.com>
2020-07-24 20:02:06 +05:30
Travis Nielsen 234877e7c0 ceph: skip first mon health check at startup
At startup there is a race condition between the goroutine checking
for mon health and the first reconcile of the mons. There is a lock
that ensures only one of them will check the mon health at a time.
The issue is that if the wrong one gets there first, it will not
have the desired mon count initialized from the cluster CR as expected
and the healther checker will use a desired mon count of 0.
The desired count of 0 leads the health checker to remove
a mon since there are extra mons. Only one mon is allowed
to be removed at a time, and adjusting to fewer than two mons is
disallowed once there are at least two mons.

After the health checker is completed, the main reconcile of the mons
will proceed with the correct desired mon count of 3 and attempt
to restore to the real desired quorum size.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-07-22 08:23:09 -06:00
Travis Nielsen c1736844c9 ceph: require the mon spec for initialization
The mon was starting with a partial spec. There was a race condition
that was causing incorrect initialization in the health check goroutine
that could remove a mon upon operator restart. Now the mon will be
properly initialized with the full spec.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-07-22 08:23:09 -06:00