Commit Graph
671 Commits
Author SHA1 Message Date
Ali Maredia 5d9c2f09a9 ceph: enable the realm pull on ceph-object-realm
This commit:
- adds the pull section on the CephObjectRealm spec
to enables realms to be pulled instead of created.

- creates the system user and generates the access
key and secret key for the user and zones in a realm.

- adds "omitempty" to fields in multisite CRs that
are not required

Signed-off-by: Ali Maredia <amaredia@redhat.com>
2020-07-21 13:30:19 -04:00
Sébastien Han 0e684460bd Merge pull request #5642 from leseb/monitoring-endpoint
ceph: add external prometheus endpoint
2020-07-17 16:20:51 +02:00
Sébastien Han 34ccb84aa3 ceph: add external prometheus endpoint
We can now connect an external prometheus exporter to Rook to collect
metrics and generate alerts from Prometheus.
Just enable this in the CephCluster CR:

```
spec:
  external:
    enable: true
   monitoring:
    enabled: true
    rulesNamespace: rook-ceph
    externalMgrEndpoints:
    - ip: 192.168.39.182
```

Closes: https://github.com/rook/rook/issues/5516
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-07-17 11:29:48 +02:00
Sébastien Han 90787daef7 ceph: do not enable the 'rook' module by default
We currently don't use the module, so let's speed up the bootstrap
progress as well as the reconcile by not always enabling it.
It can always be enabled via the spec mgr section.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-07-17 10:37:28 +02:00
Sébastien Han 1bdb369cb4 ceph: do not enable orchestrator_cli
orchestrator_cli is always-on so we don't need to do anything.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-07-17 10:36:59 +02:00
Travis Nielsen 76028f66ab ceph: load external creds from deprecated secret
The external creds have been loaded from their own key. For
backward compatibility we need to check that key if the
new key is not found.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-07-16 15:54:54 -06:00
Travis Nielsen e74c7eaef8 ceph: refactor context and clusterInfo passed to the ceph commands
To provide more context for executing commands in a ceph cluster,
the full clusterInfo is now passed to the ceph execution commands.
All information about the cluster will now be available throughout
all the areas of the operator. The namespace, ceph credentials,
mon endpoints, and other info is a core part of that cluster info.

Arguments passed through the controllers are also simplified for
mons, mgr, osds, and other daemons where the parameters had
become too complex.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-07-16 15:54:54 -06:00
Travis Nielsen 4681f9e73d ceph: consolidate ceph config and client packages
The ceph config and client packages are conceptually the same.
To avoid circular dependencies in some cases, we simplify by combining
the packages into the client package.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-07-16 15:54:43 -06:00
Travis Nielsen 631b13b906 ceph: refactor creds used by operator
The operator should only connect to ceph with a single set of creds.
In a converged cluster this will be the admin creds and in an external
cluster it will be lower-privileged creds. Independent clusters were
implemented with a separate set of creds. To simplify the code these
are now merged to a single set.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-07-16 15:54:42 -06:00
Hiroshi Muraoka e2b1a2230d ceph: ignore PodAntiAffinity for detect-version placement
When a PodAntiAffinity is set to mon, detect-version may not be scheduled because detect-version has the same PodAntiAffinity with mon.
This commit fixes this issue.

Closes #5810

Signed-off-by: Hiroshi Muraoka <h.muraoka714@gmail.com>
2020-07-16 04:27:06 +00:00
Blaine Gardner 7117fc12b7 ceph: osd: add drive groups spec to cluster CR
Add the ability to provision Ceph OSDs with Drive Groups.
This adds Drive Groups to the CephCluster CRD, and it sets code
in place for propagating this config to the OSD provisioning pod.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2020-07-14 09:43:03 -06:00
Sébastien Han ccb52b84e6 ceph: configurable status checks and livenessprobe
This commit allows us to configure status check for each daemon:

* "mon": health check on the ceph monitors (quorum)
* "osd": health check on the ceph osds
* "status": ceph health status check

Each check is controlled by the following settings:

* disabled: whether to disable the check (default: false)
* internal: interval to run the check
* timeout: only valid for mons, is the timeout for unresponsive mon
before failling over.

Example to disable the status health check:

```yaml
healthCheck:
  daemonHealth:
    status:
      disabled: true
```

As part of that, pod's livenessprobe can now be configured via the
following settings:

* disabled: whether to enable or not
* probe: override the current probe in place by a new one

```yaml
healthCheck:
  livenessProbe:
    mon:
      disabled: true
```

Closes: https://github.com/rook/rook/issues/5772
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-07-10 19:21:59 +02:00
Travis Nielsen 8f9df75f7b ceph: capture logging from the osd daemon
The OSD daemon has been missing critical flags for logging to stderr
where k8s can capture the logs. Without the --log-to-stderr=true,
all the OSD logging was essentially lost until now.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-07-06 14:38:37 -06:00
Travis Nielsen dbae36b2bb ceph: update to the latest octopus release
Update the base operator image and cluster examples to use the latest
octopus release v15.2.4. Also cleanup some old examples and unit tests
that were still based on mimic.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-07-06 12:11:47 -06:00
Travis Nielsen 99f17d3a33 ceph: force delete stuck mons in every reconcile
The stuck mons were only being force deleted in the mon health checker.
After an operator restart, the health check may not be executed
for 20 minutes until after the timeout during the main reconcile
which is holding onto a lock for mon reconcile. Now the stuck mon
will be deleted during the reconcile as well.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-06-29 10:38:16 -06:00
Travis Nielsen 1d370b1314 ceph: during mon failover remove the backing pvc
If the mon is backed by a pvc, ensure the pvc is deleted when
the mon is failed over. The pvc is being left in a bound state
even though it will no longer be used in the cluster.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-06-24 15:37:42 -06:00
Travis Nielsen 0be6fdfa2d ceph: save mon config even if failed to remove dead mon
During mon failover if the mon fails to be removed, the new mon
config was not being saved. The mon removal anyway is not retried
if it fails, so it should make a best effort to remove all the
resources related to the mon, and then always update the mon
config after the failover.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-06-24 15:37:42 -06:00
binoue 57b944abfb ceph: fix an argument name
This commit fixes an argument of the CreateCrashCollectorSecret function as
it is expected to be the namespace of the cluster.

Signed-off-by: binoue <banji-inoue@cybozu.co.jp>
2020-06-21 20:27:34 +09:00
Sébastien Han aca541803b Merge pull request #5664 from leseb/ceph-object-user-external
ceph: add external support for object store user
2020-06-18 20:10:05 +02:00
Sébastien Han 84d1e28c99 ceph: add external support for objectstoreuser
Now, the object store user is capable of creating s3 users on an
external Ceph cluster.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-06-18 16:34:01 +02:00
Travis Nielsen 0e4c5e1d58 ceph: update maxMonId if incorrect
When the operator loads the mons and checks for a valid maxMonId
the check of the id against the currently running mons was outdated
from legacy mon ids. The check now has to parse the new id which is
just the letter instead of having a number from legacy mon ids.
Without this fix, the maxMonId would stay incorrect if it was somehow
corrupted.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-06-17 17:44:58 -06:00
Sébastien Han c7f255a0e8 ceph: add the ability to set any pool property
We can now explicitly set any property on a given pool by using the new
Property field in the CephBlockPool Spec.

Also, this fixes the case where both `CephBlockPool` and `CephCluster`
are created at the same time. When Rook creates the pool, the cluster is
still being bootstrapped and the global option
`osd_pool_default_pg_autoscale_mode` has not bee set yet. So the pool
gets created but its `pg_autoscale_mode` property is set to `warn`
instead of `on`.

Closes: https://github.com/rook/rook/issues/5608V
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-06-17 10:08:00 +02:00
UMEZAWA Takeshi 6085ff3223 ceph: add schedulerName property to storageClassDeviceSet
Support schdulerName to use another scheduler for OSD pod placement.

Closes: https://github.com/rook/rook/issues/5536
Signed-off-by: UMEZAWA Takeshi <takeshi-umezawa@cybozu.co.jp>
Signed-off-by: Masayuki Ishii <masa213f@gmail.com>
2020-06-12 11:53:28 +09:00
Travis Nielsen 5a02de5621 ceph: metadata pvcs use a map instead of slice index
The volume sources for an OSD on PVC were relying on the ordering
of the volume sources slice. While this was working, it was causing
headaches and potential future developer error when being
interpreted incorrectly. The addition of the wal pvc device
would make the issue even more pronounced. Now each PVC used
for the OSD will be identified in a map by name such as data
 and metadata.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-06-11 08:58:53 -06:00
Travis Nielsen 955dcb930e Merge pull request #5377 from alimaredia/wip-object-store-multisite
Foundation for Object Store Multisite CRs
2020-06-10 16:07:53 -06:00
Ali Maredia fc579f4520 ceph: initial commit for ceph rgw multisite resources
This commit contains CR implementations for:
CephObjectRealm
CephObjectZoneGroup
CephObjectZone

Also there are changes made to the objectstore
to add rgws in the object-store to zones and
zone groups in a multisite configuration and
the removal of the --default parameter for any
realms/zonegroups/zones that are created.

Signed-off-by: Ali Maredia <amaredia@redhat.com>
2020-06-09 16:28:53 -04:00
Sébastien Han 993af443a4 ceph: always turn on balancer on octopus
When running Octopus, we still need to turn on the balancer module. The
distinction is tricky, the module is "enabled" but it has to be "turned
on" and configured.

This was only working when the `pg_autoscaler` module was explicitly
listed in the CephCluster spec mgr section module section but not
automatically configured when running Octopus (the `pg_autoscaler`
module is already enabled).

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-06-09 15:42:15 +02:00
Sébastien Han 2749035315 ceph: do not activate crash module
The mgr crash module is always enabled so we don't need to run the
command again. It saves some orchestration time.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-06-09 15:37:04 +02:00
Travis Nielsen 282a2a3741 ceph: increase osd size when pvc template increases
When the PVC template increases in size for the storageClassDeviceSet
in the cluster CR, we need to increase the size of the PVC and restart
the OSD. The OSD is restarted by changing a variable on the pod spec
that indicates the desired size of the OSD.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-06-08 14:34:43 -06:00
Travis Nielsen daeb9b56b0 ceph: remove dead osd code
The completeProvisionSkipOSDStart function is no longer used.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-06-08 14:34:43 -06:00
Sébastien Han 887836586a Merge pull request #5595 from afflom/port-naming-work
Configurable Service Port Names
2020-06-08 12:57:01 +02:00
Alex Flom 9a4d535f09 ceph: hardcode service port and container port names
When integrating rook with istio, service port names need to be prefixed by their protocol.
Trying to fix this by removing static port names in favor of configurable names.

Closes: #5587

add protocol prefix to service port names, hardcoded as discussed. renamed container ports

Signed-off-by: Alex Flom <alexander.flom@gmail.com>
2020-06-06 21:58:23 -04:00
Sébastien Han bfa3b4daf4 ceph: do not use external node IP for mon endpoint
When the cluster is using HostNetworking, Rook was either picking up the
external or internal IP of the node. This resulted in mon endpoints have
public IP addresses. Those IP are not reachable from within the cluster
so OSD/CSI couldn't access the monitors from the configmap endpoint.

Also, exposing the cluster on a public network does not seem realistic,
so sticky with private/internal IP addresses is better.

Closes: https://github.com/rook/rook/issues/5495
Signed-off-by: Sébastien Han <seb@redhat.com>
2020-06-04 14:12:45 +02:00
Travis Nielsen 3ab060965c ceph: force restart mon pods if on bad node
If a mon is stuck in terminating state on a failed node,
force delete the pod so it can attempt to restart. If
the mon is backed by a PV the mon can start on another
node as well.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-05-28 12:11:05 -06:00
Travis Nielsen ff6a3f80cd ceph: improve osd update logging
The operator should only print helpful info messages
when an OSD is going to be updated. If the OSD hasn't
changed it is just a debug message.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-05-28 10:05:51 -06:00
Travis Nielsen 07f095c772 ceph: set osd health check interval to 60s
The osd health check every 5 min is not frequent enough to respond
quickly enough to osds going down. Reducing the time interval
to one minute for faster response time.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-05-28 10:05:51 -06:00
Travis Nielsen 171910707b ceph: refactor pod deletion to k8sutil package
The pod deletion is needed by other daemons besides the osds,
so we move the helper method into the k8sutil package.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-05-28 10:05:50 -06:00
Sébastien Han c799e11032 ceph: enhancement cleanup with disks
Now, we can not only cleanup monitor data, logs and crashes but the
disks too. As part of the cleanupPolicy CR spec, we have a new setting
called sanitizeDisks which holds more details:

* confirmation: the confirmation message to sanitize disks,
use "yes-really-sanitize-disks" to confirm

This will **only** wipe the metadata, so it's a fast cleanup allowing
you to re-install later but won't remove all the data from the drive.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-05-27 18:14:02 +02:00
Sébastien Han 0ed8e73942 ceph: add missing security context to crash collector
The crash collector was missing the security context to properly run on
OCP, thus crashing when trying to post a crash dump.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-05-26 16:35:11 +02:00
Sébastien Han 062e58e053 ceph: do not enforce osd-memory-target
As of Nautilus Ceph auto-tunes its osd_memory_target on the fly so we don't need to force it.
ROok does not support Mimic anymore so this can be removed.

Signed-off-by: Sébastien Han <seb@redhat.com>
2020-05-22 11:09:02 +02:00
Travis Nielsen 8bb4f603e6 ceph: default the device set template name to data
In 1.3 we added support for the metadata pvcs, thus requiring the name
to be set on the templates to differentiate between data and metadata.
For backward compatibility we need to assume the template is for data
if it is not set.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-05-21 11:33:16 -06:00
Sébastien Han d28e2f605c Merge pull request #5490 from travisn/monitoring-goroutines
Only run the cluster monitoring goroutines once
2020-05-19 09:40:13 +02:00
Travis Nielsen 394b98cb0b ceph: cancel the host cleanup if pvc deletion pending
During cluster delection, the finalizer will not be removed until
all the PVCs are removed. Since this could wait for a long time,
as long as the deletion event is re-queued, we need to cancel
the goroutine that is watching for cluster deletion so it can
avoid running multiple goroutines as the event runs again
and again.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-05-18 13:07:35 -06:00
Travis Nielsen 4173bee7c1 ceph: stop monitoring a cluster when deleted
When a cluster is requested for deletion there is no longer
a reason to monitor the cluster health such as whether mons
are in quorum, osds are up and in, or monitor the ceph status.
The cluster will wait potentially for a long time if the
PVCs are not deleted by the admin, therefore we stop monitoring
the cluster even before the finalizer is removed.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-05-18 13:02:37 -06:00
Travis Nielsen 90fa3a6d22 ceph: set external creds if admin key is available
If the admin key is available, set the external creds struct
to use that admin username and key instead of requiring the
callers to check every time if they should use the admin
key or the external key.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-05-18 12:56:28 -06:00
Travis Nielsen 81dae96929 ceph: remove obsolete version from osd health check
The Ceph version information is not needed to be passed
to the osd health checker since mimic support was dropped.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-05-18 12:56:28 -06:00
Travis Nielsen 85b0be301e ceph: simplify ceph user for monitoring goroutines
The ceph username is all that needs to be passed to the goroutines
that are monitoring the health of Ceph. This simplifies the
initialization of the goroutines.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-05-18 12:56:27 -06:00
Travis Nielsen 52e15b6e36 ceph: only run the cluster monitoring goroutines once
The monitoring goroutines that check for mon and osd health,
the ceph cluster status, and the bucket provisioner should
only be started once for each cluster. Previous to this change
and since the controller runtime conversion the goroutines were
being started again every time a reconcile was run. This means
you could potentially have many goroutines monitoring the same
thing.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-05-18 12:55:58 -06:00
Travis Nielsen d1da12c6ac ceph: wait indefinitely for cleanup before removing cluster finalizer
During cluster deletion, we currently only retry for a couple minutes
to wait for the pvcs to be deleted. After the timeout, we proceed
with the cluster deletion. To properly protect the pvcs for proper
cleanup, the finalizer should not be removed until the pvcs
are all confirmed to be deleted. In order to not block other cluster
events, we re-queue the deletion event to run again every 10s
until the pvcs are deleted or the finalizer is manually removed.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-05-14 16:52:24 -06:00
Travis Nielsen f47bb945c2 ceph: remove duplicate controller wait setting
The controller setting for requeuing an event moved to the
opcontroller package and was no longer needed in the main
controller package.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2020-05-14 16:52:24 -06:00