Based on the feature of metadata PVC for OSD: #4818, add the wal device.
For this, you need to create a new volumeClaimTemplates, its name must be "wal", if "wal" is
set, "metadata" device will refers specially for block.db device, and "wal" refers for block.wal
device in BlueStore.
A template will look like this:
```
volumeClaimTemplates:
- metadata:
name: data
spec:
resources:
requests:
storage: 10Gi
# IMPORTANT: Change the storage class depending on your environment (e.g. local-storage, gp2)
storageClassName: gp2
volumeMode: Block
accessModes:
- ReadWriteOnce
- metadata:
name: metadata
spec:
resources:
requests:
storage: 6Gi
# IMPORTANT: Change the storage class depending on your environment (e.g. local-storage, io1)
storageClassName: io1
volumeMode: Block
accessModes:
- ReadWriteOnce
- metadata:
name: wal
spec:
resources:
requests:
storage: 6Gi
# IMPORTANT: Change the storage class depending on your environment (e.g. local-storage, io1)
storageClassName: io1
volumeMode: Block
accessModes:
- ReadWriteOnce
```
We now map block, block.db and block.wal directly inside the container instead of
running ceph-volume activate.
Signed-off-by: Chen, Tingjie <tingjie.chen@intel.com>
The cluster CR status needs to be updated with the latest object
instead of a cached copy or else it could fail if there is some
other update. Using the Rook clientset ensures it is the latest
instead of using the cached controller runtime object.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The service selector labels cannot be updated during the upgrade or else the
mon commands will not respond until the mon deployment is also updated.
Since there is a mon command to check for the ceph versions before the
deployment is updated, the operator would remain stuck during the upgrade.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
We can now encrypted OSD device that were provisioned via a storage
class using the PV interface.
The encryption works at the storageClassDeviceSets level, which means we
can have encrypted and non-encrypted sets.
Using the new key `encrypted` we can turn it on.
Signed-off-by: Sébastien Han <seb@redhat.com>
The labels cannot be updated on a deployment match selector.
In v1.4 a new label was added for ceph_daemon_type that is intended
to be on the pod labels, but cannot be applied to the match selectors.
Therefore, we suppress any new labels that are added to the daemons.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Rename PodLabels function to CephDaemonAppLabels for more clarity about
what the function's purpose is.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
this commit handles all the gosec g601
error code (i.e Implicit memory aliasing
of items from a range statement).
Signed-off-by: subhamkrai <subhamkumarrai03@gmail.com>
Instead of setting the Phase of the CR once the reconcile is done, let's
actually set in from the healthcheck so it is more accurate.
Closes: https://github.com/rook/rook/issues/5249
Signed-off-by: Sébastien Han <seb@redhat.com>
At startup there is a race condition between the goroutine checking
for mon health and the first reconcile of the mons. There is a lock
that ensures only one of them will check the mon health at a time.
The issue is that if the wrong one gets there first, it will not
have the desired mon count initialized from the cluster CR as expected
and the healther checker will use a desired mon count of 0.
The desired count of 0 leads the health checker to remove
a mon since there are extra mons. Only one mon is allowed
to be removed at a time, and adjusting to fewer than two mons is
disallowed once there are at least two mons.
After the health checker is completed, the main reconcile of the mons
will proceed with the correct desired mon count of 3 and attempt
to restore to the real desired quorum size.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The mon was starting with a partial spec. There was a race condition
that was causing incorrect initialization in the health check goroutine
that could remove a mon upon operator restart. Now the mon will be
properly initialized with the full spec.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The admin needs to start the toolbox and run a command such as ceph osd tree to see the device classes for each OSD. This PR simplifies it by storing the deviceclass info in the CephCluster CR status
closes: https://github.com/rook/rook/issues/5754
Signed-off-by: kesavan <kvellalo@redhat.com>
This commit:
- adds the pull section on the CephObjectRealm spec
to enables realms to be pulled instead of created.
- creates the system user and generates the access
key and secret key for the user and zones in a realm.
- adds "omitempty" to fields in multisite CRs that
are not required
Signed-off-by: Ali Maredia <amaredia@redhat.com>
Now, we can not only cleanup monitor data, logs and crashes but the
disks too. As part of the cleanupPolicy CR spec, we have a new setting
called sanitizeDisk which holds more details:
* method: indicates if the entire disk should be sanitized or simply ceph's metadata.
Possible choices are 'complete' or 'quick' (default)
* dataSource: indicate where to get random bytes from to write on the disk.
Possible choices are 'zero' (default) or 'random'
Using random sources will consume entropy from the system and will take much more time then the zero source
* iteration: overwrite N times instead of the default (1). Takes an integer value
See the documentation for more details.
Signed-off-by: Sébastien Han <seb@redhat.com>
The user creds should not be passed to the ceph helper methods
since the clusterInfo already has the needed context. Removing
a final reference that was missed in the refactor.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
If the healthcheck goroutine cannot check for the ceph status/times out
(if the mons are not in quorum) this should still be reflected in the
CRD status field.
Example:
```
status:
ceph:
details:
error:
message: 'failed to get status. . timed out: exit status 1'
severity: Urgent
health: HEALTH_ERR
lastChanged: "2020-07-20T14:11:05Z"
lastChecked: "2020-07-20T14:11:05Z"
previousHealth: HEALTH_WARN
```
Signed-off-by: Sébastien Han <seb@redhat.com>
A long-standing issue that was only solved via a documentation note. Now,
during the prepare pod instantiation, we check for the presence of the
lvm binary on the host. Success will indicate that we can bootstrap
OSDs where failure will refuse to prepare the OSD.
Closes: https://github.com/rook/rook/issues/5627
Signed-off-by: Sébastien Han <seb@redhat.com>
Pass desired devices to the OSD provisioning container by
JSON-marshalling/-unmarshalling a disk ID with StorageConfig settings.
This will allow disks to be specified that contain special characters.
Notably, this will support /dev/disk/by-path/pci-HHHH:HH:HH.H devices
that were unsupported previously due to the format which separated
devices from params with colons (:).
Fixes#5056Fixes#5535
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
We can now connect an external prometheus exporter to Rook to collect
metrics and generate alerts from Prometheus.
Just enable this in the CephCluster CR:
```
spec:
external:
enable: true
monitoring:
enabled: true
rulesNamespace: rook-ceph
externalMgrEndpoints:
- ip: 192.168.39.182
```
Closes: https://github.com/rook/rook/issues/5516
Signed-off-by: Sébastien Han <seb@redhat.com>
We currently don't use the module, so let's speed up the bootstrap
progress as well as the reconcile by not always enabling it.
It can always be enabled via the spec mgr section.
Signed-off-by: Sébastien Han <seb@redhat.com>
The external creds have been loaded from their own key. For
backward compatibility we need to check that key if the
new key is not found.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
To provide more context for executing commands in a ceph cluster,
the full clusterInfo is now passed to the ceph execution commands.
All information about the cluster will now be available throughout
all the areas of the operator. The namespace, ceph credentials,
mon endpoints, and other info is a core part of that cluster info.
Arguments passed through the controllers are also simplified for
mons, mgr, osds, and other daemons where the parameters had
become too complex.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The ceph config and client packages are conceptually the same.
To avoid circular dependencies in some cases, we simplify by combining
the packages into the client package.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The operator should only connect to ceph with a single set of creds.
In a converged cluster this will be the admin creds and in an external
cluster it will be lower-privileged creds. Independent clusters were
implemented with a separate set of creds. To simplify the code these
are now merged to a single set.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
When a PodAntiAffinity is set to mon, detect-version may not be scheduled because detect-version has the same PodAntiAffinity with mon.
This commit fixes this issue.
Closes#5810
Signed-off-by: Hiroshi Muraoka <h.muraoka714@gmail.com>
Add the ability to provision Ceph OSDs with Drive Groups.
This adds Drive Groups to the CephCluster CRD, and it sets code
in place for propagating this config to the OSD provisioning pod.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
This commit allows us to configure status check for each daemon:
* "mon": health check on the ceph monitors (quorum)
* "osd": health check on the ceph osds
* "status": ceph health status check
Each check is controlled by the following settings:
* disabled: whether to disable the check (default: false)
* internal: interval to run the check
* timeout: only valid for mons, is the timeout for unresponsive mon
before failling over.
Example to disable the status health check:
```yaml
healthCheck:
daemonHealth:
status:
disabled: true
```
As part of that, pod's livenessprobe can now be configured via the
following settings:
* disabled: whether to enable or not
* probe: override the current probe in place by a new one
```yaml
healthCheck:
livenessProbe:
mon:
disabled: true
```
Closes: https://github.com/rook/rook/issues/5772
Signed-off-by: Sébastien Han <seb@redhat.com>
The OSD daemon has been missing critical flags for logging to stderr
where k8s can capture the logs. Without the --log-to-stderr=true,
all the OSD logging was essentially lost until now.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Update the base operator image and cluster examples to use the latest
octopus release v15.2.4. Also cleanup some old examples and unit tests
that were still based on mimic.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The stuck mons were only being force deleted in the mon health checker.
After an operator restart, the health check may not be executed
for 20 minutes until after the timeout during the main reconcile
which is holding onto a lock for mon reconcile. Now the stuck mon
will be deleted during the reconcile as well.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
If the mon is backed by a pvc, ensure the pvc is deleted when
the mon is failed over. The pvc is being left in a bound state
even though it will no longer be used in the cluster.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
During mon failover if the mon fails to be removed, the new mon
config was not being saved. The mon removal anyway is not retried
if it fails, so it should make a best effort to remove all the
resources related to the mon, and then always update the mon
config after the failover.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
This commit fixes an argument of the CreateCrashCollectorSecret function as
it is expected to be the namespace of the cluster.
Signed-off-by: binoue <banji-inoue@cybozu.co.jp>
Chen, Tingjie
Travis Nielsen
Sébastien Han
Blaine Gardner
subhamkrai
kesavan
Ali Maredia
Hiroshi Muraoka
binoue