Just like we do for the monitors, we set an ownerref to the OSD PVC so
when the cluster is deleted we remove the PVC too.
Signed-off-by: Sébastien Han <seb@redhat.com>
When the Ceph cluster runs on PVC and the OSDs are encrypted we can
store LUKS's Key Encryption Key inside a Key Management System. Today,
Rook only supports HashiCorp Vault: https://www.vaultproject.io/
The CephCluster has now a new "security" field which will plug onto the
KMS. Here is an example:
security:
kms:
tokenSecretName: <name of the secret containing a Vault token, used
to authenticate>
connectionDetails: < a map of strings containing connection
information>
Refer to the ceph-cluster-crd documentation to lear more.
Closes: https://github.com/rook/rook/issues/6105
Signed-off-by: Sébastien Han <seb@redhat.com>
The CRUSH map does not allow duplicate values (nor keys) for the
labels. That means that if the currently hardcoded label value
`default` occurs anywhere else in the tree (for example, because
some of the topology labels provided by the cloud provider use it),
the cluster cannot sucessfully spawn.
By giving the user control over the label value used for the root
CRUSH map label, it is possible to adapt the cluster to this
situation.
To be able to actually use the `default` value, the root=default
node which is created by default by ceph itself has to be removed;
since that node is accompanied by a replicated crush rule, we have
to remove that rule, too.
The integration tests are modified to sometimes use the custom
crushRoot (based on an arbitrary criterium) to get coverage across
the various suites. Separate tests could be added, but were not
deemed necessary at this point.
Fixes#4993.
Signed-off-by: Jonas Schäfer <jonas.schaefer@cloudandheat.com>
This allows the catch-22 situation where the filesystem cannot be
reconciled because there is no MDS but there is no MDS because the
operator has not reconciled the filesystem and brought up the MDS pods.
Closes#5967, #5846
Signed-off-by: Lalit Maganti <lalitm@google.com>
We don't always get an error of the type "*exec.ExitError" so we must
validate the type before printing it otherwise the interface conversion
will fail.
Signed-off-by: Sébastien Han <seb@redhat.com>
adding fields `TotalBytes`, `UsedBytes`, `AvailableBytes`
`LastUpdated` to the ceph cluster CR's status section to
report the storage details of ceph cluster.
`TotalBytes`: Total storage capacity in bytes
`UsedBytes`: Total storage used in bytes
`AvailableBytes`: Total storage used in bytes(Total - Used)
`LastUpdated`: Last update time
for example(I have added 10GB disk):
`kubectl get cephclusters.ceph.rook.io -o yaml`
```
status:
ceph:
capacity:
bytesAvailable: 9657188352
bytesTotal: 10733223936
bytesUsed: 1076035584
lastUpdated: "2020-10-23T12:57:48Z"
```
Signed-off-by: subhamkrai <srai@redhat.com>
The mon resources may be orphaned if there is a mon failover
that was cancelled, for example a mon pvc may remain if the
original mon comes up and the operator was restarted before
the mon failover completed. The cleaning up of mon resources
was only happening was each successful mon failover, now we
perform the check with every reconcile since the resources
were not being cleaned up predictably enough.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Add unit tests for when the "Zone" is configured
in an object store so that the object store joins
the CephObjectZone and the zone's corresponding
multisite configuration.
Signed-off-by: Ali Maredia <amaredia@redhat.com>
updated mon pdb reconcile to delete mon pdb and create a new one when mon count changes in the cluster spec
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
The number of admission controller pods is 2. When there is only one
node as the CI environment, one of these becomes Pending state.
Although it's harmless, it's better to reduce this value to 1.
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
golangci-lint linter gosec showing more errors
than gosec gh. This commit resolve
new errors. And, removing
nosec comments from autogenerated files.
Signed-off-by: subhamkrai <srai@redhat.com>
The default number of provisioner pods are 2. When there is only one node,
for example the CI environment, one of these becomes Pending state.
Although it's harmless, it's better to reduce this value to 1.
Closes: https://github.com/rook/rook/issues/6294
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
The pool spec has now a new property called "replicasPerFailureDomain"
which essentially represents the number of replicas to store in each
failure domain.
Assuming the failure domain is a datacenter (if the cluster is
stretched) then you will have 2 replicas per datacenter where each
replica ends up on a different host. This gives you a total of 4
replicas and for this, the "size" must be set to 4.
Closes: https://github.com/rook/rook/issues/5591
Signed-off-by: Sébastien Han <seb@redhat.com>
The ceph mons should never be started with an even number in quorum.
The desired state should always be an odd number of mons to ensure
a healthy majority quorum. The operator now rejects a request for
an even number of mons.
If there are not a sufficient number of nodes for mons to be on a
unique host, the operator also returns an early error instead of
getting stuck waiting for a pending mon. This is not foolproof since
there might be fewer available nodes for the mons, but at least it
is a quick check for the common case.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
When a node is not tolerable, also output the node name so it's clear
which not it is which is not tolerable.
Signed-off-by: Thomas Bechtold <tbechtold@suse.com>
When deploying on non-PVC, we still need to gather c-v logs on failures.
Now if the bootstrap fails, the c-v log will be printed.
Signed-off-by: Sébastien Han <seb@redhat.com>
When rgw is not responding the aws-sdk retries using exponential
backoff. Having 20 retries means we would wait for a while (hours) to
eventually fail. This is a problem as we cannot reflect the status of
the bucket properly.
Now we use 5 max retries which means around 8 seconds.
Signed-off-by: Sébastien Han <seb@redhat.com>
In cephadm and volume/nfs module, all ganesha daemons within a cluster watch
single config object. Each ganesha daemon in rook watches it's own config
object. This patch updates the ganesha config object name to maintain
uniformity.
Signed-off-by: Varsha Rao <varao@redhat.com>
We can now tune the value of TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES by
setting an annotation on the CephCluster CRD like so:
```
annotation:
osd:
TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES: "134217728"
```
Alternatively, if nothing is specified Rook will read the value of the
file `/etc/sysconfig/ceph`.
Closes: https://github.com/rook/rook/issues/6134
Signed-off-by: Sébastien Han <seb@redhat.com>
In order to properly debug errors, we need to merge stderr inside the
`err` reported so that we don't only see the stdout.
We were doing this when executing without file output, doing the same
for the output file.
Signed-off-by: Sébastien Han <seb@redhat.com>
Once we close the channel for the monitoring daemons, we update the
monitoring status so as to not call them again.
This is to prevent "close of closed channel" panic.
Co-authored-by: Travis Nielsen <tnielsen@redhat.com>
Signed-off-by: Nitin Goyal <nigoyal@redhat.com>
Signed-off-by: Raghavendra Talur <raghavendra.talur@gmail.com>
on Azure, all SSDs are reporting as a rotational disk.
This commit will add `tuneFastDeviceClass` flag which will
tune for an SSD even if the cloud provider is reporting
rotational. We have a flag `tuneDeviceClass`
which works vice-versa.
Signed-off-by: subhamkrai <srai@redhat.com>
During the initialization encrypted OSD initialization sequence, we check
for the presence of the encrypted container. If it exists we don't try
to open it again since this will result in an error.
However, there is another case we need to handle, when the underlying
device is gone. For instance, if the pod/PV couple was drained and move
back, living the orphan dm. Once the pod comes back, the dm is still
present and perfectly matches. Unfortunately, the underlying disk is
different and thus the dm must be removed and the disk re-opened.
Signed-off-by: Sébastien Han <seb@redhat.com>
During the 1.3 cycle, we had implemented support for OBC external mode
through an Endpoint in the StorageClass parameter. In 1.4, this is not
the case anymore as we directly use the CephObjectStore external mode.
However, we must maintain backward compatibility with the cluster using
the Endpoint and not fail the upgrade.
Thus checking for a CephObjectStore is invalid and should not always be
assumed, now if we detect an Endpoint we return a simple context which
fixes the upgrade.
Signed-off-by: Sébastien Han <seb@redhat.com>
this commit handle golangci-lint linter errcheck.
`errcheck` - Errcheck is a program for checking for
unchecked errors in go programs. These unchecked errors
can be critical bugs in some cases
To see only staticcheck linter output
`golangci-lint run --disable-all -E errcheck`
Signed-off-by: subhamkrai <srai@redhat.com>
On LUKS2 format, the resize operation requires the KEK to be passed. To
maintain a single simple init container for resize, we now open the
encrypted block with "--disable-keyring" so that further operations
won't require the keyring.
Signed-off-by: Sébastien Han <seb@redhat.com>
Ganesha daemons don't need to acess ceph cluster with admin capabilities.
Instead new user with restricted caps can be used.
Signed-off-by: Varsha Rao <varao@redhat.com>
Sébastien Han
Travis Nielsen
Jonas Schäfer
Lalit Maganti
subhamkrai
Ali Maredia
Santosh Pillai
Satoru Takeuchi
Daichi Sakaue
Thomas Bechtold
Varsha Rao
Raghavendra TalurandTravis Nielsen
subhamkrai