this commit adds new crd to admission webhooks to check
if one of the port or securePort are set to non-zero
value in objectstore crd.
If both are set to 0 in crd we'll see error like
```
Error from server (invalid create: either of port or securePort
fields should be not be zero): error when creating "object-test.yaml":
admission webhook
"cephobjectstore-wh-rook-ceph-admission-controller-rook-ceph.rook.io"
denied the request: invalid create: either of port or securePort fields
should be not be zero.
```
Signed-off-by: subhamkrai <srai@redhat.com>
The conditions on the cephcluster CR were being set to false
when not in progress, which is misleading to the purpose of
conditions. The conditions will now always represent current
state of the cluster. Transient conditions that show progress of
a reconcile will be removed from the status.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Remove features and design that supports adding OSDs to Ceph clusters
via `spec:driveGroups`. Update the Ceph upgrade doc that informs users
who currently use Drive Groups (we believe there are none of these
users) how to migrate to using the `spec:storage` config.
Resolves https://github.com/rook/rook/issues/7275
Revert "ceph: fix drive group deployment failure"
This reverts commit 76f1d9944e.
Revert "ceph: osd: add drive groups spec to cluster CR"
This reverts commit 7117fc12b7.
Revert "design: ceph orchestrator module add/remove OSDs"
This reverts commit 178187d035.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
There are still references to v15.2.7 in integration tests to allow
testing OSDs on partitions.
The newest dated Ceph image is ceph/ceph:v15.2.9-20210224
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
The pool quotas were added in #7264, however the codegen CI
action did not catch that an update was needed until it
was backported.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The cleanup job will need to run wherever the ceph daemons
were running. The scheduling is done by rook on the specific
nodes where the daemons were running, so there is no need for
other placement such as node or pod affinity.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
we don't want to merge the node affinity if specified in both
storageClassDeviceSet and placement. Now, we are passing new
boolean argument in `ApplyToPodSpec` which will be false in
case of osd and prepare pods because we don't want to overlap
placement of osd on PVC's and non-PVC's.
Signed-off-by: subhamkrai <srai@redhat.com>
The YugabyteDB team has decided to move support of their operator
to a new repo at https://github.com/yugabyte/yugabyte-operator.
The operator in Rook is no longer needed. Further usage of
the YugabyteDB operator is recommended at the new location.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
With Ceph Pacific, Rook can now deploy the cephfs-mirror daemon.
This initial commit covers the deployment of a single daemon only.
Multiple mirror daemons is currently untested.
Only a single mirror daemon is recommended.
The configuration of peers will come in a later PR since the mgr module
is still pending upstream: https://github.com/ceph/ceph/pull/39050
The same goes for integration tests, they will get added later once we
start testing on Pacific.
Closes: https://github.com/rook/rook/issues/7002
Signed-off-by: Sébastien Han <seb@redhat.com>
added waitTimeoutForHealthyOSD in the cluster crd that defines the time (in minutes) the operator would wait before an OSD can be stopped for
upgrade. This PR also removes the ok-to-continue logic for osds as its already handled by ok-to-stop. The default value is 10 minutes.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
The cockroachDB operator has not had community support in Rook.
Therefore, the time has come to deprecate and remove it.
If the sources are still needed, there is always git history.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
This was the last remaining CRD to not use the controller-runtime
library.
Small additions were added with the transition:
* the Kubernetes Secret that contains the CephX key has now an owner
reference to the CephClient object
* the secret name is present in the Status field of the CephClient:
```
status:
info:
secretName: rook-ceph-client-glance
phase: Ready
```
The controller will reconcile on CR updates and also if the Kubernetes
Secret is deleted.
Closes: https://github.com/rook/rook/issues/4938
Signed-off-by: Sébastien Han <seb@redhat.com>
After #6217, internal rgw spawning on external cluster was still broken,
operator claiming:
```
ceph-object-controller: failed to reconcile failed to create object
store deployments: failed to reconcile external endpoint:
failed to create or update object store "arch-cloud" endpoint:
failed to create endpoint "rook-ceph-rgw-arch-cloud". Endpoints
"rook-ceph-rgw-arch-cloud" is invalid: subsets[0]:
Required value: must specify `addresses` or `notReadyAddresses`
```
To spawn internal rgw for external cluster, changes detection of
what mean 'internal' for rgw and only use the externalRgwEndpoints
list for that independently of the status of the cluster
Now there is multiple posibilities:
* internal cluster and internal rgw pods: "normal case"
* external cluster and internal rgw pods: <= now working with the PR
* external cluster and external rgw pods: the external case
* internal cluster and external rgw pods: <= new case that could exist
Signed-off-by: Julien Girardin <jugirardin@free.fr>
The latest octopus release is now out with v15.2.8. We update the
operator base image and the examples to run with this version.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The arbiter may have unique requirements for placement settings. Thus, if the arbiter
placement is specified in the cluster CR, that placement will be applied to the arbiter
mon and the other mons will retain their placement from the other mon or all
sections of the placement.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Stretch clusters require certain node affinity for the mons and
must also be merged with the required node affinity in the placement
of the cluster CR. All conditions, both requested and generated must
be met, rather than one overriding the other.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
When the cluster is external we want to expose manager service port
along with the endpoints.
This allows us to connect but Rook will keep on using 9283 as a facing
port for Prometheus and more.
```
monitoring:
enabled: true
...
...
externalMgrPrometheusPort: 9283
```
Signed-off-by: Sébastien Han <seb@redhat.com>
With the release of the latest octopus v15.2.7 we update the base
of the operator image and set the examples to use the same release
to pick up the security and other bug fixes.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
We can now collect logs directly into a side-car container.
A new CRD spec has been added:
spec:
logCollector:
enabled: true
periodicity: 24h
Every 24h we will rotate log files for each Ceph daemon.
Signed-off-by: Sébastien Han <seb@redhat.com>
This makes it so the label merging function returns a new Labels object
with the merged content instead of modifying the existing one, which
would cause previously merged labels to be returned for a different set
of input labels.
Signed-off-by: Alexander Trost <galexrt@googlemail.com>
Rook's crashcollector pod posts entries to the ceph cluster when a crash occurs.
Over time the number cluster may hold crash entries needlessly.
To clean up old crash entries, this PR adds a field to the ceph cluster CR for the user to specify the number of days a crash entry should be kept for.
Providing a value for the field keepXDays creates a cronjob that runs every day at midnight, calling "ceph crash prune <keepXDays>".
Closes: https://github.com/rook/rook/issues/6332
Signed-off-by: Renan Campos <rcampos@redhat.com>
-creates a single PDB (max-unavailable=1) for all OSDs. This PDB allows one OSD to go down at a given time.
-When a drain is detected, blocking PDBs (max-unavailable=0) will be created for each failure domain that is not being drained and the main PDB (max-unavilable=1) will be deleted. This will allow all the OSDs in the currently drained failure domain to be removed while blocking the deletion of OSDs in other failure domains.
-Once the PGs are healthy again, the blocking PDBs will be deleted and the main PDB will be restored.
-Add PG healthcheck timeout
-Delete any legacy node drain pods and blocking OSD PDBs
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
Now, we can schedule snapshots on pools from the CephBlockPool CR when
the pool is mirrored.
It can be enabled like this:
```
mirroring:
enabled: true
mode: pool
snapshotSchedules:
- interval: 24h # daily snapshots
startTime: 14:00:00-05:00
```
Multiple schedules are supported since snapshotSchedules is a list.
Signed-off-by: Sébastien Han <seb@redhat.com>
Found by running the following command:
codespell -S .git,*.png,*.jpg -L \
aks,keyserver,atleast,dne,ser,ist,files\',ba,dum,iam,te -f -H
Signed-off-by: Mateusz Gozdek <mgozdekof@gmail.com>
In clusters where only two datacenters (or similar failure domains)
are available, a different mon and osd approach is needed to deal
with the network partitions or some other reason for one of the failure
domains going down. The Ceph stretched cluster makes the mons aware
of the failure domains by configuring one as the arbiter in a third
zone, while keeping two replicas of the data in each of the data
zoens.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The validation code for the admission controller is found in the
ceph.rook.io/v1 package, corresponding to each of the types.
To be better organized, the validators are now found in a source
file of the corresponding type name.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Due to #6492, preservePoolsOnDelete is not useful at all for CephFS;
after the filesystem is deleted, the leftover pools cannot be
reassocaited with a newly created filesystem without wiping all
metadata. The only way we can actually preserve data is keeping around
the entire filesystem.
This commit implements a `preserveFilesystemOnDelete` option which work
similar to the existing pool preservation option but instead keeps the
whole CephFS while taking it down and removing all MDSes.
This commit also changes all documentation to refer to this new option
with the intent of essentially deprecating `preservePoolsOnDelete`. IMO,
keeping around `preservePoolsOnDelete` is actively harmful because it
lulls users into thinking their data will be safe but, in reality,
recovering from this situation is highly complex and has large potential
for data loss.
Signed-off-by: Lalit Maganti <lalitm@google.com>
When the Ceph cluster runs on PVC and the OSDs are encrypted we can
store LUKS's Key Encryption Key inside a Key Management System. Today,
Rook only supports HashiCorp Vault: https://www.vaultproject.io/
The CephCluster has now a new "security" field which will plug onto the
KMS. Here is an example:
security:
kms:
tokenSecretName: <name of the secret containing a Vault token, used
to authenticate>
connectionDetails: < a map of strings containing connection
information>
Refer to the ceph-cluster-crd documentation to lear more.
Closes: https://github.com/rook/rook/issues/6105
Signed-off-by: Sébastien Han <seb@redhat.com>
To clarify the spec of ApplyToObjectMeta This also fixes the misleading
comments on the method.
Signed-off-by: Yusei Nishiyama <yusei.nishiyama@soundcloud.com>
adding fields `TotalBytes`, `UsedBytes`, `AvailableBytes`
`LastUpdated` to the ceph cluster CR's status section to
report the storage details of ceph cluster.
`TotalBytes`: Total storage capacity in bytes
`UsedBytes`: Total storage used in bytes
`AvailableBytes`: Total storage used in bytes(Total - Used)
`LastUpdated`: Last update time
for example(I have added 10GB disk):
`kubectl get cephclusters.ceph.rook.io -o yaml`
```
status:
ceph:
capacity:
bytesAvailable: 9657188352
bytesTotal: 10733223936
bytesUsed: 1076035584
lastUpdated: "2020-10-23T12:57:48Z"
```
Signed-off-by: subhamkrai <srai@redhat.com>
golangci-lint linter gosec showing more errors
than gosec gh. This commit resolve
new errors. And, removing
nosec comments from autogenerated files.
Signed-off-by: subhamkrai <srai@redhat.com>
The pool spec has now a new property called "replicasPerFailureDomain"
which essentially represents the number of replicas to store in each
failure domain.
Assuming the failure domain is a datacenter (if the cluster is
stretched) then you will have 2 replicas per datacenter where each
replica ends up on a different host. This gives you a total of 4
replicas and for this, the "size" must be set to 4.
Closes: https://github.com/rook/rook/issues/5591
Signed-off-by: Sébastien Han <seb@redhat.com>
The ceph mons should never be started with an even number in quorum.
The desired state should always be an odd number of mons to ensure
a healthy majority quorum. The operator now rejects a request for
an even number of mons.
If there are not a sufficient number of nodes for mons to be on a
unique host, the operator also returns an early error instead of
getting stuck waiting for a pending mon. This is not foolproof since
there might be fewer available nodes for the mons, but at least it
is a quick check for the common case.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
on Azure, all SSDs are reporting as a rotational disk.
This commit will add `tuneFastDeviceClass` flag which will
tune for an SSD even if the cloud provider is reporting
rotational. We have a flag `tuneDeviceClass`
which works vice-versa.
Signed-off-by: subhamkrai <srai@redhat.com>
subhamkrai
Travis Nielsen
Blaine Gardner
Frank Ritchie
Sébastien Han
Santosh Pillai
ushen
Julien Girardin
Blaine Gardner
Alexander Trost
Renan Campos
Mateusz Gozdek
Lalit Maganti
Yusei Nishiyama
subhamkrai