if admin secret is created stop creating csi secret from
the rook operator
the csi creds and secret will still be created by the python and
the import script
Signed-off-by: parth-gr <partharora1010@gmail.com>
When writing to the operator log, where possible
the namespace of the resource on which the operator
is working will be logged as a standard prefix
to more easily troubleshoot what resource is being reconciled
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Use a sync map for the clusterMap in the cluster controller
and fix other issues to allow for concurrent cluster
reconcile.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
external mode was dependednt on the ceph status to
update the cluster connected condition,
BUt as now the status check will only update the ceph status,
so we will explicitly update the ceph status condition in the
end of the reconcile
Signed-off-by: parth-gr <partharora1010@gmail.com>
CephConnection CR is created with the
kubernetes namespace name as its the
unique name in the entire cluster.
This PR removes unwanted input
parameters and adjust the namespace
name for the cephConnection CR.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
This PR implements the CSI cephx key rotation feature,
which follows overlapping rotation for non-daemon keys.
Refer to the cephx rotation API from design PR 15915 for details.
Signed-off-by: subhamkrai <srai@redhat.com>
today, csi operator cephConnection and client profile
CR's where not getting created in external mode. So, let's
start creating those.
Signed-off-by: subhamkrai <srai@redhat.com>
Implements #14733. Allows to set IDs of external mons to
Cluster CRD. Rook will not remove external mons from quorum
and will add external mon addresses to mon endpoints.
Use-case for external mon is to maintain quorum for 2-AZ
k8s cluster in case of zone outage.
Signed-off-by: Artem Torubarov <artem.torubarov@clyso.com>
Fix the issue when using with external ceph cluster, ceph exporter
secret isn't created which cause ceph-exporter fail to run. By
default this option is false means ceph-exporter will run for
external cluster.
monitoring
metricsDisabled: false
This patch also adds metricsDisabled option to helm values.yaml.
fixes#14275
Signed-off-by: Yaguang Tang <heut20008@gmail.com>
Someone testing the Multus holder pod removal feature encountered an
issue where the migration process failed to lead to a system state where
PVCs could be created successfully.
The root cause was found to be a ceph csi config map wherein the primary
CephCluster entry was lacking a value for the "namespace" field.
I observed this once in my development on the holder pod removal
feature, but I was unable to reproduce and assumed it was my own error.
Since this has been seen in a user environment, it must be that the
error is a race condition, and I am unable to determine the exact source
of the bug.
I do not believe this bug would be present if the code that updates the
CSI configmap were properly idempotent, but it has many conditions based
on prior states, and I was unable to determine how to resolve this
underlying impelementation pattern issue.
Instead, I opted to separate the `clusterKey` parameter into two clear
parts:
1. `clusterID` for when `clusterKey` is used as an analogue for
`clusterID`
2. `clusterNamespace` for when `clusterKey` is used as an analogue for
`clusterNamespace`
I added unit tests to ensure that SaveClusterConfig() will be able to
detect when the namespace is currently missing, and using the new
`clusterNamespace` field, it should always know what value to use as
input when correcting the bug in already-installed systems.
I also verified that this update works when the function simultaneously
removes netNamespaceFilePath entries, and that those entries are removed
properly.
Finally, manual testing also verifies the change.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
There have been some issues with non-CSI Rook controllers that are
creating the CSI config configmap (`rook-ceph-csi-config`). This causes
problems with the K8s OwnerReference. The primary CSI reconciler
(controller) creates the configmap with the correct owner reference,
which is supposed to be the operator deployment.
Other instances were creating the configmap with owner references set to
the CephCluster. This is a minor bug, but it can result in this
configmap being deleted along with the first CephCluster that initially
created it.
To fix this issue, remove all instances of `CreateCsiConfigMap()` except
the single usage which the CSI reconcile uses to initially create the
configmap. Other controllers that might have attempted to create this
configmap previously will return an error indicating that it is waiting
for the configmap to be created.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
This commit uses an csi config map API exposed by Ceph-CSI.
This will reduce the efforts of keeping the configuration in
sync with Ceph-CSI.
Signed-off-by: Praveen M <m.praveen@ibm.com>
Currently if we use the msgr2 port,
cs-cluster-config-json looks like
"monitors":["192.59.109.11:3300","192.59.109.13:6789","192.59.109.12:6789"]
The first monitor is passed by python script and rest are
Discovered by rook operator, so if the first mon is using the v2 port
lets update the other mons also by setting up the RequireMsgr2 flag
Signed-off-by: parth-gr <paarora@redhat.com>
When msgr2 is required, ensure the mon endpoints passed to
the csi configmap are on port 3300. The mons will be
listening on both 6789 and 3300. Existing volumes can
continue using port 6789 while new volumes will use
port 3300.
Signed-off-by: travisn <tnielsen@redhat.com>
With the new mgr HA implementation (based on readiness probe) we
don't need the mgr sidecar (live-watch) anymore. This commit is
intended to remove all the related code.
Signed-off-by: Redouane Kachach <rkachach@redhat.com>
The default configmap does not have any labels or annotations
applied. If the helm chart is later updated to configure the
ceph config overrides, the configmap will fail to be created
by the chart since it doesn't have the needed label and helm
annotations so helm will know it can take over the resource
management.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The crash collector controller is designed for watching nodes where
ceph daemons are running, and ensuring a special daemon is running
on that node to provide additional support for ceph on that node.
The crash collector is the first example of a daemon that should be
running on all the ceph daemon nodes. The next example of such a
node daemon will be the ceph exporter that will listen for the
ceph metrics as described in the design doc.
https://github.com/rook/rook/blob/master/design/ceph/ceph-exporter.md
Now the crash collector controller is renamed to the node daemon controller
so the ceph exporter daemon can also be managed by the same controller.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
if Multus is enabled the clusterinfo should be updated with
network as multus as to run the ceph cmds in remote
executor
Signed-off-by: parth-gr <paarora@redhat.com>
The CSI package needs to load clusterInfo, today this code is in the mon
package which makes the call of LoadClusterInfo impossible without
having a circular import.
Signed-off-by: Sébastien Han <seb@redhat.com>
If a mon is added or removed in a ceph
cluster all the CSI clusterID's belongs
to the ceph cluster should also gets updated.
Added a new cluster identifier to the csi configmap,
i.e ceph clusterNamespace which helps to identify
the clusterID's belongs to that ceph cluster.
Note:- to support the upgraded clusters,we
can ask users to update the mon IP's for the
problamatic clusterID as it wont be updated
by the operator.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
The prometheus rules had been previously created if the cephcluster CR
setting monitoring.enabled was set to true. The rules were not customizable
and therefore not flexible enough. Now the rules are installed by the helm
chart. To customize the rules, a post-processor can be applied to the helm
chart.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
adding observedGeneration field in the cephcluster cr
status for having better control on reconciling,
as observedGeneration field will be updated by the controller
Closes: https://github.com/rook/rook/issues/9673
Signed-off-by: parth-gr <paarora@redhat.com>
This is fixing a potential race where the context is canceled and the
external cluster is trying to fetch the cluster info. If the context is
canceled then the loop will be stuck forever. Now we check for the
canceled context and return appropriately.
Signed-off-by: Sébastien Han <seb@redhat.com>
The rook-ceph-csi-config config map now includes CephFS subvolumegroups
when a subvolumegroup CR is created. It is useful for ceph-csi to
understand which subvolumegroup to use for a given cluster.
Signed-off-by: Sébastien Han <seb@redhat.com>
This commit adds context parameter to k8sutil service functions. By
this, we can handle cancellation during API call of service resource.
Signed-off-by: Yuichiro Ueno <y1r.ueno@gmail.com>
This commit adds context parameter to utilities in opcontroller to
remove context.TODO use in opcontroller. By this, we can handle
cancellation of reconcilers in a fine-grained way.
Signed-off-by: Yuichiro Ueno <y1r.ueno@gmail.com>
This commit is a large refactor on how the operator starts, stops and
how it starts various sub-components such as the ceph-csi driver. It
also refines the way we cancel orchestrations. We don't use breakpoints
anymore but send our self a SIGUP to reload our controller runtime
manager.
The reload will happen under different circonstances like:
* a new adminission controller secret is created/deleted/changed
* a CephCluster CR is edited
As mentioned earlier, the csi driver now has its own controller, just
like flex. It reacts to change in the operator config map for particular
ROOK_CSI_ fields.
A second new controller for the operator's general config has been
created, it manages:
* the logging level
* the ceph CLI command timeout
* the discovery daemon
The operator reacts much more rapidly to cancellation events by stopping
the manager's context and reloading it.
Signed-off-by: Sébastien Han <seb@redhat.com>
If the cluster is external we want to periodically rehydrate the mgr
endpoint. This handles the scenarion where the active manager changes,
so we need to update the endpoint with the new IP address.
The create-external-cluster-resources.py script now requires an extra
permission to query the manager service so Rook can discover the active
one and its IP.
Testing:
```
[leseb@tarox~/go/src/github.com/rook/rook][external-active-mgr-change] minikube kubectl -- exec -n rook-ceph deploy/rook-ceph-tools -ti -- ceph mgr stat
{
"epoch": 37,
"available": true,
"active_name": "b",
"num_standby": 1
}
[leseb@tarox~/go/src/github.com/rook/rook][external-active-mgr-change] kubectl -n rook-ceph-external get ep
NAME ENDPOINTS AGE
rook-ceph-mgr-external 172.17.0.12:9283 3m10s
[leseb@tarox~/go/src/github.com/rook/rook][external-active-mgr-change] k scale --replicas=0 deployment rook-ceph-mgr-b
deployment.apps/rook-ceph-mgr-b scaled
[leseb@tarox~/go/src/github.com/rook/rook][external-active-mgr-change] minikube kubectl -- exec -n rook-ceph deploy/rook-ceph-tools -ti -- ceph mgr stat
{
"epoch": 40,
"available": true,
"active_name": "a",
"num_standby": 0
}
[leseb@tarox~/go/src/github.com/rook/rook][external-active-mgr-change] kubectl -n rook-ceph-external get ep
NAME ENDPOINTS AGE
rook-ceph-mgr-external 172.17.0.13:9283 3m55s
```
Signed-off-by: Sébastien Han <seb@redhat.com>
Because of the recent CRD changes made, the mon count had a minimum of
1, making the configuration of the external cluster impossible. The
operator would fail to add the finalizer:
```
2021-04-29 16:57:38.429451 E | ceph-cluster-controller: failed to reconcile. failed to add finalizer: failed to add finalizer "cephcluster.ceph.rook.io" on "test-external": CephCluster.ceph.rook.io "test-external" is invalid: spec.mon.count: Invalid value: 0: spec.mon.count in body should be greater than or equal to 1
```
We now fixed the CRD as well as adding the CR status.
Signed-off-by: Sébastien Han <seb@redhat.com>
It's better to validate ownerReferences when setting them. In addition, we should use
controllerrutil.Set{Controller,Owner}Reference, that have such validation, as possible.
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
The active mgr should match the labels on the services that
are available for prometheus and the dashboard. The selector
labels must be updated whenever there is a new active mgr.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The conditions on the cephcluster CR were being set to false
when not in progress, which is misleading to the purpose of
conditions. The conditions will now always represent current
state of the cluster. Transient conditions that show progress of
a reconcile will be removed from the status.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
When the cluster is external we want to expose manager service port
along with the endpoints.
This allows us to connect but Rook will keep on using 9283 as a facing
port for Prometheus and more.
```
monitoring:
enabled: true
...
...
externalMgrPrometheusPort: 9283
```
Signed-off-by: Sébastien Han <seb@redhat.com>
Since we moved to the controller-runtime, events are processed one by
one and so are reconciles. This means we won't have multiple
orchestrations happening at the same time. Thus removing this code.
Also removing one unused variable.
Signed-off-by: Sébastien Han <seb@redhat.com>
We only need to read the ceph cluster version if the cluster external
has external monitoring enabled.
This avoids any issue during upgrades from 1.3 to 1.4 when monitoring is
enabled and ceph caps have not been updated.
Signed-off-by: Sébastien Han <seb@redhat.com>
Instead of setting the Phase of the CR once the reconcile is done, let's
actually set in from the healthcheck so it is more accurate.
Closes: https://github.com/rook/rook/issues/5249
Signed-off-by: Sébastien Han <seb@redhat.com>
We can now connect an external prometheus exporter to Rook to collect
metrics and generate alerts from Prometheus.
Just enable this in the CephCluster CR:
```
spec:
external:
enable: true
monitoring:
enabled: true
rulesNamespace: rook-ceph
externalMgrEndpoints:
- ip: 192.168.39.182
```
Closes: https://github.com/rook/rook/issues/5516
Signed-off-by: Sébastien Han <seb@redhat.com>
To provide more context for executing commands in a ceph cluster,
the full clusterInfo is now passed to the ceph execution commands.
All information about the cluster will now be available throughout
all the areas of the operator. The namespace, ceph credentials,
mon endpoints, and other info is a core part of that cluster info.
Arguments passed through the controllers are also simplified for
mons, mgr, osds, and other daemons where the parameters had
become too complex.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The operator should only connect to ceph with a single set of creds.
In a converged cluster this will be the admin creds and in an external
cluster it will be lower-privileged creds. Independent clusters were
implemented with a separate set of creds. To simplify the code these
are now merged to a single set.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
parth-gr
Travis Nielsen
Madhu Rajanna
subhamkrai
subham rai
Artem Torubarov
Yaguang Tang
Blaine Gardner
Praveen M
parth-gr
Redouane Kachach
Sébastien Han
Josh Soref
Divyansh Kamboj
Yuichiro Ueno
Satoru Takeuchi
Arun Kumar Mohan