To enable multiple mgrs to run properly, we need to set the respective server_addr path with the mgr's name for the modules of each mgr.
When multiple mgrs are running, the modules( e.g. dashboard and Prometheus ) of each mgr need to bind its own server_addr when it is started. The way to get server_addr when ceph mgr starts the module is as follows: first, get the server_addr value by "mgr/<module-name>/<mgr-daemonid>/server_addr" path; and if this path does not exist, then get the server_addr value by" mgr/<module-name>/server_addr" path.
According to the original rook setting for server_addr, the modules of all mgrs take the same server_addr on ceph luminous and mimic, this produces a wrong result that only one mgr's modules can run properly ( the modules of other mgrs failed to start with the wrong server_addr).
This change is made to be compatible with ceph luminous, mimic, and nautilus. and Modify according to: the "HOST NAME AND PORT" description of http://docs.ceph.com/docs/master/mgr/dashboard/ .
Signed-off-by: wangxiao86 <wangxiao1@sensetime.com>
this skips orchestration when the device configmap receives an update
but the cluster hasn't been fully configured, which may leave some state
undefined and cause nil pointer dereferences accessing the cluster info
structure.
fixes: #3001
Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
this is an attempt to reduce the number of false positive config map
updates used to trigger orchestration for hotplugging.
Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
This commits fixes the upgrade for cluster running on 0.9.x and thus
have monitors listening on 6790.
The only downside is that the new messengers 2 protocol can not be
enabled because Ceph expects monitors to listen on 6789.
In order to enable messengers 2 your monitors will have to failover so
they can pick up both 6789 and 3300 ports, then and only then messenger
2 can be enabled.
Fixes: https://github.com/rook/rook/issues/3048 and
https://github.com/rook/rook/issues/3049
Signed-off-by: Sébastien Han <seb@redhat.com>
This reverts commit b7a564e929, reversing
changes made to 6996571b5c.
Reverting: Linking ceph noout to a node taint.
The taint is applied for reasons other than maintenance as well.
This also needs a way to respect user-set noout.
Need to revisit this with a better design.
Signed-off-by: Rohan CJ <rohantmp@gmail.com>
This commits allows the upgrade from Mimic to Nautilus to work by:
* removing the v2 brackets on the operator ceph config file generation,
we stick with v1 and can revert this back once we deprecate mimic there
is no rush since mons keep on listening to v1.
* enable messengers 2 when the cluster runs on Nautilus
Fixes: https://github.com/rook/rook/issues/2973
Signed-off-by: Sébastien Han <seb@redhat.com>
this adds a new operator environment variable that allows automatic
orchestration triggered by device discovery to be disabled.
Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
The CephCluster CR contains settings that are needed by other
CRs to configure the Ceph daemons. When the CephCluster CR
is updated, the updates will now be passed on to each of the
CR controllers to ensure the daemons are updated properly
without requiring an operator restart.
When calling the controllers from another controller,
we ensure that only a single goroutine is handling
CRs at any given time to prevent contention across
multiple CRs of the same type.
Signed-off-by: travisn <tnielsen@redhat.com>
Add a 'ceph-version' label to application controllers in the same manner
as the prior 'rook-version' label to help with upgrades.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
This code path is no longer necessary, and removing it allows removing
the `replicasets` RBAC permission for the operator.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
As per https://github.com/ceph/ceph/pull/26856 Ceph tunes the value of
osd-memory-target when running the OSD process so there is no need to
enforce it.
We also set a safety factor to that value which has been calculated by
the Ceph eng/perf team. Without this factor, the OSD will get OOM killed
since it is consuming memory for other purposes too.
Signed-off-by: Sébastien Han <seb@redhat.com>
Allow the user to set annotations in the CRDs of the following operators which will then be set on
the resulting Pods (Deployments, StatefulSets and so on).
Operators:
* Cassandra
* Ceph
* CockroachDB
* EdgeFS
* Minio
* NFS
The Annotations related structures have been added to the
rook.io/v1alpha2 pkg.
Signed-off-by: Alexander Trost <galexrt@googlemail.com>
When Ceph OSD spins up using ceph-volume, `ceph-volume lvm activate
--no-systemd --bluestore...` runs `lvs` which can cause significant
delays when /run/udev is not mounted from the host.
Before:
[root@rook-ceph-osd-0-858f49b5-md4t9 /]# time lvs
WARNING: Failed to connect to lvmetad. Falling back to device scanning.
WARNING: Device /dev/sda not initialized in udev database even after waiting 10000000 microseconds.
...
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
osd-data-729c1fb6-440f-434c-be49-f199fa59f1c0 ceph-3bf7823b-6db3-42b6-bf8f-c36301e85238 -wi-ao---- <10.00g
real 1m30.545s
After:
[root@rook-ceph-osd-0-d8c4698b8-5vqnk /]# time lvs
WARNING: Failed to connect to lvmetad. Falling back to device scanning.
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
osd-data-6c7249cc-cbd1-4f3b-9027-03725886bb30 ceph-9808319f-8567-4930-803a-41f70e537c1f -wi-ao---- <10.00g
real 0m0.043s
Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
the k8s Node add/update events can fire before a cluster is fully
configured, which may leave some state undefined. skip orchestration
when the cluster is in this state.
fixes: #3001
Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
Because the deployment update-and-wait function waits for the deployment
to be ready, and that cannot happen due to the keyring not
having been updated yet. The keyring has its
owner set to the deployment so that no special keyring handling must
be done, but it must be created after the deployment UID is known.
Change the order of operations so that the existing deployment's UID can
be retrieved if it exists and the keyring created/updated before the
update-and-wait function is called.
This order-of-operations issue currently only applies to the mds. Other
daemons whose keyrings are owned by the replication controller do not
use the update-and-wait function.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
We must expose the ceph version to the osd pod so that we can generate
the proper ceph configuration file. If we don't do this all the version
check will assume they run the opposite of the desired version and the
formatting of the `ceph.conf` will be wrong.
Signed-off-by: Sébastien Han <seb@redhat.com>
Let's return the expected pattern so that can be consumed by
ExtractCephVersion later. We don't modify the built-in String() so we
can more easily remove that once we switch the osd to be configured from
the operator.
Signed-off-by: Sébastien Han <seb@redhat.com>
We now run all the daemon with a new option from Nautilus 14.2.1 which
allows us to tell to a daemon to not log on file. However, we can decide
to activate logging by editing the configuration flag 'log_to_file' via
the centralized config store like this for a particular daemon:
ceph config set mon.a log_to_file true
This is useful when a daemon keeps crashing and we want to collect log
files on the system.
Fixes: https://github.com/rook/rook/issues/2881
Signed-off-by: Sébastien Han <seb@redhat.com>
It's quite convinient to expose /var/log/ceph so that we can decide to
activate logs locally on the machine and see what's going on.
This is only a placeholder when a daemon is stuck crashlooping and we
want to allow administrator to gather log files.
We still do not log on file but this can be activated via a config
option passed to the centralized config option store.
For some daemons, which typically do not store any data (rgw, rbd-mirror
and mds) we had to propagate dataDirHostPath from the cluster spec to
each creation call so that the bindmount can happen.
Fixes: https://github.com/rook/rook/issues/2881
Signed-off-by: Sébastien Han <seb@redhat.com>
When instanciating an osd, we now pass clusterInfo which basically
represents a collection of information about a particular Ceph cluster. Rook uses information
// about the cluster to configure daemons to connect to the desired cluster.
From clusterInfo, we use the CephVersion which helps us putting
conditions for our code execution.
Signed-off-by: Sébastien Han <seb@redhat.com>
Add a `rook-version` label to all controller resources used by Ceph:
deployments, daemonsets, and jobs. Labels are added to controller
resources only and not to the pod templates within because changing pod
templates causes updates due to the label change. The controller
resources themselves are not updated with the label addition and
therefore don't cause an unnecessary upgrade.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
This fixes issues for users trying to delete volumes and failing because
they only have `clusterNames` set on them in Rook earlier v0.8.
Signed-off-by: Alexander Trost <galexrt@googlemail.com>
A ceph filesystem was brought down by marking the filesystem
as down and individually failing the MDSes. From nautilus onwards,
this could be more simply and efficiently done by using
`fs fail` command. See,
https://github.com/ceph/ceph/commit/4c49f165ec1Resolves: #2841
Signed-off-by: Ramana Raja <rraja@redhat.com>
All usages of k8s go client are now also using the versioned `AppsV1() `
call for the client.
Updated MySQL and Wordpress, and Kube Registy examples to use apps/v1
Deployments.
Signed-off-by: Alexander Trost <galexrt@googlemail.com>
Make the Ceph operator more cautious about when it decides to remove
nodes from the Rook-Ceph cluster which are acting as osd hosts.
When `useAllNodes` is set to `true` we assume that the user wants to
have the most hands-off experience. Node removals are allowed when a
node is delted from Kubernetes and when a node has its taints/affinities
modified by the user (but not by automatic k8s modification as much as
possible).
When `useAllnodes` is set to `false` the only time a node is removed is
if it is removed from the Ceph cluster definition.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
wangxiao86
Noah Watkins
Travis Nielsen
Sébastien Han
Rohan CJ
Maximilian Mack
Louis
Blaine Gardner
Alexander Trost
Ramana Raja
Stefan Haas