Also check for the empty namespace instead of an
empty name which was incorrect.
Co-authored-by: Nikita Sivukhin <sivukhin@turso.tech>
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
After an update, the mgr pod will be restarted. Since the
readiness probe will indicate the standby mgr pod is never
ready, we need to check simply for the running mgr pod
and not whether or not it is ready.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Ceph will use a random nonce for the msgr daemons as long as
they are running as PID 1 or have the environment variable
CEPH_USE_RANDOM_NONCE set to indicate running in a containerized
environment. While the Ceph daemons in Rook are expected to
run in PID 1, there are some environments where the container
runtime may choose to run something else such as
/usr/bin/pod as PID 1. The side effect then is that ceph uses
the pid as the nonce and causes problems when the daemons
restart with the same PID.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
adding observedGeneration field in the cephcluster cr
status for having better control on reconciling,
as observedGeneration field will be updated by the controller
Closes: https://github.com/rook/rook/issues/9673
Signed-off-by: parth-gr <paarora@redhat.com>
This commit adds context parameter to k8sutil pod functions. By this, we
can handle cancellation during API call of pod resource.
Signed-off-by: Yuichiro Ueno <y1r.ueno@gmail.com>
Events like node drain can take more than 10-15 minutes. If the node is not update the default monTimeOut of 10 minutes, then rook will attempt to fail over the mon. This failover won't work as the node is still down.
This PR retries once before the mon failover if the mon pod is not scheduled
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
If there are multiple mgrs running, the operator will automatically
add pod antiaffinity across hosts by default. For stretch clusters,
antiaffinity across the stretch failure domain (e.g. zones)
will be added. Required antiaffinity will be created unless the
test setting allowMultiplePerNode is set to true.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
If there are two mgr daemons, they should be running on different
hosts or zones, depending on the failure domain.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The mgr daemon may be failed over by ceph if the active mgr is not
responding and the standby mgr is available. If the active mgr changes
the services for the dashboard and metrics will be updated with a
label selector for the new active mgr. The services cannot direct
traffic to the standby mgr or else they will be incorrectly redirected
to the active mgr.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
we don't want to merge the node affinity if specified in both
storageClassDeviceSet and placement. Now, we are passing new
boolean argument in `ApplyToPodSpec` which will be false in
case of osd and prepare pods because we don't want to overlap
placement of osd on PVC's and non-PVC's.
Signed-off-by: subhamkrai <srai@redhat.com>
Currently, mgr deployment has two overlapping environment variables, ROOK_POD_IP.
This causes kube-apiserver to create error logs
Signed-off-by: binoue <banji-inoue@cybozu.co.jp>
The arbiter can only be configured with the stretch cluster if the
CRUSH map is balanced and there are two zones in the CRUSH map.
After the OSDs are configured, we wait for all the OSD pods to be
running and that the CRUSH map is balanced. If it takes more than
two minutes, we fail the reconcile and try again. This is only done
the first time the stretch cluster is configured. In future reconciles
we first check if the stretch cluster is already enabled before
enabling it again.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
In clusters where only two datacenters (or similar failure domains)
are available, a different mon and osd approach is needed to deal
with the network partitions or some other reason for one of the failure
domains going down. The Ceph stretched cluster makes the mons aware
of the failure domains by configuring one as the arbiter in a third
zone, while keeping two replicas of the data in each of the data
zoens.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
this commit handle golangci-lint linter staticcheck error.
`staticcheck` - Staticcheck is a go vet on steroids,
applying a ton of static analysis checks.
To see only `staticcheck` linter output
`golangci-lint run --disable-all -E staticcheck`
Signed-off-by: subhamkrai <srai@redhat.com>
this commit handles all the gosec g601
error code (i.e Implicit memory aliasing
of items from a range statement).
Signed-off-by: subhamkrai <subhamkumarrai03@gmail.com>
The pod deletion is needed by other daemons besides the osds,
so we move the helper method into the k8sutil package.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The rook types used across the storage providers moved from the v1alpha2
package to the v1 package. This commit points the packages at their new
location. Implementation is expected to remain unchanged.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Code for pod anti affinity mechanism on ceph gateways is a duplica of the one used by ceph mon.
This is a refacto to use same code on both components
Signed-off-by: n.fraison <n.fraison@criteo.com>
Reduce from 5 minutes to less than 50 seconds the time needed to restart a
mgr and osds(PVCs based),rgw,mds,rbd,nfs and toolbox pod that was running
in a k8s "NotReady" Node.
A new Rook Operator setting has been added to allow the user to set the
time used in pod's toleration <node.kubernetes.io/unreachable>:
ROOK_UNREACHABLE_NODE_TOLERATION_SECONDS = <5>
New setting added in release notes
Addressed @travisn,@leseb, and @blaineEXE suggestions
Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Add missing docstrings for container image
functions in pod.go, noting the behavior
when getting a container image from a pod
which only has a single container defined.
Signed-off-by: Kristoffer Grönlund <kgronlund@suse.com>
The Ceph mon flag `--public-bind-addr` does not keep the same port as
the previous daemon run when the port is left off the flag's value. A
port is undesired since that could interfere with Nautilus' msgr2
protocol, so instead configure the mon's services to forward the default
bind port on pods (6789) to the mon endpoint's port. The endpoint port
will be 6789 for new mons, but it may be 6790 for legacy clusters.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
We now expose resource as environment variable in the pod.
They will be used by Ceph to report as a daemon metadata.
Signed-off-by: Sébastien Han <seb@redhat.com>
The mgr orchestrator modules need the container image to run
the ceph image for blinking the lights when a disk is down
Signed-off-by: travisn <tnielsen@redhat.com>
This commit introduces the necessary changes to support the new
messenger feature coming with Ceph Nautilus (currently in development).
What changes? Now the monitor listens on two port:
* old 6789 for messenger v1, which will help us support older client
(e,g: krbd)
* new 3300 for messengers v2, which brings new improvement in the
messaging layer. This new transport layer brings numerous advantages
such as encryption improvement, speed improvement, pluggable nature to
support different network stack than TCP and many more.
We still have one Service IP, however it has 2 ports, see:
```
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
rook-ceph-mon-a ClusterIP 10.106.217.160 <none> 3300/TCP,6789/TCP 4h
rook-ceph-mon-b ClusterIP 10.99.36.175 <none> 3300/TCP,6789/TCP 4h
rook-ceph-mon-c ClusterIP 10.108.220.74 <none> 3300/TCP,6789/TCP 4h
````
The `ceph.conf` has changed and we don't force the port when using an IP
address (public addr etc). Ceph, depending on its version will naturally
start the monitors on their right port, 6789.
A new --ceph-version-name CLI argument has been added to the Rook binary
so that when the pod starts it passes the ceph version name and the
configuration of the ceph.conf, as well as daemon startup flags, happen
properly.
Given that the Rook Operator remembers the port of all the monitors it
deployed (through Pod definition), this change is not an issue and will
maintain backward compatibility.
Note that to test this you must build rook with dev container image,
which contains the dev Nautilus version. So you should do something
like:
`make -j4 BASEIMAGE='ceph/daemon-base:latest-master' IMAGES='ceph' build`
Resolves: #2525
Signed-off-by: Sébastien Han <seb@redhat.com>
Make the Rook config-init unnecessary for mons, and remove that init
container. Perform all mon configuration steps in the operator, and set
up the mon pods and k8s environment such that only Ceph containers are
needed for running mons.
This should help streamline changes to the mons, as there will be no
need to change the `daemon/mon` code or `cmd/rook/ceph` code with mon
changes in the future.
This work starts to lay the groundwork for supporting the
`design/ceph-config-updates.md` design.
Notable new bits:
Create a keyring secret store helper for storing dameon keyrings, and
use it to store the mon keyring. Mon pods mount the keyring into a
k8s secret-backed volume.
Create a configmap store for the Ceph config file which can be mounted
into pods/containers directly to /etc/ceph/ceph.conf. Also store
individual mon_host and mon_initial_members values which can be mapped
into pods as environment variables and used in Ceph commandline flags,
enabling the mon pods to have the most up-to-date information about the
mon cluster when restarting and without need for operator intervention.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
Ceph expects to be able to reason about physical hardware / nodes.
Before this patch the NODE_NAME that was available referred to the pod
name. This patch exposes the underlying node name, pod, and pod
namespace for the ceph daemons to consume.
fixes: #2078
Signed-off-by: Noah Watkins <nwatkins@redhat.com>
deprecate NetworkInfo.PublicAddrIPv4, use NetworkInfo.PublicAddr instead
deprecate NetworkInfo.ClusterAddrIPv4, use NetworkInfo.ClusterAddr instead
deprecate command flag public-ipv4, use public-ip instead
deprecate command flag private-ipv4, use private-ip instead
change ROOK_PUBLIC_IPV4 to ROOK_PUBLIC_IP
change ROOK_PRIVATE_IPV4 to ROOK_PRIVATE_IP
Signed-off-by: Zhang Miaolei <zmlcc@outlook.com>
- Operator is deployed under namespace rook-system
- Operator deploys rook-agent on all nodes as daemonset also on
rook-system
- Rook-agent installs flexvolume driver on hosts
- Rook-agent listens on unix socket for driver requests
- Rook-agent perform attach/detach on its node
- Rook-agent creates/delete CRD volumeattach objects
- Added fencing to support ROX and RWO
- Added unit and integration tests
- Updated examples and docks
fixes#432
- multiple file systems are allowed
- multiple mds instances are allowed per file system.
- a keyring is created per mds instance
- double the desired mds instances are started, half in standby mode
Travis NielsenandNikita Sivukhin
Joshua Hoblitt
Redouane Kachach
parth-gr
Yuichiro Ueno
Andy Bursavich
Santosh Pillai
subhamkrai
binoue
Arun Kumar Mohan
subhamkrai
n.fraison
Juan Miguel Olmo Martínez
Kristoffer Grönlund
Blaine Gardner
Sébastien Han
Huamin Chen
Noah Watkins
Jared Watts
Zhang Miaolei
Tony Allen
Travis Nielsen
Steve Leon
Bassam Tabbara