The MGR watch-active sidecar needs Kubernetes API access to
read configmaps and monitor active-standby state. The API
server is host-networked and cannot be targeted by
pod/namespace selectors, making egress restrictions
impractical.
Switch the rook-ceph-mgr NetworkPolicy from egress-only to
ingress-only:
- Allow MON/OSD/MDS/tools pods on ports 6800-7300
- Allow Prometheus scraping on port 9283
- Remove egress rules that blocked API server access and
caused CrashLoopBackOff in the watch-active container
Signed-off-by: Oded Viner <oviner@redhat.com>