Implement an allow list mechanism that disables potentially unsafe OBC
fields by default. OBC fields beyond `maxObjects` and `maxSize` don't
neatly fit into the OBC framework as it was originally envisioned and
implemented.
Some of the newly added configs could allow users to cause confusion for
themselves. Others might allow users to hijack others buckets. Some
might allow bricking the entire S3 store.
Out of an abundance of safety, allow-list the known-safe options by
default, and require administrators to enable potentially troublesome
options via the new operator-level config
`ROOK_OBC_ALLOW_ADDITIONAL_CONFIG_FIELDS`.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
the rgw operations for s3 can now be accessible using sidecar
rgw-ops-log availabe in json form that can be further filtered logging
for observability, this will set the rgw_enable_ops_log setting
Signed-off-by: Deepika Upadhyay <deepika.upadhyay@clyso.com>
The ci was using a pretty old version og golangci-lint.
This updates to the latest version.
Additionally, it silences some
gosec integer conversion overflow false positves
and fixes some real errors of this category
and string format errors found by golangci-lint, while at it.
Co-authored-by: Blaine Gardner <b.blaine.gardner@gmail.com>
Co-authored-by: Travis Nielsen <tnielsen@redhat.com>
Signed-off-by: Michael Adam <obnox@samba.org>
Fix the issue when using with external ceph cluster, ceph exporter
secret isn't created which cause ceph-exporter fail to run. By
default this option is false means ceph-exporter will run for
external cluster.
monitoring
metricsDisabled: false
This patch also adds metricsDisabled option to helm values.yaml.
fixes#14275
Signed-off-by: Yaguang Tang <heut20008@gmail.com>
Resolves CIS benchmark 5.2.8 rule by adding capabilities with
explicitly defined empty "add" list (where it needed) and with
NET_RAW capability in "drop" list.
5.2.8 Minimize the admission of containers with added capabilities
Containers must drop the `NET_RAW` capability and are not permitted
to add back any capabilities.
Signed-off-by: Peter Razumovsky <prazumovsky@mirantis.com>
If the cmd-reporter key is set under the resources
element in the CephCluster CR, the memory and cpu request
and limits will be set on the ceph detect version job
and network job accordingly.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Resolves CIS benchmark rule 5.7.3, Pods part. SecurityContext
should be explicitly defined in pod level of Pod spec section.
It is sufficient to specify empty securityContext to satisfy
CIS 5.7.3 rule.
5.7.3 Apply Security Context to Your Pods and Containers
When designing your containers and pods, make sure
that you configure the security context for your pods,
containers, and volumes.
Signed-off-by: Peter Razumovsky <prazumovsky@mirantis.com>
Given that Ceph Quincy (v17) is past end of life,
remove Quincy from the supported Ceph versions,
examples, and documentation.
Supported versions now include only Reef and Squid.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
This adds an operator config setting ROOK_REVISION_HISTORY_LIMIT
defaulting to kubernetes'value for RevisionHistoryLimit.
If configured, the provided value will be used as RevisionHistoryLimit
for all Deployments rook creates.
Fixes: #12722
Signed-off-by: Michael Adam <obnox@samba.org>
This new setting is of Boolean type and defaults to "false".
When set to "true", it changes the behavior of the
rook operator to
nable host network on all pods created by the cephcluster controller
new method to check the setting: opcontroller.EnForceHostNetwork()
Signed-off-by: Michael Adam <obnox@samba.org>
The jobs to detect the ceph and csi versions now can have
custom annotationso and labels added to them. They are
very short-lived jobs, but they may need custom annotations
in some environments to run.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
1) Make the csi rbd container logs persisted in a file
(csi plugin, csi provisioner, csi addons sidecar)
2) Use the cephcluster api specs to configure the log rotate
3) Add log rotation to rotate the log file and
Add a sidecar log collector container
part-of: https://github.com/rook/rook/issues/12809
Signed-off-by: parth-gr <partharora1010@gmail.com>
The CreateOrLoadClusterInfo (and therefore LoadClusterInfo) methods were
not loading the ClusterInfo.OwnerInfo. This should help future CRD
controllers get the full ClusterInfo struct without having missing
information. Current controllers that need ClusterInfo fill the field
themselves.
During testing, I observed one corner case where an upgraded cluster was
missing the `rook-ceph-csi-config` configmap. The cluster had a
CephFilesystemSubVolumeGroup resource created, and the reconcile for
that resource was attempting to create the missing CSI configmap and
failing with a nil pointer exception due to the missing OwnerInfo field
in ClusterInfo. This cluster condition hasn't been reproduced in healthy
environments, and it is unknown how the CSI configmap came to be
missing. However, the case did expose the missing loaded info as a
potential for causing nil pointer exceptions during corner cases when
code is otherwise correct.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
Cleanup the resources created by subvolumegroup
when its deleted. Following resources will be cleaned up:
- OMAP value
- OMAP keys
- Clones
- Snapshots
- Subvolumes
Signed-off-by: sp98 <sapillai@redhat.com>
If the configmap rook-config-override is empty,
there is no need to trigger the reconcile to update
the ceph daemons. This configmap update is causing
unnecessary reconciles periodically in some clusters
even when it is empty.
Signed-off-by: travisn <tnielsen@redhat.com>
The changes made to cosi driver are reflected. Add the cosidriver crd to
predicate workflow.
Fixes: #12545
Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
The exporter pod is ephemeral and frequently being deleted
and re-created when ceph daemons are created. Therefore,
we need to skip reconciling based on the deletion
of the exporter deployment.
Signed-off-by: travisn <tnielsen@redhat.com>
This commit adds new CSIDriverOptions section in
cephCluster CR. This section contains settings
for read affinity and kernel+fuse Mount options
These settings will be injected directly into
rook-ceph-csi-config cm to be applicable per
ceph cluster.
Signed-off-by: Rakshith R <rar@redhat.com>
During certain maintenance tasks the admin will own running
operations on the ceph mgr, rgw, mds and rbd-mirror daemons
and the operator should not interfere with those operations.
Co-authored-by: gauravsitlani <gaurav.sitlani@live.com>
Signed-off-by: subhamkrai <srai@redhat.com>
Fix an issue in the network address detection job where placement was
only retreived from osd and not merged with all.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
Use K8s LivenessProbe mechanism to check OK-status of nfs-ganesha
container. A user may define his own lineness-probe, or a default one
which expects NFS TCP-port 2049 to be active; that is, willing to accept
new connections: for Ceph>=18.2.1 issue 'rpcinfo' call on local pod;
otherwise use standard K8s TCP-socket liveness probe mechanism.
Define permissive values to liveness-probe to ensure that the NFS
service is defined in failed-state only when it has non-recoverable
error.
The current default definition of LivenessProbe is expected to guard the
nfs pod from at least the following two cases:
- Deadlocks: where an nfs-ganesha server is running, but unable serve
new connections due to internal bad-state.
- Resource exhaustion on the host node (e.g. OOM) which prevents the
server from accepting new connections and reply to NULL RPC request.
In both cases we expect K8s to reschedule the nfs pod, most likely on
different host node.
Refs rook issue #12719
Signed-off-by: Shachar Sharon <ssharon@redhat.com>
Use the Rook image (defined by the operator pod) to detect the Multus
network address ranges. It is reasonable for users to want to have a
minimal Ceph image that does not have the `ip` utility installed, which
is used for detecting the address ranges of multus interfaces. Instead,
use the Rook image, which Rook can ensure has the `ip` tool if Ceph ever
removes it from their image.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
When OSDs flap, ceph stops the OSD daemon if its marked down greater than
5 times in 600 seconds. But OSD pod restarts and marks the OSD `up` again.
This causes the PGs mapped to these OSDs to peer. While the PGs are peering,
IO to these PGs are blocked.
So we need to ensure that if ceph is marking OSD `down` due to flapping, OSD pod
should not restart to mark the OSDs `up` again.
This PR adds a sleep to the OSD pod if the container returned with a 0 exit code
Default behavior is to sleep for 6 hrs. But user can configure it from the
ceph cluster spec.
Signed-off-by: sp98 <sapillai@redhat.com>
Change how Rook detects network CIDRs for Multus networks. The IPAM
configuration is only defined as an arbitrary string JSON blob with a
"type" field and nothing more. Rook's detection of CIDRs for whereabouts
had already grown out of date since the initial implementation.
Additionally, Rook did not support DHCP IPAM, which is a reasonable
choice for users. And more, Rook did not support CNI plugin chaining,
which further complicates NADs. Based on the CNI spec, network chaning
can result in any changes to network CIDRs from the first-given plugin.
All these problems make it more and more difficult for Rook to support
Multus by inspecting the NAD itself to predict network CIDRs. Instead,
it is better for Rook to treat the CNI process as a black box. To
preserve legacy functionality of auto-detecting networks and to make
that as robust as possible, change to a canary-style architecture like
that used for Ceph mons, from which Rook will detect the network CIDRs
if possible.
Also allow users to specify overrides for CIDR ranges. This allows Rook
to still support esoteric and unexpected NAD or network configurations
where a CIDR range is not detectable or where the range detected would
be incomplete. Because it may be impossible for Rook to understand the
network CIDRs wholistically while residing only on a portion of the
network, this feature should have been present from Multus's inception.
Improving CIDR auto-detection and allowing users to specify overrides
for auto-detected CIDRs rounds out Rook's Multus support for CephCluster
(core/RADOS) installations. No further architectural changes should be
needed for CephClusters as regards application of public/cluster network
CIDRs for Multus networks.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
We need to restart all the ceph daemons whenever
cephCluster network settings are modified like
requiremsgr2, encryption and compression. This
required for Ceph to consider the new settings
it require new ceph daemons all over.
Signed-off-by: subhamkrai <srai@redhat.com>
golangci linter was throughing error `k8s.io/utils/pointer`
package is deprecated. So, I have removed that.
Signed-off-by: subhamkrai <srai@redhat.com>
Signed-off-by: avanthakkar <avanjohn@gmail.com>
Volume "ceph-daemons-sock-dir" is coming empty in case if dataDirHostPath,
which is the case for osd onPVC. Fix the volume creation by using the
ceph cluster spec dataDirHostPath, which allows to run socket commands
on osd containers.
enable flags with --default prefix for --log-to-stderr, --mon-cluster-log-to-stderr, --err-to-stderr, and --log-stderr-prefix
Signed-off-by: Javier <sjavierlopez@gmail.com>
few functions got change as they were deprecated
for ex: ioutil.Readfile change to os.Readfile
ioutil.TempFile change to os.CreateTemp
And fixed golang-ci-lint-issues
Signed-off-by: parth-gr <paarora@redhat.com>
This fixes the the MultiClusterDeploySuite CI failure. The operator
is timing out waiting for the mgr deployments to be
ready, according to the WaitForDeploymentToStart() method. After
the reconcile times out after about five minutes, the next
reconcile succeeds since the wait is only done for new mgr
deployments.
Closes: https://github.com/rook/rook/issues/11685
Signed-off-by: Redouane Kachach <rkachach@redhat.com>
The idea behind this change is to use the readiness probe to implement
the mgr HA mechanism. In the current ceph mgr implementation only the
active instance offers the command 'mgr_status' through the admin
socket. We use this command combined with a Readiness Exec Probe to
detect which manager is active. Kubernetes will automatically mark
it as ready and redirect any service traffic to the active instance.
Closes: https://github.com/rook/rook/issues/11640
Closes: https://github.com/rook/rook/issues/11638
Signed-off-by: Redouane Kachach <rkachach@redhat.com>
increasing the rotation from default 7 to 28 as
in case of rbdmirror logs seems not enough in some cases
with maxLogSize 500 so it's better to increase the rotation
for rbdmirror specific.
Signed-off-by: subhamkrai <srai@redhat.com>
Blaine Gardner
Deepika Upadhyay
Travis Nielsen
Yaguang Tang
Peter Razumovsky
parth-gr
sp98
Jiffin Tony Thottan
Rakshith R
gauravsitlani
Shachar Sharon
guoguangwu
subhamkrai
Tarun Gupta Akirala
avanthakkar
Javier
Redouane Kachach
parth-gr
Avan Thakkar