Sometime the kube engine needs a bit of time to return the logs of a
given job and fails to read the stream.
Retrying up to detect the Ceph version seems reasonnable to
overcome this issue.
Fixes: https://github.com/rook/rook/issues/3227
Signed-off-by: Sébastien Han <seb@redhat.com>
the only exception to a naive device list comparison had been to ignore
drive UUID information which was unreliable when a device wasn't
formatted / partitioned. however various users have reported different
type of false positives that resulted in orchestration being run
continuously due to the wrong observation that devices were changing.
this patch fixes the cases we have observed and attempts to be slightly
more conservative in the calculation.
1. the devlinks is ignored. when a device is setup for lvm, for example,
the devlinks will be updated with different paths that point to the
device in addition to its standard paths addressable by pci address.
2. in the lvm case, the "model" field and "filesystem" field may also
change.
3. we ignore devices with devlinks that contain "usb" to avoid issues
when using usb drives.
4. be smart about detecting device availability. if a device transitions
from a non-empty (or has-partitions) state to an empty (or unpartitioned)
state then orchestration is triggered. this like observing that a device
is now available (e.g. in the allDevices case). however, when a device
transistions from empty to non-empty, then this is ignored as while it
is a change, it's generally a change associated with the new consumption
of the device.
fixes: #3059fixes: #3185fixes: #3131
Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
The memory check was missing and will be trigger if resources limit are
configured for the rbdmirror pod.
Signed-off-by: Sébastien Han <seb@redhat.com>
This commit fixes the second test case where limit and request are
either identical or different but still we use limit as a value.
Signed-off-by: Sébastien Han <seb@redhat.com>
If setting the fsgroup recursively on a shared filesystem mount is
not desirable, the fsgroup capability on the flex driver
should first be disabled in the operator env vars. Now the driver
will apply the fsgroup only at the top level instead of
recursively for the entire shared filesystem.
Signed-off-by: travisn <tnielsen@redhat.com>
- Updated getCephVolumeOSDs method to use fsid filter while retriving devices.
- This solves "failed to fetch mon config (--no-mon-config to skip)" error when multiple ceph clusters are running on same Node
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
for the osd recovery case (DOWN->UP) log at a matching log level as the
message indicating the OSD went down.
fixes: #2904
Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
- Updated code to use deviceClass property when a pool is created for both "replicated" and "erasure code".
- Updated "ceph crush rule create-..." command to use "create-replicated" instead of "create-simple"
- Updated unit tests
- Updated (ceph-pool-crd.md) documentation to reflect the changes.
- Updated pending release notes.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
The mons require the --public-bind-addr when host networking is not in use
since the pod ip will be different from the service ip that will be
advertised to clients. When host networking is enabled, there is no
need for this argument since the endpoints will be the same to bind
inside the pod as what is advertised to clients.
Signed-off-by: travisn <tnielsen@redhat.com>
Clusters using host networking will not work if the non-default port is
being used by the mons. Previous to rook 1.0 the mons were all
using the non-default port 6790 and now they are using the default port
of 6789 from 1.0. Now host networking will preserve the non-default port
to allow upgrades to continue working.
Signed-off-by: travisn <tnielsen@redhat.com>
provision OSDs with ceph-volume when metadtaDevice is set (#3108)
always print ceph-volume report before executing ceph-volume.
Signed-off-by: Michael Vollman <michael.b.vollman@gmail.com>
this adds a structure to hold various settings related to executing ceph
cli commands, and introduces a common interface for configuring and
running such commands.
Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
the existing exec interface with timeout is effectively the same as
ExecuteCommandWithOutput plus a timeout. this patch adds a variant of
ExecuteCommandWithOutputFile that uses a timeout.
Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
Before performing the copy, the OSD init container now checks
destination path exists. If the destination exists, it skips the copy.
The OSD pods copy the rook and tini binaries into a shared volume using
an init container. In restarting pods, this copy could fail with 'text
file busy'(ETXTBUSY), because the destination file in the shared volume
could be under execution. The restarting pods will enter a
'Init:CrashLoopBackOff' state after this, leaving the ceph cluster
degraded. Skipping the copy will allow the init containers to complete
successfully, allowing the pod to restart.
Fixes#2674.
Signed-off-by: Kaushal M <kshlmster@gmail.com>
This removes the unnecessary check for taints on nodes in the
`GetNodeSchedulable()` function. This fixes that even though the user
has specified `tolerations` for, e.g., `NoSchedule` taints, that would
be ignored and the node(s) directly be "marked" as unschedulable.
Signed-off-by: Alexander Trost <galexrt@googlemail.com>
when upgrading to a newer version of ceph that doesn't need the http
binding fix, this patch unwinds the configuration fix which may still be
present when upgrading from a cluster without the fix. if the
configuration fix remains then the dashboard and prometheus servers will
not be able to properly configure themselves.
Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
The list of nodes to be queried for configuring OSDs
should include all of the K8s nodes before the placement
or other filters are applied. The operator was sometimes
missing configuring OSDs on nodes if the discover pod
hadn't finished running on a node. Now the osd prepare
job will be triggered on a node even if the discover pod
hasn't yet run there.
Signed-off-by: travisn <tnielsen@redhat.com>
The flex settings cannot be passed to the driver with environment variables.
There is no context available for returning the flex settings except
that the flex driver will look in a config file in the same directory.
These settings must be valid or else the driver will return the default
settings.
Signed-off-by: travisn <tnielsen@redhat.com>
The flex driver no longer needs to check whether a kubelet
restart is needed or whether the namespace-specific driver
can be supported. The copy of k8s 1.13 types can also
be removed since we have updated to a newer k8s client.
Signed-off-by: travisn <tnielsen@redhat.com>
fixes: #2492
the http bind address fix in rook address an upstream issue in ceph in
which cherrypy break when binding to `::` when the host doesn't have an
ipv6 address. it fixes the issue by always binding prometheus and the
dashboard webservers to the pod address. by doing this, it prevents
using the kubectl port-forward technique, which some users would like.
a patch to cherrypy is upstream in ceph. this PR checks the ceph version
and selectively disables the fix so that both port-forward and the
normal service method works. this is the verison matrix:
FIX VER
--- ---
luminous merged (>= 12.2.12)
mimic merged (>= 13.2.6)
nautilus merged (>= 14.1.1)
when all the containers for these versions exists and sufficient time
passes the unreachable code guarded by the version checks in this patch
can be removed.
Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
Sébastien Han
Travis Nielsen
Noah Watkins
Dmitry Yusupov
DL186038
rohan47
Santosh Pillai
Blaine Gardner
Alexander TrostandMaksim Nabokikh
Christian Hüning
Anton Skriptsov
Jose A. Rivera
Michael Vollman
Kaushal M