Example of failure scenarios:
- requested PVC 500M = 500,000,000 bytes
- PV is created as:
(500M / 2**20)Mi = "476"Mi
But 476Mi = 476 * 2**20 = 499122176 < 500M
From the k8s documentation:
Claims will remain unbound indefinitely if a matching volume does not
exist. Claims will be bound as matching volumes become available. For
example, a cluster provisioned with many 50Gi PVs would not match a
PVC requesting 100Gi. The PVC can be bound when a 100Gi PV is added
to the cluster.
which is inline with the behavior observed in which a 500M PVC is never
bound. However, the documentation also states:
If a PV was dynamically provisioned for a new PVC, the loop will
always bind that PV to the PVC. Otherwise, the user will always get
at least what they asked for
which suggests that an _explicit_ binding can be created, and perhaps
something has changed with respect to this binding that has caused the
failures in newer versions of k8s.
fixes: #3391fixes: #2922
Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
Previously, Rook Agent and Discovery DaemonSet deployment didn't allow
adding nodeAffinity. This commit adds nodeAffinity spec to daemonSet
deployment, which can be configured through environment variables in
operator deployment yaml.
+ Support multiple LabelKey, each with multiple LabelValue
+ Support multiple LabelKey with no value
Signed-off-by: Umanga Chapagain <chapagainumanga@gmail.com>
If we are running a Ceph cluster that has an unsupported version and if
we have set allowUnsupported: false, a newly bootstrap operator will
fail to start. To fix this, the user will update the CR and set
allowUnsupported: true, however at this point clusterInfo hasn't been
initialized so the operator will panic because we were referencing a nil
pointer.
To fix this, we re-populate clusterInfo if necessary and things go
smoothly.
Closes: https://github.com/rook/rook/issues/3137
Signed-off-by: Sébastien Han <seb@redhat.com>
Somehow during startup the arg got misinterpreted by ceph-osd and was
failing with:
2019-06-19 15:23:09.098967 I | exec: Running command: ceph-osd --foreground --id 0 --osd-uuid feceb7d7-3148-4a0d-af7a-753ba2f21a92 --conf /var/lib/rook/osd0/rook-ceph.config --cluster ceph --default-log-to-file false --ms-learn-addr-from-peer false
2019-06-19 15:23:09.135921 I | unrecognized arg false
failed to start osd. Failed to complete '': exit status 1.
Now we concatenate the whole flag to avoid that.
Signed-off-by: Sébastien Han <seb@redhat.com>
OwnerReferences are now created with the fully qualified apiVersion
such that the references will work properly on OpenShift.
Signed-off-by: travisn <tnielsen@redhat.com>
The check for whether to set the OwnerReferences is no longer
necessary. When passing the correct value to the apiVersion,
the OwnerReferences can be successfully set for OpenShift.
Signed-off-by: travisn <tnielsen@redhat.com>
Signed-off-by: Guy Margalit <guymguym@gmail.com>
Co-Authored-By: Sébastien Han <seb@redhat.com>
Co-Authored-By: Travis Nielsen <tnielsen@redhat.com>
This change is meant to allow running operators locally on a developer machine.
The idea is to allow faster development cycles by reducing the time and complexity of building -> deploying -> debugging on cluster.
For operators that rely only on kubernetes API this works easily - see cockroachdb and minio examples in development-flow doc.
The change includes:
- rook.NewContext() - Refactored to remove repeating initialization code that was copy-pasted in most of the operators in order to create the clusterd.Context and the Clientsets. Also it detects the mode of working in-cluster vs external and sets up the external mode with standard user config (~/.kube/config) and a job executor.
- rook.GetOperatorImage() - Refactor this repeating code in many operators to detect the operator pod image. Also added a global flag --operator-image that developers can use to override this when running locally.
- rook.TerminateOnError() - Added a convenient function.
When the operator first starts, the only operation needed
is to watch for new cephcluster crds to be created and
start the discovery to find available devices. The flexvolume
agent, the csi driver, and the volume provisioning can all be delayed
starting until the first cluster is created.
Signed-off-by: travisn <tnielsen@redhat.com>
- When an osd is marked out, and it is safe to be destroyed then delete the osds deployment.
- Removed osdGracePeriod
- Updated unit tests to test osd marked out action
Signed-off-by: rohan47 <rohgupta@redhat.com>
This commit does multiple things:
* remove support for AllNodes where we would deploy one rgw per node on
all the nodes.
* a transition path is implemented in the code so that if someone has an
existing deployment, daemonsets will be removed and replaced by an
deployments.
* when using "instances", each rgw deployed has its own key which makes
Ceph reporting the exact number of rgw running, see:
```
[root@rook-ceph-operator-775cf575c5-bh4sr /]# ceph -s
cluster:
id: 611fcf39-0669-4864-9a12-debb35c0397a
health: HEALTH_OK
services:
mon: 3 daemons, quorum a,b,c (age 12h)
mgr: a(active, since 12h)
osd: 3 osds: 3 up (since 12h), 3 in (since 12h)
rgw: 3 daemons active (my.store.a, my.store.b, my.store.c)
data:
pools: 6 pools, 600 pgs
objects: 235 objects, 3.8 KiB
usage: 3.0 GiB used, 84 GiB / 87 GiB avail
pgs: 600 active+clean
```
Closes: https://github.com/rook/rook/issues/2474, https://github.com/rook/rook/issues/2957 and https://github.com/rook/rook/issues/3245
Signed-off-by: Sébastien Han <seb@redhat.com>
Create and maintain a config map that meets the requirements of
the ceph csi such that Rook can maintain the contents of config
map with up-to-date mon information to be used later by csi.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
When OSDs are running on Nautilus we always disable old osd features and
aplpy the onces for Nautilus as described in the upgrade doc.
During an upgrade or the next time an orchestration will be called the
command will be applied. The command is idempotent so we can run it each
time.
This can be backported for 1.0.3
Closes: https://github.com/rook/rook/issues/2960
Signed-off-by: Sébastien Han <seb@redhat.com>
As per: ceph/ceph#26599, Beast is now the
default fronted for rados gateway.
Newly created cluster as of Nautilus will use it by default.
Re-added version of 03587352d5Resolves: #2707
Signed-off-by: Sébastien Han <seb@redhat.com>
Sometime the kube engine needs a bit of time to return the logs of a
given job and fails to read the stream.
Retrying up to detect the Ceph version seems reasonnable to
overcome this issue.
Fixes: https://github.com/rook/rook/issues/3227
Signed-off-by: Sébastien Han <seb@redhat.com>
The memory check was missing and will be trigger if resources limit are
configured for the rbdmirror pod.
Signed-off-by: Sébastien Han <seb@redhat.com>
This commit fixes the second test case where limit and request are
either identical or different but still we use limit as a value.
Signed-off-by: Sébastien Han <seb@redhat.com>
for the osd recovery case (DOWN->UP) log at a matching log level as the
message indicating the OSD went down.
fixes: #2904
Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
- Updated code to use deviceClass property when a pool is created for both "replicated" and "erasure code".
- Updated "ceph crush rule create-..." command to use "create-replicated" instead of "create-simple"
- Updated unit tests
- Updated (ceph-pool-crd.md) documentation to reflect the changes.
- Updated pending release notes.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
Sébastien Han
Noah Watkins
Umanga Chapagain
Dmitry Yusupov
Dmitry Yusupov
Travis Nielsen
Anton Skriptsov
Madhu Rajanna
Guy Margalit
xiaorui.zou
rohan47
John Mulligan
DL186038
Blaine Gardner
Christian Hüning
Santosh Pillai