Creating an object user was always waiting at least 15 seconds before
attempting to create the user. With this change the operator will check
immediately if the object store is already initialized instead
of waiting for the initial 15 seconds delay to elapse.
Signed-off-by: travisn <tnielsen@redhat.com>
Following the Ceph OSD rework, my test environment regularly failed to
start OSDs, as they would start before the postStart lifecyle hook which
was chown'ing the log and data directories could finish. Pods failed to
start for an arbitrarily long time before starting successfully. This
was a manifestation of a race condition identified in Ceph monitors in
https://github.com/rook/rook/pull/3594#discussion_r312279176
Create a Ceph operator-level method to create an init container that
will chown the necessary directories for all Ceph daemons (any daemon
which runs as the ceph:ceph user-and-group). Also add this container to
expectations for Ceph pod templates in unit tests.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
The resources for rgw, mds, and nfs were being cleaned up explicitly when
the CR was deleted. This is legacy from before the owner references were
set on the CRs. Now the owner references are set to the appropriate CRs so
the resources will be cleaned up upon deletion of the specific CR
rather than the cluster CR.
Signed-off-by: travisn <tnielsen@redhat.com>
The access of user will be revoked by Droppolicy(). And user unlink
need to perform only if it is the owner. For Delete() it makes sense
to call user unlink, but it is not required for Revoke()
Signed-off-by: Jiffin Tony Thottan <jthottan@redhat.com>
Following corrections are needed for bucket policy to work properly
* Change Principal:{"AWS":["<username>"]} to Principal:{"AWS":["arn:aws:iam:::user/<username>"]}
* Add missing "PutObject" policy in Action[]
* Add <bucket/*> to Resource[] for accessing contents inside the bucket
Signed-off-by: Jiffin Tony Thottan <jthottan@redhat.com>
We now differentiate the cases where:
* we only consume the external cluster
* we consume the external cluster as well as creating stateless
resources in Kubernetes (bootstrap mds,rgw, nfs)
This is mostly controlled via the image property spec. If not defined,
not extra CRs won't be able to be created.
Now the external cluster feature supports Ceph cluster as of Luminous 12.2.
Signed-off-by: Sébastien Han <seb@redhat.com>
A new CRD property `PreservePoolsOnDelete` has been added to Filesystem(fs) and
Object Store(os) resources in order to increase protection against data loss.
If it is set to `true`, associated pools won't be deleted when the main
resource(fs/os) is deleted. Creating again the deleted fs/os with the same name
will reuse the preserved pools.
Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
currently we are not checking the pool is empty or
not before deleting, This PR adds a check to check
if any images/snapshots using the pool, if yes it will
not delete the pool
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
On rare occasion, the ceph checks might not be 100% correct and the user
might decide to force an upgrade anyway.
Correctly, if we are in an upgrade, we perform `ok-to-stop` check for
each daemon, if one of them fails, we retry and eventually fail.
They are corner-cases where users would like to force the upgrade anyway.
Enhance the new CR property: "skipUpgradeChecks: true".
Closes: https://github.com/rook/rook/issues/3872
Signed-off-by: Sébastien Han <seb@redhat.com>
Reduce the duplication of detecting the version of the external cluster
by refactoring to a single method that retrieves and validates the
version
Signed-off-by: travisn <tnielsen@redhat.com>
Add various upgrade tests so that the local cluster does not fail if
an upgrade is available on the external cluster.
Also, add better upgrades handling on update and on add. We now rely as
much as possible on a unique function that works both local and external
cluster 'detectAndValidateCephVersion()'. So the code will act the same
way wether a CR update is triggered or if the operator is restarted.
The health check now reports any topology changes on the external
cluster such as odd number of mons and low number. It will also detect
if the external cluster has been upgraded and will log that information,
helping the admin in his/her decision to trigger an image upgrade.
Finally, we prevent the creation of new CRs if the external cluster
version has a higher major number than the local one.
Closes: https://github.com/rook/rook/issues/3731
Signed-off-by: Sébastien Han <seb@redhat.com>
Allow users to specify overrides in the CephCluster CRD. The override
ConfigMap still exists for emergency situations and is mounted into
daemon pods directly instead of being merged into a Rook-created config
file.
Rook Ceph no longer uses a config file for managing daemons with the
exception of the OSDs which still generate a config in an init
container.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
From now on, Rook will only perform check before upgrades when there is
an actual upgrade. So if the Ceph image changed and a new version is
desired Rook will go through all the daemons and update them one by one
and perform checks in between.
Closes: https://github.com/rook/rook/issues/3583
Signed-off-by: Sébastien Han <seb@redhat.com>
creating an objectstore user requires that the objectstore is
created first. This commits adds changes to wait for the object
store to be created before the objectstore user can be created.
It'll retry every 15sec for 5mins.
Signed-off-by: Umanga Chapagain <chapagainumanga@gmail.com>
ceph: Use the new NetworkSpec api
ceph: Change all tests to use the NetworkSpec
ceph: Remove TypeMeta field from NetworkSpec
ceph: Change IsHost network func signature
Signed-off-by: giovanism <giovanism@outlook.co.id>
We can now deploy all the other Rook's CRDs. They will be deployed in
Kubernetes but will consume the external cluster storage.
Signed-off-by: Sébastien Han <seb@redhat.com>
When configuring an external cluster the orchestration of discover and all the ceph
daemons will be skipped. The operator will still watch for the creation of
crds for this namespace. When a filesystem, object, object user, or
ganesha crd are created, the operator will simply print an error
to the log.
Signed-off-by: travisn <tnielsen@redhat.com>
Signed-off-by: Sébastien Han <seb@redhat.com>
When configuring an external cluster the orchestration of discover and all the ceph
daemons will be skipped. The operator will still watch for the creation of
crds for this namespace. When a filesystem, object, object user, or
ganesha crd are created, the operator will simply print an error
to the log.
Signed-off-by: travisn <tnielsen@redhat.com>
Signed-off-by: Sébastien Han <seb@redhat.com>
Before each pod starts, the daemon changes the ownership of
/var/log/ceph which is bindmounted on a hostPath.
On OpenShift envionments, the container must be privileged in order to
successfully change the owernship.
Closes: https://github.com/rook/rook/issues/3485
Signed-off-by: Sébastien Han <seb@redhat.com>
At the beginning of every container startup we now change the ownership
of every known ceph directories to ceph:ceph.
This will allow daemons to start properly as well as allowing daemon to
writes their logs on files.
This is needed because ceph daemon now run as 'ceph' instead of 'root'
Signed-off-by: Sébastien Han <seb@redhat.com>
We do not determine the daemon type and name by string parsing anymore
but passing that info directly in the update call, it's more reliable.
Also removing a bunch of unused functions.
Signed-off-by: Sébastien Han <seb@redhat.com>
Do not consider clusterInfo Ceph version since it's the one picked by
the spec image.
The CephVersion provided here needs to be the current running version
of Ceph for when it is later passed to OkToStop and checked with
IsAtLeast. If you are upgrading from a pre 14.2.1 version to a
version 14.2.1 or newer, IsAtLeast will incorrectly return true and
then fail trying to run mon ok-to-stop on a mon pod running a pre 14.2.1 version.
Signed-off-by: Sébastien Han <seb@redhat.com>
This method does not start anything but creates the deployment. So let's
rename it to createDeployment() instead.
Signed-off-by: Sébastien Han <seb@redhat.com>
rgw wasn't using UpdateDeploymentAndWait() to update its deployment
configuration. We now use it so that it's consistent with the other
daemons configuration.
Signed-off-by: Sébastien Han <seb@redhat.com>
In the 0.9 release rook converted the v1beta1 CRD resources
to v1 resources. This conversion code is no longer necessary
as we will be using v1 resources going forward.
The code paths will not be triggered anymore, thus
removing the dead code.
Signed-off-by: travisn <tnielsen@redhat.com>
The beast fronted configuration for rgw is quite different than civetweb
so we adapted the construct of the arg line based on that configuration
requirement.
Closes: https://github.com/rook/rook/issues/3411
Signed-off-by: Sébastien Han <seb@redhat.com>
The check for whether to set the OwnerReferences is no longer
necessary. When passing the correct value to the apiVersion,
the OwnerReferences can be successfully set for OpenShift.
Signed-off-by: travisn <tnielsen@redhat.com>
This commit does multiple things:
* remove support for AllNodes where we would deploy one rgw per node on
all the nodes.
* a transition path is implemented in the code so that if someone has an
existing deployment, daemonsets will be removed and replaced by an
deployments.
* when using "instances", each rgw deployed has its own key which makes
Ceph reporting the exact number of rgw running, see:
```
[root@rook-ceph-operator-775cf575c5-bh4sr /]# ceph -s
cluster:
id: 611fcf39-0669-4864-9a12-debb35c0397a
health: HEALTH_OK
services:
mon: 3 daemons, quorum a,b,c (age 12h)
mgr: a(active, since 12h)
osd: 3 osds: 3 up (since 12h), 3 in (since 12h)
rgw: 3 daemons active (my.store.a, my.store.b, my.store.c)
data:
pools: 6 pools, 600 pgs
objects: 235 objects, 3.8 KiB
usage: 3.0 GiB used, 84 GiB / 87 GiB avail
pgs: 600 active+clean
```
Closes: https://github.com/rook/rook/issues/2474, https://github.com/rook/rook/issues/2957 and https://github.com/rook/rook/issues/3245
Signed-off-by: Sébastien Han <seb@redhat.com>
As per: ceph/ceph#26599, Beast is now the
default fronted for rados gateway.
Newly created cluster as of Nautilus will use it by default.
Re-added version of 03587352d5Resolves: #2707
Signed-off-by: Sébastien Han <seb@redhat.com>
- Updated code to use deviceClass property when a pool is created for both "replicated" and "erasure code".
- Updated "ceph crush rule create-..." command to use "create-replicated" instead of "create-simple"
- Updated unit tests
- Updated (ceph-pool-crd.md) documentation to reflect the changes.
- Updated pending release notes.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
The CephCluster CR contains settings that are needed by other
CRs to configure the Ceph daemons. When the CephCluster CR
is updated, the updates will now be passed on to each of the
CR controllers to ensure the daemons are updated properly
without requiring an operator restart.
When calling the controllers from another controller,
we ensure that only a single goroutine is handling
CRs at any given time to prevent contention across
multiple CRs of the same type.
Signed-off-by: travisn <tnielsen@redhat.com>
Add a 'ceph-version' label to application controllers in the same manner
as the prior 'rook-version' label to help with upgrades.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
Allow the user to set annotations in the CRDs of the following operators which will then be set on
the resulting Pods (Deployments, StatefulSets and so on).
Operators:
* Cassandra
* Ceph
* CockroachDB
* EdgeFS
* Minio
* NFS
The Annotations related structures have been added to the
rook.io/v1alpha2 pkg.
Signed-off-by: Alexander Trost <galexrt@googlemail.com>
It's quite convinient to expose /var/log/ceph so that we can decide to
activate logs locally on the machine and see what's going on.
This is only a placeholder when a daemon is stuck crashlooping and we
want to allow administrator to gather log files.
We still do not log on file but this can be activated via a config
option passed to the centralized config option store.
For some daemons, which typically do not store any data (rgw, rbd-mirror
and mds) we had to propagate dataDirHostPath from the cluster spec to
each creation call so that the bindmount can happen.
Fixes: https://github.com/rook/rook/issues/2881
Signed-off-by: Sébastien Han <seb@redhat.com>
Add a `rook-version` label to all controller resources used by Ceph:
deployments, daemonsets, and jobs. Labels are added to controller
resources only and not to the pod templates within because changing pod
templates causes updates due to the label change. The controller
resources themselves are not updated with the label addition and
therefore don't cause an unnecessary upgrade.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
All usages of k8s go client are now also using the versioned `AppsV1() `
call for the client.
Updated MySQL and Wordpress, and Kube Registy examples to use apps/v1
Deployments.
Signed-off-by: Alexander Trost <galexrt@googlemail.com>
When deleting a CephObjectStoreUser the deleteUser function
attempts to delete the stored secret using the wrong name. This
patch updates deleteUser to use the same fmt.Sprintf call used in
the createUser function.
Updated CI to delete the correct CRD and check the secret is deleted.
Signed-off-by: Keith Schincke <keith.schincke@gmail.com>
Upgrading a Ceph objectstore was failing the upgrade test because the
existing method for handling the case where update errors occurred
because labels and label selectors have changed was not robust enough.
Make this more robust by attempting delete-and-create if a normal update
does not work on an rgw deployment.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
The basic idea here is to replace the release name in the ClusterInfo
object with the fine grained version information queried at runtime.
This patch also removes the Name field from the cluster spec, which was
also runtime determined and was redundant with ClusterInfo relase name.
Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
If someone sets limits to pod, we want to ensure the possible
experience, so we want to make sure that people do not configure
inapropriate values for certain daemons.
We decide to fail if the memory.limit is too low.
Signed-off-by: Sébastien Han <seb@redhat.com>
In the form of mon, mgr, mds, and rgw, convert the rbd-mirror to be
configured completely from the operator. This is a straightforward
conversion with one functional addition: the rbd-mirror daemon stores
*no* data and has no default data dir, so the concept of a `Dataless`
daemon is here introduced.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
Configure the Ceph rgw daemon completely from the operator a la the
recent changes to the Ceph mon, mgr, and mds operators.
Create the rgw deployment or daemonset first, and then create the
keyring secret for the object store with its owner reference as the
corresponding deployment or daemonset. When the replication controller
is deleted, the secret is also deleted.
The RGW's mime.types file is now stored in a configmap with a different
file created for each object store. This is primarily just a means to
get the mime.types file into the rgw pod, but the added benefit is that
the administrator can modify the configmap, which could reduce
susceptibility to file type execution vulnerabilities (worst case).
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
travisn
Owen Tuz
Blaine Gardner
Jiffin Tony Thottan
Sébastien Han
Juan Miguel Olmo Martínez
Madhu Rajanna
Umanga Chapagain
giovanism
jiffin
Santosh Pillai
Alexander Trost
Keith Schincke
Noah Watkins