The mgr modules take some time to run and have also been known
to hang. While the hang issues should have been resolved separately,
an improvement is still to allow the configuration to happen
in parallel for unrelated modules. All of the module configuration
will be completed before continuing with osd configuration
in order to keep the logging simpler.
Signed-off-by: travisn <tnielsen@redhat.com>
The pools always claimed to be watching all namespaces
when in fact only a single namespace is watched for pools.
Signed-off-by: travisn <tnielsen@redhat.com>
OSD on PVC does not upgrade when the user upgrades the ceph version on cluster-on-pvc yaml.
The solution incudes:
- upgrade osd prepare and daemon pods on upgrade
- skip c-v prepare if filesystem is already present on the pvc device.
- skip lvm release in case of upgrade.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
OSDs can be removed automatically with the current mechanism if a new
setting removeOSDsIfOutAndSafeToRemove is set to true. The default for
all new or upgraded clusters should be false.
Signed-off-by: rohan47 <rohgupta@redhat.com>
If mon deployments need to be created, the operator should
go ahead and create all of them instead of waiting for quorum
in between checking each mon. In particular, if a cluster is
being restored in a disaster recovery situation, none of the mons
deployments will exist. All of them should be created before
checking for quorum.
Signed-off-by: travisn <tnielsen@redhat.com>
Requests to enable mgr modules or call mgr modules should
timeout rather than hang. For example, if creating a self
signed cert request hangs, the mgr should continue with other
actions in the orchestration
Signed-off-by: travisn <tnielsen@redhat.com>
The command `fs new` already enable the application pool settings for
the cephfs pools so we don't need to run that command again.
Signed-off-by: Sébastien Han <seb@redhat.com>
When both PARTNAME and ID_PART_ENTRY_NAME are specified, rook takes
PARTNAME in account, overriding ID_PART_ENTRY_NAME. As PARTNAME is not
always updated by the kernel after sgdisk --change-name, it's better to
do the opposite. It also makes GetDevicePartitions coherent with
parsePartLabel (called by GetPartitionLabel).
Signed-off-by: Mikaël Cluseau <mikael.cluseau@gmail.com>
Checking for clean PGs is a recurring event when reconciling nodes
for upgrades and fencing. No need to log every request.
Signed-off-by: travisn <tnielsen@redhat.com>
Creating an object user was always waiting at least 15 seconds before
attempting to create the user. With this change the operator will check
immediately if the object store is already initialized instead
of waiting for the initial 15 seconds delay to elapse.
Signed-off-by: travisn <tnielsen@redhat.com>
The osd-prepare jobs should not inherit osd resources since this might
cause scheduling issues.
Instead, we can now use the new CRD property `prepareosd` in the resource section.
Refer to the documentation for more details.
Closes: https://github.com/rook/rook/issues/2502
Signed-off-by: Sébastien Han <seb@redhat.com>
- OSD on PVC provisioning fails if k8s version is less than 1.13.0
- updated document to reflect minimum k8s version required.
- Added check to skip OSD on pvc provisoning if the minimum k8s version requirement is not met.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
This issue is seen in brownfield cases in which max_bucket
size for the user is set to 0. In rgw, if the max_bucket is 0
then the user can create an unlimited no of buckets.
So for preventing users creating new buckets, set the value to
any negative value
Signed-off-by: Jiffin Tony Thottan <jthottan@redhat.com>
The current implementation was overriding any previous annotations set on
the object meta.
Moving the logic to its own method as well as adding unit tests.
Closes: https://github.com/rook/rook/issues/4106
Signed-off-by: Sébastien Han <seb@redhat.com>
The clusterdisruption starts up before the Ceph cluster is created and
during the initilization, it'll display invalid error messages such as:
2019-10-14T04:22:21.243679454Z 2019-10-14 04:22:21.243668 E | clusterdisruption-controller: could not check cluster health: failed to get status: exit status 1
2019-10-14T04:22:22.524222494Z 2019-10-14 04:22:22.524179 I | exec: Running command: ceph status --connect-timeout=15 --cluster=openshift-storage --conf=/var/lib/rook/openshift-storage/openshift-storage.config --keyring=/var/lib/rook/openshift-storage/client.admin.keyring --format json --out-file /tmp/605071670
2019-10-14T04:22:22.617370007Z 2019-10-14 04:22:22.617321 I | exec: Error initializing cluster client: ObjectNotFound('error calling conf_read_file',)
That error message is invalid since the cluster is not up and running
YET.
So let's simply log a message.
Signed-off-by: Sébastien Han <seb@redhat.com>
Following the Ceph OSD rework, my test environment regularly failed to
start OSDs, as they would start before the postStart lifecyle hook which
was chown'ing the log and data directories could finish. Pods failed to
start for an arbitrarily long time before starting successfully. This
was a manifestation of a race condition identified in Ceph monitors in
https://github.com/rook/rook/pull/3594#discussion_r312279176
Create a Ceph operator-level method to create an init container that
will chown the necessary directories for all Ceph daemons (any daemon
which runs as the ceph:ceph user-and-group). Also add this container to
expectations for Ceph pod templates in unit tests.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
Just as with the mon, mgr, mds, rgw, and rbd-mirror daemons, do not
generate a ceph.conf in an init container for directory-based OSDs.
Instead use the mon config database and the commandline to supply all
the needed arguments for running these OSDs.
Also generate a keyring secret to mount to OSD pods. Use this
secret for directory-based OSDs for now with the intention to
use this for other OSDs in the future.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
As with the rest of the Ceph daemons, use the new config patters for
dir-based OSDs. Do not generate a config to use in an init container;
instead use CLI flags and the mon config database.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
This is just one small unit of work that still results in usable
dir-based OSDs. The goal is to build on this to run dir-based OSDs using
the new config patterns used by the other Ceph daemons. This also
removes an unnecessary mount of the dataDirHostPath in directory-based
OSD containers.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
Log file path on host for OSDs was dataDirHostPath/log/<namespace>
instead of dataDirHostPath/<namespace>/log as with other daemons.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
PopulateDeviceInfo() in pkg/clusterd/disk.go returns nil as *sys.LocalDisk
if an error has occurred. This causes a nil pointer exception at
getAvailableDevices() in pkg/daemon/ceph/osd/daemon.go. At least the returned
value should be checked at the caller.
I added an explicit error to the return value of PopulateDeviceInfo(). This
naturally revokes an error check at the caller.
I modified PopulateDeviceUdevInfo() in the same file too.
Signed-off-by: morimoto-cybozu <kenji_morimoto@cybozu.co.jp>
The cluster finalizer in some scenarios was not being removed
during cluster removal. Specifically, if the cluster CR had been
modified, the operator would always fail to remove the finalizer.
This could occur when the CR status is updated around the same time
that the cluster is deleted. Therefore, we need to remove the finalizer
from a freshly retrieved instance of the cluster.
Signed-off-by: travisn <tnielsen@redhat.com>
The owner references should be set based on the name, otherwise the
garbage collector will delete the resources at some point.
These references to yugabyte, minio, and cockroach were
missed in the original fix for these references.
Signed-off-by: travisn <tnielsen@redhat.com>
The resources for rgw, mds, and nfs were being cleaned up explicitly when
the CR was deleted. This is legacy from before the owner references were
set on the CRs. Now the owner references are set to the appropriate CRs so
the resources will be cleaned up upon deletion of the specific CR
rather than the cluster CR.
Signed-off-by: travisn <tnielsen@redhat.com>
The full path to vgchange should not be necessary. For some
downstream releases the path may even be in a different path.
Therefore we require that vgchange be in the path.
Signed-off-by: travisn <tnielsen@redhat.com>
The access of user will be revoked by Droppolicy(). And user unlink
need to perform only if it is the owner. For Delete() it makes sense
to call user unlink, but it is not required for Revoke()
Signed-off-by: Jiffin Tony Thottan <jthottan@redhat.com>
Following corrections are needed for bucket policy to work properly
* Change Principal:{"AWS":["<username>"]} to Principal:{"AWS":["arn:aws:iam:::user/<username>"]}
* Add missing "PutObject" policy in Action[]
* Add <bucket/*> to Resource[] for accessing contents inside the bucket
Signed-off-by: Jiffin Tony Thottan <jthottan@redhat.com>
We now differentiate the cases where:
* we only consume the external cluster
* we consume the external cluster as well as creating stateless
resources in Kubernetes (bootstrap mds,rgw, nfs)
This is mostly controlled via the image property spec. If not defined,
not extra CRs won't be able to be created.
Now the external cluster feature supports Ceph cluster as of Luminous 12.2.
Signed-off-by: Sébastien Han <seb@redhat.com>
Update pkg/operator/ceph/cluster/osd/spec.go
Rook v1.1.2 uses a different command to start an OSD, and seems to
have problems when --osd-memory-target contains a number with a
decimal point.
Co-Authored-By: Sébastien Han <seb@redhat.com>
Signed-off-by: Erik Knudtson <eknudtson@discogsinc.com>
When osd nodes do not finish the prepare jobs, the user remove the
nodes manully and it causes nodes always is
'OrchestrationStatusStarting' status.
Try to remove them after completeProvisionTimeout.
Signed-off-by: James Lu <jamesluhz@gmail.com>
travisn
Santosh Pillai
rohan47
Sébastien Han
Mikaël Cluseau
Owen Tuz
Anton Skriptsov
Jiffin Tony Thottan
Mateusz Los
Blaine Gardner
morimoto-cybozu
Rohan CJ
Juan Miguel Olmo Martínez
Erik KnudtsonandSébastien Han
James Lu