If a volume is being attached, it should be verified that the volume
is safe to attach. The volume was assumed to be safe to attach if it was
for the same pod. But this was assuming the pod of the same name would be
on the same node. This is true for pods created from deployments, but not
for pods that are part of a stateful set. A stateful set will maintain the
pod name even as the pod is failed over to a new node. Therefore, the fencing
much check if the pod is from the same node before allowing the attach to
continue. Otherwise, we need to wait for the volume to be detached from the
other node.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The owner reference for the NFS custom resource was using the namespace
instead of the name. This causes the k8s garbage collector to remove the
nfs resources in some scenarios such as restarting etcd.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
This will be helpful to list the CSI drivers deployed
in the kubernetes cluster. if we have N number of
CSI drivers deployed in the cluster, the user can
query kubectl get csidriver and get the list of
registered CSI drivers and it requires kube 1.14+
and also the functionality is disabled by default
More info here: https://github.com/ceph/ceph-csi/pull/698
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
In Octopus, this commit
https://github.com/ceph/ceph/pull/30859/commits/c09b82a80a392ccd0da7677c7b424ce5cd3fa5d6
moved the quorum_status command to a socket call. Thus, the `ceph
quorum_status` command does not exist anymore, to use it we need to do
`ceph daemon mon.$ID quorum_status`. However, this command only works
locally so it won't work if we run it from the operator.
The `ceph mon_status` command is a general command that run from
anywhere and returns the mon map, let's use this one on the operator
code.
This is not a breaking change.
Signed-off-by: Sébastien Han <seb@redhat.com>
Even legacy osds were still running with this flag. In my previous patch
I was trying to minimize the impact of the change and was afraid the
config file was holding important information but that appears not to be
true.
The only option in the config could have the keyring path, but it's
passed in the CLI startup line.
Closes: https://github.com/rook/rook/issues/4063
Signed-off-by: Sébastien Han <seb@redhat.com>
The mgr modules take some time to run and have also been known
to hang. While the hang issues should have been resolved separately,
an improvement is still to allow the configuration to happen
in parallel for unrelated modules. All of the module configuration
will be completed before continuing with osd configuration
in order to keep the logging simpler.
Signed-off-by: travisn <tnielsen@redhat.com>
The pools always claimed to be watching all namespaces
when in fact only a single namespace is watched for pools.
Signed-off-by: travisn <tnielsen@redhat.com>
OSD on PVC does not upgrade when the user upgrades the ceph version on cluster-on-pvc yaml.
The solution incudes:
- upgrade osd prepare and daemon pods on upgrade
- skip c-v prepare if filesystem is already present on the pvc device.
- skip lvm release in case of upgrade.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
OSDs can be removed automatically with the current mechanism if a new
setting removeOSDsIfOutAndSafeToRemove is set to true. The default for
all new or upgraded clusters should be false.
Signed-off-by: rohan47 <rohgupta@redhat.com>
If mon deployments need to be created, the operator should
go ahead and create all of them instead of waiting for quorum
in between checking each mon. In particular, if a cluster is
being restored in a disaster recovery situation, none of the mons
deployments will exist. All of them should be created before
checking for quorum.
Signed-off-by: travisn <tnielsen@redhat.com>
Requests to enable mgr modules or call mgr modules should
timeout rather than hang. For example, if creating a self
signed cert request hangs, the mgr should continue with other
actions in the orchestration
Signed-off-by: travisn <tnielsen@redhat.com>
The command `fs new` already enable the application pool settings for
the cephfs pools so we don't need to run that command again.
Signed-off-by: Sébastien Han <seb@redhat.com>
When both PARTNAME and ID_PART_ENTRY_NAME are specified, rook takes
PARTNAME in account, overriding ID_PART_ENTRY_NAME. As PARTNAME is not
always updated by the kernel after sgdisk --change-name, it's better to
do the opposite. It also makes GetDevicePartitions coherent with
parsePartLabel (called by GetPartitionLabel).
Signed-off-by: Mikaël Cluseau <mikael.cluseau@gmail.com>
Checking for clean PGs is a recurring event when reconciling nodes
for upgrades and fencing. No need to log every request.
Signed-off-by: travisn <tnielsen@redhat.com>
Creating an object user was always waiting at least 15 seconds before
attempting to create the user. With this change the operator will check
immediately if the object store is already initialized instead
of waiting for the initial 15 seconds delay to elapse.
Signed-off-by: travisn <tnielsen@redhat.com>
The osd-prepare jobs should not inherit osd resources since this might
cause scheduling issues.
Instead, we can now use the new CRD property `prepareosd` in the resource section.
Refer to the documentation for more details.
Closes: https://github.com/rook/rook/issues/2502
Signed-off-by: Sébastien Han <seb@redhat.com>
- OSD on PVC provisioning fails if k8s version is less than 1.13.0
- updated document to reflect minimum k8s version required.
- Added check to skip OSD on pvc provisoning if the minimum k8s version requirement is not met.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
This issue is seen in brownfield cases in which max_bucket
size for the user is set to 0. In rgw, if the max_bucket is 0
then the user can create an unlimited no of buckets.
So for preventing users creating new buckets, set the value to
any negative value
Signed-off-by: Jiffin Tony Thottan <jthottan@redhat.com>
The current implementation was overriding any previous annotations set on
the object meta.
Moving the logic to its own method as well as adding unit tests.
Closes: https://github.com/rook/rook/issues/4106
Signed-off-by: Sébastien Han <seb@redhat.com>
The clusterdisruption starts up before the Ceph cluster is created and
during the initilization, it'll display invalid error messages such as:
2019-10-14T04:22:21.243679454Z 2019-10-14 04:22:21.243668 E | clusterdisruption-controller: could not check cluster health: failed to get status: exit status 1
2019-10-14T04:22:22.524222494Z 2019-10-14 04:22:22.524179 I | exec: Running command: ceph status --connect-timeout=15 --cluster=openshift-storage --conf=/var/lib/rook/openshift-storage/openshift-storage.config --keyring=/var/lib/rook/openshift-storage/client.admin.keyring --format json --out-file /tmp/605071670
2019-10-14T04:22:22.617370007Z 2019-10-14 04:22:22.617321 I | exec: Error initializing cluster client: ObjectNotFound('error calling conf_read_file',)
That error message is invalid since the cluster is not up and running
YET.
So let's simply log a message.
Signed-off-by: Sébastien Han <seb@redhat.com>
Following the Ceph OSD rework, my test environment regularly failed to
start OSDs, as they would start before the postStart lifecyle hook which
was chown'ing the log and data directories could finish. Pods failed to
start for an arbitrarily long time before starting successfully. This
was a manifestation of a race condition identified in Ceph monitors in
https://github.com/rook/rook/pull/3594#discussion_r312279176
Create a Ceph operator-level method to create an init container that
will chown the necessary directories for all Ceph daemons (any daemon
which runs as the ceph:ceph user-and-group). Also add this container to
expectations for Ceph pod templates in unit tests.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
Just as with the mon, mgr, mds, rgw, and rbd-mirror daemons, do not
generate a ceph.conf in an init container for directory-based OSDs.
Instead use the mon config database and the commandline to supply all
the needed arguments for running these OSDs.
Also generate a keyring secret to mount to OSD pods. Use this
secret for directory-based OSDs for now with the intention to
use this for other OSDs in the future.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
As with the rest of the Ceph daemons, use the new config patters for
dir-based OSDs. Do not generate a config to use in an init container;
instead use CLI flags and the mon config database.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
This is just one small unit of work that still results in usable
dir-based OSDs. The goal is to build on this to run dir-based OSDs using
the new config patterns used by the other Ceph daemons. This also
removes an unnecessary mount of the dataDirHostPath in directory-based
OSD containers.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
Log file path on host for OSDs was dataDirHostPath/log/<namespace>
instead of dataDirHostPath/<namespace>/log as with other daemons.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
Sébastien Han
Travis Nielsen
Dmitry Yusupov
Anton Skriptsov
Song Gao
Madhu Rajanna
Santosh Pillai
rohan47
Mikaël Cluseau
Owen Tuz
Jiffin Tony Thottan
Mateusz Los
Blaine Gardner