Update pkg/operator/ceph/cluster/osd/spec.go
Rook v1.1.2 uses a different command to start an OSD, and seems to
have problems when --osd-memory-target contains a number with a
decimal point.
Co-Authored-By: Sébastien Han <seb@redhat.com>
Signed-off-by: Erik Knudtson <eknudtson@discogsinc.com>
When osd nodes do not finish the prepare jobs, the user remove the
nodes manully and it causes nodes always is
'OrchestrationStatusStarting' status.
Try to remove them after completeProvisionTimeout.
Signed-off-by: James Lu <jamesluhz@gmail.com>
Add a finalizer to the CephCluster to check
the PV created by CSI so that we can block the
cluster deletion till the PVC are deleted.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
We don't need to force any Ceph config file for ceph-volume based OSDs
since the 'ceph-volume activate' command mount everything at start time
in the right place.
So now, OSD will keep running normally and they will also read back
normally the default location of the ceph configuration:
/etc/ceph/ceph.conf which will allow OSDs to read any configuration
overrides.
We cannot remove the generation of that file in /var/lib/rook/osd* since
we cannot tell at the stage that triggers the generation if we run
ceph-volume or not. So let's keep it around for some time.
If you are wondering how the OSD connects to the mons this is done via
the CEPH_ARGS env variable which was introduced in
https://github.com/rook/rook/pull/3894.
Closes: https://github.com/rook/rook/issues/3926
Signed-off-by: Sébastien Han <seb@redhat.com>
A job is started to detect the Ceph version. This job now allows setting
the node affinity and tolerations with the same setting that is
specified for the mon placement. No new placement spec is required, but
we will simply use the same spec that the mons use.
Signed-off-by: travisn <tnielsen@redhat.com>
Previously the debug flag was turned on for the 'activate' call only,
now 'prepare' will also take advantage of it.
Signed-off-by: Sébastien Han <seb@redhat.com>
When the flex driver is disabled, the check for
volume attachments will fail and the finalizer
is never removed. To avoid this, just log the
failure to list volumes and remove the
finalizer anyway.
Resolves#3912
Signed-off-by: Kristoffer Grönlund <kgronlund@suse.com>
Encrypted OSDs are failing because ceph-volume is unable to determine
how to contact the monitors. Setting the CEPH_CONF env variable for OSD
pods fixes this issue. (#3846)
Signed-off-by: Michael Vollman <michael.b.vollman@gmail.com>
When running osds on pvc, the lvm is still holding the device when the cluster is deleted. As a result the PVs go into `failed` state
when the cluster is deleted. On deactivating the device, the 'failed' pvs get deleted as well. Solution involves:
- Running `lvchange -an <vg group name>` to deactivate the device.
- This command runs after 'osd prepare' completes.
- This command also runs after the 'osd daemon pod' is deleted or if the ceph-osd process terminates for some reasons.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
The node selector for running OSDs on PVCs was using the node name
rather than the node's hostname. In clusters where the node name
is different from the hostname this would cause the osd daemons
to be in pending state indefinitely. Now the node selector will
use the hostname as expected.
Signed-off-by: travisn <tnielsen@redhat.com>
On rare occasion, the ceph checks might not be 100% correct and the user
might decide to force an upgrade anyway.
Correctly, if we are in an upgrade, we perform `ok-to-stop` check for
each daemon, if one of them fails, we retry and eventually fail.
They are corner-cases where users would like to force the upgrade anyway.
Enhance the new CR property: "skipUpgradeChecks: true".
Closes: https://github.com/rook/rook/issues/3872
Signed-off-by: Sébastien Han <seb@redhat.com>
The CSI secret was only being created for new clusters.
Upgraded clusters did not have the secret generated. To support
upgraded clusters to use CSI, we always ensure the csi secret
exists when the orchestration runs.
Signed-off-by: travisn <tnielsen@redhat.com>
Several commands were not using the cli finalizer function which builds
up the right argument to run a ceph command. This was working on the
operator since it used to have an /etc/ceph/ceph.conf file so we could
ran the command without specifying the ceph conf and key paths.
However, we recently removed that and it silently broke this "already"
broken code :).
Signed-off-by: Sébastien Han <seb@redhat.com>
Using port 8443 / dashboardHTTPS when not running HTTPS
is confusing, so use the dashboardHTTP port (7000 by
default) instead.
Also don't expose the dashboard port as "https-dashboard"
when not using https, instead expose as "dashboard".
Resolves#3810
Signed-off-by: Kristoffer Grönlund <kgronlund@suse.com>
Now that the pg_autoscaler is on by default, it is "normal" (and okay) to
have a small number of PGs in the cluster if the overall cluster usage is
also low. This setting just results in a health warning out of the box
when you create a pool and haven't written any data yet.
fyi: ceph upstream https://github.com/ceph/ceph/pull/30352
Signed-off-by: Sébastien Han <seb@redhat.com>
This adds a list of unusable directories paths to the Ceph Cluster CRD
doc page and the Ceph operator validation code. Reason for adding these
directories is that they are causing errors during Ceph OSD deployment.
Example: When an user wants to use `/rook` as a directory for an OSD,
the cluster deployment fails because the `mountPath`s must be unique:
```
Job.batch "rook-ceph-osd-prepare-NODE" is invalid: spec.template.spec.containers[1].volumeMounts[4].mountPath: Invalid value: "/rook": must be unique
```
Signed-off-by: Alexander Trost <galexrt@googlemail.com>
If a Ceph container image is specified which has a entrypoint then the
Init container for osd prepare job blkdevmapper will fail. So set the
Command property.
Signed-off-by: rohan47 <rohgupta@redhat.com>
Starting with Nautilus 14.2.1 the dashboard SSL port is readed from <<mgr/dashboard/ssl_server_port>> ceph setting.
Previous versions use the port specified in <<mgr/dashboard/server_port>>
Addressed @travisn suggestion about use of <ssl> attribute in conditional.
This suggestion drove me to change the type of the SSl attribute (from *bool to bool) in the DashboardSpec Struct.
This makes code more clean/safe and make sure that the SSL atribute has always a meaningful value.
Now if you want to run the <secure dashboard> you need to set explicitly SSL to true, as is demonstrated in unit tests.
[x] Code generation (make codegen) has been run to update object specifications
Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
We can now enable via the cluster CR any manager module with a
setting under the new mgr element
mgr:
modules:
- name: pg_autoscaler
enabled: true
Co-authored-by: Sébastien Han <seb@redhat.com>
Co-authored-by: travisn <tnielsen@redhat.com>
Signed-off-by: travisn <tnielsen@redhat.com>
Some users have reported issues running Ceph OSDs in environments
without beefy systems. After consulting with Ceph experts, we believe
2Gb is still a safe minimum for Ceph for users with more modest systems
that won't result in disfunction in Ceph.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
After further consideration and working with the feature for a few
weeks, the `configOverride` functionality seems to be more complexity
than is worth it for the expected benefit. Users still have the most
preferred option of setting configs in Ceph's config database using
`ceph cofig set` or the dashboard, and the override ConfigMap still
exists for fallback scenarios. The config database can still be used
by Rook for setting necessary configs, but the user has no way of
telling Rook to do so.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
OSD pods when running on PVC fail randomly due to some race condition:
- OSD daemon pod was activating the volume group using vgchange -an without any volume group name.
- This caused the pods to fail randomly when number of OSDs is more, say greater than 10.
- This fix adds Volume group name while activating/deactivating volume groups to fix this
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
Reduce the duplication of detecting the version of the external cluster
by refactoring to a single method that retrieves and validates the
version
Signed-off-by: travisn <tnielsen@redhat.com>
Add various upgrade tests so that the local cluster does not fail if
an upgrade is available on the external cluster.
Also, add better upgrades handling on update and on add. We now rely as
much as possible on a unique function that works both local and external
cluster 'detectAndValidateCephVersion()'. So the code will act the same
way wether a CR update is triggered or if the operator is restarted.
The health check now reports any topology changes on the external
cluster such as odd number of mons and low number. It will also detect
if the external cluster has been upgraded and will log that information,
helping the admin in his/her decision to trigger an image upgrade.
Finally, we prevent the creation of new CRs if the external cluster
version has a higher major number than the local one.
Closes: https://github.com/rook/rook/issues/3731
Signed-off-by: Sébastien Han <seb@redhat.com>
The log messages for orchestrations being skipped due to nodes still
being schedulable or unschedulable is filling the logs and not necessary.
The log message for reconciling is now debug, and also fixed another
formatting log message.
Signed-off-by: travisn <tnielsen@redhat.com>
Erik KnudtsonandSébastien Han
James Lu
Travis Nielsen
Madhu Rajanna
Kristoffer Grönlund
Michael Vollman
Santosh Pillai
Blaine Gardner
Alexander Trost
rohan47
Juan Miguel Olmo Martínez
Rohan CJ