From d3b9b74165de1471b7841360e21ff37409b05add Mon Sep 17 00:00:00 2001 From: Blaine Gardner Date: Thu, 12 Dec 2024 11:38:24 -0700 Subject: [PATCH] doc: design ceph config passthru for object store Add a design for CephObjectStore to allow users to pass through Ceph configs (e.g., `rgw_enable_apis`, `debug_rgw`) to the object store. This design will help users set configs that they want for their environments that aren't covered under Rook's umbrella of managed configs. This will benefit users by allowing them to make use of rare RGW features that Rook devs don't have ability to prioritize, as well as newly-released RGW features that Rook devs haven't had time to plan API interfaces for. This frees up Rook developers to focus effort on implementing APIs that have integration benefits with Kubernetes or other resources and less effort on simple configs. Signed-off-by: Blaine Gardner --- design/ceph/object/store.md | 80 +++++++++++++++++++++++++++++++++++++ 1 file changed, 80 insertions(+) diff --git a/design/ceph/object/store.md b/design/ceph/object/store.md index 2fcb8dc88..129b6ac91 100644 --- a/design/ceph/object/store.md +++ b/design/ceph/object/store.md @@ -712,3 +712,83 @@ For external CephCephObjectStores (i.e., when `spec.gateway.externalRgwEndpoints vhost-style addressing should be configured on the host cluster, and `hosting.dnsNames` is irrelevant. The default `advertiseEndpoint` for external CephObjectStores is the first entry in the `spec.gateway.externalRgwEndpoints` list, which users should be able to override if desired. + +### Advanced user configuration + +RGW is a very active project with many existing features and regular new feature additions. Rook +maintainers have a hard time catching up with RGW's latest features. In an effort to help users who +want access to rare and new features, Rook can allow users to set arbitrary Ceph configs in their +CephObjectStore CR. RGW config reference: https://docs.ceph.com/en/reef/radosgw/config-ref/ + +This is an advanced feature. It allows users to modify many untested aspects of the RGW and object +store. Any user configs that break the object store should be documented to be a responsibility of +the user. + +```yaml +spec: + metadataPool: # ... + dataPool: # ... + gateway: + rgwConfig: + conf-name: value + other_conf: value + debug_rgw: "20" + rgwCommandFlags: + flag-name: value + other_flag: value + rgw_enable_apis: "s3, s3website, swift, swift_auth, sts, iam, notifications" +``` + +CephCluster has a configuration section called `config` that sets mon config store values. `config` +may be too vague in the context of the CephObjectStore CR. `rgwConfig` is a clearer naming for users +and gives context that Ceph RGW configs are expected. A search engine search for "rgw config" also +yields the correct Ceph documentation for users, which suggest the naming is apt. + +CephObjectStore will also add an `rgwCommandFlags` variable to allow users to set CLI flags to the +`radosgw` deployment. Changes to these parameters will result in RGW daemon restarts, which can be +desired behavior for config values like `rgw_enable_apis`. In testing, the `radosgw` command process +supports adding multiple flags, and the last flag sequentially is used as the input source. + +Rook users can already use the `CephCluster.spec.config` section to control individual +CephObjectStore configurations today. However, this requires users to know the name of the Ceph +client user for each CephObjectStore, and it only allows setting object store fields after the +object store is provisioned. Adding this feature improves config override usability for object store +users, and it keeps config overrides in the same place as the resource. This is a clear quality of +life improvement for users managing object stores. + +Rook will apply `rgwConfig` values to the mon config store after its own values are set. Similarly, +Rook will append `rgwCommandFlags` to the CLI flags it sets normally. This allows users to override +anything Rook is doing to configure the RGW. This tradeoff favors user flexibility at the risk of +less advanced users having more ability to break their RGW deployments. Rook can't make guarantees +about data security when configs are overridden. + +For debugging, Rook should log when it overrides its own ceph mon store config with a user setting. +This is not easily feasible when Rook applies CLI flags to the RGW deployment, but users can see +when when their values conflict by inspecting the RGW deployment; this is sufficient for debugging. + +Rook will apply each configuration under the client scope of the shared RGW client. +i.e., `client.rook-ceph-rgw-.a`. This will also help ensure that configs changed don't +affect other Rook-managed Ceph daemons, critically mons, osds, and mds. + +Configs set via `rgwConfig` will be equivalent to a Ceph config section like the following ini. +Config keys can use spaces, hyphens, or underscores. CLI flags must use hyphens, and Rook can +auto-convert spaces/underscores for ease of use. + +```ini +[client.rook-ceph-rgw-.a] +conf_name = value +other_conf = value +debug_rgw = "20" +``` + +Because the subject of the configs are limited to RGWs using the +`client.rook-ceph-rgw-.a` ID, risk of affecting other Ceph components is low. Rook has +the option to allow-/deny-list the configs users are able to set, but this limits user flexibility. +Since it will be documented that any breakage or bugs related to options set via this mechanism is +their responsibility, we can be further sure Rook maintainers won't be obligated to stress +themselves for abuse or misuse of this feature. + +Rook will also not perform validation to check if CephObjectStore configs conflict with CephCluster +`config` options. Users should refrain from setting `client.rook-ceph-rgw-*` configs in the +CephCluster when this feature is used. Documentation should note this for users. Rook can revisit +this decision in the future to add cross-CRD validation if it becomes a strong user desire/need.