From 7887f03c5c29aa4e94acf181c0cb24f461dcecd8 Mon Sep 17 00:00:00 2001 From: Joshua Hoblitt Date: Wed, 12 Mar 2025 10:24:44 -0700 Subject: [PATCH] object: add CephBucketTopic .spec.endpoint.kafka.{user,password}SecretRef userSecretRef and passwordSecretRef fields are added to allow the Kafka endpoint username and password to be supplied from a Kubernetes Secret, rather than exposed as plaintext as part of the endpoint URI. If the endpoint URI has HTTP basic auth user-id and user-pass components, they are overridden by userSecretRef and passwordSecretRef. Squid added bucket topic attributes for configuring the user-name and password for pushing notifications to Kafka as an alternative to encoding credentials into the URI. However, these attributes are not supported under Reef. Thus, this initial implementation relies on URI mangling for compatible with both Reef and Squid. Future work could using Ceph version detection and switch to the new attributes to be used with Squid and/or the implementation could be converted exclusively to use the new attributes once Rook has dropped support for Reef. Signed-off-by: Joshua Hoblitt --- Documentation/CRDs/specification.md | 43 +- .../ceph-object-bucket-notifications.md | 57 +- .../charts/rook-ceph/templates/resources.yaml | 64 ++ deploy/examples/crds.yaml | 64 ++ go.mod | 17 +- go.sum | 14 +- pkg/apis/ceph.rook.io/v1/types.go | 9 + .../ceph.rook.io/v1/zz_generated.deepcopy.go | 17 +- pkg/operator/ceph/object/topic/controller.go | 140 +++- .../ceph/object/topic/controller_test.go | 5 +- pkg/operator/ceph/object/topic/provisioner.go | 86 ++- .../ceph/object/topic/provisioner_test.go | 18 +- tests/framework/installer/ceph_manifests.go | 1 + tests/integration/ceph_object_test.go | 2 + tests/integration/object/topic/kafka/kafka.go | 701 ++++++++++++++++++ tests/integration/object/util/sns/sns.go | 147 ++++ 16 files changed, 1327 insertions(+), 58 deletions(-) create mode 100644 tests/integration/object/topic/kafka/kafka.go create mode 100644 tests/integration/object/util/sns/sns.go diff --git a/Documentation/CRDs/specification.md b/Documentation/CRDs/specification.md index 6cf3bcde8..dcd304780 100644 --- a/Documentation/CRDs/specification.md +++ b/Documentation/CRDs/specification.md @@ -2948,6 +2948,19 @@ int64

ObservedGeneration is the latest generation observed by the controller.

+ + +secrets
+ + +[]SecretReference + + + + +(Optional) + +

CIDR @@ -7533,6 +7546,34 @@ string

The authentication mechanism for this topic (PLAIN/SCRAM-SHA-512/SCRAM-SHA-256/GSSAPI/OAUTHBEARER)

+ + +UserSecretRef
+ + +Kubernetes core/v1.SecretKeySelector + + + + +(Optional) +

The kafka user name to use for authentication

+ + + + +PasswordSecretRef
+ + +Kubernetes core/v1.SecretKeySelector + + + + +(Optional) +

The kafka password to use for authentication

+ +

KerberosConfigFiles @@ -12487,7 +12528,7 @@ int32

SecretReference

-(Appears on:ObjectStoreUserStatus) +(Appears on:BucketTopicStatus, ObjectStoreUserStatus)

diff --git a/Documentation/Storage-Configuration/Object-Storage-RGW/ceph-object-bucket-notifications.md b/Documentation/Storage-Configuration/Object-Storage-RGW/ceph-object-bucket-notifications.md index 7653b0947..8aa76d68d 100644 --- a/Documentation/Storage-Configuration/Object-Storage-RGW/ceph-object-bucket-notifications.md +++ b/Documentation/Storage-Configuration/Object-Storage-RGW/ceph-object-bucket-notifications.md @@ -66,50 +66,67 @@ spec: # ackLevel: broker [17] # useSSL: false [18] # mechanism: SCRAM-SHA-512 [19] +# userSecretRef: [20] +# name: foo-kafka +# key: user +# passwordSecretRef: [21] +# name: foo-kafka +# key: pass +``` + +```yaml +--- +apiVersion: v1 +kind: Secret +metadata: + name: foo-kafka + namespace: rook-ceph +stringData: + user: bar + pass: baz ``` 1. `name` of the `CephBucketTopic` - + In case of AMQP endpoint, the name is used for the AMQP topic (“routing key” for a topic exchange) + + In case of AMQP endpoint, the name is used for the AMQP topic ("routing key" for a topic exchange) + In case of Kafka endpoint, the name is used as the Kafka topic 2. `namespace`(optional) of the `CephBucketTopic`. Should match the namespace of the CephBucketNotification associated with this CephBucketTopic, and the OBC with the label referencing the CephBucketNotification 3. `objectStoreName` is the name of the object store in which the topic should be created. This must be the same object store used for the buckets associated with the notifications referencing this topic. 4. `objectStoreNamespace` is the namespace of the object store in which the topic should be created 5. `opaqueData` (optional) is added to all notifications triggered by a notifications associated with the topic -6. `persistent` (optional) indicates whether notifications to this endpoint are persistent (=asynchronous) or sent synchronously (“false” by default) +6. `persistent` (optional) indicates whether notifications to this endpoint are persistent (=asynchronous) or sent synchronously (`false` by default) 7. `endpoint` to which to send the notifications to. Exactly one of the endpoints must be defined: `http`, `amqp`, `kafka` 8. `http` (optional) hold the spec for an HTTP endpoint. The format of the URI would be: `http[s]://[:][/]` + port defaults to: 80/443 for HTTP/S accordingly -9. `disableVerifySSL` indicates whether the RGW is going to verify the SSL certificate of the HTTP server in case HTTPS is used ("false" by default) -10. `sendCloudEvents`: (optional) send the notifications with the [CloudEvents header](https://github.com/cloudevents/spec/blob/main/cloudevents/adapters/aws-s3.md). ("false" by default) +9. `disableVerifySSL` indicates whether the RGW is going to verify the SSL certificate of the HTTP server in case HTTPS is used (`false` by default) +10. `sendCloudEvents`: (optional) send the notifications with the [CloudEvents header](https://github.com/cloudevents/spec/blob/main/cloudevents/adapters/aws-s3.md). (`false` by default) 11. `amqp` (optional) hold the spec for an AMQP endpoint. The format of the URI would be: `amqp[s]://[:@][:][/]` + port defaults to: 5672/5671 for AMQP/S accordingly + user/password defaults to: guest/guest + user/password may only be provided if HTTPS is used with the RGW. If not, topic creation request will be rejected + vhost defaults to: “/” -12. `disableVerifySSL` (optional) indicates whether the RGW is going to verify the SSL certificate of the AMQP server in case AMQPS is used ("false" by default) +12. `disableVerifySSL` (optional) indicates whether the RGW is going to verify the SSL certificate of the AMQP server in case AMQPS is used (`false` by default) 13. `ackLevel` (optional) indicates what kind of ack the RGW is waiting for after sending the notifications: - + “none”: message is considered “delivered” if sent to broker - + “broker”: message is considered “delivered” if acked by broker (default) - + “routable”: message is considered “delivered” if broker can route to a consumer + + `none`: message is considered "delivered" if sent to broker + + `broker`: message is considered "delivered" if acked by broker (default) + + `routable`: message is considered "delivered" if broker can route to a consumer 14. `exchange` in the AMQP broker that would route the notifications. Different topics pointing to the same endpoint must use the same exchange 15. `kafka` (optional) hold the spec for a Kafka endpoint. The format of the URI would be: `kafka://[:@][: