docs: update vault kms with RGW details

The KMS encryption via HashiCorp Vault can be consumed for RGW, adding those details
in the doc.

Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>

Documentation/ceph-cluster-crd.md

@@ -1385,6 +1385,10 @@ By default, the Key Encryption Keys (also known as Data Encryption Keys) are sto
 However, if a Key Management System exists Rook is capable of using it. HashiCorp Vault is the only KMS currently supported by Rook.
 Please refer to the next section.
 
+Ceph RGW supports encryption via KMS using HashiCorp Vault. If the below settings are defined, then RGW establish a connection between Vault
+and whenever S3 client sends a request with Server Side Encryption, it encrypts that using the key specified by the client.
+For more details w.r.t RGW, please refer [Ceph Vault documentation](https://docs.ceph.com/en/latest/radosgw/vault/)
+
 The `security` section contains settings related to encryption of the cluster.
 
 * `security`:
@@ -1404,6 +1408,7 @@ security:
       KMS_PROVIDER: vault
       VAULT_ADDR: https://vault.default.svc.cluster.local:8200
       VAULT_BACKEND_PATH: rook
+      VAULT_BACKEND: kv
     # name of the k8s secret containing the kms authentication token
     tokenSecretName: rook-vault-token
 ```
@@ -1498,3 +1503,16 @@ data:
 
 Note: if you are using self-signed certificates (not known/approved by a proper CA) you must pass `VAULT_SKIP_VERIFY: true`.
 Communications will remain encrypted but the validity of the certificate will not be verified.
+
+For RGW, please note the following:
+
+* `VAULT_BACKEND` option is specifically for RGW to mention about the secret engine which can be used, currently supports two: [kv](https://www.vaultproject.io/docs/secrets/kv) and [transit](https://www.vaultproject.io/docs/secrets/transit).
+* The Storage administrator needs to create a secret in the Vault server so that S3 clients use that key for encryption
+```console
+# kv engine
+vault kv put rook/mybucketkey key=$(openssl rand -base64 32)
+
+# transit engine
+vault write -f transit/keys/mybucketkey exportable=true
+```
+* TLS authentication with custom certs between Vault and RGW are yet to support.

PendingReleaseNotes.md

@@ -17,3 +17,4 @@ v1.6...
 ### Ceph
 
 * CephClient CRD has been converted to use the controller-runtime library
+* Extending the support of vault KMS configuration for Ceph RGW

cluster/examples/kubernetes/ceph/cluster-on-pvc.yaml

@@ -180,6 +180,7 @@ spec:
          #KMS_PROVIDER: "vault"
          #VAULT_ADDR: VAULT_ADDR_CHANGE_ME # e,g: https://vault.my-domain.com:8200
          #VAULT_BACKEND_PATH: "rook"
+         #VAULT_BACKEND: "kv"
   #     # name of the secret containing the kms authentication token
   #     tokenSecretName: rook-vault-token
 # UNCOMMENT THIS TO ENABLE A KMS CONNECTION