ceph: update nfs ganesha pool recommendation

Add following don'ts note for storing ganesha recovery object:
* Don't use fs pool, it can cause this dashboard issue https://tracker.ceph.com/issues/46176
* Don't use erasure coded pool, it will not work due to ganesha's omap requirements

Signed-off-by: Varsha Rao <varao@redhat.com>

Documentation/ceph-nfs-crd.md

@@ -73,6 +73,8 @@ spec:
 * `pool`: The pool where ganesha recovery backend and supplemental configuration objects will be stored
 * `namespace`: The namespace in `pool` where ganesha recovery backend and supplemental configuration objects will be stored
 
+> **NOTE**: Don't use EC pools for NFS because ganesha uses omap in the recovery objects and grace db. EC pools do not support omap.
+
 ## EXPORT Block Configuration
 
 All daemons within a cluster will share configuration with no exports defined, and that includes a RADOS object via:

cluster/examples/kubernetes/ceph/nfs.yaml

@@ -5,8 +5,9 @@ metadata:
   namespace: rook-ceph # namespace:cluster
 spec:
   rados:
-    # RADOS pool where NFS client recovery data is stored.
-    # In this example the data pool for the "myfs" filesystem is used.
+    # RADOS pool where NFS client recovery data is stored, must be a replica pool. EC pools don't support omap which is required by ganesha.
+    # In this example the data pool for the "myfs" filesystem is used. Separate pool for storing ganesha recovery data is recommended.
+    # Due to this dashboard issue https://tracker.ceph.com/issues/46176.
     # If using the object store example, the data pool would be "my-store.rgw.buckets.data".
     pool: myfs-data0
     # RADOS namespace where NFS client recovery data is stored in the pool.