feedc0de/my-rook-config
1
0 Fork 0
forked from rook/rook
Code Pull Requests Activity

edgefs: docs: unified formatting and style

Browse Source 
This is the first phase EdgeFS part of three phases to improve and
rework the Rook documentation for better user accessibility to the Rook
projects and  with that the storage providers.
Phase two is about automatic generation of CRD documentation and phase
three is about reworking the docs themselves and, e.g., examples for
CRDs and certain use cases.

Signed-off-by: Alexander Trost <galexrt@googlemail.com>
This commit is contained in:
Alexander Trost committed 2019-11-01 18:11:00 +01:00
1 parent eb0d8699d1
commit a91d83d833
14 files changed
+500 -424
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

Documentation/edgefs-cluster-crd.md
+55 -24
View File
@@ -5,9 +5,11 @@ indent: true
---
# EdgeFS Cluster CRD
Rook allows creation and customization of storage clusters through the custom resource definitions (CRDs).
## Sample
To get you started, here is a simple example of a CRD to configure a EdgeFS cluster with just one local per-host directory /data:
```yaml
@@ -58,14 +60,17 @@ spec:
In addition to the CRD, you will also need to create a namespace, role, and role binding as seen in the [common cluster resources](#common-cluster-resources) below.
## Settings
Settings can be specified at the global level to apply to the cluster as a whole, while other settings can be specified at more fine-grained levels, e.g. individual nodes. If any setting is unspecified, a suitable default will be used automatically.
### Cluster metadata
- `name`: The name that will be used internally for the EdgeFS cluster. Most commonly the name is the same as the namespace since multiple clusters are not supported in the same namespace.
- `namespace`: The Kubernetes namespace that will be created for the Rook cluster. The services, pods, and other resources created by the operator will be added to this namespace. The common scenario is to create a single Rook cluster. If multiple clusters are created, they must not have conflicting devices or host paths.
- `edgefsImageName`: EdgeFS image to use. If not specified then `edgefs/edgefs:latest` is used. We recommend to specify particular image version for production use, for example `edgefs/edgefs:1.2.64`.
### Cluster Settings
- `dataDirHostPath`: The path on the host ([hostPath](https://kubernetes.io/docs/concepts/storage/volumes/#hostpath)) where config and data should be stored for each of the services. If the directory does not exist, it will be created. Because this directory persists on the host, it will remain after pods are deleted. If `storage` settings not provided then provisioned hostPath will also be used as a storage device for Target pods (automatic provisioning via `rtlfs`).
- On **Minikube** environments, use `/data/rook`. Minikube boots into a tmpfs but it provides some [directories](https://github.com/kubernetes/minikube/blob/master/docs/persistent_volumes.md) where files can be persisted across reboots. Using one of these directories will ensure that Rook's data and configuration files are persisted and that enough storage space is available.
- **WARNING**: For test scenarios, if you delete a cluster and start a new cluster on the same hosts, the path used by `dataDirHostPath` must be deleted. Otherwise, stale information and other config will remain from the previous cluster and the new target will fail to start.
@@ -97,21 +102,27 @@ If this value is empty, each pod will get an ephemeral directory to store their
- `trlogKeepDays`: Controls for how many days cluster need to keep transaction log interval batches with version manifest references. If you planning to have cluster disconnected from ISGW downlinks for longer period time, consider to increase this value. Default is 3. This is cluster wide setting and cannot be easily changed after cluster is created.
- `maxContainerCapacity`: Overrides default total disks capacity per target container. Default is "132Ti".
- `useHostLocalTime`: Force usage of the host's /etc/localtime inside EdgeFS containers. Default is `false`.
#### Node Updates
Nodes can be added and removed over time by updating the Cluster CRD, for example with `kubectl -n rook-edgefs edit cluster.edgefs.rook.io rook-edgefs`.
This will bring up your default text editor and allow you to add and remove storage nodes from the cluster.
This feature is only available when `useAllNodes` has been set to `false`.
### Node Settings
In addition to the cluster level settings specified above, each individual node can also specify configuration to override the cluster level settings and defaults.
If a node does not specify any configuration then it will inherit the cluster level settings.
- `name`: The name of the node, which should match its `kubernetes.io/hostname` label.
- `config`: Config settings applied to all VDEVs on the node unless overridden by `devices` or `directories`. See the [config settings](#storage-configuration-settings) below.
- [storage selection settings](#storage-selection-settings)
- [storage configuration settings](#storage-configuration-settings)
### Storage Selection Settings
Below are the settings available, both at the cluster and individual node level, for selecting which storage resources will be included in the cluster.
- `useAllDevices`: `true` or `false`, indicating whether all devices found on nodes in the cluster should be automatically consumed by Targets. This is recommended for controlled environments where you will not risk formatting of devices with existing data. When `true`, all devices will be used except those with partitions created or a local filesystem. This can be overridden by `deviceFilter`. **warning** Don't set this option to `true` for RTKVS disk backend.
- `deviceFilter`: A regular expression that allows selection of devices to be consumed by target. If individual devices have been specified for a node then this filter will be ignored. This field uses [golang regular expression syntax](https://golang.org/pkg/regexp/syntax/). For example:
- `sdb`: Only selects the `sdb` device if found
@@ -127,35 +138,39 @@ Below are the settings available, both at the cluster and individual node level,
- `path`: The path on disk of the directory (e.g., `/rook/storage-dir`).
- `config`: Directory-specific config settings. See the [config settings](#target-configuration-settings) below.
### Storage Configuration Settings
The following storage selection settings are specific to EdgeFS and do not apply to other backends. All variables are key-value pairs represented as strings. While EdgeFS supports multiple backends, it is not recommended to mix them within same cluster. In case of `devices` (physical or emulated raw disks), EdgeFS will automatically use `rtrd` backend unless `useRtkvsBackend` is specified. In the latter case, the `rtkvs` engine will be chosen. In all other cases `rtlfs` (local file system) will be used.
**IMPORTANT** Keys needs to be case-sensitive and values has to be provided as strings.
- `useRtkvsBackend`: forces the cluster use the `rtkvs` disk engine and setting's value selects a key-value backend to be used. At the moment there is only backend named the `kvssd` for Samsung's KV SSD. The usage of `rtkvs` engine implies definition of one or several `device.name` or `device.fullpath` settings which have to point to backend's disk entries (see cluster_kvsdd.yaml).
- `walMode`: allows to enable/disable the Write-Ahead log (WAL). For `rtlfs` and `rtkvs` there are two options: `on` to enable or `off` to disable. It's better to keep it `on` unless `useAllSSD` or `useRtkvsBackend` are used. For `rtkvs`, there is an extra option: `metadata` which implies usage of WAL for data types which aren't stored on `rtkvs` backend (a KVSSD).
- `useMetadataOffload`: Dynamically detect appropriate SSD/NVMe device to use for the metadata on each node. Performance can be improved by using a low latency device as the metadata device, while other spinning platter (HDD) devices on a node are used to store data. Typical and recommended proportion is in range of 1:1 - 1:6. Default is false. Applicable only to `rtrd`.
- `useMetadataMask`: Defines what parts of metadata needs to be stored on offloaded devices. Default is 0xff, offload all metadata. To save SSD/NVMe capacity, set it to 0x7d to offload all except second level manifests. Applicable only to `rtrd`.
- `useBCache`: When `useMetadataOffload` is true, enable use of BCache. Default is false. Applicable only to `rtrd` and when host has "bcache" kernel module preloaded.
- `useBCacheWB`: When `useMetadataOffload` and `useBCache` is true, this option can enable use of BCache write-back cache. By default BCache only used as read cache in front of HDD. Applicable only to `rtrd`.
- `useAllSSD`: When set to true, only SSD/NVMe non rotational devices will be used. Default is false and if `useMetadataOffload` not defined then only rotational devices (HDDs) will be picked up during node provisioning phase. Is not applicabel to `rtkvs`.
- `rtPLevelOverride`: In case of large devices or directories, it will be automatically partitioned into smaller parts around 500GB each. In case of embedded use cases, lowering the value would allow to operate with smaller memory footprint devices at the cost of performance. This option allows partitioning number override. Default is automatic. Typical and recommended range is 1 - 32. For `rtkvs` recomended plevel is 16.
- `hddReadAhead`: For all HDD or hybrid (SSD/HDD) use cases, adjusting hddReadAhead may provide significant boost in performance. Set to a value higher then 0, in KBs. Not applicable to `rtkvs`
- `mdReserved`: For hybrid (SSD/HDD) use case, adjusting mdReserved can be necessary when combined with BCache read/write caches. Allowed range 10-99% of automatically calcuated slice. Not applicable to `rtkvs`
- `rtVerifyChid`: Verify transferred or read payload. Payload can be data or metadata chunk of flexible size between 4K and 8MB. EdgeFS uses SHA-3 variant to cryptographically sign each chunk and uses it for self validation, self healing and FlexHash addressing. In case of low CPU systems verification after networking transfer prior to write can be disabled by setting this parameter to 0. In case of high CPU systems, verification after read but before networking transfer can be enabled by setting this parameter to 2. Default is 1, i.e. verify after networking transfer only. Setting it to 0 may improve CPU utilization at the cost of reduced availability. However, for objects with 3 or more replicas, availability isn't going to be visibly affected.
- `lmdbPageSize`: Defines default LMDB page size in bytes. Default is 16384. For capacity (all HDD) or hybrid (HDD/SSD) systems consider to increase this value to 32768 to achieve higher throughput performance. For all SSD and small database workloads, consider to decrease this to 8192 to achieve lower latency and higher IOPS. Please be advised that smaller values MAY cause fragmentation. Acceptable values are 4096, 8192, 16384 and 32768. Not applicable to `rtkvs`
- `lmdbMdPageSize`: Defines SSD metadata offload LMDB page size in bytes. Default is 8192. For large amount of small objects or files, consider to decrease this to 4096 to achieve better SSD capacity utilization. Acceptable values are 4096, 8192, 16384 and 32768. Not applicable to `rtkvs`
- `sync`: Defines default behavior of write operations at device or directory level. Acceptable values are 0, 1 (default), 2, 3.
- `0`: No syncing will happen. Highest performance possible and good for HPC scratch types of deployments. This option will still sustain crash of pods or software bugs. It will not sustain server power loss an may cause node / device level inconsistency.
- `1`: Default method. Will guarantee node / device consistency in case of power loss with reduced durability.
- `2`: Provides better durability in case of power loss at the cost of extra metadata syncing.
- `3`: Most durable and reliable option at the cost of significant performance impact.
- `maxSizeGB`: For `rtlfs`, defines maximum allowed size to use per directory in gigabytes. For `rtkvs` this is the maximum space the disk's metadata table can occupy.
- `zone`: Enables the node's failure domain number. Default value is 0 (no zoning). Zoning number is a logical failure domain tagging mechanism and if enabled then it has to be set for all the nodes in the cluster. See also, the `failureDomain`
The following storage selection settings are specific to EdgeFS and do not apply to other backends. All variables are key-value pairs represented as strings. While EdgeFS supports multiple backends, it is not recommended to mix them within same cluster. In case of `devices` (physical or emulated raw disks), EdgeFS will automatically use `rtrd` backend unless `useRtkvsBackend` is specified. In the latter case, the `rtkvs` engine will be chosen. In all other cases `rtlfs` (local filesystem) will be used.
> **IMPORTANT**: Keys needs to be case-sensitive and values has to be provided as strings.
- `useRtkvsBackend`: forces the cluster use the `rtkvs` disk engine and setting's value selects a key-value backend to be used. At the moment there is only backend named the `kvssd` for Samsung's KV SSD. The usage of `rtkvs` engine implies definition of one or several `device.name` or `device.fullpath` settings which have to point to backend's disk entries (see cluster_kvsdd.yaml).
- `walMode`: allows to enable/disable the Write-Ahead log (WAL). For `rtlfs` and `rtkvs` there are two options: `on` to enable or `off` to disable. It's better to keep it `on` unless `useAllSSD` or `useRtkvsBackend` are used. For `rtkvs`, there is an extra option: `metadata` which implies usage of WAL for data types which aren't stored on `rtkvs` backend (a KVSSD).
- `useMetadataOffload`: Dynamically detect appropriate SSD/NVMe device to use for the metadata on each node. Performance can be improved by using a low latency device as the metadata device, while other spinning platter (HDD) devices on a node are used to store data. Typical and recommended proportion is in range of 1:1 - 1:6. Default is false. Applicable only to `rtrd`.
- `useMetadataMask`: Defines what parts of metadata needs to be stored on offloaded devices. Default is 0xff, offload all metadata. To save SSD/NVMe capacity, set it to 0x7d to offload all except second level manifests. Applicable only to `rtrd`.
- `useBCache`: When `useMetadataOffload` is true, enable use of BCache. Default is false. Applicable only to `rtrd` and when host has "bcache" kernel module preloaded.
- `useBCacheWB`: When `useMetadataOffload` and `useBCache` is true, this option can enable use of BCache write-back cache. By default BCache only used as read cache in front of HDD. Applicable only to `rtrd`.
- `useAllSSD`: When set to true, only SSD/NVMe non rotational devices will be used. Default is false and if `useMetadataOffload` not defined then only rotational devices (HDDs) will be picked up during node provisioning phase. Is not applicabel to `rtkvs`.
- `rtPLevelOverride`: In case of large devices or directories, it will be automatically partitioned into smaller parts around 500GB each. In case of embedded use cases, lowering the value would allow to operate with smaller memory footprint devices at the cost of performance. This option allows partitioning number override. Default is automatic. Typical and recommended range is 1 - 32. For `rtkvs` recomended plevel is 16.
- `hddReadAhead`: For all HDD or hybrid (SSD/HDD) use cases, adjusting hddReadAhead may provide significant boost in performance. Set to a value higher then 0, in KBs. Not applicable to `rtkvs`
- `mdReserved`: For hybrid (SSD/HDD) use case, adjusting mdReserved can be necessary when combined with BCache read/write caches. Allowed range 10-99% of automatically calcuated slice. Not applicable to `rtkvs`
- `rtVerifyChid`: Verify transferred or read payload. Payload can be data or metadata chunk of flexible size between 4K and 8MB. EdgeFS uses SHA-3 variant to cryptographically sign each chunk and uses it for self validation, self healing and FlexHash addressing. In case of low CPU systems verification after networking transfer prior to write can be disabled by setting this parameter to 0. In case of high CPU systems, verification after read but before networking transfer can be enabled by setting this parameter to 2. Default is 1, i.e. verify after networking transfer only. Setting it to 0 may improve CPU utilization at the cost of reduced availability. However, for objects with 3 or more replicas, availability isn't going to be visibly affected.
- `lmdbPageSize`: Defines default LMDB page size in bytes. Default is 16384. For capacity (all HDD) or hybrid (HDD/SSD) systems consider to increase this value to 32768 to achieve higher throughput performance. For all SSD and small database workloads, consider to decrease this to 8192 to achieve lower latency and higher IOPS. Please be advised that smaller values MAY cause fragmentation. Acceptable values are 4096, 8192, 16384 and 32768. Not applicable to `rtkvs`
- `lmdbMdPageSize`: Defines SSD metadata offload LMDB page size in bytes. Default is 8192. For large amount of small objects or files, consider to decrease this to 4096 to achieve better SSD capacity utilization. Acceptable values are 4096, 8192, 16384 and 32768. Not applicable to `rtkvs`
- `sync`: Defines default behavior of write operations at device or directory level. Acceptable values are 0, 1 (default), 2, 3.
- `0`: No syncing will happen. Highest performance possible and good for HPC scratch types of deployments. This option will still sustain crash of pods or software bugs. It will not sustain server power loss an may cause node / device level inconsistency.
- `1`: Default method. Will guarantee node / device consistency in case of power loss with reduced durability.
- `2`: Provides better durability in case of power loss at the cost of extra metadata syncing.
- `3`: Most durable and reliable option at the cost of significant performance impact.
- `maxSizeGB`: For `rtlfs`, defines maximum allowed size to use per directory in gigabytes. For `rtkvs` this is the maximum space the disk's metadata table can occupy.
- `zone`: Enables the node's failure domain number. Default value is 0 (no zoning). Zoning number is a logical failure domain tagging mechanism and if enabled then it has to be set for all the nodes in the cluster. See also, the `failureDomain`
### Placement Configuration Settings
Placement configuration for the cluster services. It includes the following keys: `mgr`, `target` and `all`. Each service will have its placement configuration generated by merging the generic configuration under `all` with the most specific one (which will override any attributes).
A Placement configuration is specified (according to the Kubernetes PodSpec) as:
- `nodeAffinity`: Kubernetes [NodeAffinity](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#node-affinity-beta-feature)
- `podAffinity`: Kubernetes [PodAffinity](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#inter-pod-affinity-and-anti-affinity-beta-feature)
- `podAntiAffinity`: Kubernetes [PodAntiAffinity](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#inter-pod-affinity-and-anti-affinity-beta-feature)
@@ -164,7 +179,9 @@ A Placement configuration is specified (according to the Kubernetes PodSpec) as:
The `mgr` pod does not allow `Pod` affinity or anti-affinity. This is because of the mgrs having built-in anti-affinity with each other through the operator. The operator chooses which nodes are to run a mgr on. Each mgr is then tied to a node with a node selector using a hostname.
### Network Configuration Settings
Configure the network that will be enabled for the cluster and services. This is optional and if not defined then the cluster default network's `eth0` will be used to construct cluster bucket network.
- `provider`: Specifies the network provider that will be used to connect the network interface. You can choose between `host`, and `multus`.
- `selectors`: List the network selector that will be used associated by a key. The available keys are `server` and `broker`.
- `server`: Specifies data daemon host's networking interface name or multus's network attachment selection annotation.
@@ -174,15 +191,19 @@ For `multus` network provider, an already working cluster with multus networking
You can add the multus network attachment selection annotation selecting the created network attachment definition on `selectors`. Make sure to define the interface name that will be assigned by multus and choose only one syntax either the short or JSON form to define all the available keys.
### Cluster-wide Resources Configuration Settings
Resources should be specified so that the rook components are handled after [Kubernetes Pod Quality of Service classes](https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/).
This allows to keep rook components running when for example a node runs out of memory and the rook components are not killed depending on their Quality of Service class.
You can set resource requests/limits for rook components through the [Resource Requirements/Limits](#resource-requirementslimits) structure in the following keys:
- `mgr`: Set resource requests/limits for Mgrs.
- `target`: Set resource requests/limits for Targets.
### Resource Requirements/Limits
For more information on resource requests/limits see the official Kubernetes documentation: [Kubernetes - Managing Compute Resources for Containers](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#resource-requests-and-limits-of-pod-and-container)
For more information on resource requests/limits see the official Kubernetes documentation: [Kubernetes - Managing Compute Resources for Containers](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#resource-requests-and-limits-of-pod-and-container).
- `requests`: Requests for cpu or memory.
- `cpu`: Request for CPU (example: one CPU core `1`, 50% of one CPU core `500m`).
- `memory`: Limit for Memory (example: one gigabyte of memory `1Gi`, half a gigabyte of memory `512Mi`).
@@ -191,15 +212,18 @@ For more information on resource requests/limits see the official Kubernetes doc
- `memory`: Limit for Memory (example: one gigabyte of memory `1Gi`, half a gigabyte of memory `512Mi`).
### Kubernetes node labeling and selection
By default each Kubernetes node, available to deploy EdgeFS over it, will be treated as "target" EdgeFS instance. But cluster administrator able to label node as "gateway" node, such node will have no devices prepared for EdgeFS and will be used as EdgeFS dedicated service node.
To mark a node as a "gateway", the administrator can add a specific label to a node.
Label format: `<edgefs-namespace>-nodetype=gateway`, where `edgefs-namespace` is current namespace for EdgeFS cluster deployment, by default is `rook-edgefs`.
Example: `kubectl label node "k8s node name" rook-edgefs-nodetype=gateway`
## Samples
Here are several samples for configuring EdgeFS clusters. Each of the samples must also include the namespace and corresponding access granted for management by the EdgeFS operator. See the [common cluster resources](#common-cluster-resources) below.
### Storage configuration: All devices, All SSD/NVMes.
```yaml
apiVersion: edgefs.rook.io/v1
kind: Cluster
@@ -221,6 +245,7 @@ spec:
```
### Storage Configuration: Specific devices
Individual nodes and their config can be specified so that only the named nodes below will be used as storage resources.
Each node's 'name' field should match their 'kubernetes.io/hostname' label.
@@ -252,7 +277,9 @@ spec:
- name: "172.17.4.301"
deviceFilter: "^sd."
```
### Storage Configuration: Samsung's KV SSD
A single nodes configuration with 2 KV SSDs. The host's /media directory should have at least 64GB of free space for metadata.
```yaml
@@ -281,6 +308,7 @@ spec:
```
### Node Affinity
To control where various services will be scheduled by Kubernetes, use the placement configuration sections below.
The example under 'all' would have all services scheduled on Kubernetes nodes labeled with 'role=storage' and
tolerate taints with a key of 'storage-node'.
@@ -317,6 +345,7 @@ spec:
```
### Resource requests/Limits
To control how many resources the rook components can request/use, you can set requests and limits in Kubernetes for them.
You can override these requests/limits for Targts per node when using `useAllNodes: false` in the `node` item in the `nodes` list.
@@ -346,6 +375,7 @@ spec:
```
### Network Configuration: Multus network
An example on how to configure the cluster network to use multus network. Here, a NetworkAttachmentDefinition named flannel on rook-edgefs namespace is assumed.
```yaml
@@ -366,6 +396,7 @@ spec:
```
## Common Cluster Resources
Each EdgeFS cluster must be created in a namespace and also give access to the Rook operator to manage the cluster in the namespace. Creating the namespace and these controls must be added to each of the examples previously shown.
```yaml
Documentation/edgefs-csi.md
+154 -137
View File
@@ -4,7 +4,7 @@ weight: 4700
indent: true
---
## EdgeFS Rook integrated CSI driver, provisioner, attacher and snapshotter
# EdgeFS Rook integrated CSI driver, provisioner, attacher and snapshotter
[Container Storage Interface (CSI)](https://github.com/container-storage-interface/) driver, provisioner, attacher and snapshotter for EdgeFS Scale-Out NFS/ISCSI services
@@ -14,33 +14,42 @@ EdgeFS CSI plugins implement an interface between CSI enabled Container Orchestr
## Prerequisites
- Ensure your kubernetes cluster version is 1.13+
- Kubernetes cluster must allow privileged pods, this flag must be set for the API server and the kubelet
* Ensure your kubernetes cluster version is 1.13+
* Kubernetes cluster must allow privileged pods, this flag must be set for the API server and the kubelet
([instructions](https://github.com/kubernetes-csi/docs/blob/735f1ef4adfcb157afce47c64d750b71012c8151/book/src/Setup.md#enable-privileged-pods)):
```
```console
--allow-privileged=true
```
- Required the API server and the kubelet feature gates
```
* Required the API server and the kubelet feature gates
([instructions](https://github.com/kubernetes-csi/docs/blob/735f1ef4adfcb157afce47c64d750b71012c8151/book/src/Setup.md#enabling-features)):
```
```console
--feature-gates=VolumeSnapshotDataSource=true,CSIDriverRegistry=true
```
- Mount propagation must be enabled, the Docker daemon for the cluster must allow shared mounts
```
* Mount propagation must be enabled, the Docker daemon for the cluster must allow shared mounts
([instructions](https://github.com/kubernetes-csi/docs/blob/735f1ef4adfcb157afce47c64d750b71012c8151/book/src/Setup.md#enabling-mount-propagation))
- Kubernetes CSI drivers require `CSIDriver` and `CSINodeInfo` resource types
* Kubernetes CSI drivers require `CSIDriver` and `CSINodeInfo` resource types
[to be defined on the cluster](https://github.com/kubernetes-csi/docs/blob/460a49286fe164a78fde3114e893c48b572a36c8/book/src/Setup.md#csidriver-custom-resource-alpha).
Check if they are already defined:
```bash
```console
kubectl get customresourcedefinition.apiextensions.k8s.io/csidrivers.csi.storage.k8s.io
kubectl get customresourcedefinition.apiextensions.k8s.io/csinodeinfos.csi.storage.k8s.io
```
If the cluster doesn't have "csidrivers" and "csinodeinfos" resource types, create them:
```bash
```console
kubectl create -f https://raw.githubusercontent.com/kubernetes/csi-api/release-1.13/pkg/crd/manifests/csidriver.yaml
kubectl create -f https://raw.githubusercontent.com/kubernetes/csi-api/release-1.13/pkg/crd/manifests/csinodeinfo.yaml
```
- Depends on preferred CSI driver type, following utilities must be installed on each Kubernetes node (For Debian/Ubuntu based systems):
```bash
* Depends on preferred CSI driver type, following utilities must be installed on each Kubernetes node (For Debian/Ubuntu based systems):
```console
# for NFS
apt install -y nfs-common rpcbind
# for ISCSI
@@ -48,15 +57,15 @@ EdgeFS CSI plugins implement an interface between CSI enabled Container Orchestr
```
## EdgeFS CSI drivers configuration
For each driver type (NFS/ISCSI) we have already prepared configuration files examples, there are:
[EdgeFS CSI NFS driver config](https://github.com/rook/rook/tree/master/cluster/examples/kubernetes/edgefs/csi/nfs/edgefs-nfs-csi-driver-config.yaml)
[EdgeFS CSI ISCSI driver config](https://github.com/rook/rook/tree/master/cluster/examples/kubernetes/edgefs/csi/iscsi/edgefs-iscsi-csi-driver-config.yaml)
* [EdgeFS CSI NFS driver config](https://github.com/rook/rook/tree/master/cluster/examples/kubernetes/edgefs/csi/nfs/edgefs-nfs-csi-driver-config.yaml)
* [EdgeFS CSI ISCSI driver config](https://github.com/rook/rook/tree/master/cluster/examples/kubernetes/edgefs/csi/iscsi/edgefs-iscsi-csi-driver-config.yaml)
Secret file configuration options example:
```
```console
# EdgeFS k8s cluster options
k8sEdgefsNamespaces: ["rook-edgefs"] # edgefs cluster namespace
k8sEdgefsMgmtPrefix: rook-edgefs-mgr # edgefs cluster management prefix
@@ -73,43 +82,45 @@ password: admin # edgefs k8s cluster grpc service password
Options for NFS and ISCSI configuration files
| `Name` | `Description` | `Default value` | `Required` | `Type` |
|---------------------|-----------------------|---------------|----------|------|
| k8sEdgefsNamespaces | Array of Kubernetes cluster's namespaces for EdgeFS service discovery | rook-edgefs | true | both |
| k8sEdgefsMgmtPrefix | Rook EdgeFS cluster mgmt service prefix | rook-edgefs-mgr | true | both |
| username | EdgeFS gRPC API server privileged user | "admin" | true | both |
| password | EdgeFS gRPC API server password | "admin" | true | both |
| cluster | EdgeFS cluster namespace also known as 'region' | | false | both |
| tenant | EdgeFS tenant isolated namespace | | false | both |
| bucket | EdgeFS tenant bucket to use as a default | | false | ISCSI only |
| serviceFilter | Comma delimited list of allowed service names for filtering | "" means all services allowed | false | both |
| serviceBalancerPolicy | Service selection policy [`minexportspolicy`, `randomservicepolicy`] | `minexportspolicy` |false | both |
| chunksize | Chunk size for actual volume, in bytes | 16384, should be power of two | false | both |
| blocksize | Block size for actual volume, in bytes | 4096, should be power of two | false | iSCSI only |
| fsType | New volume's filesystem type | ext4, ext3, xfs | ext4 | ISCSI only |
| forceVolumeDeletion | Automatically deletes EdgeFS volume after usage | false | false | both|
| `Name` | `Description` | `Default value` | `Required` | `Type` |
| ----------------------- | --------------------------------------------------------------------- | ------------------------------- | ---------- | ---------- |
| `k8sEdgefsNamespaces` | Array of Kubernetes cluster's namespaces for EdgeFS service discovery | `rook-edgefs` | true | both |
| `k8sEdgefsMgmtPrefix` | Rook EdgeFS cluster mgmt service prefix | `rook-edgefs-mgr` | true | both |
| `username` | EdgeFS gRPC API server privileged user | `admin` | true | both |
| `password` | EdgeFS gRPC API server password | `admin` | true | both |
| `cluster` | EdgeFS cluster namespace also known as 'region' | | false | both |
| `tenant` | EdgeFS tenant isolated namespace | | false | both |
| `bucket` | EdgeFS tenant bucket to use as a default | | false | ISCSI only |
| `serviceFilter` | Comma delimited list of allowed service names for filtering | `""` means all services allowed | false | both |
| `serviceBalancerPolicy` | Service selection policy [`minexportspolicy`, `randomservicepolicy`] | `minexportspolicy` | false | both |
| `chunksize` | Chunk size for actual volume, in bytes | `16384`, should be power of two | false | both |
| `blocksize` | Block size for actual volume, in bytes | `4096`, should be power of two | false | iSCSI only |
| `fsType` | New volume's filesystem type | `ext4`, `ext3`, `xfs` | ext4 | ISCSI only |
| `forceVolumeDeletion` | Automatically deletes EdgeFS volume after usage | `false` | false | both |
By using `k8sEdgefsNamespaces` and `k8sEdgefsMgmtPrefix` parameters, driver is capable of detecting ClusterIPs and Endpoint IPs to provision and attach volumes.
## Apply EdgeFS CSI NFS driver configuration
Check configuration options and create kubernetes secret for Edgefs CSI NFS plugin
```
```console
cd cluster/examples/kubernetes/edgefs/csi/nfs
kubectl create secret generic edgefs-nfs-csi-driver-config --from-file=./edgefs-nfs-csi-driver-config.yaml
```
## Deploy EdgeFS CSI NFS driver
After secret is created successfully, deploy EdgeFS CSI plugin, provisioner and attacher using the following command
```
```console
cd cluster/examples/kubernetes/edgefs/csi/nfs
kubectl apply -f edgefs-nfs-csi-driver.yaml
```
There should be number of EdgeFS CSI plugin PODs available running as a DaemonSet
```
There should be number of EdgeFS CSI plugin PODs available running as a DaemonSet:
```console
...
NAMESPACE NAME READY STATUS RESTARTS AGE
default edgefs-nfs-csi-controller-0 4/4 Running 0 33s
@@ -118,6 +129,7 @@ default edgefs-nfs-csi-node-js7jp 2/2 Running 0 33s
default edgefs-nfs-csi-node-lhjgr 2/2 Running 0 33s
...
```
At this point configuration is all ready and available for consumption by applications.
## Pre-provisioned volumes (NFS)
@@ -125,15 +137,15 @@ At this point configuration is all ready and available for consumption by applic
This method allows to use already created exports in EdgeFS services. This method keeps exports provisioned after application PODs terminated.
Read more on how to create PersistentVolume specification for pre-provisioned volumes:
[link to Pre-provisioned volumes manifest specification](https://kubernetes-csi.github.io/docs/Usage.html#pre-provisioned-volumes)
[Link to Pre-provisioned volumes manifest specification](https://kubernetes-csi.github.io/docs/Usage.html#pre-provisioned-volumes)
To test creation and mount pre-provisioned volume to pod execute example
#### Note:
Make sure that `volumeHandle: segment:service@cluster/tenant/bucket` in nginx.yaml already exist on EdgeFS cluster and served via any Edgefs NFS service. Any volumeHandle's parameters may be omitted and will be substituted via CSI configuration file parameters.
> **NOTE**: Make sure that `volumeHandle: segment:service@cluster/tenant/bucket` in nginx.yaml already exist on EdgeFS cluster and served via any Edgefs NFS service. Any volumeHandle's parameters may be omitted and will be substituted via CSI configuration file parameters.
Examples:
```
```console
cd cluster/examples/kubernetes/edgefs/csi/nfs/examples
kubectl apply -f ./preprovisioned-edgefs-volume-nginx.yaml
```
@@ -142,12 +154,13 @@ kubectl apply -f ./preprovisioned-edgefs-volume-nginx.yaml
To setup the system for dynamic provisioning, administrator needs to setup a StorageClass pointing to the CSI driver’s external-provisioner and specifying any parameters required by the driver
[link to dynamically provisioned volumes specification](https://kubernetes-csi.github.io/docs/Usage.html#dynamic-provisioning)
[Link to dynamically provisioned volumes specification](https://kubernetes-csi.github.io/docs/Usage.html#dynamic-provisioning)
### Note
#### Note:
For dynamically provisioned volumes kubernetes will generate volume name automatically
(for example pvc-871068ed-8b5d-11e8-9dae-005056b37cb2)
Additional creation options should be passed as parameters in storage class definition i.e :
Additional creation options should be passed as parameters in StorageClass definition i.e :
```yaml
apiVersion: storage.k8s.io/v1
@@ -162,35 +175,35 @@ parameters:
encryption: true
```
### Options:
### Parameters
| `Name` | `Description` | `Allowed values` | `Default value` |
|-----------|-----------------------|---------------------------|---------------|
| segment | Edgefs cluster namespace for current StorageClass or PV. | | rook-edgefs |
| service | Edgefs cluster service if not defined in secret | | |
| cluster | Edgefs cluster namespace if not defined in secret | | |
| tenant | Edgefs tenant namespace if not defined in secret | | |
| chunksize | Chunk size for actual volume, in bytes | should be power of two | 16384 bytes |
| blocksize | Block size for actual volume, in bytes | should be power of two | 4096 bytes |
| acl | Volume acl restrictions | | all |
| ec | Enables ccow erasure coding for volume | true, false, 0, 1 | false |
| ecmode | Set ccow erasure mode data mode (If 'ec' option enabled) | "3:1:xor", "2:2:rs", 4:2:rs" ,"6:2:rs", "9:3:rs" | 6:2:rs |
| encryption | Enables encryption for volume | true, false, 0, 1 | false |
> **NOTE**: Parameters and their options are case sensitive and should be in lower case.
#### Note:
Options are case sensitive and should be in lower case
| `Name` | `Description` | `Allowed values` | `Default value` |
| ------------ | -------------------------------------------------------- | ------------------------------------------------- | --------------- |
| `segment` | Edgefs cluster namespace for current StorageClass or PV. | | rook-edgefs |
| `service` | Edgefs cluster service if not defined in secret | | |
| `cluster` | Edgefs cluster namespace if not defined in secret | | |
| `tenant` | Edgefs tenant namespace if not defined in secret | | |
| `chunksize` | Chunk size for actual volume, in bytes | should be power of two | 16384 bytes |
| `blocksize` | Block size for actual volume, in bytes | should be power of two | 4096 bytes |
| `acl` | Volume acl restrictions | | all |
| `ec` | Enables ccow erasure coding for volume | `true`, `false`, `0`, `1` | `false` |
| `ecmode` | Set ccow erasure mode data mode (If 'ec' option enabled) | `3:1:xor`, `2:2:rs`, `4:2:rs`, `6:2:rs`, `9:3:rs` | `6:2:rs` |
| `encryption` | Enables encryption for volume | `true`, `false`, `0`, `1` | `false` |
Example:
```
```console
cd cluster/examples/kubernetes/edgefs/csi/nfs/examples
kubectl apply -f ./dynamic-nginx.yaml
```
## Apply Edgefs CSI ISCSI driver configuration
Check configuration options and create kubernetes secret for Edgefs CSI ISCSI plugin
```
```console
cd cluster/examples/kubernetes/edgefs/csi/iscsi
kubectl create secret generic edgefs-iscsi-csi-driver-config --from-file=./edgefs-iscsi-csi-driver-config.yaml
```
@@ -198,13 +211,15 @@ kubectl create secret generic edgefs-iscsi-csi-driver-config --from-file=./edgef
## Deploy Edgefs CSI ISCSI driver
After secret is created successfully, deploy EdgeFS CSI plugin, provisioner, attacher and snapshotter using the following command
```
```console
cd cluster/examples/kubernetes/edgefs/csi/iscsi
kubectl apply -f edgefs-iscsi-csi-driver.yaml
```
There should be number of EdgeFS CSI ISCSI plugin PODs available running as a DaemonSet
```
There should be number of EdgeFS CSI ISCSI plugin PODs available running as a DaemonSet:
```console
...
NAMESPACE NAME READY STATUS RESTARTS AGE
default edgefs-iscsi-csi-controller-0 4/4 Running 0 12s
@@ -221,15 +236,15 @@ At this point configuration is all ready and available for consumption by applic
This method allows to use already created exports in EdgeFS ISCSI services. This method keeps exports provisioned after application PODs terminated.
Read more on how to create PersistentVolume specification for pre-provisioned volumes:
[link to Pre-provisioned volumes manifest specification](https://kubernetes-csi.github.io/docs/Usage.html#pre-provisioned-volumes)
[Link to Pre-provisioned volumes manifest specification](https://kubernetes-csi.github.io/docs/Usage.html#pre-provisioned-volumes)
To test creation and mount pre-provisioned volume to pod execute example
To test creation and mount pre-provisioned volume to pod execute example:
#### Note:
Make sure that `volumeHandle: segment:service@cluster/tenant/bucket/lun` in nginx.yaml already exist on EdgeFS cluster and served via any Edgefs ISCSI service. Any volumeHandle's parameters may be omitted and will be substituted via CSI configuration file parameters.
> **NOTE**: Make sure that `volumeHandle: segment:service@cluster/tenant/bucket/lun` in nginx.yaml already exist on EdgeFS cluster and served via any Edgefs ISCSI service. Any volumeHandle's parameters may be omitted and will be substituted via CSI configuration file parameters.
Examples:
```
Example:
```console
cd cluster/examples/kubernetes/edgefs/csi/iscsi/examples
kubectl apply -f ./preprovisioned-edgefs-volume-nginx.yaml
```
@@ -240,12 +255,15 @@ For dynamic volume provisioning, the administrator needs to set up a _StorageCla
In this case Kubernetes generates volume name automatically (for example `pvc-ns-cfc67950-fe3c-11e8-a3ca-005056b857f8`).
Default driver configuration may be overwritten in `parameters` section:
[link to dynamically provisioned volumes specification](https://kubernetes-csi.github.io/docs/Usage.html#dynamic-provisioning)
[Link to dynamically provisioned volumes specification](https://kubernetes-csi.github.io/docs/Usage.html#dynamic-provisioning)
#### Note:
For dynamically provisioned volumes kubernetes will generate volume name automatically
(for example pvc-871068ed-8b5d-11e8-9dae-005056b37cb2)
Additional creation options should be passed as parameters in storage class definition i.e :
### Note for dynamically provisioned volumes
For dynamically provisioned volumes, Kubernetes will generate volume names automatically
(for example pvc-871068ed-8b5d-11e8-9dae-005056b37cb2).
To pass additional creation parameters, you can add them as parameters to your StorageClass definition.
Example:
```yaml
apiVersion: storage.k8s.io/v1
@@ -262,28 +280,28 @@ parameters:
encryption: true
```
### Parameters:
### Parameters
| `Name` | `Description` | `Allowed values` | `Default value` |
|-----------|-----------------------|---------------------------|---------------|
| segment | Edgefs cluster namespace for specific StorageClass or PV.| | rook-edgefs |
| service | Edgefs cluster service if not defined in secret | | |
| cluster | Edgefs cluster namespace if not defined in secret | | |
| tenant | Edgefs tenant namespace if not defined in secret | | |
| bucket | Edgefs bucket namespace if not defined in secret | | |
| chunksize | Chunk size for actual volume, in bytes | should be power of two | 16384 |
| blocksize | Blocksize size for actual volume, in bytes | should be power of two | 4096 |
| fsType | New volume's filesystem type | ext4, ext3, xfs | ext4 |
| acl | Volume acl restrictions | | all |
| ec | Enables ccow erasure coding for volume | true, false, 0, 1 | false |
| ecmode | Set ccow erasure mode data mode (If 'ec' option enabled) | "4:2:rs" ,"6:2:rs", "9:3:rs" | 4:2:rs |
| encryption | Enables encryption for volume | true, false, 0, 1 | false |
> **NOTE**: Parameters and their options are case sensitive and should be in lower case.
#### Note:
Options are case sensitive and should be in lower case
| `Name` | `Description` | `Allowed values` | `Default value` |
| ------------ | --------------------------------------------------------- | -------------------------------------- | --------------- |
| `segment` | Edgefs cluster namespace for specific StorageClass or PV. | | `rook-edgefs` |
| `service` | Edgefs cluster service if not defined in secret | | |
| `cluster` | Edgefs cluster namespace if not defined in secret | | |
| `tenant` | Edgefs tenant namespace if not defined in secret | | |
| `bucket` | Edgefs bucket namespace if not defined in secret | | |
| `chunksize` | Chunk size for actual volume, in bytes | should be power of two | `16384` |
| `blocksize` | Blocksize size for actual volume, in bytes | should be power of two | `4096` |
| `fsType` | New volume's filesystem type | `ext4`, `ext3`, `xfs` | `ext4` |
| `acl` | Volume acl restrictions | | `all` |
| `ec` | Enables ccow erasure coding for volume | `true`, `false`, `0`, `1` | `false` |
| `ecmode` | Set ccow erasure mode data mode (If 'ec' option enabled) | `4:2:rs` ,`6:2:rs`, `4:2:rs`, `9:3:rs` | `4:2:rs` |
| `encryption` | Enables encryption for volume | `true`, `false`, `0`, `1` | `false` |
Example:
```
```console
cd cluster/examples/kubernetes/edgefs/csi/nfs/examples
kubectl apply -f ./dynamic-nginx.yaml
```
@@ -292,7 +310,7 @@ kubectl apply -f ./dynamic-nginx.yaml
### Getting information about existing snapshots
```bash
```console
# snapshot classes
kubectl get volumesnapshotclasses.snapshot.storage.k8s.io
@@ -305,11 +323,10 @@ kubectl get volumesnapshotcontents.snapshot.storage.k8s.io
### To create volume's clone from existing snapshot you should:
- Create snapshotter storage class [Example yaml](https://github.com/rook/rook/tree/master/cluster/examples/kubernetes/edgefs/csi/iscsi/examples/snapshots/snapshot-class.yaml)
- Have an existing PVC based on EdgeFS ISCSI LUN
- Take snapshot from volume [Example yaml](https://github.com/rook/rook/tree/master/cluster/examples/kubernetes/edgefs/csi/iscsi/examples/snapshots/create-snapshot.yaml)
- Clone volume from existing snapshot [Example yaml](https://github.com/rook/rook/tree/master/cluster/examples/kubernetes/edgefs/csi/iscsi/examples/snapshots/nginx-snapshot-clone-volume.yaml)
* Create snapshotter StorageClass [Example yaml](https://github.com/rook/rook/tree/master/cluster/examples/kubernetes/edgefs/csi/iscsi/examples/snapshots/snapshot-class.yaml)
* Have an existing PVC based on EdgeFS ISCSI LUN
* Take snapshot from volume [Example yaml](https://github.com/rook/rook/tree/master/cluster/examples/kubernetes/edgefs/csi/iscsi/examples/snapshots/create-snapshot.yaml)
* Clone volume from existing snapshot [Example yaml](https://github.com/rook/rook/tree/master/cluster/examples/kubernetes/edgefs/csi/iscsi/examples/snapshots/nginx-snapshot-clone-volume.yaml)
## Troubleshooting and log collection
@@ -321,50 +338,50 @@ Please submit an issue at: [Issues](https://github.com/Nexenta/edgefs-csi/issues
## Troubleshooting
- Show installed drivers:
* Show installed drivers:
```bash
kubectl get csidrivers.csi.storage.k8s.io
kubectl describe csidrivers.csi.storage.k8s.io
```
```console
kubectl get csidrivers.csi.storage.k8s.io
kubectl describe csidrivers.csi.storage.k8s.io
```
- Error:
* Error:
```
MountVolume.MountDevice failed for volume "pvc-ns-<...>" :
driver name io.edgefs.csi.iscsi not found in the list of registered CSI drivers
```
```console
MountVolume.MountDevice failed for volume "pvc-ns-<...>" :
driver name io.edgefs.csi.iscsi not found in the list of registered CSI drivers
```
Make sure _kubelet_ is configured with `--root-dir=/var/lib/kubelet`, otherwise update paths in the driver yaml file
([all requirements](https://github.com/kubernetes-csi/docs/blob/387dce893e59c1fcf3f4192cbea254440b6f0f07/book/src/Setup.md#enabling-features)).
- "VolumeSnapshotDataSource" feature gate is disabled:
* "VolumeSnapshotDataSource" feature gate is disabled:
```bash
vim /var/lib/kubelet/config.yaml
# ...
# featureGates:
# VolumeSnapshotDataSource: true
# ...
vim /etc/kubernetes/manifests/kube-apiserver.yaml
# ...
# - --feature-gates=VolumeSnapshotDataSource=true
# ...
```
```console
vim /var/lib/kubelet/config.yaml
# ...
# featureGates:
# VolumeSnapshotDataSource: true
# ...
vim /etc/kubernetes/manifests/kube-apiserver.yaml
# ...
# - --feature-gates=VolumeSnapshotDataSource=true
# ...
```
- Driver logs (for ISCSI driver, to get NFS driver logs substitute iscsi to nfs)
* Driver logs (for ISCSI driver, to get NFS driver logs substitute iscsi to nfs)
```bash
kubectl logs -f edgefs-iscsi-csi-controller-0 driver
kubectl logs -f $(kubectl get pods | awk '/edgefs-iscsi-csi-node-/ {print $1;exit}') driver
# combine all pods:
kubectl get pods | awk '/edgefs-iscsi-csi-node-/ {system("kubectl logs " $1 " driver &")}'
```
```console
kubectl logs -f edgefs-iscsi-csi-controller-0 driver
kubectl logs -f $(kubectl get pods | awk '/edgefs-iscsi-csi-node-/ {print $1;exit}') driver
# combine all pods:
kubectl get pods | awk '/edgefs-iscsi-csi-node-/ {system("kubectl logs " $1 " driver &")}'
```
- Show termination message in case driver failed to run:
* Show termination message in case driver failed to run:
<!-- {% raw %} -->
```bash
kubectl get pod edgefs-iscsi-csi-controller-0 -o go-template="{{range .status.containerStatuses}}{{.lastState.terminated.message}}{{end}}"
```
<!-- {% endraw %} -->
<!-- {% raw %} -->
```console
kubectl get pod edgefs-iscsi-csi-controller-0 -o go-template="{{range .status.containerStatuses}}{{.lastState.terminated.message}}{{end}}"
```
<!-- {% endraw %} -->
Documentation/edgefs-iscsi-crd.md
+28 -28
View File
@@ -46,25 +46,25 @@ spec:
# key: value
```
### Metadata
## Metadata
- `name`: The name of the iSCSI target service to create, which must match existing EdgeFS service.
- `namespace`: The namespace of the Rook cluster where the iSCSI Target service is created.
- `targetName`: The name for iSCSI target name. Default is iqn.2018-11.edgefs.io.
- `targetParams`: If specified, then some of iSCSI target protocol parameters can be overridden.
- `MaxRecvDataSegmentLength`: Value in range value range 512..16777215. Default is 524288.
- `DefaultTime2Retain`: Value in range 0..3600. Default is 60.
- `DefaultTime2Wait`: Value in range 0..3600. Default is 30.
- `FirstBurstLength`: Value in range 512..16777215. Default is 524288.
- `MaxBurstLength`: Value in range 512..16777215. Default is 1048576.
- `MaxQueueCmd`: Value in range 1..128. Default is 64.
- `chunkCacheSize`: Limit amount of memory allocated for dynamic chunk cache. By default iSCSI pod uses up to 75% of available memory as chunk caching area. This option can influence this allocation strategy.
- `annotations`: Key value pair list of annotations to add.
- `placement`: The iSCSI pods can be given standard Kubernetes placement restrictions with `nodeAffinity`, `tolerations`, `podAffinity`, and `podAntiAffinity` similar to placement defined for daemons configured by the [cluster CRD](/cluster/examples/kubernetes/edgefs/cluster.yaml).
- `resourceProfile`: iSCSI pod resource utilization profile (Memory and CPU). Can be `embedded` or `performance` (default). In case of `performance` an iSCSI pod trying to increase amount of internal I/O resources that results in higher performance at the cost of additional memory allocation and more CPU load. In `embedded` profile case, iSCSI pod gives preference to preserving memory over I/O and limiting chunk cache (see `chunkCacheSize` option). The `performance` profile is the default unless cluster wide `embedded` option is defined.
- `resources`: Set resource requests/limits for the iSCSI pods, see [Resource Requirements/Limits](edgefs-cluster-crd.md#resource-requirementslimits).
* `name`: The name of the iSCSI target service to create, which must match existing EdgeFS service.
* `namespace`: The namespace of the Rook cluster where the iSCSI Target service is created.
* `targetName`: The name for iSCSI target name. Default is iqn.2018-11.edgefs.io.
* `targetParams`: If specified, then some of iSCSI target protocol parameters can be overridden.
* `MaxRecvDataSegmentLength`: Value in range value range 512..16777215. Default is 524288.
* `DefaultTime2Retain`: Value in range 0..3600. Default is 60.
* `DefaultTime2Wait`: Value in range 0..3600. Default is 30.
* `FirstBurstLength`: Value in range 512..16777215. Default is 524288.
* `MaxBurstLength`: Value in range 512..16777215. Default is 1048576.
* `MaxQueueCmd`: Value in range 1..128. Default is 64.
* `chunkCacheSize`: Limit amount of memory allocated for dynamic chunk cache. By default iSCSI pod uses up to 75% of available memory as chunk caching area. This option can influence this allocation strategy.
* `annotations`: Key value pair list of annotations to add.
* `placement`: The iSCSI pods can be given standard Kubernetes placement restrictions with `nodeAffinity`, `tolerations`, `podAffinity`, and `podAntiAffinity` similar to placement defined for daemons configured by the [cluster CRD](/cluster/examples/kubernetes/edgefs/cluster.yaml).
* `resourceProfile`: iSCSI pod resource utilization profile (Memory and CPU). Can be `embedded` or `performance` (default). In case of `performance` an iSCSI pod trying to increase amount of internal I/O resources that results in higher performance at the cost of additional memory allocation and more CPU load. In `embedded` profile case, iSCSI pod gives preference to preserving memory over I/O and limiting chunk cache (see `chunkCacheSize` option). The `performance` profile is the default unless cluster wide `embedded` option is defined.
* `resources`: Set resource requests/limits for the iSCSI pods, see [Resource Requirements/Limits](edgefs-cluster-crd.md#resource-requirementslimits).
### Setting up EdgeFS namespace and tenant
## Setting up EdgeFS namespace and tenant
For more detailed instructions please refer to [EdgeFS Wiki](https://github.com/Nexenta/edgefs/wiki).
@@ -78,37 +78,37 @@ Root object holds system information and table of namespaces registered to a loc
To initialize system and prepare logical definitions, login to the toolbox as shown in this example:
```
```console
kubectl get po --all-namespaces | grep edgefs-mgr
kubectl exec -it -n rook-edgefs rook-edgefs-mgr-6cb9598469-czr7p -- env COLUMNS=$COLUMNS LINES=$LINES TERM=linux toolbox
```
Assumption at this point is that nodes are all configured and can be seen via the following command:
```
```console
efscli system status
```
1. Initialize cluster
1. Initialize EdgeFS cluster:
Verify that HW (or better say emulated in this case) configuration look normal and accept it
```
```console
efscli system init
```
At this point new dynamically discovered configuration checkpoint will be created at $NEDGE_HOME/var/run/flexhash-checkpoint.json
This will also create system "root" object, holding Site's Namespace. Namespace may consist of more then single region.
2. Create new local namespace (or we also call it "Region" or "Segment")
2. Create new local namespace (or we also call it "Region" or "Segment"):
```
```console
efscli cluster create Hawaii
```
3. Create logical tenants of cluster namespace "Hawaii", also buckets if needed
3. Create logical tenants of cluster namespace "Hawaii", also buckets if needed:
```
```console
efscli tenant create Hawaii/Cola
efscli bucket create Hawaii/Cola/bk1
efscli tenant create Hawaii/Pepsi
@@ -117,9 +117,9 @@ efscli bucket create Hawaii/Pepsi/bk1
Now cluster is setup, services can be now created.
4. Create iSCSI Target services objects for tenants
4. Create iSCSI Target services objects for tenants:
```
```console
efscli service create iscsi isc-cola
efscli service serve isc-cola Hawaii/Cola/bk1/lun1 X-volsize=10G,ccow-chunkmap-chunk-size=16384
efscli service serve isc-cola Hawaii/Cola/bk1/lun2 X-volsize=20G,ccow-chunkmap-chunk-size=131072
@@ -127,7 +127,7 @@ efscli service create iscsi isc-pepsi
efscli service serve isc-pepsi Hawaii/Pepsi/bk1/lun1 X-volsize=20G
```
5. Create ISCSI CRDs
5. Create ISCSI CRDs:
```yaml
apiVersion: edgefs.rook.io/v1
Documentation/edgefs-isgw-crd.md
+29 -29
View File
@@ -81,24 +81,24 @@ spec:
# key: value
```
### Metadata
## Metadata
- `name`: The name of the ISGW link service to create, which must match existing EdgeFS service. Provided CRD metadata can override existing EdgeFS service metadata.
- `namespace`: The namespace of the Rook cluster where the ISGW service is created.
- `direction`: I/O flow direction: `send`, `receive` or `send+receive` (default).
- `remoteURL`: I/O flow with destination pointing to the other ISGW endpoint service, locally mounted directory or other (non-EdgeFS) S3 compatible service. Format should be of form PROTOCOL://LOCATION, where PROTOCOL can be of "ccow", "file", "s3" and LOCATION depends on the PROTOCOL. In case of "ccow" LOCATION needs to specify a HOST:PORT of the other EdgeFS site, e.g. ccow://192.168.1.43:14000. In case of "file" LOCATION needs to be a path to a local POSIX directory, e.g. file://volumeDir. In cae of "s3" LOCATION can either point to an existing bucket, e.g. s3://mybucket.aws.com or to the other S3 compatible server IP:HOST, e.g. s3://214.4.34.22:4567. It is important to note that file:// and s3:// protocols only provided for easy integration with existing infrastructure.
- `replicationType`: Defines how tenant's buckets needs to be replicated. Can be specified as `initial`, `continuous` or `initial+continuous` (default). In case of `initial` replication type, ISGW service will only sync served buckets once with ability to manually trigger re-sync again. In case of `continuous` replication type, ISGW service will enable asynchronous syncing from the moment a bucket added to a served list of buckets, so that only recent changes will be synchronized, continuously. In case of `initial+continuous` replication type, ISGW service will first initiate syncing of all existing served buckets contents and once all is copied, it will automatically enable `continuous` mode of operation, guaranteeing that none of the source objects are skipped.
- `metadataOnly`: If set, then only immutable metadata changes will be transferred. This also enables on-demand data fetch and E-LRU cache capability. In case of metadata only transfers, dynamically fetched data will be cached at the accessing endpoints ISGW links locations. If an application would want to read an offset of a data which isn't found, global search request will be formed and distributed across all the connected ISGW bi-directional links serving source bucket. Missing data chunks will be found and fetched, delivered transparently to application. Can be set to either `all` or `versions`. Setting it to `versions` enables extremely fast replication (next second!) of data modifications.
- `dynamicFetchAddr`: Defines local cluster pod service host:port. Used by EdgeFS gateways (S3, S3X, NFS or iSCSI) to distribute missing chunk fetch requests. It needs to be set to other than 0.0.0.0 on access endpoints so to enable dynamic fetch feature. Default is 0.0.0.0:49678.
- `localAddr`: Defines local cluster pod service address and port. Used by ISGW link endpoint. Kubernetes ExternalIP will be automatically constructed if address is provided. Default is 0.0.0.0:14000.
- `useEncryptedTunnel`: If true, enables usage of encrypted SSH tunnel. By default, transferred metadata and data chunks are just compressed but unencrypted. Enable encryption if ISGW links crossing boundaries of single data center or multiple clouds.
- `chunkCacheSize`: Limit amount of memory allocated for dynamic chunk cache. By default ISGW pod uses up to 75% of available memory as chunk caching area. This option can influence this allocation strategy.
- `annotations`: Key value pair list of annotations to add.
- `placement`: The ISGW pods can be given standard Kubernetes placement restrictions with `nodeAffinity`, `tolerations`, `podAffinity`, and `podAntiAffinity` similar to placement defined for daemons configured by the [cluster CRD](/cluster/examples/kubernetes/edgefs/cluster.yaml).
- `resourceProfile`: ISGW pod resource utilization profile (Memory and CPU). Can be `embedded` or `performance` (default). In case of `performance` an ISGW pod trying to increase amount of internal I/O resources that results in higher performance at the cost of additional memory allocation and more CPU load. In `embedded` profile case, ISGW pod gives preference to preserving memory over I/O and limiting chunk cache (see `chunkCacheSize` option). The `performance` profile is the default unless cluster wide `embedded` option is defined.
- `resources`: Set resource requests/limits for the ISGW pods, see [Resource Requirements/Limits](edgefs-cluster-crd.md#resource-requirementslimits).
* `name`: The name of the ISGW link service to create, which must match existing EdgeFS service. Provided CRD metadata can override existing EdgeFS service metadata.
* `namespace`: The namespace of the Rook cluster where the ISGW service is created.
* `direction`: I/O flow direction: `send`, `receive` or `send+receive` (default).
* `remoteURL`: I/O flow with destination pointing to the other ISGW endpoint service, locally mounted directory or other (non-EdgeFS) S3 compatible service. Format should be of form PROTOCOL://LOCATION, where PROTOCOL can be of "ccow", "file", "s3" and LOCATION depends on the PROTOCOL. In case of "ccow" LOCATION needs to specify a HOST:PORT of the other EdgeFS site, e.g. ccow://192.168.1.43:14000. In case of "file" LOCATION needs to be a path to a local POSIX directory, e.g. file://volumeDir. In cae of "s3" LOCATION can either point to an existing bucket, e.g. s3://mybucket.aws.com or to the other S3 compatible server IP:HOST, e.g. s3://214.4.34.22:4567. It is important to note that file:// and s3:// protocols only provided for easy integration with existing infrastructure.
* `replicationType`: Defines how tenant's buckets needs to be replicated. Can be specified as `initial`, `continuous` or `initial+continuous` (default). In case of `initial` replication type, ISGW service will only sync served buckets once with ability to manually trigger re-sync again. In case of `continuous` replication type, ISGW service will enable asynchronous syncing from the moment a bucket added to a served list of buckets, so that only recent changes will be synchronized, continuously. In case of `initial+continuous` replication type, ISGW service will first initiate syncing of all existing served buckets contents and once all is copied, it will automatically enable `continuous` mode of operation, guaranteeing that none of the source objects are skipped.
* `metadataOnly`: If set, then only immutable metadata changes will be transferred. This also enables on-demand data fetch and E-LRU cache capability. In case of metadata only transfers, dynamically fetched data will be cached at the accessing endpoints ISGW links locations. If an application would want to read an offset of a data which isn't found, global search request will be formed and distributed across all the connected ISGW bi-directional links serving source bucket. Missing data chunks will be found and fetched, delivered transparently to application. Can be set to either `all` or `versions`. Setting it to `versions` enables extremely fast replication (next second!) of data modifications.
* `dynamicFetchAddr`: Defines local cluster pod service host:port. Used by EdgeFS gateways (S3, S3X, NFS or iSCSI) to distribute missing chunk fetch requests. It needs to be set to other than 0.0.0.0 on access endpoints so to enable dynamic fetch feature. Default is 0.0.0.0:49678.
* `localAddr`: Defines local cluster pod service address and port. Used by ISGW link endpoint. Kubernetes ExternalIP will be automatically constructed if address is provided. Default is 0.0.0.0:14000.
* `useEncryptedTunnel`: If true, enables usage of encrypted SSH tunnel. By default, transferred metadata and data chunks are just compressed but unencrypted. Enable encryption if ISGW links crossing boundaries of single data center or multiple clouds.
* `chunkCacheSize`: Limit amount of memory allocated for dynamic chunk cache. By default ISGW pod uses up to 75% of available memory as chunk caching area. This option can influence this allocation strategy.
* `annotations`: Key value pair list of annotations to add.
* `placement`: The ISGW pods can be given standard Kubernetes placement restrictions with `nodeAffinity`, `tolerations`, `podAffinity`, and `podAntiAffinity` similar to placement defined for daemons configured by the [cluster CRD](/cluster/examples/kubernetes/edgefs/cluster.yaml).
* `resourceProfile`: ISGW pod resource utilization profile (Memory and CPU). Can be `embedded` or `performance` (default). In case of `performance` an ISGW pod trying to increase amount of internal I/O resources that results in higher performance at the cost of additional memory allocation and more CPU load. In `embedded` profile case, ISGW pod gives preference to preserving memory over I/O and limiting chunk cache (see `chunkCacheSize` option). The `performance` profile is the default unless cluster wide `embedded` option is defined.
* `resources`: Set resource requests/limits for the ISGW pods, see [Resource Requirements/Limits](edgefs-cluster-crd.md#resource-requirementslimits).
### Setting up two-sites bi-directional syncing EdgeFS namespace and tenants
## Setting up two-sites bi-directional syncing EdgeFS namespace and tenants
This example will demonstrate creation of simple syncing namespace between two sites.
@@ -114,37 +114,37 @@ Root object holds system information and table of namespaces registered to a loc
For each cluster, to initialize system and prepare logical definitions, login to the toolbox as shown in this example:
```
```console
kubectl get po --all-namespaces | grep edgefs-mgr
kubectl exec -it -n rook-edgefs rook-edgefs-mgr-6cb9598469-czr7p -- env COLUMNS=$COLUMNS LINES=$LINES TERM=linux toolbox
```
Assumption at this point is that nodes are all configured and can be seen via the following command:
```
```console
efscli system status
```
1. Initialize cluster
1. Initialize EdgeFS cluster:
Verify that HW (or better say emulated in this case) configuration look normal and accept it
```
```console
efscli system init
```
At this point new dynamically discovered configuration checkpoint will be created at `$NEDGE_HOME/var/run/flexhash-checkpoint.json`.
This will also create system "root" object, holding Site's Namespace. Namespace may consist of more then single region.
2. Create new local namespace (or we also call it "Region" or "Segment")
2. Create new local namespace (or we also call it "Region" or "Segment"):
```
```console
efscli cluster create Hawaii
```
3. Create logical tenants of cluster namespace "Hawaii", also buckets if needed
3. Create logical tenants of cluster namespace "Hawaii", also buckets if needed:
```
```console
efscli tenant create Hawaii/Cola
efscli bucket create Hawaii/Cola/bk1
efscli tenant create Hawaii/Pepsi
@@ -153,7 +153,7 @@ efscli bucket create Hawaii/Pepsi/bk1
Now that Hawaii serving namespace cluster is setup, repeat steps on the secondary site:
```
```console
efscli cluster create Hawaii
efscli tenant create Hawaii/Cola -r 2
efscli tenant create Hawaii/Pepsi -r 2
@@ -162,14 +162,14 @@ efscli tenant create Hawaii/Pepsi -r 2
This steps also includes a possibility to override destination defaults on per-tenant (Cola and Pepsi) basis.
In the example above we setting replication count to 2 on the secondary site.
4. Create ISGW link services on both EdgeFS sites
4. Create ISGW link services on both EdgeFS sites:
Order of creation of ISGW links doesn't matter.
Link status change will be noticed automatically and syncing can be delayed for up to 7 days by default.
On the primary site:
```
```console
efscli service create isgw hawaii
efscli service serve hawaii Hawaii/Cola/bk1
efscli service serve hawaii Hawaii/Pepsi/bk1
@@ -177,7 +177,7 @@ efscli service serve hawaii Hawaii/Pepsi/bk1
Because we creating bi-directional link, on the secondary site we would need to create exactly same configuration except X-ISGW-Remote (remoteURL in CRD) where it should point back to primary. Synchronization is enabled on per-bucket level. Buckets which do not yet exist will be automatically created with tenant parameter overrides taken into an account.
5. Create ISGW Link CRDs on both EdgeFS sites
5. Create ISGW Link CRDs on both EdgeFS sites:
```yaml
apiVersion: edgefs.rook.io/v1
@@ -211,6 +211,6 @@ EdgeFS UI Dashboard automates the process of CRD composing but also has a capabi
At this point two ISGW link services should be operational. Verify its operation with `show` command:
```
```console
efscli service show hawaii -s
```
Documentation/edgefs-monitoring.md
+22 -13
View File
@@ -11,44 +11,51 @@ If you do not have Prometheus running, follow the steps below to enable monitori
contains a Prometheus instance, it will automatically discover Rooks scrape endpoint using the standard
`prometheus.io/scrape` and `prometheus.io/port` annotations.
## Prometheus Operator
First the Prometheus operator needs to be started in the cluster so it can watch for our requests to start monitoring Rook and respond by deploying the correct Prometheus pods and configuration.
A full explanation can be found in the [Prometheus operator repository on GitHub](https://github.com/coreos/prometheus-operator), but the quick instructions can be found here:
```bash
```console
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/v0.27.0/bundle.yaml
```
This will start the Prometheus operator, but before moving on, wait until the operator is in the `Running` state:
```bash
```console
kubectl get pod
```
Once the Prometheus operator is in the `Running` state, proceed to the next section.
## Prometheus Instances
With the Prometheus operator running, we can create a service monitor that will watch the Rook cluster and collect metrics regularly.
From the root of your locally cloned Rook repo, go the monitoring directory:
```bash
```console
cd cluster/examples/kubernetes/edgefs/monitoring
```
Create the service monitor as well as the Prometheus server pod and service:
```bash
```console
kubectl create -f service-monitor.yaml
kubectl create -f prometheus.yaml
kubectl create -f prometheus-service.yaml
```
Ensure that the Prometheus server pod gets created and advances to the `Running` state before moving on:
```bash
```console
kubectl -n rook-edgefs get pod prometheus-rook-prometheus-0
```
## Prometheus Web Console
Once the Prometheus server is running, you can open a web browser and go to the URL that is output from this command:
```bash
```console
echo "http://$(kubectl -n rook-edgefs -o jsonpath={.status.hostIP} get pod prometheus-rook-prometheus-0):30900"
```
@@ -56,34 +63,36 @@ You should now see the Prometheus monitoring website.
![Prometheus Monitoring Website](media/prometheus-monitor.png)
Click on `Graph` in the top navigation bar.
![Prometheus Add graph](media/prometheus-graph.png)
In the dropdown that says `insert metric at cursor`, select any metric you would like to see, for example `nedge_cluster_objects`.
Below the `Execute` button, ensure the `Graph` tab is selected and you should now see a graph of your chosen metric over time.
## Prometheus Consoles
A guide to how you can write your own Prometheus consoles can be found on the official Prometheus site here: https://prometheus.io/docs/visualization/consoles/.
A guide to how you can write your own Prometheus consoles can be found on the official Prometheus site here: [Prometheus.io Documentation - Console Templates](https://prometheus.io/docs/visualization/consoles/).
## Grafana Dashboards
For feedback on the dashboards please reach out to him on the [Rook.io Slack](https://slack.rook.io).
> **NOTE** The dashboard only compatible with Grafana 5.0.3 or higher.
> **NOTE**: The dashboard only compatible with Grafana 5.0.3 or higher.
The following Grafana dashboard is available:
* [EdgeFS Cluster](https://grafana.com/dashboards/9683)
## Teardown
To clean up all the artifacts created by the monitoring walkthrough, copy/paste the entire block below (note that errors about resources "not found" can be ignored):
```bash
```console
kubectl delete -f service-monitor.yaml
kubectl delete -f prometheus.yaml
kubectl delete -f prometheus-service.yaml
kubectl delete -f https://raw.githubusercontent.com/coreos/prometheus-operator/v0.27.0/bundle.yaml
```
Then the rest of the instructions in the [Prometheus Operator docs](https://github.com/coreos/prometheus-operator#removal) can be followed to finish cleaning up.
Documentation/edgefs-nfs-crd.md
+25 -25
View File
@@ -6,7 +6,7 @@ indent: true
# EdgeFS Scale-Out NFS CRD
Rook allows creation and customization of EdgeFS NFS file systems through the custom resource definitions (CRDs).
Rook allows creation and customization of EdgeFS NFS filesystems through the custom resource definitions (CRDs).
The following settings are available for customization of EdgeFS NFS services.
## Sample
@@ -48,19 +48,19 @@ spec:
# memory: "1024Mi"
```
### Metadata
## Metadata
- `name`: The name of the NFS system to create, which must match existing EdgeFS service.
- `namespace`: The namespace of the Rook cluster where the NFS service is created.
- `instances`: The number of active NFS service instances. EdgeFS NFS service is Multi-Head capable, such so that multiple PODs can mount same tenant's buckets via different endpoints. [EdgeFS CSI provisioner](edgefs-csi.md) orchestrates distribution and load balancing across NFS service instances in round-robin or random policy ways.
- `relaxedDirUpdates`: If set to `true` then it will significantly improve performance of directory operations by defering updates, guaranteeing eventual directory consistency. This option is recommended when a bucket exported via single NFS instance and it is not a destination for ISGW Link synchronization.
- `chunkCacheSize`: Limit amount of memory allocated for dynamic chunk cache. By default NFS pod uses up to 75% of available memory as chunk caching area. This option can influence this allocation strategy.
- `annotations`: Key value pair list of annotations to add.
- `placement`: The NFS PODs can be given standard Kubernetes placement restrictions with `nodeAffinity`, `tolerations`, `podAffinity`, and `podAntiAffinity` similar to placement defined for daemons configured by the [cluster CRD](/cluster/examples/kubernetes/edgefs/cluster.yaml).
- `resourceProfile`: NFS pod resource utilization profile (Memory and CPU). Can be `embedded` or `performance` (default). In case of `performance` an NFS pod trying to increase amount of internal I/O resources that results in higher performance at the cost of additional memory allocation and more CPU load. In `embedded` profile case, NFS pod gives preference to preserving memory over I/O and limiting chunk cache (see `chunkCacheSize` option). The `performance` profile is the default unless cluster wide `embedded` option is defined.
- `resources`: Set resource requests/limits for the NFS Pod(s), see [Resource Requirements/Limits](edgefs-cluster-crd.md#resource-requirementslimits).
* `name`: The name of the NFS system to create, which must match existing EdgeFS service.
* `namespace`: The namespace of the Rook cluster where the NFS service is created.
* `instances`: The number of active NFS service instances. EdgeFS NFS service is Multi-Head capable, such so that multiple PODs can mount same tenant's buckets via different endpoints. [EdgeFS CSI provisioner](edgefs-csi.md) orchestrates distribution and load balancing across NFS service instances in round-robin or random policy ways.
* `relaxedDirUpdates`: If set to `true` then it will significantly improve performance of directory operations by defering updates, guaranteeing eventual directory consistency. This option is recommended when a bucket exported via single NFS instance and it is not a destination for ISGW Link synchronization.
* `chunkCacheSize`: Limit amount of memory allocated for dynamic chunk cache. By default NFS pod uses up to 75% of available memory as chunk caching area. This option can influence this allocation strategy.
* `annotations`: Key value pair list of annotations to add.
* `placement`: The NFS PODs can be given standard Kubernetes placement restrictions with `nodeAffinity`, `tolerations`, `podAffinity`, and `podAntiAffinity` similar to placement defined for daemons configured by the [cluster CRD](/cluster/examples/kubernetes/edgefs/cluster.yaml).
* `resourceProfile`: NFS pod resource utilization profile (Memory and CPU). Can be `embedded` or `performance` (default). In case of `performance` an NFS pod trying to increase amount of internal I/O resources that results in higher performance at the cost of additional memory allocation and more CPU load. In `embedded` profile case, NFS pod gives preference to preserving memory over I/O and limiting chunk cache (see `chunkCacheSize` option). The `performance` profile is the default unless cluster wide `embedded` option is defined.
* `resources`: Set resource requests/limits for the NFS Pod(s), see [Resource Requirements/Limits](edgefs-cluster-crd.md#resource-requirementslimits).
### Setting up EdgeFS namespace and tenant
## Setting up EdgeFS namespace and tenant
For more detailed instructions please refer to [EdgeFS Wiki](https://github.com/Nexenta/edgefs/wiki).
@@ -74,37 +74,37 @@ Root object holds system information and table of namespaces registered to a loc
To initialize system and prepare logical definitions, login to the toolbox as shown in this example:
```
```console
kubectl get po --all-namespaces | grep edgefs-mgr
kubectl exec -it -n rook-edgefs rook-edgefs-mgr-6cb9598469-czr7p -- env COLUMNS=$COLUMNS LINES=$LINES TERM=linux toolbox
```
Assumption at this point is that nodes are all configured and can be seen via the following command:
```
```console
efscli system status
```
1. Initialize cluster
1. Initialize EdgeFS cluster:
Verify that HW (or better say emulated in this case) configuration look normal and accept it
Verify that HW (or better say emulated in this case) configuration look normal and accept it:
```
```console
efscli system init
```
At this point new dynamically discovered configuration checkpoint will be created at $NEDGE_HOME/var/run/flexhash-checkpoint.json
This will also create system "root" object, holding Site's Namespace. Namespace may consist of more then single region.
2. Create new local namespace (or we also call it "Region" or "Segment")
2. Create new local namespace (or we also call it "Region" or "Segment"):
```
```console
efscli cluster create Hawaii
```
3. Create logical tenants of cluster namespace "Hawaii", also buckets if needed
3. Create logical tenants of cluster namespace "Hawaii", also buckets if needed:
```
```console
efscli tenant create Hawaii/Cola
efscli bucket create Hawaii/Cola/bk1
efscli tenant create Hawaii/Pepsi
@@ -113,16 +113,16 @@ efscli bucket create Hawaii/Pepsi/bk1
Now cluster is setup, services can be now created and attached to CSI provisioner.
4. Create NFS service objects for tenants
4. Create NFS service objects for tenants:
```
```console
efscli service create nfs nfs-cola
efscli service serve nfs-cola Hawaii/Cola/bk1
efscli service create nfs nfs-pepsi
efscli service serve nfs-pepsi Hawaii/Pepsi/bk1
```
5. Create NFS CRDs
5. Create your EdgeFS NFS objects:
```yaml
apiVersion: edgefs.rook.io/v1
@@ -146,7 +146,7 @@ spec:
At this point two NFS services should be available. Verify that showmount command can see service (substitute CLUSTERIP with corresponding entry from `kubectl get svc` command):
```
```console
kubectl get svc --all-namespaces
showmount -e CLUSTERIP
```
Documentation/edgefs-quickstart.md
+19 -17
View File
@@ -23,9 +23,9 @@ If you are using `dataDirHostPath` to persist rook data on kubernetes hosts, mak
We recommend you to configure EdgeFS to use of raw devices and equal distribution of available storage capacity.
**IMPORTANT** EdgeFS will automatically adjust deployment nodes to use larger then 128KB data chunks, with the following addition to /etc/sysctl.conf:
> **IMPORTANT**: EdgeFS will automatically adjust deployment nodes to use larger then 128KB data chunks, with the following addition to /etc/sysctl.conf:
```
```ini
net.core.rmem_default = 80331648
net.core.rmem_max = 80331648
net.core.wmem_default = 33554432
@@ -34,12 +34,14 @@ vm.dirty_ratio = 10
vm.dirty_background_ratio = 5
vm.swappiness = 15
```
To turn off this node adjustment need to enable `skipHostPrepare` option in cluster CRD [configuring the cluster](edgefs-cluster-crd.md)
To turn off this node adjustment need to enable `skipHostPrepare` option in cluster CRD [configuring the cluster](edgefs-cluster-crd.md).
## TL;DR
If you're feeling lucky, a simple EdgeFS Rook cluster can be created with the following kubectl commands. For the more detailed install, skip to the next section to [deploy the Rook operator](#deploy-the-rook-operator).
```
```console
cd cluster/examples/kubernetes/edgefs
kubectl create -f operator.yaml
kubectl create -f cluster.yaml
@@ -53,13 +55,13 @@ The first step is to deploy the Rook system components, which include the Rook a
Note that Google Cloud users need to explicitly grant user permission to create roles:
```bash
```console
kubectl create clusterrolebinding cluster-admin-binding --clusterrole cluster-admin --user $(gcloud config get-value account)
```
Now you ready to create operator:
```bash
```console
cd cluster/examples/kubernetes/edgefs
kubectl create -f operator.yaml
@@ -74,19 +76,18 @@ You can also deploy the operator with the [Rook EdgeFS Helm Chart](edgefs-helm-o
Now that the Rook operator, and discover pods are running, we can create the Rook cluster. For the cluster to survive reboots,
make sure you set the `dataDirHostPath` property. For more settings, see the documentation on [configuring the cluster](edgefs-cluster-crd.md).
Edit the cluster spec in `cluster.yaml` file.
Create the cluster:
```bash
```console
kubectl create -f cluster.yaml
```
Use `kubectl` to list pods in the `rook-edgefs` namespace. You should be able to see the following pods once they are all running.
The number of target pods will depend on the number of nodes in the cluster and the number of devices and directories configured.
```bash
```console
$ kubectl -n rook-edgefs get pod
rook-edgefs rook-edgefs-mgr-7c76cb564d-56sxb 1/1 Running 0 24s
rook-edgefs rook-edgefs-target-0 3/3 Running 0 24s
@@ -96,24 +97,25 @@ rook-edgefs rook-edgefs-target-2 3/3 Running 0
Notice that EdgeFS Targets are running as StatefulSet.
# Storage
## Storage
For a walkthrough of the types of Storage CRDs exposed by EdgeFS Rook, see the guides for:
- **[NFS Server](edgefs-nfs-crd.md)**: Create Scale-Out NFS storage to be consumed by multiple pods, simultaneously
- **[S3X](edgefs-s3x-crd.md)**: Create an Extended S3 HTTP/2 compatible object and key-value store that is accessible inside or outside the Kubernetes cluster
- **[AWS S3](edgefs-s3-crd.md)**: Create an AWS S3 compatible object store that is accessible inside or outside the Kubernetes cluster
- **[iSCSI Target](edgefs-iscsi-crd.md)**: Create low-latency and high-throughput iSCSI block to be consumed by a pod
# CSI Integration
* **[NFS Server](edgefs-nfs-crd.md)**: Create Scale-Out NFS storage to be consumed by multiple pods, simultaneously
* **[S3X](edgefs-s3x-crd.md)**: Create an Extended S3 HTTP/2 compatible object and key-value store that is accessible inside or outside the Kubernetes cluster
* **[AWS S3](edgefs-s3-crd.md)**: Create an AWS S3 compatible object store that is accessible inside or outside the Kubernetes cluster
* **[iSCSI Target](edgefs-iscsi-crd.md)**: Create low-latency and high-throughput iSCSI block to be consumed by a pod
## CSI Integration
EdgeFS comes with built-in gRPC management services, which are tightly integrated into Kubernetes provisioning and attaching CSI framework. Please see the [EdgeFS CSI](edgefs-csi.md) for setup and usage information.
# EdgeFS Dashboard and Monitoring
## EdgeFS Dashboard and Monitoring
Each Rook cluster has some built in metrics collectors/exporters for monitoring with [Prometheus](https://prometheus.io/).
In additional to classic monitoring parameters, it has a built-in multi-tenancy, multi-service capability. For instance, you can quickly discover most loaded tenants, buckets or services.
To learn how to set up monitoring for your Rook cluster, you can follow the steps in the [monitoring guide](./edgefs-monitoring.md).
# Teardown
## Teardown
When you are done with the cluster, simply delete CRDs in reverse order. You may want to re-format your raw disks with `wipefs -a` command. Or if you using raw devices and want to keep same storage configuration but change some resource or networking parameters, consider to use `devicesResurrectMode`.
Documentation/edgefs-s3-crd.md
+26 -26
View File
@@ -48,22 +48,22 @@ spec:
# memory: "1024Mi"
```
### Metadata
## Metadata
- `name`: The name of the S3 system to create, which must match existing EdgeFS service.
- `namespace`: The namespace of the Rook cluster where the S3 service is created.
- `s3type`: The type of S3 service to be created. It can be one of the following: `s3` (default, path style) or `s3s` (buckets as DNS style)
- `sslCertificateRef`: If the certificate is not specified, SSL will use default crt and key files. If specified, this is the name of the Kubernetes secret that contains the SSL certificate to be used for secure connections. Please see [secret YAML file example](/cluster/examples/kubernetes/edgefs/sslKeyCertificate.yaml) on how to setup Kuberenetes secret. Notice that base64 encoding is required.
- `port`: The port on which the S3 pods and the S3 service will be listening (not encrypted). Default port is 9982 for `s3` and 9983 for `s3s`.
- `securePort`: The secure port on which S3 pods will be listening. If not defined then default SSL certificates will be used. Default port is 8443 for `s3` and 8444 for `s3s`.
- `chunkCacheSize`: Limit amount of memory allocated for dynamic chunk cache. By default S3 pod uses up to 75% of available memory as chunk caching area. This option can influence this allocation strategy.
- `instances`: The number of active S3 service instances. For load balancing we recommend to use nginx and the like solutions.
- `annotations`: Key value pair list of annotations to add.
- `placement`: The S3 pods can be given standard Kubernetes placement restrictions with `nodeAffinity`, `tolerations`, `podAffinity`, and `podAntiAffinity` similar to placement defined for daemons configured by the [cluster CRD](/cluster/examples/kubernetes/edgefs/cluster.yaml).
- `resourceProfile`: S3 pod resource utilization profile (Memory and CPU). Can be `embedded` or `performance` (default). In case of `performance` an S3 pod trying to increase amount of internal I/O resources that results in higher performance at the cost of additional memory allocation and more CPU load. In `embedded` profile case, S3 pod gives preference to preserving memory over I/O and limiting chunk cache (see `chunkCacheSize` option). The `performance` profile is the default unless cluster wide `embedded` option is defined.
- `resources`: Set resource requests/limits for the S3 pods, see [Resource Requirements/Limits](edgefs-cluster-crd.md#resource-requirementslimits).
* `name`: The name of the S3 system to create, which must match existing EdgeFS service.
* `namespace`: The namespace of the Rook cluster where the S3 service is created.
* `s3type`: The type of S3 service to be created. It can be one of the following: `s3` (default, path style) or `s3s` (buckets as DNS style)
* `sslCertificateRef`: If the certificate is not specified, SSL will use default crt and key files. If specified, this is the name of the Kubernetes secret that contains the SSL certificate to be used for secure connections. Please see [secret YAML file example](/cluster/examples/kubernetes/edgefs/sslKeyCertificate.yaml) on how to setup Kuberenetes secret. Notice that base64 encoding is required.
* `port`: The port on which the S3 pods and the S3 service will be listening (not encrypted). Default port is 9982 for `s3` and 9983 for `s3s`.
* `securePort`: The secure port on which S3 pods will be listening. If not defined then default SSL certificates will be used. Default port is 8443 for `s3` and 8444 for `s3s`.
* `chunkCacheSize`: Limit amount of memory allocated for dynamic chunk cache. By default S3 pod uses up to 75% of available memory as chunk caching area. This option can influence this allocation strategy.
* `instances`: The number of active S3 service instances. For load balancing we recommend to use nginx and the like solutions.
* `annotations`: Key value pair list of annotations to add.
* `placement`: The S3 pods can be given standard Kubernetes placement restrictions with `nodeAffinity`, `tolerations`, `podAffinity`, and `podAntiAffinity` similar to placement defined for daemons configured by the [cluster CRD](/cluster/examples/kubernetes/edgefs/cluster.yaml).
* `resourceProfile`: S3 pod resource utilization profile (Memory and CPU). Can be `embedded` or `performance` (default). In case of `performance` an S3 pod trying to increase amount of internal I/O resources that results in higher performance at the cost of additional memory allocation and more CPU load. In `embedded` profile case, S3 pod gives preference to preserving memory over I/O and limiting chunk cache (see `chunkCacheSize` option). The `performance` profile is the default unless cluster wide `embedded` option is defined.
* `resources`: Set resource requests/limits for the S3 pods, see [Resource Requirements/Limits](edgefs-cluster-crd.md#resource-requirementslimits).
### Setting up EdgeFS namespace and tenant
## Setting up EdgeFS namespace and tenant
For more detailed instructions please refer to [EdgeFS Wiki](https://github.com/Nexenta/edgefs/wiki).
@@ -77,37 +77,37 @@ Root object holds system information and table of namespaces registered to a loc
To initialize system and prepare logical definitions, login to the toolbox as shown in this example:
```
```console
kubectl get po --all-namespaces | grep edgefs-mgr
kubectl exec -it -n rook-edgefs rook-edgefs-mgr-6cb9597469-czr7p -- env COLUMNS=$COLUMNS LINES=$LINES TERM=linux toolbox
```
Assumption at this point is that nodes are all configured and can be seen via the following command:
```
```console
efscli system status
```
1. Initialize cluster
1. Initialize EdgeFS cluster:
Verify that HW (or better say emulated in this case) configuration look normal and accept it
```
```console
efscli system init
```
At this point new dynamically discovered configuration checkpoint will be created at $NEDGE_HOME/var/run/flexhash-checkpoint.json
This will also create system "root" object, holding Site's Namespace. Namespace may consist of more then single region.
2. Create new local namespace (or we also call it "Region" or "Segment")
2. Create new local namespace (or we also call it "Region" or "Segment"):
```
```console
efscli cluster create Hawaii
```
3. Create logical tenants of cluster namespace "Hawaii", also buckets if needed
3. Create logical tenants of cluster namespace "Hawaii", also buckets if needed:
```
```console
efscli tenant create Hawaii/Cola
efscli bucket create Hawaii/Cola/bk1
efscli tenant create Hawaii/Pepsi
@@ -116,9 +116,9 @@ efscli bucket create Hawaii/Pepsi/bk1
Now cluster is setup, services can be now created.
4. Create S3 services objects for tenants
4. Create S3 services objects for tenants:
```
```console
efscli service create s3 s3-cola
efscli service serve s3-cola Hawaii/Cola
efscli service create s3 s3-pepsi
@@ -127,12 +127,12 @@ efscli service serve s3-pepsi Hawaii/Pepsi
In case of s3type set to `s3`, do not forget to configure default domain name:
```
```console
efscli service config s3-cola X-Domain cola.com
efscli service config s3-pepsi X-Domain pepsi.com
```
5. Create S3 CRDs
5. Create EdgeFS S3 objects:
```yaml
apiVersion: edgefs.rook.io/v1
Documentation/edgefs-s3x-crd.md
+24 -24
View File
@@ -51,21 +51,21 @@ spec:
# memory: "1024Mi"
```
### Metadata
## Metadata
- `name`: The name of the Edge-X S3 system to create, which must match existing EdgeFS service.
- `namespace`: The namespace of the Rook cluster where the Edge-X S3 service is created.
- `sslCertificateRef`: If the certificate is not specified, SSL will use default crt and key files. If specified, this is the name of the Kubernetes secret that contains the SSL certificate to be used for secure connections. Please see [secret YAML file example](/cluster/examples/kubernetes/edgefs/sslKeyCertificate.yaml) on how to setup Kuberenetes secret. Notice that base64 encoding is required.
- `port`: The port on which the Edge-X S3 pods and the Edge-X S3 service will be listening (not encrypted). Default port is 3000.
- `securePort`: The secure port on which Edge-X S3 pods will be listening. If not defined then default SSL certificates will be used. Default port is 3001.
- `instances`: The number of active Edge-X S3 service instances. For load balancing we recommend to use nginx and the like solutions.
- `chunkCacheSize`: Limit amount of memory allocated for dynamic chunk cache. By default S3X pod uses up to 75% of available memory as chunk caching area. This option can influence this allocation strategy.
- `annotations`: Key value pair list of annotations to add.
- `placement`: The Edge-X S3 PODs can be given standard Kubernetes placement restrictions with `nodeAffinity`, `tolerations`, `podAffinity`, and `podAntiAffinity` similar to placement defined for daemons configured by the [cluster CRD](/cluster/examples/kubernetes/edgefs/cluster.yaml).
- `resourceProfile`: S3X pod resource utilization profile (Memory and CPU). Can be `embedded` or `performance` (default). In case of `performance` an S3X pod trying to increase amount of internal I/O resources that results in higher performance at the cost of additional memory allocation and more CPU load. In `embedded` profile case, S3X pod gives preference to preserving memory over I/O and limiting chunk cache (see `chunkCacheSize` option). The `performance` profile is the default unless cluster wide `embedded` option is defined.
- `resources`: Set resource requests/limits for the Edge-X S3 Pod(s), see [Resource Requirements/Limits](edgefs-cluster-crd.md#resource-requirementslimits).
* `name`: The name of the Edge-X S3 system to create, which must match existing EdgeFS service.
* `namespace`: The namespace of the Rook cluster where the Edge-X S3 service is created.
* `sslCertificateRef`: If the certificate is not specified, SSL will use default crt and key files. If specified, this is the name of the Kubernetes secret that contains the SSL certificate to be used for secure connections. Please see [secret YAML file example](/cluster/examples/kubernetes/edgefs/sslKeyCertificate.yaml) on how to setup Kuberenetes secret. Notice that base64 encoding is required.
* `port`: The port on which the Edge-X S3 pods and the Edge-X S3 service will be listening (not encrypted). Default port is 3000.
* `securePort`: The secure port on which Edge-X S3 pods will be listening. If not defined then default SSL certificates will be used. Default port is 3001.
* `instances`: The number of active Edge-X S3 service instances. For load balancing we recommend to use nginx and the like solutions.
* `chunkCacheSize`: Limit amount of memory allocated for dynamic chunk cache. By default S3X pod uses up to 75% of available memory as chunk caching area. This option can influence this allocation strategy.
* `annotations`: Key value pair list of annotations to add.
* `placement`: The Edge-X S3 PODs can be given standard Kubernetes placement restrictions with `nodeAffinity`, `tolerations`, `podAffinity`, and `podAntiAffinity` similar to placement defined for daemons configured by the [cluster CRD](/cluster/examples/kubernetes/edgefs/cluster.yaml).
* `resourceProfile`: S3X pod resource utilization profile (Memory and CPU). Can be `embedded` or `performance` (default). In case of `performance` an S3X pod trying to increase amount of internal I/O resources that results in higher performance at the cost of additional memory allocation and more CPU load. In `embedded` profile case, S3X pod gives preference to preserving memory over I/O and limiting chunk cache (see `chunkCacheSize` option). The `performance` profile is the default unless cluster wide `embedded` option is defined.
* `resources`: Set resource requests/limits for the Edge-X S3 Pod(s), see [Resource Requirements/Limits](edgefs-cluster-crd.md#resource-requirementslimits).
### Setting up EdgeFS namespace and tenant
## Setting up EdgeFS namespace and tenant
For more detailed instructions please refer to [EdgeFS Wiki](https://github.com/Nexenta/edgefs/wiki).
@@ -79,37 +79,37 @@ Root object holds system information and table of namespaces registered to a loc
To initialize system and prepare logical definitions, login to the toolbox as shown in this example:
```
```console
kubectl get po --all-namespaces | grep edgefs-mgr
kubectl exec -it -n rook-edgefs rook-edgefs-mgr-6cb9598469-czr7p -- env COLUMNS=$COLUMNS LINES=$LINES TERM=linux toolbox
```
Assumption at this point is that nodes are all configured and can be seen via the following command:
```
```console
efscli system status
```
1. Initialize cluster
1. Initialize EdgeFS cluster:
Verify that HW (or better say emulated in this case) configuration look normal and accept it
```
```console
efscli system init
```
At this point new dynamically discovered configuration checkpoint will be created at $NEDGE_HOME/var/run/flexhash-checkpoint.json
This will also create system "root" object, holding Site's Namespace. Namespace may consist of more then single region.
2. Create new local namespace (or we also call it "Region" or "Segment")
2. Create new local namespace (or we also call it "Region" or "Segment"):
```
```console
efscli cluster create Hawaii
```
3. Create logical tenants of cluster namespace "Hawaii", also buckets if needed
3. Create logical tenants of cluster namespace "Hawaii", also buckets if needed:
```
```console
efscli tenant create Hawaii/Cola
efscli bucket create Hawaii/Cola/bk1
efscli tenant create Hawaii/Pepsi
@@ -118,16 +118,16 @@ efscli bucket create Hawaii/Pepsi/bk1
Now cluster is setup, services can be now created.
4. Create Edge-X S3 services objects for tenants
4. Create Edge-X S3 services objects for tenants:
```
```console
efscli service create s3x s3x-cola
efscli service serve s3x-cola Hawaii/Cola
efscli service create s3x s3x-pepsi
efscli service serve s3x-pepsi Hawaii/Pepsi
```
5. Create S3X CRDs
5. Create EdgeFS S3X CRDs:
```yaml
apiVersion: edgefs.rook.io/v1
Documentation/edgefs-storage.md
+1 -1
View File
@@ -11,7 +11,7 @@ EdgeFS is capable of spanning unlimited number of geographically distributed sit
At each Geo-site, EdgeFS nodes deployed as containers (StatefulSet) on physical or virtual Kubernetes nodes, pooling available storage capacity and presenting it via compatible S3/NFS/iSCSI/etc storage emulated protocols for cloud-native applications running on the same or dedicated servers.
### How it works, in a Nutshell?
## How it works, in a Nutshell?
If you familiar with "git", where all modifications are fully versioned and globally immutable, it is highly likely you already know how it works at its core. Think of it as a world-scale copy-on-write technique. Now, if we can make a parallel for you to understand it better - what EdgeFS does, it expands "git" paradigm to object storage and making Kubernetes Persistent Volumes accessible via emulated storage standard protocols e.g. S3, NFS and even block devices such as iSCSI, in a high-performance and low-latency ways. With fully versioned modifications, fully immutable metadata and data, users data can be transparently replicated, distributed and dynamically pre-fetched across many Geo-sites.
Documentation/edgefs-swift-crd.md
+23 -23
View File
@@ -47,21 +47,21 @@ spec:
# memory: "1024Mi"
```
### Metadata
## Metadata
- `name`: The name of the SWIFT service to create, which must match existing EdgeFS service.
- `namespace`: The namespace of the Rook cluster where the SWIFT service is created.
- `sslCertificateRef`: If the certificate is not specified, SSL will use default crt and key files. If specified, this is the name of the Kubernetes secret that contains the SSL certificate to be used for secure connections. Please see [secret YAML file example](/cluster/examples/kubernetes/edgefs/sslKeyCertificate.yaml) on how to setup Kuberenetes secret. Notice that base64 encoding is required.
- `port`: The port on which the SWIFT pods and the SWIFT service will be listening (not encrypted). Default port is 9981.
- `securePort`: The secure port on which SWIFT pods will be listening. If not defined then default SSL certificates will be used. Default port is 443.
- `instances`: The number of active SWIFT service instances. For load balancing we recommend to use nginx and the like solutions.
- `chunkCacheSize`: Limit amount of memory allocated for dynamic chunk cache. By default SWIFT pod uses up to 75% of available memory as chunk caching area. This option can influence this allocation strategy.
- `annotations`: Key value pair list of annotations to add.
- `placement`: The SWIFT pods can be given standard Kubernetes placement restrictions with `nodeAffinity`, `tolerations`, `podAffinity`, and `podAntiAffinity` similar to placement defined for daemons configured by the [cluster CRD](/cluster/examples/kubernetes/edgefs/cluster.yaml).
- `resourceProfile`: SWIFT pod resource utilization profile (Memory and CPU). Can be `embedded` or `performance` (default). In case of `performance` an SWIFT pod trying to increase amount of internal I/O resources that results in higher performance at the cost of additional memory allocation and more CPU load. In `embedded` profile case, SWIFT pod gives preference to preserving memory over I/O and limiting chunk cache (see `chunkCacheSize` option). The `performance` profile is the default unless cluster wide `embedded` option is defined.
- `resources`: Set resource requests/limits for the SWIFT pods, see [Resource Requirements/Limits](edgefs-cluster-crd.md#resource-requirementslimits).
* `name`: The name of the SWIFT service to create, which must match existing EdgeFS service.
* `namespace`: The namespace of the Rook cluster where the SWIFT service is created.
* `sslCertificateRef`: If the certificate is not specified, SSL will use default crt and key files. If specified, this is the name of the Kubernetes secret that contains the SSL certificate to be used for secure connections. Please see [secret YAML file example](/cluster/examples/kubernetes/edgefs/sslKeyCertificate.yaml) on how to setup Kuberenetes secret. Notice that base64 encoding is required.
* `port`: The port on which the SWIFT pods and the SWIFT service will be listening (not encrypted). Default port is 9981.
* `securePort`: The secure port on which SWIFT pods will be listening. If not defined then default SSL certificates will be used. Default port is 443.
* `instances`: The number of active SWIFT service instances. For load balancing we recommend to use nginx and the like solutions.
* `chunkCacheSize`: Limit amount of memory allocated for dynamic chunk cache. By default SWIFT pod uses up to 75% of available memory as chunk caching area. This option can influence this allocation strategy.
* `annotations`: Key value pair list of annotations to add.
* `placement`: The SWIFT pods can be given standard Kubernetes placement restrictions with `nodeAffinity`, `tolerations`, `podAffinity`, and `podAntiAffinity` similar to placement defined for daemons configured by the [cluster CRD](/cluster/examples/kubernetes/edgefs/cluster.yaml).
* `resourceProfile`: SWIFT pod resource utilization profile (Memory and CPU). Can be `embedded` or `performance` (default). In case of `performance` an SWIFT pod trying to increase amount of internal I/O resources that results in higher performance at the cost of additional memory allocation and more CPU load. In `embedded` profile case, SWIFT pod gives preference to preserving memory over I/O and limiting chunk cache (see `chunkCacheSize` option). The `performance` profile is the default unless cluster wide `embedded` option is defined.
* `resources`: Set resource requests/limits for the SWIFT pods, see [Resource Requirements/Limits](edgefs-cluster-crd.md#resource-requirementslimits).
### Setting up EdgeFS namespace and tenant
## Setting up EdgeFS namespace and tenant
For more detailed instructions please refer to [EdgeFS Wiki](https://github.com/Nexenta/edgefs/wiki).
@@ -75,35 +75,35 @@ Root object holds system information and table of namespaces registered to a loc
To initialize system and prepare logical definitions, login to the toolbox as shown in this example:
```
```console
kubectl get po --all-namespaces | grep edgefs-mgr
kubectl exec -it -n rook-edgefs rook-edgefs-mgr-6cb9598469-czr7p -- env COLUMNS=$COLUMNS LINES=$LINES TERM=linux toolbox
```
Assumption at this point is that nodes are all configured and can be seen via the following command:
```
```console
efscli system status
```
1. Initialize cluster
1. Initialize EdgeFS cluster:
Verify that HW (or better say emulated in this case) configuration look normal and accept it
```
```console
efscli system init
```
At this point new dynamically discovered configuration checkpoint will be created at $NEDGE_HOME/var/run/flexhash-checkpoint.json
This will also create system "root" object, holding Site's Namespace. Namespace may consist of more then single region.
2. Create new local namespace (or we also call it "Region" or "Segment")
2. Create new local namespace (or we also call it "Region" or "Segment"):
```
```console
efscli cluster create Hawaii
```
3. Create logical tenants of cluster namespace "Hawaii", also buckets if needed
3. Create logical tenants of cluster namespace "Hawaii", also buckets if needed:
```
efscli tenant create Hawaii/Cola
@@ -114,16 +114,16 @@ efscli bucket create Hawaii/Pepsi/bk1
Now cluster is setup, services can be now created.
4. Create SWIFT services objects for tenants
4. Create SWIFT services objects for tenants:
```
```console
efscli service create swift swift-cola
efscli service serve swift-cola Hawaii
efscli service create swift swiftPepsi
efscli service serve swift-pepsi Hawaii
```
5. Create SWIFT CRDs
5. Create EdgeFS SWIFT objects:
```yaml
apiVersion: edgefs.rook.io/v1
Documentation/edgefs-ui.md
+2 -1
View File
@@ -17,7 +17,8 @@ Rook EdgeFS Operator enables the dashboard by default.
A K8s service will be created to expose HTTP/S ports inside the cluster.
This example shows what ports was configured for EdgeFS right after successful cluster deployment:
```bash
```console
# kubectl get svc -n rook-edgefs
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
rook-edgefs-restapi NodePort 10.105.177.93 <none> 8881:32135/TCP,8080:31363/TCP,4443:31485/TCP 3h14m
Documentation/edgefs-upgrade.md
+29 -20
View File
@@ -5,6 +5,7 @@ indent: true
---
# EdgeFS Upgrades
This guide will walk you through the manual steps to upgrade the software in a Rook EdgeFS cluster
from one version to the next. Rook EdgeFS is a multi-cloud distributed software system and
therefore there are multiple components to individually upgrade in the sequence defined in this
@@ -14,6 +15,7 @@ healthy and fully functional state.
We welcome feedback and opening issues!
## Supported Versions
The supported version for this upgrade guide is **from a 1.0 release to a 1.x releases**.
Build-to-build upgrades are not guaranteed to work. This guide is to perform upgrades only between
the official releases.
@@ -21,12 +23,13 @@ the official releases.
Upgrades from Alpha to Beta not supported. However, please see migration procedure below.
## EdgeFS Migration
EdgeFS Operator provides a way of preserving data on disks or directories while moving to a
new version (like Alpha to Beta transitioning) or reconfiguring (like full re-start).
We will do all our work in the EdgeFS example manifests directory.
```sh
```console
cd $YOUR_ROOK_REPO/cluster/examples/kubernetes/edgefs/
```
@@ -34,7 +37,7 @@ Example of migration from existing `v1beta1` EdgeFS cluster to stable `v1` EdgeF
For already existing EdgeFS manifests we should replace `v1beta1` version to `v1`.
We can use a few simple `sed` commands to do this for all manifests at once.
```sh
```console
sed -i.bak -e "s/edgefs.rook.io\/v1beta1/edgefs.rook.io\/v1beta1/g" *.yaml
sed -i -e "s/edgefs.rook.io\/v1beta1/edgefs.rook.io\/v1/g" *.yaml
mkdir -p backups
@@ -42,13 +45,14 @@ mv *.bak backups/
```
Then we should add new `v1` version specification to existing EdgeFS' CRDs.
```sh
```console
kubectl apply -f upgrade-from-v1beta1-create.yaml
```
And replace EdgeFS operator to new stable 'v1' version
```sh
```console
kubectl -n rook-edgefs-system set image deploy/rook-edgefs-operator rook-edgefs-operator=rook/edgefs:v1.1.0
```
@@ -56,10 +60,11 @@ Then you could update your cluster's EdgeFS image for latest one as discribed be
## EdgeFS Version Upgrade
### EdgeFS images
Official EddgeFS container images can be found on [Docker Hub](https://hub.docker.com/r/edgefs/edgefs/tags).
```sh
1. Update the EdgeFS image used for the EdgeFS cluster:
```console
# Parameterize the environment
export ROOK_SYSTEM_NAMESPACE="rook-edgefs-system"
export CLUSTER_NAME="rook-edgefs"
@@ -67,25 +72,27 @@ export CLUSTER_NAME="rook-edgefs"
The majority of the upgrade will be handled by the Rook operator. Begin the upgrade by changing the
EdgeFS image field in the cluster CRD (`spec:edgefsImageName`).
```sh
```console
NEW_EDGEFS_IMAGE='edgefs/edgefs:1.2.64'
kubectl -n $CLUSTER_NAME patch Cluster $CLUSTER_NAME --type=merge \
-p "{\"spec\": {\"edgefsImageName\": \"$NEW_EDGEFS_IMAGE\"}}"
```
or via console editor fix `edgefsImageName` property
or via console editor update the `edgefsImageName` property:
```sh
```console
kubectl edit -n $CLUSTER_NAME Cluster $CLUSTER_NAME
```
and save results.
#### 2. Wait for the pod updates to complete
2. Wait for the pod updates to complete:
As with upgrading Rook, you must now wait for the upgrade to complete. Determining when the EdgeFS
version has fully updated is rather simple.
```sh
```console
kubectl -n $CLUSTER_NAME describe pods | grep "Image:" | sort | uniq
# This cluster is not yet finished:
# Image: edgefs/edgefs:1.2.31
@@ -97,24 +104,26 @@ kubectl -n $CLUSTER_NAME describe pods | grep "Image:" | sort | uniq
# Image: edgefs/edgefs-restapi:1.2.50
# Image: edgefs/edgefs-ui:1.2.50
```
#### 3. Verify the updated cluster
3. Verify the updated EdgeFS cluster:
Access to EdgeFS mgr pod and check EdgeFS system status
```sh
```console
kubectl exec -it -n $CLUSTER_NAME rook-edgefs-mgr-xxxx-xxx -- toolbox
efscli system status -v 1
```
## EdgeFS Nodes update
## EdgeFS Nodes Update
Nodes can be added and removed over time by updating the Cluster CRD, for example with `kubectl edit Cluster -n rook-edgefs`.
This will bring up your default text editor and allow you to add and remove storage nodes from the cluster.
This feature is only available when `useAllNodes` has been set to `false` and `resurrect` mode is not used.
### 1. Add node example
#### a. Edit Cluster CRD `kubectl edit Cluster -n rook-edgefs`
### Add node Example
#### b. Add new node section with desired configuration in storage section of Cluster CRD
1. Edit Cluster CRD `kubectl edit Cluster -n rook-edgefs`
2. Add new node section with desired configuration in storage section of Cluster CRD:
Currently we adding new node `node3072ub16` with two drives `sdb` and `sdc` on it.
@@ -130,6 +139,6 @@ Currently we adding new node `node3072ub16` with two drives `sdb` and `sdc` on i
name: node3072ub16
resources: {}
```
#### c. Save CRD and operator will update all target nodes and related pods of the EdgeFS cluster.
#### d. Login to EdgeFS mgr toolbox and adjust FlexHash table to a new configuration using `efscli system fhtable` command.
* Save CRD and operator will update all target nodes and related pods of the EdgeFS cluster.
* Login to EdgeFS mgr toolbox and adjust FlexHash table to a new configuration using `efscli system fhtable` command.
Documentation/edgefs-vdev-management.md
+63 -56
View File
@@ -6,8 +6,8 @@ indent: true
# EdgeFS VDEV Management
EdgeFS can run on top of any block device. It can be a raw physical or virtual disk (RT-RD type). Or it can be a directory on a local file system (RT-LFS type).
In case of a local file system, VDEV function will be emulated via memory-mapped files and as such, management of the files is a responsibility of underlying filesystem of choice. (e.g. ext4, xfs, zfs, etc).
EdgeFS can run on top of any block device. It can be a raw physical or virtual disk (RT-RD type). Or it can be a directory on a local filesystem (RT-LFS type).
In case of a local filesystem, VDEV function will be emulated via memory-mapped files and as such, management of the files is a responsibility of underlying filesystem of choice. (e.g. ext4, xfs, zfs, etc).
This document describes the management of VDEVs built on top of raw disks (RT-RD type).
## EdgeFS on-disk organization
@@ -26,51 +26,53 @@ To define an object assembly order there are two manifest metadata types: `TT_CH
And the third group is *the temporary metadata*. It includes on-disk data queues whose entries are removed after being processed by server's background jobs: `TT_VERIFICATION_QUEUE`, `TT_BATCH_QUEUE`, `TT_INCOMING_BATCH_QUEUE`, `TT_ENCODING_QUEUE`, `TT_REPLICATION_QUEUE` and `TT_TRANSACTION_LOG`.
Persistent data | Persistent metadata | Temporary metadata
--- | --- | ---
_TT_CHUNK_PAYLOAD_<br/>_TT_HASHCOUNT_ | _TT_CHUNK_MANIFEST_<br/>_TT_VERSION_MANIFEST_<br/>_TT_PARTIY_MANIFEST_<br/>_TT_VERIFIED_BACKREF_<br/>_TT_NAMEINDEX_ | _TT_VERIFICATION_QUEUE_<br/>_TT_BATCH_QUEUE_<br/>_TT_INCOMING_BATCH_QUEUE_<br/>_TT_ENCODING_QUEUE_<br/>_TT_REPLICATION_QUEUE_<br/>_TT_TRANSACTION_LOG_
| Persistent data | Persistent metadata | Temporary metadata |
| ----------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `_TT_CHUNK_PAYLOAD_`<br/>`_TT_HASHCOUNT_` | `_TT_CHUNK_MANIFEST_`<br/>`_TT_VERSION_MANIFEST_`<br/>`_TT_PARTIY_MANIFEST_`<br/>`_TT_VERIFIED_BACKREF_`<br/>`_TT_NAMEINDEX_` | `_TT_VERIFICATION_QUEUE_`<br/>`_TT_BATCH_QUEUE_`<br/>`_TT_INCOMING_BATCH_QUEUE_`<br/>`_TT_ENCODING_QUEUE_`<br/>`_TT_REPLICATION_QUEUE_`<br/>`_TT_TRANSACTION_LOG_` |
### Consistency considerations
Each VDEV in the cluster holds a certain number of entries of each data type. Damage of data table of different groups has a different impact on VDEV's consistency. Usually, if a damaged data type belongs to _the temporary metadata_, then such data table can be dropped without a noticeable influence on the cluster. If a persistent data or persistent metadata gets corrupted, then impact will take place, however, it will never be vital for the cluster thanks to data protection approaches the EdgeFS implements: data redundancy or erasure coding. Moreover, the underlying device management layer also tries to reduce the cost of data loss by means of different techniques, like data sharding.
_IMPORTANT_: In EdgeFS design, loss of a data or metadata chunk generally do not affect front I/O performance due to unique dynamic placement and retrieval technique. Data placement or retrieval gets negotiated prior to each chunk I/O. This allows EdgeFS to select the most optimal location for I/O rather then hard-coded. With this, failing device(s) or temporary disconnected/busy devices will not affect overall local cluster performance.
> **IMPORTANT**: In EdgeFS design, loss of a data or metadata chunk generally do not affect front I/O performance due to unique dynamic placement and retrieval technique. Data placement or retrieval gets negotiated prior to each chunk I/O. This allows EdgeFS to select the most optimal location for I/O rather then hard-coded. With this, failing device(s) or temporary disconnected/busy devices will not affect overall local cluster performance.
Stronger consistency/durability comes with a performance price and it highly depends on an use cases. EdgeFS provides needed flexibility of selecting most optimal consistency model via usage of `sync` parameter that defines default behavior of write operations at device or directory level. Acceptable values are 0, 1 (default), 2, 3:
- `0`: No syncing will happen. Highest performance possible and good for HPC scratch types of deployments. This option will still sustain crash of pods or software bugs. It will not sustain server power loss an may cause node/device level inconsistency.
- `1`: Default method. Will guarantee node / device consistency in case of power loss with reduced durability.
- `2`: Provides better durability in case of power loss at the cost of extra metadata syncing.
- `3`: Most durable and reliable option at the cost of significant performance impact.
* `0`: No syncing will happen. Highest performance possible and good for HPC scratch types of deployments. This option will still sustain crash of pods or software bugs. It will not sustain server power loss an may cause node/device level inconsistency.
* `1`: Default method. Will guarantee node / device consistency in case of power loss with reduced durability.
* `2`: Provides better durability in case of power loss at the cost of extra metadata syncing.
* `3`: Most durable and reliable option at the cost of significant performance impact.
## EdgeFS RT-RD Architecture
The main metrics of the RT-RD device is a partition level (the _plevel_). The _plevel_ defines a number of disk partitions user's data will be split across: for each key-value pair, the key identifies the _plevel index_. The RT-RD splits entire raw HDD (or SSD) into several partitions. The following partition types are defined:
- Main partitions. Resides on the HDD and used as the main data store. A number of partitions equal to disk _plevel_. In a all SSD or all HDD configurations, main partitions keep all data types.
- Write-ahead-log (_WAL_) partitions. The _WAL_ drastically improves write performance, especially for a hybrid (HDDs+SSD) configuration. There is one _WAL_ partition per _plevel_ Its location depends on configuration type. In a hybrid configuration, it's on SSD, otherwise on the same disk as the corresponding _main_ partition.
- A metadata offload partition (_mdoffload_). For the hybrid configuration-only. One partition per an HDD. It is used to store temporary metadata, indexes and some persistent metadata types (optionally).
- A configuration partition. A tiny partition per device which keeps a disk configuration options (a _metaloc entry_).
* Main partitions. Resides on the HDD and used as the main data store. A number of partitions equal to disk _plevel_. In a all SSD or all HDD configurations, main partitions keep all data types.
* Write-ahead-log (_WAL_) partitions. The _WAL_ drastically improves write performance, especially for a hybrid (HDDs+SSD) configuration. There is one _WAL_ partition per _plevel_ Its location depends on configuration type. In a hybrid configuration, it's on SSD, otherwise on the same disk as the corresponding _main_ partition.
* A metadata offload partition (_mdoffload_). For the hybrid configuration-only. One partition per an HDD. It is used to store temporary metadata, indexes and some persistent metadata types (optionally).
* A configuration partition. A tiny partition per device which keeps a disk configuration options (a _metaloc entry_).
Regardless of location, each partition keeps an instance of a key-value database (_the environment_) and a number sub-databases for different data types within the environment. A sub-database is referenced by its Data Base Identifier(_DBI_) which contains data type name as a part of the _DBI_ and depends on the environment's location. Along with 3 aforementioned data type categories, the RT-RD keeps an extra one for a hybrid device - the _mdoffload index_. The _mdoffload index_ is used to improve the performance of GET operations that target the data types on HDD without data retrieval: chunk stat or an attribute get. A damaged _mdoffload index_ can always be restored from data on HDD.
Below is a data type to DBI mapping table
Group | Data type | DBI | Location
--- | --- | --- | ---
Persistent data | TT_CHUNK_PAYLOD | bd-part-TT_CHUNK_PAYLOAD-0 | HDD
&nbsp; | TT_HASHCOUNT | bd-part-TT_HASHCOUNT-0 | HDD
Persistent metadata | TT_CHUNK_MANIFEST | TT_CHUNK_MANIFEST<br/>bd-part-TT_CHUNK_MANIFEST-0 | SSD<br/>HDD
&nbsp; | TT_VERSION_MANIFEST | TT_VERSION_MANIFEST<br/>bd-part-TT_VERSION_MANIFEST-0 | SSD<br/>HDD
&nbsp; | TT_PARITY_MANIFEST | TT_PARITY_MANIFEST<br/>bd-part-TT_PARITY_MANIFEST-0 | SSD<br/>HDD
&nbsp; | TT_NAMEINDEX | TT_NAMEINDEX<br/>bd-part-TT_NAMEINDEX-0 | SSD<br/>HDD
&nbsp; | TT_VERIFIED_BACKREF | TT_VERIFIED_BACKREF<br/>bd-part-TT_VERIFIED_BACKREF-0 | SSD<br/>HDD
Temporary metadata | TT_VERIFICATION_QUEUE | TT_VERIFICATION_QUEUE<br/>bd-part-TT_VERIFICATION_QUEUE-0 | SSD<br/>HDD
&nbsp; | TT_BATCH_QUEUE | TT_BATCH_QUEUE<br/>bd-part-TT_BATCH_QUEUE-0 | SSD<br/>HDD
&nbsp; | TT_INCOMING_BATCH_QUEUE | TT_INCOMING_BATCH_QUEUE<br/>bd-part-TT_INCOMING_BATCH_QUEUE-0 | SSD<br/>HDD
&nbsp; | TT_ENCODING_QUEUE | TT_ENCODING_QUEUE<br/>bd-part-TT_ENCODING_QUEUE-0 | SSD<br/>HDD
&nbsp; | TT_REPLICATION_QUEUE | TT_REPLICATION_QUEUE<br/>bd-part-TT_REPLICATION_QUEUE-0 | SSD<br/>HDD
&nbsp; | TT_TRANSACTION_LOG | TT_TRANSACTION_LOG<br/>bd-part-TT_TRANSACTION_LOG-0 | SSD<br/>HDD
Mdoffload index | - | keys-TT_CHUNK_MANIFEST<br/>keys-TT_CHUNK_PAYLOAD<br/>keys-TT_VERSION_MANIFEST | SSD<br/>SSD<br/>SSD
Mdoffload cache | - | mdcache-TT_CHUNK_MANIFEST<br/>mdcache-TT_VERSION_MANIFEST | SSD<br/>SSD
| Group | Data type | DBI | Location |
| ------------------- | ------------------------- | ----------------------------------------------------------------------------------- | ------------------- |
| Persistent data | `TT_CHUNK_PAYLOD` | `bd-part-TT_CHUNK_PAYLOAD-0` | HDD |
| &nbsp; | `TT_HASHCOUNT` | `bd-part-TT_HASHCOUNT-0` | HDD |
| Persistent metadata | `TT_CHUNK_MANIFEST` | `TT_CHUNK_MANIFEST`<br/>`bd-part-TT_CHUNK_MANIFEST-0` | SSD<br/>HDD |
| &nbsp; | `TT_VERSION_MANIFEST` | `TT_VERSION_MANIFEST`<br/>`bd-part-TT_VERSION_MANIFEST-0` | SSD<br/>HDD |
| &nbsp; | `TT_PARITY_MANIFEST` | `TT_PARITY_MANIFEST`<br/>`bd-part-TT_PARITY_MANIFEST-0` | SSD<br/>HDD |
| &nbsp; | `TT_NAMEINDEX` | `TT_NAMEINDEX`<br/>`bd-part-TT_NAMEINDEX-0` | SSD<br/>HDD |
| &nbsp; | `TT_VERIFIED_BACKREF` | `TT_VERIFIED_BACKREF`<br/>`bd-part-TT_VERIFIED_BACKREF-0` | SSD<br/>HDD |
| Temporary metadata | `TT_VERIFICATION_QUEUE` | `TT_VERIFICATION_QUEUE`<br/>`bd-part-TT_VERIFICATION_QUEUE-0` | SSD<br/>HDD |
| &nbsp; | `TT_BATCH_QUEUE` | `TT_BATCH_QUEUE`<br/>`bd-part-TT_BATCH_QUEUE-0` | SSD<br/>HDD |
| &nbsp; | `TT_INCOMING_BATCH_QUEUE` | `TT_INCOMING_BATCH_QUEUE`<br/>`bd-part-TT_INCOMING_BATCH_QUEUE-0` | SSD<br/>HDD |
| &nbsp; | `TT_ENCODING_QUEUE` | `TT_ENCODING_QUEUE`<br/>`bd-part-TT_ENCODING_QUEUE-0` | SSD<br/>HDD |
| &nbsp; | `TT_REPLICATION_QUEUE` | `TT_REPLICATION_QUEUE`<br/>`bd-part-TT_REPLICATION_QUEUE-0` | SSD<br/>HDD |
| &nbsp; | `TT_TRANSACTION_LOG` | `TT_TRANSACTION_LOG`<br/>`bd-part-TT_TRANSACTION_LOG-0` | SSD<br/>HDD |
| Mdoffload index | `-` | `keys-TT_CHUNK_MANIFEST`<br/>`keys-TT_CHUNK_PAYLOAD`<br/>`keys-TT_VERSION_MANIFEST` | SSD<br/>SSD<br/>SSD |
| Mdoffload cache | `-` | `mdcache-TT_CHUNK_MANIFEST`<br/>`mdcache-TT_VERSION_MANIFEST` | SSD<br/>SSD |
So now you are informed enough to understand next paragraphs.
@@ -78,13 +80,13 @@ So now you are informed enough to understand next paragraphs.
Login to the toolbox as shown in this example:
```
kubectl exec -it -n rook-edgefs 'kubectl get po --all-namespaces | awk '{print($2)}' |grep edgefs-mgr' -- env COLUMNS=$COLUMNS LINES=$LINES TERM=linux toolbox
```console
kubectl exec -it -n rook-edgefs 'kubectl get po --all-namespaces | awk '{print($2)}' | grep edgefs-mgr' -- env COLUMNS=$COLUMNS LINES=$LINES TERM=linux toolbox
```
To find out what device needs to go into the maintenance state, run the following command:
```
```console
# efscli system status -v1
ServerID CFAE1E62652370A93769378B2C862F23 ubuntu1632243:rook-edgefs-target-0 DEGRADED
VDEVID D76242575CAA2662862CEEBC65D0B69F ata-ST1000NX0423_W470M48T ONLINE
@@ -102,13 +104,13 @@ From this command you will see that pod 'rook-edgefs-target-0' is in degraded st
Login to the affected target pod as shown in this example:
```
```console
kubectl exec -it -n rook-edgefs rook-edgefs-target-0 -- env COLUMNS=$COLUMNS LINES=$LINES TERM=linux toolbox
```
Make sure the disk is faulted:
```
```console
# efscli device list
NAME | VDEV ID | PATH | STATUS
@@ -119,11 +121,11 @@ ata-ST1000NX0423_W470P9XJ | BB287F0C6747B1E59A85872B2C7F39B3 | /dev/sdd | ONLI
ata-ST1000NX0423_W470NQ7A | B2BB69729BC3EB9ECE5F0DCB3DB4D0D6 | /dev/sde | UNAVAILABLE
```
Use `efscli device detach ata-ST1000NX0423_W470NQ7A` for detaching a disk. It will be marked a faulted and won't be attached at the next ccow-daemon restart. Also, the preserved detached state can be cleared by a command `nezap --disk=diskID --restore-metaloc`.
Use `efscli device detach ata-ST1000NX0423_W470NQ7A` for detaching a disk. It will be marked a faulted and won't be attached at the next `ccow-daemon` restart. Also, the preserved detached state can be cleared by a command `nezap --disk=diskID --restore-metaloc`.
Once detached, related HDD/SSD partitions can be inspected/fixed by means of `efscli device check` command or zapped. When maintenance is done, the VDEV(s) can become operational again by invoking a command `efscli device attach`.
Assuming that your disk is detached and your verification procedure can be initiated. The tool is accessible via `efscli device check` command. It provides an interactive user interface for device validation and recovery. Before getting started, a user needs to define a scratch area location. It will be used as a temporary store for environments being compacted or recovered. The scratch area can be defined in terms of data path within filesystem, a path to a raw disk (or its partition) in the /dev/ folder or raw disk/partition ID. User has to make sure the filesystem can provide enough free space to keep data from single _plevel_. The same requirement is for raw disk/partition size: at least 600GB, 1TB is recommended. A user can define the scratch area path in $(NEDGE_HOME)/etc/ccow/rt-rd.json as follow:
Assuming that your disk is detached and your verification procedure can be initiated. The tool is accessible via `efscli device check` command. It provides an interactive user interface for device validation and recovery. Before getting started, a user needs to define a scratch area location. It will be used as a temporary store for environments being compacted or recovered. The scratch area can be defined in terms of data path within filesystem, a path to a raw disk (or its partition) in the `/dev/` folder or raw disk/partition ID. User has to make sure the filesystem can provide enough free space to keep data from single `plevel`. The same requirement is for raw disk/partition size: at least 600GB, 1TB is recommended. A user can define the scratch area path in `$(NEDGE_HOME)/etc/ccow/rt-rd.json` as follow:
```json
{
@@ -136,7 +138,8 @@ Alternatively, the path can be specified by `-s <path>` flag. For example: `efsc
The following actions will be completely interactive and a user's confirmation will be required before any important disk changes. The command output might look like follow. We split it into parts for convenience
```
```console
$ efscli device check -s /scratch.db ata-ST1000NX0423_W470NQ7A
INFO: checking disk /dev/sde. Stored metaloc record info:
NAME | TYPE | ID | PATH | CAPACITY | USED | PSIZE | BCACHE | STATUS
@@ -148,9 +151,9 @@ PLEVEL2 | Main | ata-ST1000NX0423_W470NQ7A-part2 | /dev/sde2 | 465.76G | 0%
OFFLOAD | | scsi-35000c5003021f02f-part12 | /dev/sdf12 | 180.97G | 0% | 8k | n/a | CORRUPTED
```
The first table displays the VDEV configuration and its known errors (if any). In our example the VDEV is a hybrid one (The OFFLOAD partition signals that) with 2 _plevels_ situated on the HDD /dev/sde. Main partitions are /dev/sde1 and /dev/sde2, WAL partitions are on SSD (/dev/sdf7 and /dev/sdf8) as well as the mdoffload partition /dev/sdf12. For each partition we see its capacity, utilization, logical paze size (internal for the key-value engine), bcache presence and healthy status. 'None' means the partition doesn't have know errors. However, the /dev/sdf12 is marked as a 'CORRUPTED' and will be check by the verification algorithm. If there known errors are absent, then user will be asked for extended data validation of each partition. It's important to mention, that if a VDEV is online and user wants to validate it, then such VDEV will be set READ-ONLY until validation is in progress. If there are any errors, then the VDEV will be detached for maintenance. However, in our case the problem is known and it requires further validation in order to find a proper recovery solution.
The first table displays the VDEV configuration and its known errors (if any). In our example the VDEV is a hybrid one (The OFFLOAD partition signals that) with 2 _plevels_ situated on the HDD `/dev/sde`. Main partitions are `/dev/sde1` and `/dev/sde2`, WAL partitions are on SSD (`/dev/sdf7` and `/dev/sdf8`) as well as the mdoffload partition `/dev/sdf12`. For each partition we see its capacity, utilization, logical paze size (internal for the key-value engine), bcache presence and healthy status. `None` means the partition doesn't have know errors. However, the `/dev/sdf12` is marked as a `CORRUPTED` and will be check by the verification algorithm. If there known errors are absent, then user will be asked for extended data validation of each partition. It's important to mention, that if a VDEV is online and user wants to validate it, then such VDEV will be set `READ-ONLY` until validation is in progress. If there are any errors, then the VDEV will be detached for maintenance. However, in our case the problem is known and it requires further validation in order to find a proper recovery solution.
```text
```console
WARN: a fault record is detected for mdoffload (/dev/sdf12)
INFO: disk ata-ST1000NX0423_W470NQ7A is UNAVAILABLE
INFO: locking the device
@@ -181,19 +184,19 @@ Press 'Y' to start [y/n]: y
The validation is performed on a sub-database (DB) basis. For each DB there are up to 3 tests: open, read and modify. The last (modify) is disabled by default but can be activated in a policy file. The open and read tests are able to discover an overwhelming majority of structural errors without detaching a disk from ccow-daemon: if an environment is online, the user will be asked for permission to switch the device to read-only mode before any tests. The modify test requires the device to be set unavailable. Validation result is shown as a table for each DBI.
The table has a column named 'CORRUPTED' which may show number key-value pairs whose value's hash ID doesn't match expected one. For certain data types, it's acceptable to have a limited number of damaged entries. It's a compromise between data lost cost and data integrity. Often we don't want to mark the whole DB as faulted due to just a few damaged values which will be detected and removed by ccow-daemon soon or later.
The table has a column named `CORRUPTED` which may show number key-value pairs whose value's hash ID doesn't match expected one. For certain data types, it's acceptable to have a limited number of damaged entries. It's a compromise between data lost cost and data integrity. Often we don't want to mark the whole DB as faulted due to just a few damaged values which will be detected and removed by ccow-daemon soon or later.
Further behaviour depends on validation results:
* if there are no corrupted DB(s), the environment is considered healthy.
* if there are damaged DB(s), then the behaviour depends on DB's error handling policy. By default it implies the following:
* If there are no corrupted DB(s), the environment is considered healthy.
* If there are damaged DB(s), then the behaviour depends on DB's error handling policy. By default it implies the following:
* if corrupted only temporary metadata or those data can be reconstructed (the _mdoffload index_), then a selective recovery will be suggested to the user. The selective recovery makes copies of non-damaged DBs only. Corrupted ones will be re-created or re-constructed when the device is attached.
* If corrupted only temporary metadata or those data can be reconstructed (the _mdoffload index_), then a selective recovery will be suggested to the user. The selective recovery makes copies of non-damaged DBs only. Corrupted ones will be re-created or re-constructed when the device is attached.
* Corrupted data or metadata table with sensitive data. In this case, a _plevel_ or whole device needs to be formatted. All damaged environments will be added to a format queue that will be processed on the final stage.
As we can see 3 DBs are corrupted. One of them (keys-TT_CHUNK_PAYLOAD) can be easily recovered (the _mdoffload index_), however _TT_NAMEINDEX_ and _TT_VERSION_MANIFEST_ cannot be reconstructed and they reside on a mdoffload partition. So the whole device needs to be formatted.
As we can see 3 DBs are corrupted. One of them (`keys-TT_CHUNK_PAYLOAD`) can be easily recovered (the _mdoffload index_), however `TT_NAMEINDEX` and `TT_VERSION_MANIFEST` cannot be reconstructed and they reside on a mdoffload partition. So the whole device needs to be formatted.
```
```console
WARN: the entire device is about to be formatted.
ALL the data on it will be LOST. Do you want to proceed? [y/n]: y
INFO: formatting entire device ata-ST1000NX0423_W470NQ7A
@@ -212,17 +215,17 @@ A user confirmed and the VDEV has been formatted. Check is done. The VDEV can be
## VDEV Replacement Procedure
A command `efscli disk replace [-f] [-y] <old-name> <new-name>` performs on-the-fly disk substitution.
The disk with name 'old-name' will be detached and disk with name 'new-name' will be configured and used instead of the old one. If the 'new-disk' has a partition table on it, then the command will fail unless the '-f' flag is specified. When the flag set, the user will be asked for permission to destroy the partition table. The '-y' flags forces destruction of the partition table without confirmation. The 'replace' command can be used when EdgeFS service is down. In this case, the new disk will be attached upon the next service start.
The disk with name `old-name` will be detached and disk with name `new-name` will be configured and used instead of the old one. If the `new-disk` has a partition table on it, then the command will fail unless the `-f` flag is specified. When the flag set, the user will be asked for permission to destroy the partition table. The '-y' flags forces destruction of the partition table without confirmation. The `replace` command can be used when EdgeFS service is down. In this case, the new disk will be attached upon the next service start.
Login to the affected target pod where disk needs to be replaced as shown in this example:
```
```console
kubectl exec -it -n rook-edgefs rook-edgefs-target-0 -- env COLUMNS=$COLUMNS LINES=$LINES TERM=linux toolbox
```
List available and unused disks:
```
```console
# efscli device list -s
NAME | VDEV ID | PATH | STATUS
+---------------------------+----------------------------------+----------+-------------+
@@ -233,9 +236,9 @@ ata-ST1000NX0423_W470P9XJ | C82B78AC845989DC731BF59FE705256A | /dev/sdd | ONLI
ata-ST1000NX0423_W470NQ7A | BCA4C8F096DE96B4B350DF4F92E30F19 | /dev/sde | UNAVAILABLE
```
There is a device with an 'UNUSED' or 'PARTITIONED' status. The disk ST1000NX0423_W470M4BZ is not used and can replace a faulted one.
There is a device with an `UNUSED` or `PARTITIONED` status. The disk `ST1000NX0423_W470M4BZ` is not used and can replace a faulted one.
```
```console
# efscli device replace ata-ST1000NX0423_W470NQ7A ata-ST1000NX0423_W470M4BZ
INFO: Probbing disk ata-ST1000NX0423_W470M4BZ
@@ -247,14 +250,18 @@ INFO: The disk is replaced succefully
If a database partition gets corrupted, then there is a small probability for the VDEV container to enter an infinite restart loop. In order to prevent a container from restart use the following command:
```
```console
kubectl exec -it -n rook-edgefs rook-edgefs-target-<n> -c auditd -- touch /opt/nedge/var/run/.edgefs-start-block-ccowd
```
where `rook-edgefs rook-edgefs-target-<n>` is the affected pod ID. Once command is done, you must be able to reach a toolbox console:
where `rook-edgefs rook-edgefs-target-<n>` is the affected pod ID.
```
Once command is done, you must be able to reach a toolbox console:
```console
kubectl exec -it -n rook-edgefs rook-edgefs rook-edgefs-target-<n> -- env COLUMNS=$COLUMNS LINES=$LINES TERM=linux toolbox
```
and run the `efscli device check ...` command. When you are done, before exiting the toolbox, do not forget to remove the file /opt/nedge/var/run/.edgefs-start-block-ccowd
and run the `efscli device check ...` command.
**When you are done**, before exiting the toolbox, do not forget to remove the file /opt/nedge/var/run/.edgefs-start-block-ccowd