forked from rook/rook
127 lines
6.0 KiB
Markdown
127 lines
6.0 KiB
Markdown
---
|
|
title: EdgeFS Data Fabric
|
|
weight: 550
|
|
indent: true
|
|
---
|
|
|
|
{% include_relative branch.liquid %}
|
|
|
|
# EdgeFS Data Fabric Quickstart
|
|
|
|
EdgeFS Data Fabric virtualzing common storage protocols and enables multi-cluster, multi-region data flow topologies.
|
|
|
|
This guide will walk you through the basic setup of a EdgeFS cluster namespaces and enable you to consume S3 object, NFS file access, and iSCSI block storage
|
|
from other pods running in your cluster, in decentralized ways.
|
|
|
|
## Minimum Version
|
|
|
|
EdgeFS operator, CSI plugin and CRDs were tested with Kubernetes **v1.11** or higher.
|
|
|
|
## Prerequisites
|
|
|
|
To make sure you have a Kubernetes cluster that is ready for `Rook`, you can [follow these instructions](k8s-pre-reqs.md).
|
|
|
|
A minimum of 3 storage devices are required with the default data replication count of 3. (To test EdgeFS on a single-node cluster with only one device or storage directory, set `sysRepCount` to `1` under the `rook-edgefs` `Cluster` object manifest in `cluster/examples/kubernetes/edgefs/cluster.yml`)
|
|
|
|
To operate efficiently EdgeFS requires 1 CPU core and 1GB of memory per storage device. Minimal memory requirement for EdgeFS target pod is 4GB. To get maximum out of SSD/NVMe device we recommend to double requirements to 2 CPU and 2GB per device.
|
|
|
|
If you are using `dataDirHostPath` to persist rook data on kubernetes hosts, make sure your host has at least 5GB of space available on the specified path.
|
|
|
|
We recommend you to configure EdgeFS to use of raw devices and equal distribution of available storage capacity.
|
|
|
|
> **IMPORTANT**: EdgeFS will automatically adjust deployment nodes to use larger then 128KB data chunks, with the following addition to /etc/sysctl.conf:
|
|
|
|
```ini
|
|
net.core.rmem_default = 80331648
|
|
net.core.rmem_max = 80331648
|
|
net.core.wmem_default = 33554432
|
|
net.core.wmem_max = 50331648
|
|
vm.dirty_ratio = 10
|
|
vm.dirty_background_ratio = 5
|
|
vm.swappiness = 15
|
|
```
|
|
|
|
To turn off this node adjustment need to enable `skipHostPrepare` option in cluster CRD [configuring the cluster](edgefs-cluster-crd.md).
|
|
|
|
## TL;DR
|
|
|
|
If you're feeling lucky, a simple EdgeFS Rook cluster can be created with the following kubectl commands. For the more detailed install, skip to the next section to [deploy the Rook operator](#deploy-the-rook-operator).
|
|
|
|
```console
|
|
git clone --single-branch --branch {{ branchName }} https://github.com/rook/rook.git
|
|
cd cluster/examples/kubernetes/edgefs
|
|
kubectl create -f operator.yaml
|
|
kubectl create -f cluster.yaml
|
|
```
|
|
|
|
After the cluster is running, you can create [NFS, S3, or iSCSI](#storage) storage to be consumed by other applications in your cluster.
|
|
|
|
## Deploy the Rook Operator
|
|
|
|
The first step is to deploy the Rook system components, which include the Rook agent running on each node in your cluster as well as Rook operator pod.
|
|
|
|
Note that Google Cloud users need to explicitly grant user permission to create roles:
|
|
|
|
```console
|
|
kubectl create clusterrolebinding cluster-admin-binding --clusterrole cluster-admin --user $(gcloud config get-value account)
|
|
```
|
|
|
|
Now you ready to create operator:
|
|
|
|
```console
|
|
cd cluster/examples/kubernetes/edgefs
|
|
kubectl create -f operator.yaml
|
|
|
|
# verify the rook-edgefs-operator, and rook-discover pods are in the `Running` state before proceeding
|
|
kubectl -n rook-edgefs-system get pod
|
|
```
|
|
|
|
## Create a Rook Cluster
|
|
|
|
Now that the Rook operator, and discover pods are running, we can create the Rook cluster. For the cluster to survive reboots,
|
|
make sure you set the `dataDirHostPath` property. For more settings, see the documentation on [configuring the cluster](edgefs-cluster-crd.md).
|
|
|
|
Edit the cluster spec in `cluster.yaml` file.
|
|
|
|
Create the cluster:
|
|
|
|
```console
|
|
kubectl create -f cluster.yaml
|
|
```
|
|
|
|
Use `kubectl` to list pods in the `rook-edgefs` namespace. You should be able to see the following pods once they are all running.
|
|
The number of target pods will depend on the number of nodes in the cluster and the number of devices and directories configured.
|
|
|
|
```console
|
|
$ kubectl -n rook-edgefs get pod
|
|
rook-edgefs rook-edgefs-mgr-7c76cb564d-56sxb 1/1 Running 0 24s
|
|
rook-edgefs rook-edgefs-target-0 3/3 Running 0 24s
|
|
rook-edgefs rook-edgefs-target-1 3/3 Running 0 24s
|
|
rook-edgefs rook-edgefs-target-2 3/3 Running 0 24s
|
|
```
|
|
|
|
Notice that EdgeFS Targets are running as StatefulSet.
|
|
|
|
## Storage
|
|
|
|
For a walkthrough of the types of Storage CRDs exposed by EdgeFS Rook, see the guides for:
|
|
|
|
* **[NFS Server](edgefs-nfs-crd.md)**: Create Scale-Out NFS storage to be consumed by multiple pods, simultaneously
|
|
* **[S3X](edgefs-s3x-crd.md)**: Create an Extended S3 HTTP/2 compatible object and key-value store that is accessible inside or outside the Kubernetes cluster
|
|
* **[AWS S3](edgefs-s3-crd.md)**: Create an AWS S3 compatible object store that is accessible inside or outside the Kubernetes cluster
|
|
* **[iSCSI Target](edgefs-iscsi-crd.md)**: Create low-latency and high-throughput iSCSI block to be consumed by a pod
|
|
|
|
## CSI Integration
|
|
|
|
EdgeFS comes with built-in gRPC management services, which are tightly integrated into Kubernetes provisioning and attaching CSI framework. Please see the [EdgeFS CSI](edgefs-csi.md) for setup and usage information.
|
|
|
|
## EdgeFS Dashboard and Monitoring
|
|
|
|
Each Rook cluster has some built in metrics collectors/exporters for monitoring with [Prometheus](https://prometheus.io/).
|
|
In additional to classic monitoring parameters, it has a built-in multi-tenancy, multi-service capability. For instance, you can quickly discover most loaded tenants, buckets or services.
|
|
To learn how to set up monitoring for your Rook cluster, you can follow the steps in the [monitoring guide](./edgefs-monitoring.md).
|
|
|
|
## Teardown
|
|
|
|
When you are done with the cluster, simply delete CRDs in reverse order. You may want to re-format your raw disks with `wipefs -a` command. Or if you using raw devices and want to keep same storage configuration but change some resource or networking parameters, consider to use `devicesResurrectMode`.
|