documentation/design: Removed unnecessary whitespaces/new lines

Signed-off-by: Alexander Trost <galexrt@googlemail.com>
This commit is contained in:
Alexander Trost committed 2018-06-24 18:39:07 +02:00
1 parent 90644bb00b
commit d7795f30d0
8 files changed
+38 -42

No files matched your search

+9 -9
View File
@@ -14,16 +14,16 @@ and more. Rook makes it simple to enable the dashboard.
## Enable the Dashboard
The [dashboard](http://docs.ceph.com/docs/luminous/mgr/dashboard/) can be enabled with settings in the cluster CRD. The cluster CRD must have the dashboard `enabled` setting set to `true`.
This is the default setting in the example manifests.
The [dashboard](http://docs.ceph.com/docs/luminous/mgr/dashboard/) can be enabled with settings in the cluster CRD. The cluster CRD must have the dashboard `enabled` setting set to `true`.
This is the default setting in the example manifests.
```yaml
spec:
dashboard:
enabled: true
```
The Rook operator will enable the ceph-mgr dashboard module to listen on the default port 7000.
A K8s service will also be created to expose that port inside the cluster.
The Rook operator will enable the ceph-mgr dashboard module to listen on the default port 7000.
A K8s service will also be created to expose that port inside the cluster.
```bash
kubectl -n rook-ceph get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
@@ -31,15 +31,15 @@ rook-ceph-mgr ClusterIP 10.108.111.192 <none> 92
rook-ceph-mgr-dashboard ClusterIP 10.110.113.240 <none> 7000/TCP 3h
```
The first service is for reporting the [Prometheus metrics](monitoring.md), while the latter service is for the dashboard.
If you are on a node in the cluster, you will be able to connect to the dashboard by using either the
DNS name of the service at `http://rook-ceph-mgr-dashboard:7000` or by connecting to the cluster IP,
The first service is for reporting the [Prometheus metrics](monitoring.md), while the latter service is for the dashboard.
If you are on a node in the cluster, you will be able to connect to the dashboard by using either the
DNS name of the service at `http://rook-ceph-mgr-dashboard:7000` or by connecting to the cluster IP,
in this example at `http://10.110.113.240:7000`.
## Viewing the Dashboard External to the Cluster
Commonly you will want to view the dashboard from outside the cluster. For example, on a development machine with the
cluster running inside minikube you will want to access the dashboard from the host.
cluster running inside minikube you will want to access the dashboard from the host.
There are several ways to expose a service that will depend on the environment you are running in.
You can use an [Ingress Controller](https://kubernetes.io/docs/concepts/services-networking/ingress/) or [other methods](https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types) for exposing services such as
@@ -84,5 +84,5 @@ rook-ceph-mgr-dashboard-external NodePort 10.101.209.6 <none> 70
```
In this example, port `31176` will be opened to expose port `7000` from the ceph-mgr pod. Find the ip address
of the VM. If using minikube, you can `minikube ssh` to the machine and `ifconfig` to find the ip address.
of the VM. If using minikube, you can `minikube ssh` to the machine and `ifconfig` to find the ip address.
Now you can enter the URL in your browser such as `http://192.168.99.110:31176` and the dashboard will appear.
+5 -5
View File
@@ -57,11 +57,11 @@ The number of data and coding chunks you choose will depend on your resiliency t
Here are some examples to illustrate how the number of chunks affects the storage and loss toleration.
| Data chunks (k) | Coding chunks (m) | Total storage | Losses Tolerated | OSDs required |
|---|---|---|---|---|
| 2 | 1 | 1.5x | 1 | 3 |
| 2 | 2 | 2x | 2 | 4 |
| 4 | 2 | 1.5x | 2 | 6 |
| 16 | 4 | 1.25x | 4 | 20 |
| --------------- | ----------------- | ------------- | ---------------- | ------------- |
| 2 | 1 | 1.5x | 1 | 3 |
| 2 | 2 | 2x | 2 | 4 |
| 4 | 2 | 1.5x | 2 | 6 |
| 16 | 4 | 1.25x | 4 | 20 |
The `failureDomain` must be also be taken into account when determining the number of chunks. The failure domain determines the level in the Ceph CRUSH hierarchy where the chunks must be uniquely distributed. This decision will impact whether node losses or disk losses are tolerated. There could also be performance differences of placing the data across nodes or osds.
- `host`: All chunks will be placed on unique hosts
+5 -5
View File
@@ -8,10 +8,10 @@ indent: true
## Restoring Mon Quorum
Under extenuating circumstances, the mons may lose quorum. If the mons cannot form quorum again,
Under extenuating circumstances, the mons may lose quorum. If the mons cannot form quorum again,
there is a manual procedure to get the quorum going again. The only requirement is that at least one mon
is still healthy. The following steps will remove the unhealthy
mons from quorum and allow you to form a quorum again with a single mon, then grow the quorum back to the original size.
mons from quorum and allow you to form a quorum again with a single mon, then grow the quorum back to the original size.
For example, if you have three mons and lose quorum, you will need to remove the two bad mons from quorum, notify the good mon
that it is the only mon in quorum, and then restart the good mon.
@@ -48,7 +48,7 @@ ceph-mon -i ${good_mon_id} --extract-monmap ${monmap_path} \
--monmap=/var/lib/rook/${good_mon_id}/monmap
# review the contents of the monmap
monmaptool --print /tmp/monmap
monmaptool --print /tmp/monmap
# remove the bad mon(s) from the monmap
monmaptool ${monmap_path} --rm <bad_mon>
@@ -91,7 +91,7 @@ You will need to restart the good mon pod to pick up the changes. Delete the goo
kubectl -n rook-ceph delete pod -l mon=rook-ceph-mon1
```
Start the rook [toolbox](/Documentation/toolbox.md) and verify the status of the cluster.
Start the rook [toolbox](/Documentation/toolbox.md) and verify the status of the cluster.
```bash
ceph -s
```
@@ -99,7 +99,7 @@ ceph -s
The status should show one mon in quorum. If the status looks good, your cluster should be healthy again.
### Restart the operator
Start the rook operator again to resume monitoring the health of the cluster.
Start the rook operator again to resume monitoring the health of the cluster.
```bash
# create the operator. it is safe to ignore the errors that a number of resources already exist.
kubectl create -f operator.yaml
+1 -1
View File
@@ -24,7 +24,7 @@ Replication can be 1 or more
Erasure coding requires k >= 2 and m >= 1, where k is data chunks and m is coding chunks
Erasure coding specifies a plugin (default=jerasure)
Erasure coding specifies an encoding algorithm (default=reed_sol_van)
A pool can set its failure domain using a CRUSH rule (default=host)
A pool can set its failure domain using a CRUSH rule (default=host)
# Object Store
The object store CRD defines desired settings for an object store
+8 -9
View File
@@ -14,7 +14,7 @@ As observed, all Ceph OSDs are running in the same Pod.
The limitations of current design are:
- Reliability issue. One Pod for all OSDs doesn't have the highest reliability nor efficiency. If the Pod is deleted, accidentally or during maintenence, all OSDs are down till the ReplicaSet restart.
- Reliability issue. One Pod for all OSDs doesn't have the highest reliability nor efficiency. If the Pod is deleted, accidentally or during maintenence, all OSDs are down till the ReplicaSet restart.
- Efficiency issue. Resource limits cannot be set effectively on the OSDs since the number of osds per in the pod could vary from node to node. The operator cannot make decisions about the topology because it doesn't know in advance what devices are available on the nodes.
- Tight Ceph coupling. The monolithic device discovery and provisioning code cannot be reused for other backends.
- Process management issue. Rook's process management is very simple. Using Kubernetes pod management is much more reliable.
@@ -24,7 +24,7 @@ A more comprehensive discussions can be found at [this issue](https://github.com
## Terms
- Device Discovery. A DaemonSet that discovers unformatted devices on the host. The DaemonSet populates a per node Raw Device Configmap with device information. The deamonSet is running on nodes that are labelled as storage nodes. The DaemonSet can start independently of Rook Operator. Device Discovery is storage backend agnostic.
- Device Discovery. A DaemonSet that discovers unformatted devices on the host. The DaemonSet populates a per node Raw Device Configmap with device information. The deamonSet is running on nodes that are labelled as storage nodes. The DaemonSet can start independently of Rook Operator. Device Discovery is storage backend agnostic.
- Device Provisioner. A Pod that is given device or directory paths upon start and make backend specific storage types. For instance, the provisioner prepares OSDs for Ceph backend. It is a Kubernetes batch job and exits after the devices are prepared.
@@ -34,7 +34,7 @@ We propose the following change to address the limitations.
### Create new OSDs
| Sequence |Rook Operator | Device Discovery | Device Provisioner | Ceph OSD Deployment |
| Sequence |Rook Operator | Device Discovery | Device Provisioner | Ceph OSD Deployment |
|---|---|---|---|---|
| 0 | | Start on labeled storage nodes, discover unformatted devices and store device information in per node Raw Device Configmap | |
| 1 | Read devices and node filters from cluster CRD | | |
@@ -52,7 +52,7 @@ This change addresses the above limitations in the follow ways:
### Detailed Device Discovery Process
Each `Device Discovery` DaemonSet walks through device trees to unformatted block devices and stores the device information in a per node `Raw Device Configmap`.
Each `Device Discovery` DaemonSet walks through device trees to unformatted block devices and stores the device information in a per node `Raw Device Configmap`.
A sample of `Raw Device Configmap` from Node `node1` is as the following:
@@ -72,14 +72,14 @@ data:
size: 214748364800 # size in byte
rotational: 1 # 0 for ssd/nvme, 1 for hdd, based on reading from sysfs
```
### Discussions
It is expected Device Discovery will be merged into Rook Operator once local PVs are supported in Rook Cluster CRD. Rook Operator can infer the device topology from local PV Configmaps. However, as long as raw devices or directories are still in use, a dedicated Device Discovery Pod is still needed.
If the storage nodes are also compute nodes, it is possible that dynamically attached and unformatted devices to those nodes are discovered by Device Discovery DaemonSet. To avoid this race condition, admin can choose to use separate device tree directories: one for devices used for Rook and the other for compute. Or the Cluster CRD should explicitly identify which devices should be used for Rook.
Alternatively, since `rook agent` is currently running as a DaemonSet on all nodes, it is conceivable to make `rook agent` to poll devices and update device orchestration Configmap. This approach, however, needs to give `rook agent` the privilege to modify Configmaps. Moreover, `Device Discovery` Pod doesn't need privileged mode, host network, or write access to hosts' `/dev` directory, all of which are required by `rook agent`.
Alternatively, since `rook agent` is currently running as a DaemonSet on all nodes, it is conceivable to make `rook agent` to poll devices and update device orchestration Configmap. This approach, however, needs to give `rook agent` the privilege to modify Configmaps. Moreover, `Device Discovery` Pod doesn't need privileged mode, host network, or write access to hosts' `/dev` directory, all of which are required by `rook agent`.
## Impact
@@ -89,9 +89,8 @@ Alternatively, since `rook agent` is currently running as a DaemonSet on all nod
- Device Discovery. It is a new long running process in a DaemonSet that runs on each node that has matching labels. It discovers storage devices on the nodes and populates the raw devices Configmaps.
- Device Provisioner. Device Provisioner becomes a batch job, it no longer exec Ceph OSD daemon.
- Device Provisioner. Device Provisioner becomes a batch job, it no longer exec Ceph OSD daemon.
- Ceph OSD Daemon. `ceph-osd` is no longer exec'ed by Device Provisioner, it becomes the Pod entrypoint.
- Ceph OSD Daemon. `ceph-osd` is no longer exec'ed by Device Provisioner, it becomes the Pod entrypoint.
- Ceph OSD Pod naming. Rook Operator creates Ceph OSD Pod metadata using cluster name, node name, and OSD ID.
+2 -2
View File
@@ -48,11 +48,11 @@ After the MDS pods start, the file system is ready to be mounted.
## File System CRD
The file system settings are exposed to Rook as a Custom Resource Definition (CRD). The CRD is the Kubernetes-native means by which the Rook operator can watch for new resources. The operator stays in a control loop to watch for a new file system, changes to an existing file system, or requests to delete a file system.
The file system settings are exposed to Rook as a Custom Resource Definition (CRD). The CRD is the Kubernetes-native means by which the Rook operator can watch for new resources. The operator stays in a control loop to watch for a new file system, changes to an existing file system, or requests to delete a file system.
### Pools
The pools are the backing data store for the file system and are created with specific names to be private to a file system. Pools can be configured with all of the settings that can be specified in the [Pool CRD](/Documentation/ceph-pool-crd.md). The underlying schema for pools defined by a pool CRD is the same as the schema under the `metadataPool` element and the `dataPools` elements of the file system CRD.
The pools are the backing data store for the file system and are created with specific names to be private to a file system. Pools can be configured with all of the settings that can be specified in the [Pool CRD](/Documentation/ceph-pool-crd.md). The underlying schema for pools defined by a pool CRD is the same as the schema under the `metadataPool` element and the `dataPools` elements of the file system CRD.
```yaml
metadataPool:
+8 -8
View File
@@ -45,7 +45,7 @@ kubectl create -f object-store.yaml
At this point the Rook operator recognizes that a new object store resource needs to be configured. The operator will create all of the resources to start the object store.
1. Metadata pools are created (`.rgw.root`, `my-store.rgw.control`, `my-store.rgw.meta`, `my-store.rgw.log`, `my-store.rgw.buckets.index`)
1. The data pool is created (`my-store.rgw.buckets.data`)
1. A Ceph realm is created
1. A Ceph realm is created
1. A Ceph zone group is created in the new realm
1. A Ceph zone is created in the new zone group
1. A cephx key is created for the rgw daemon
@@ -57,11 +57,11 @@ When the RGW pods start, the object store is ready to receive the http or https
## Object Store CRD
The object store settings are exposed to Rook as a Custom Resource Definition (CRD). The CRD is the Kubernetes-native means by which the Rook operator can watch for new resources. The operator stays in a control loop to watch for a new object store, changes to an existing object store, or requests to delete an object store.
The object store settings are exposed to Rook as a Custom Resource Definition (CRD). The CRD is the Kubernetes-native means by which the Rook operator can watch for new resources. The operator stays in a control loop to watch for a new object store, changes to an existing object store, or requests to delete an object store.
### Pools
The pools are the backing data store for the object store and are created with specific names to be private to an object store. Pools can be configured with all of the settings that can be specified in the [Pool CRD](/Documentation/ceph-pool-crd.md). The underlying schema for pools defined by a pool CRD is the same as the schema under the `metadataPool` and `dataPool` elements of the object store CRD. All metadata pools are created with the same settings, while the data pool can be created with independent settings. The metadata pools must use replication, while the data pool can use replication or erasure coding.
The pools are the backing data store for the object store and are created with specific names to be private to an object store. Pools can be configured with all of the settings that can be specified in the [Pool CRD](/Documentation/ceph-pool-crd.md). The underlying schema for pools defined by a pool CRD is the same as the schema under the `metadataPool` and `dataPool` elements of the object store CRD. All metadata pools are created with the same settings, while the data pool can be created with independent settings. The metadata pools must use replication, while the data pool can use replication or erasure coding.
```yaml
metadataPool:
@@ -77,9 +77,9 @@ The pools are the backing data store for the object store and are created with s
### Gateway
The gateway settings correspond to the RGW service.
The gateway settings correspond to the RGW service.
- `type`: Can be `s3`. In the future support for `swift` can be added.
- `sslCertificateRef`: If specified, this is the name of the Kubernetes secret that contains the SSL certificate to be used for secure connections to the object store. The secret must be in the same namespace as the Rook cluster. Rook will look in the secret provided at the `cert` key name. The value of the `cert` key must be in the format expected by the [RGW service](http://docs.ceph.com/docs/master/install/install-ceph-gateway/#using-ssl-with-civetweb): "The server key, server certificate, and any other CA or intermediate certificates be supplied in one file. Each of these items must be in pem form." If the certificate is not specified, SSL will not be configured.
- `sslCertificateRef`: If specified, this is the name of the Kubernetes secret that contains the SSL certificate to be used for secure connections to the object store. The secret must be in the same namespace as the Rook cluster. Rook will look in the secret provided at the `cert` key name. The value of the `cert` key must be in the format expected by the [RGW service](http://docs.ceph.com/docs/master/install/install-ceph-gateway/#using-ssl-with-civetweb): "The server key, server certificate, and any other CA or intermediate certificates be supplied in one file. Each of these items must be in pem form." If the certificate is not specified, SSL will not be configured.
- `port`: The service port where the RGW service will be listening (http)
- `securePort`: The service port where the RGW service will be listening (https)
- `instances`: The number of RGW pods that will be started for this object store (ignored if allNodes=true)
@@ -102,7 +102,7 @@ The RGW service can be configured to listen on both http and https by specifying
By default, the object store will be created independently from any other object stores and replication to another object store will not be configured. This done by creating a new realm, zone group, and zone all with the name of the new object store. The zone group and zone are tagged as the `master`. If this is the first object store in the cluster, the realm, zone group, and zone will also be marked as the default.
By implementing on the independent realms, zone groups, and zones, Rook supports multiple objects stores in a cluster. The set of users with access to the object store, the metadata, and the data are isolated from other object stores.
By implementing on the independent realms, zone groups, and zones, Rook supports multiple objects stores in a cluster. The set of users with access to the object store, the metadata, and the data are isolated from other object stores.
If desired to configure the object store to replicate from another cluster or zone, the following settings would be specified on a new object store that is *not* the master. (This feature is not yet implemented.)
- `realm`: If specified, the new zone will be created in the existing realm with that name
@@ -120,7 +120,7 @@ If desired to configure the object store to replicate from another cluster or zo
Failing over the master could be handled by updating the affected object store CRDs, although more design is needed here.
See the ceph docs [here](http://docs.ceph.com/docs/master/radosgw/multisite/) on the concepts around zones and replicating between zones. For reference, a diagram of two zones working across different cluster can be found on page 5 of [this doc](http://ceph.com/wp-content/uploads/2017/01/Understanding-a-Multi-Site-Ceph-Gateway-Installation-170119.pdf).
See the ceph docs [here](http://docs.ceph.com/docs/master/radosgw/multisite/) on the concepts around zones and replicating between zones. For reference, a diagram of two zones working across different cluster can be found on page 5 of [this doc](http://ceph.com/wp-content/uploads/2017/01/Understanding-a-Multi-Site-Ceph-Gateway-Installation-170119.pdf).
### Ceph multi-site object store data model
@@ -145,4 +145,4 @@ Zone group metadata is replicated to other zone groups in the realm
A zone belongs to one cluster
A zone has a set of pools that store the user and object metadata and object data
Zone data and metadata is replicated to other zones in the zone group
```
```
-3
View File
@@ -355,6 +355,3 @@ metadata:
namespace: mycluster
...
```