From b09ca55f3c248a5d9ae197abf19187664345eee1 Mon Sep 17 00:00:00 2001 From: travisn Date: Thu, 9 May 2019 00:48:41 -0400 Subject: [PATCH] ceph: upgrade guide describes failing over mons for new port The default mon port has changed from 6790 to 6789. With the msgr2 work in nautilus, the mons need to be listening on the default ports. These instructions will direct the users to failover the mons to the new default port. Signed-off-by: travisn --- Documentation/ceph-upgrade.md | 37 +++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) diff --git a/Documentation/ceph-upgrade.md b/Documentation/ceph-upgrade.md index 0234db556..f56218c60 100644 --- a/Documentation/ceph-upgrade.md +++ b/Documentation/ceph-upgrade.md @@ -259,6 +259,29 @@ At this point, your Rook operator should be running version `rook/ceph:v1.0.0` Verify the Ceph cluster's health using the [health verification section](#health-verification). +### 6. Update the Mon Ports + +The port on which the mons are listening has changed in the 1.0 release in order to support the new +msgr2 protocol in Nautilus. Previously, the mons were listening on port 6790 by default. +In 1.0 now they are expected to be listening on port 6789. To ensure an uninterrupted +upgrade to 1.0, the operator configures the existing mons to continue listening on port 6790. + +It is recommended that the mons be re-configured to listen on the new default port **before upgrading to Ceph Nautilus**; +however, the steps outlined below will still work after an upgrade to Nautilus. +Instead of changing the port, the recommended process is to failover the mons, +which will create new mons on port 6789. While the operator will automate most of this process, there are +several steps required to induce the operator to failover a mon. +1. Cause the mon to fail by setting the replicas on the mon deployment to zero. For example, if your mon is named `mon-a`: + - `kubectl -n rook-ceph edit deploy rook-ceph-mon-a` +1. Find the line with `replicas: 1` and change it to `replicas: 0`. Save the change and exit the editor. +1. Wait for 5-10 minutes for the operator to fail over the mon. You will see messages in the operator log that the mon is down and will be failed over after a timeout. +The length of the timeout is dependent on the setting `ROOK_MON_OUT_INTERVAL` in the Rook operator deployment (operator.yaml). +1. After the timeout, a new mon will be started and the old mon deployment will be automatically removed. +1. Confirm in the toolbox that all mons are in quorum: `ceph status`. Do not continue if they are not in quorum. +1. Repeat steps 1-5 for each of the old mons + +Note that clients connected with the Rook flex driver will be automatically updated when the mons are failed over. +No intervention is needed for the clients to find the new mons. ## Ceph Version Upgrades Rook 1.0 is the last Rook release which will support Ceph's Luminous (v12.x.x) version. Users are @@ -312,3 +335,17 @@ kubectl -n $ROOK_NAMESPACE exec -it $TOOLS_POD -- ceph osd require-osd-release n #### 4. Verify the updated cluster Verify the Ceph cluster's health using the [health verification section](#health-verification). + +If you see a health warning about enabling msgr2, please see the section above on [Updating the Mon Ports](#6-update-the-mon-ports). +``` +[root@minikube /]# ceph -s + cluster: + id: b02807da-986a-40b0-ab7a-fa57582b1e4f + health: HEALTH_WARN + 3 monitors have not enabled msgr2 +``` + +Alternatively, this warning can suppressed if a temporary workaround is needed. +``` +ceph config set global mon_warn_on_msgr2_not_enabled false +```