Travis Nielsen
09c90655a7
When removing a mon from quorum, there is a race condition that
can result in mon quorum going being lost at least temporarily.
The mon pod was being deleted first, and then the mon removed
from quorum. If any other mon went down between the time the
pod of the bad mon was deleted and when the mon was removed from
quorum, there may not be sufficient quorum to complete the
action of removing the mon from quorum and the operator would
be stuck.
For example, there could be 4 mons temporarily due to timing
of upgrading K8s nodes where mons may be taken down for some
number of minutes. Say a new mon is started while the down
mon also comes back up. Now the operator sees it can remove
the 4th mon from quorum, so it starts to remove it. Now say
another mon goes down on another node that is being updated
or otherwise drained. Since the 4th mon pod was deleted
and another mon is down, there are only two mons remaining
in quorum, but 3 mons are required in quorum when there
are 4 mons. Therefore, the quorum is stuck until the
third mon comes back up.
The solution is to first remove the extra mon from quorum
before taking down the mon pod.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
(cherry picked from commit 8987d26a6d)
What is Rook?
Rook is an open source cloud-native storage orchestrator for Kubernetes, providing the platform, framework, and support for Ceph storage to natively integrate with Kubernetes.
Ceph is a distributed storage system that provides file, block and object storage and is deployed in large scale production clusters.
Rook automates deployment and management of Ceph to provide self-managing, self-scaling, and self-healing storage services. The Rook operator does this by building on Kubernetes resources to deploy, configure, provision, scale, upgrade, and monitor Ceph.
The status of the Ceph storage provider is Stable. Features and improvements will be planned for many future versions. Upgrades between versions are provided to ensure backward compatibility between releases.
Rook is hosted by the Cloud Native Computing Foundation (CNCF) as a graduated level project. If you are a company that wants to help shape the evolution of technologies that are container-packaged, dynamically-scheduled and microservices-oriented, consider joining the CNCF. For details about who's involved and how Rook plays a role, read the CNCF announcement.
Getting Started and Documentation
For installation, deployment, and administration, see our Documentation and QuickStart Guide.
Contributing
We welcome contributions. See Contributing to get started.
Report a Bug
For filing bugs, suggesting improvements, or requesting new features, please open an issue.
Reporting Security Vulnerabilities
If you find a vulnerability or a potential vulnerability in Rook please let us know immediately at cncf-rook-security@lists.cncf.io. We'll send a confirmation email to acknowledge your report, and we'll send an additional email when we've identified the issues positively or negatively.
For further details, please see the complete security release process.
Contact
Please use the following to reach members of the community:
- Slack: Join our slack channel
- GitHub: Start a discussion or open an issue
- Twitter: @rook_io
- Security topics: cncf-rook-security@lists.cncf.io
Community Meeting
A regular community meeting takes place the 2nd Tuesday of every month at 9:00 AM PT (Pacific Time). Convert to your local timezone.
Any changes to the meeting schedule will be added to the agenda doc and posted to Slack #announcements.
Anyone who wants to discuss the direction of the project, design and implementation reviews, or general questions with the broader community is welcome and encouraged to join.
- Meeting link: https://zoom.us/j/98052644520?pwd=K0R4RUZCc3NhQisyMnA5VlV2MVBhQT09
- Current agenda and past meeting notes
- Past meeting recordings
Official Releases
Official releases of Rook can be found on the releases page. Please note that it is strongly recommended that you use official releases of Rook, as unreleased versions from the master branch are subject to changes and incompatibilities that will not be supported in the official releases. Builds from the master branch can have functionality changed and even removed at any time without compatibility support and without prior notice.
Licensing
Rook is under the Apache 2.0 license.
