Fixes#2239
Set Ceph logging to `/dev/stderr`, and run rgw setup commands without
combined output so that logging (no on stderr) isn't captured as part of
the result of commands.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
The background is that there are at least two distributions with very
different host fs layouts. Those would be NixOS and GNU Guix.
For example, on NixOS, /lib/modules does not exist, it can instead
be found at /run/current-system/kernel-modules/lib/modules where
/run/current-system/kernel-modules is a symlink into /nix/store.
Within the modules directory, again on NixOS, there are more links
into the /nix store.
So to support alternate host fs layouts it is now possible to specify
the lib/modules host path and in addition specify extra mounts.
Signed-off-by: John Axel Eriksson <john@insane.se>
Previous code tried to interpret the string output as a `fmt.Sprintf`
string formatter, mangling `%` into `%!(MISSING)`, etc.
Thanks to go, I don't belive these are exploitable (unlike the similar
error in C).
Since this seemed to be a common error in the codebase, I did a quick
audit by visually inspecting the results of `git grep 'f([^"]'`. I
don't have a good suggestion for automated tests to prevent this in
future :(
Example error (look for `(MISSING)`):
```
E0927 05:31:07.618429 11227 driver-call.go:237] Failed to unmarshal output for command: unmount, output: "2018-09-27 05:31:07.191711 I | exec: Running command: df --type ceph /var/lib/kubelet/pods/95461479-c216-11e8-bcf0-02030782ac80/volumes/ceph.rook.io~rook/oe-scratch\n2018-09-27 05:46:43.808596 I | Filesystem 1K-blocks Used Available Use%!M(MISSING)ounted on\n2018-09-27 05:46:43.808659 I | 10.107.25.147:6790,10.109.173.79:6790,10.104.85.255:6790:/ 151678976 49410048 102268928 33%!/(MISSING)var/lib/kubelet/pods/95461479-c216-11e8-bcf0-02030782ac80/volumes/ceph.rook.io~rook/oe-scratch\n{\"status\":\"Success\"}\n", error: invalid character '-' after top-level value
```
Signed-off-by: Angus Lees <gus@inodes.org>
fix#2193
Fix regression introduced by PR allowing MDS config creation in init
container. This allows the new deployment-based mds clusters to scale
down where currently they can only remain constant or scale up.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
We need this environment variable in the main container, not the init
container, as it's used by the rook ceph-mgr module.
Also fix up the env var counts in the testcase.
Fixes: ca71b7a311 (Ceph mgr: Set up config in init container)
Signed-off-by: Jeff Layton <jlayton@redhat.com>
Progress toward issue #2003.
Includes design from design doc PR #1578
Use init containers to create configuration for Ceph mgrs. There is only
one init container in this design. The init container calls the Rook
binary to create Ceph config files which are then shared with the mds
daemon main container.
Once this init is run, the main mds daemon is run. Leaving room to use
the Ceph-versioned image in the future, call `ceph-mds --foreground ...`
to run the Ceph mds.
The refactor to using an init container also necessitated refactoring
the mdses replicaset implementation to a deployment-per-pod
implementation due to a chicken-egg problem. With a single container (in
the before times) the Rook binary was able to call the ceph-mds daemon
with an id generated from the pod name. Since the pod name is not known
before runtime, and the id is one of the few params that must be
specified to Ceph daemons on run, it is necessary to know the id
beforehand; thus the move to a deployment architecture following the
likes of the mon and mgr daemons.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
Ceph expects to be able to reason about physical hardware / nodes.
Before this patch the NODE_NAME that was available referred to the pod
name. This patch exposes the underlying node name, pod, and pod
namespace for the ceph daemons to consume.
fixes: #2078
Signed-off-by: Noah Watkins <nwatkins@redhat.com>
Progress toward issue #2003.
Includes design from design doc PR #1578
Use init containers to create configuration for Ceph mgrs. There is only
1 init container in this design:
1. Using the Rook image, call the Rook binary to create Ceph config
files shared with the mgr daemgr container.
Once this init is run, the main mgr daemgr is run. Leaving room to use
the Ceph-versioned image in the future, call `ceph-mgr --foreground ...`
to run the Ceph mgr.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
Add a 'test' package to the Ceph operator and define a test to verify
that containers produced by Rook-Ceph match what is expected. Because
this is for 'containers' and not strictly for 'ceph containers', this
could be moved a level up to the operator test package; however, if
other backends wish to use the container tests, they will likely need to
make modifications, and there is concern that this might make the Ceph
tests brittle.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
Add a package called 'spec' to the Ceph operator to store resource
definition specs that are shared by all Ceph daemons.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
Add functions for helping test Kubernetes volumes and volume mounts.
- Create functions for testing the existence of volumes/mounts by
name in a list of vols/mounts without needing to know the index of
the vol/mount in the list.
- Create functions for printing vols/mounts in a human-readable format
so that tests may output more useful errors.
- Create a test definition for ensuring that all the volumes a pod
provides and all the mounts from each pod container match. For each
volume, there must be at least one container which mounts the volume,
and for each mount, there must be a source volume.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
Progress toward issue #2003.
Includes design from design doc PR #1578
Use init containers to create configuration for Ceph mons. There are 3
init containers in this design:
1. Using the Rook image, call the Rook binary to create Ceph config
files ultimately stored in a k8s EmptyDir which is shared between all
containers in a mon pod.
2. Leaving room to use the Ceph-versioned image in the future, call
`monmaptool` to generate the monmap, stored in a shared config dir.
3. Leaving room to use the Ceph-versioned image in the future, call
`ceph-mon --mkfs ...` to set up the mon filesystem on the host's disk.
Once this init is run, the main mon daemon is run. Leaving room to use
the Ceph-versioned image in the future, call `ceph-mon --foreground ...`
to run the Ceph mon.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
To help with code readability, include daemons from `pkg/daemon/ceph`
as `<dmn>daemon "github.com/rook/rook/pkg/daemon/ceph/<dmn>`. This will
make it easier to understand code lines without needing to scroll up to
check the includes, as there are mon/mds/mgr/osd/rgw packages for
daemons as well as operator and cmd.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
Travis Nielsen
tsullens
Blaine Gardner
John Axel Eriksson
Angus Lees
Jeff Layton
Alexander Trost
Noah Watkins
Ratnadeep Debnath
Huamin Chen
Ganesh Maharaj Mahalingam