rook command doesn't interpret `logtostderr` option. It's OK to just
remove this option because `capnslog` outputs all logs to stdout
by default. It's better to keep `AddGoFlagSet()` call because
some libraries might define their own flags with Go's `flag` package.
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
The previous exit 32 check for loop device is 5 years old. Also, if the
device cannot be read it will be skipped anyway so let's report the
error and not hide it.
Signed-off-by: Sébastien Han <seb@redhat.com>
Create a new log level for Rook that is hidden from users. This is the
most verbose log level, and it is the level developers would like to use
to get debug logs that are important for debugging but that could leak
senstivie information like credentials in production use.
If a user sets their debug level to "TRACE", they will merely get
"DEBUG" level logs. Only if they set "TRACE_INSECURE" will they get
trace logs, and those are likely to include insecure information. Rook
tries very hard not to leak sensitive information in logs even with
verbose "DEBUG" logs.
Resolves#8778
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
Rook should update the RGW object store's period if the period doesn't
yet exist. This protects us from the case where the
'radosgw-admin period update --commit` command fails and the
CephObjectStore controller reconciles again.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
In ExtractExitCode, there is one error type that can be valid as a
pointer or not-as-a-pointer. Add a case to the type check for the
non-pointer condition.
Resolves#8280
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
This commit is a large refactor on how the operator starts, stops and
how it starts various sub-components such as the ceph-csi driver. It
also refines the way we cancel orchestrations. We don't use breakpoints
anymore but send our self a SIGUP to reload our controller runtime
manager.
The reload will happen under different circonstances like:
* a new adminission controller secret is created/deleted/changed
* a CephCluster CR is edited
As mentioned earlier, the csi driver now has its own controller, just
like flex. It reacts to change in the operator config map for particular
ROOK_CSI_ fields.
A second new controller for the operator's general config has been
created, it manages:
* the logging level
* the ceph CLI command timeout
* the discovery daemon
The operator reacts much more rapidly to cancellation events by stopping
the manager's context and reloading it.
Signed-off-by: Sébastien Han <seb@redhat.com>
Sometimes the error does not tell much, so as `exit status 1` and
printing the output along returning the error is useful.
For instance, I saw a job failing with no osd and the prepare job had
those lines:
```
exec: Running command: lsblk /dev/sdb1 --bytes --nodeps --pairs ....
inventory: skipping device "sdb1". exit status 1
```
We need to understand more about the lsblk issue.
Signed-off-by: Sébastien Han <seb@redhat.com>
During disaster recovery/migration of a cluster, as part of the failover, the
kubernetes artifacts like deployment, PVC, PV, etc will be restored to a new
cluster by the admin. Even if the kubernetes objects are restored the
corresponding RBD/CephFS subvolume cannot be retrieved during CSI operations as
the clusterID and poolID are not the same in both clusters
This PR creates a mapping between Cluster ID and RBD Pool ID between
local cluster and peer cluster.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
We should not use %+v when printing errors and wrapping it in the
caller. Passing %+v to errors.Wrap() returns a strack trace.
See:
```
2021-08-03 09:42:40.460946 E | ceph-file-controller: failed to reconcile failed to create filesystem "myfs2": failed to start deployment for MDS "a" for filesystem "myfs2": failed to update mds deployment "rook-ceph-mds-myfs2-a": failed to check if deployment "rook-ceph-mds-myfs2-a" can continue: failed to check if we can continue the deployment rook-ceph-mds-myfs2-a: failed to check if rook-ceph-mds-myfs2-a was ok to continue: max retries exceeded, last err: mds myfs2-a is up:creating, bad state
github.com/rook/rook/pkg/daemon/ceph/client.MdsActiveOrStandbyReplay
/home/leseb/go/src/github.com/rook/rook/pkg/daemon/ceph/client/status.go:286
github.com/rook/rook/pkg/daemon/ceph/client.okToContinueMDSDaemon.func1
/home/leseb/go/src/github.com/rook/rook/pkg/daemon/ceph/client/upgrade.go:215
github.com/rook/rook/pkg/util.Retry
/home/leseb/go/src/github.com/rook/rook/pkg/util/retry.go:31
github.com/rook/rook/pkg/daemon/ceph/client.okToContinueMDSDaemon
/home/leseb/go/src/github.com/rook/rook/pkg/daemon/ceph/client/upgrade.go:214
github.com/rook/rook/pkg/daemon/ceph/client.OkToContinue
/home/leseb/go/src/github.com/rook/rook/pkg/daemon/ceph/client/upgrade.go:181
github.com/rook/rook/pkg/operator/ceph/cluster/mon.UpdateCephDeploymentAndWait.func1
/home/leseb/go/src/github.com/rook/rook/pkg/operator/ceph/cluster/mon/spec.go:370
github.com/rook/rook/pkg/operator/k8sutil.UpdateDeploymentAndWait
/home/leseb/go/src/github.com/rook/rook/pkg/operator/k8sutil/deployment.go:113
github.com/rook/rook/pkg/operator/ceph/cluster/mon.UpdateCephDeploymentAndWait
/home/leseb/go/src/github.com/rook/rook/pkg/operator/ceph/cluster/mon/spec.go:383
github.com/rook/rook/pkg/operator/ceph/file/mds.(*Cluster).startDeployment
/home/leseb/go/src/github.com/rook/rook/pkg/operator/ceph/file/mds/mds.go:216
github.com/rook/rook/pkg/operator/ceph/file/mds.(*Cluster).Start
/home/leseb/go/src/github.com/rook/rook/pkg/operator/ceph/file/mds/mds.go:138
github.com/rook/rook/pkg/operator/ceph/file.createFilesystem
/home/leseb/go/src/github.com/rook/rook/pkg/operator/ceph/file/filesystem.go:81
github.com/rook/rook/pkg/operator/ceph/file.(*ReconcileCephFilesystem).reconcileCreateFilesystem
/home/leseb/go/src/github.com/rook/rook/pkg/operator/ceph/file/controller.go:373
github.com/rook/rook/pkg/operator/ceph/file.(*ReconcileCephFilesystem).reconcile
/home/leseb/go/src/github.com/rook/rook/pkg/operator/ceph/file/controller.go:288
github.com/rook/rook/pkg/operator/ceph/file.(*ReconcileCephFilesystem).Reconcile
/home/leseb/go/src/github.com/rook/rook/pkg/operator/ceph/file/controller.go:167
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/home/leseb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:298
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/home/leseb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/home/leseb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1371
```
Signed-off-by: Sébastien Han <seb@redhat.com>
Sometimes the default 15s is not enough for timeout of ceph commands. For examples,
I encountered that `radosgw-admin` command took dozens of seconds under heavy load.
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
Both `ExecuteCommandWithOutputFileTimeout()` and
`ExecuteCommandWithOutputFile()` generate unnecessary system calls by
creating/reading/removing files where the stream output of the command
can simply be used. So sticking with `ExecuteCommandWithOutput()` and
`ExecuteCommandWithCombinedOutput()` for reading outputs is sufficient.
Closes: https://github.com/rook/rook/issues/8343
Signed-off-by: Sébastien Han <seb@redhat.com>
Even if we can use raw mode, do NOT use raw mode on disks. Ceph bluestore disks can
sometimes appear as though they have "phantom" Atari (AHDI) partitions created on them
when they don't in reality. This is due to a series of bugs in the Linux kernel when it
is built with Atari support enabled. This behavior does not appear for raw mode OSDs on
partitions, and we need the raw mode to create partition-based OSDs. We cannot merely
skip creating OSDs on "phantom" partitions due to a bug in `ceph-volume raw inventory`
which reports only the phantom partitions (and malformed OSD info) when they exist and
ignores the original (correct) OSDs created on the raw disk.
Resolves https://github.com/rook/rook/issues/7940
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
When the CephCluster is configured with Multus and multiple networks are
used to deploy Ceph some commands are failing to be executed from the
Operator. These commands, in particular, `radosgw-admin` ones need access
to the "ceph public network" to talk to OSDs. Unfortunately, the
Rook-Ceph Operator does not have the network annotations and thus
doesn't have the networks available and cannot reach OSDs. So the commands end
up hanging and eventually time out.
Applying the annotations to the Operator pod is possible but will result
in restarting the operator too and this should be avoided at all costs.
Also, applying the annotations beforehand is not possible since the
Multus declaration is in the CephCluster specification. So we would have
no idea what to do.
So the current approach runs a new sidecar container in the mgr pod to
act as a proxy for "some" ceph commands, only the `radosgw-admin` ones
for multi-site setup. This is a small container with admin access
running idle waiting for commands to be executed. In a sense, it is
similar to the toolbox but we didn't want to clearly expose it, so
running as a sidecar is quite nice.
Proxying command is obviously not always recommended since we add an
extra hop in the network path. Now each request has to go from the
operator pod to the API server to the remote pod to Ceph. Previously,
the command only goes from the operator to Ceph.
It's worth noting that external mode is not impacted since no rgw pod
is configured. This scenario is flexible and allows us to scale
pretty well since any CephCluster with Multus will see its mgr sidecar
deployed and can then talk to Ceph. We are not limited.
Signed-off-by: Sébastien Han <seb@redhat.com>
Ceph bluestore raw disks can sometimes appear as though they have Atari
(AHDI) partitions on them. If a disk has Atari partitions, we just
ignore them as though they don't exist. This should be a safe assumption
since the hardware was last manufactured in 1992 and likely can't run
Kubernetes.
If we don't ignore the Atari partitions, Rook can create a new OSD on a
disk that is already running an OSD, corrupting the first and possibly
also the latest OSD. This can happen an arbitrary number of times per
disk.
More info: https://github.com/rook/rook/issues/7940
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
In e8f9cfcb71, the logging was replaced by
Debug which is probably a mistake given the intent of the commit to
enable debug logging on the prepare job.
However, this code is only triggered when running a ceph-osd with the
rook binary, so we must run Info and run Debug since Debug is not
activated.
Signed-off-by: Sébastien Han <seb@redhat.com>
Implement the first step of `design/ceph/resource-dependencies.md` to
add dependency checking when deleting a CephCluster.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
Currently it is not possible to set resource limits based on device classes
for different OSDs. Now adding an ability to use predefined keys for main
cluster spec Resource section to reflect resource limits for different OSDs
with different device classes.
Closes: https://github.com/rook/rook/issues/8007
Signed-off-by: Denis Egorenko <degorenko@mirantis.com>
Work around issue https://github.com/rook/rook/issues/7573
and make sure integration tests check for regressions.
Eventually we should use the RADOS Gateway admin REST API, but for now
we need to work around an issue where the built version of
'radosgw-admin' has incompatibilities with the RADOS Gateway version
running in the Ceph cluster.
Of note, Rook built on the Ceph Pacific image will not support
some 'radosgw-admin' commands to Ceph Nautilus (v14) or Octopus (v15)
clusters.
Further complicating matters, the flag used for the workaround changes
between Ceph v16.2.0 and v16.2.1 (both Pacific).
This bug needs to be treated a little differently than most of the ways
Rook handles different commands for different Ceph versions because this
is based on the Ceph version that is installed in the container with the
Rook operator primarily and not the version of Ceph running in the
cluster.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
Update OSDs in parallel per the design in
design/ceph/update-osds-in-parallel.md
The max number of OSDs updated in parallel is currently fixed at 20.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
Somehow the combined output does not contain the stderr in the errorn
only in the buffer.
For example:
err = "error exit 1"
buf = "Error EBUSY: not enough monitors would be available () after stopping mons [a]"
So we now embed the buf in the error so that the caller does not need to
print the buffer as well as the error.
Also, the caller can more easily decide to ignore the error by
instropecting the string.
Signed-off-by: Sébastien Han <seb@redhat.com>
Do not force unit tests to be run while the rook repo is in the GOPATH.
Now that Rook uses go modules, this is no longer necessary. Allow this
by creating a helper function that unit tests can use that return the
on-host path to the `rook` project root.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
Found by running the following command:
codespell -S .git,*.png,*.jpg -L \
aks,keyserver,atleast,dne,ser,ist,files\',ba,dum,iam,te -f -H
Signed-off-by: Mateusz Gozdek <mgozdekof@gmail.com>
We don't always get an error of the type "*exec.ExitError" so we must
validate the type before printing it otherwise the interface conversion
will fail.
Signed-off-by: Sébastien Han <seb@redhat.com>
In order to properly debug errors, we need to merge stderr inside the
`err` reported so that we don't only see the stdout.
We were doing this when executing without file output, doing the same
for the output file.
Signed-off-by: Sébastien Han <seb@redhat.com>
this commit handle golangci-lint linter errcheck.
`errcheck` - Errcheck is a program for checking for
unchecked errors in go programs. These unchecked errors
can be critical bugs in some cases
To see only staticcheck linter output
`golangci-lint run --disable-all -E errcheck`
Signed-off-by: subhamkrai <srai@redhat.com>
this commit will enable one more linter ineffassign
in golangci-lint.
This linter throws an error when variable is assigned and never used.
`golangci-lint run --disable-all -E ineffassign` is used detects ineffassign
errors only.
Signed-off-by: subhamkrai <subhamkumarrai03@gmail.com>
We can now encrypted OSD device that were provisioned via a storage
class using the PV interface.
The encryption works at the storageClassDeviceSets level, which means we
can have encrypted and non-encrypted sets.
Using the new key `encrypted` we can turn it on.
Signed-off-by: Sébastien Han <seb@redhat.com>
using defer for closing file which are open
for writing is not safe. so closing file again
following below steps:
1. open files
2. defer file.close()
3. write
4.file.close()
these will make sure files are closed.
Signed-off-by: subhamkrai <subhamkumarrai03@gmail.com>
this commit suppress the gosec errors for
g204: Audit use of command execution.
g304: File path provided as taint input.
g101: Look for hard coded credentials.
Signed-off-by: subhamkrai <subhamkumarrai03@gmail.com>
To ensure a file handle is closed, we defer the close command
so it is guaranteed to run when the method returns. The closing
of the file handle is not going to fail in our usage since we aren't
using the SetDeadline on the files that would cancel a request
and return an error.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Add mpath to the list of supported device types,
although it will only work for OSDs on PVCs.
OSDs on a raw device without PVCs have not
yet been tested.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
This commit adds support for LVs to the device availability check
in the OSD prepare pod.
The availability of an LV is checked by "ceph-volume lvm list".
If it returns non-empty result, the LV is in use and not available.
Closes: https://github.com/rook/rook/issues/5075
Signed-off-by: morimoto-cybozu <kenji_morimoto@cybozu.co.jp>
This commit fixes the argument for "ceph-volume inventory".
When a device "/dev/mapper/foo" is being checked for its availability,
the argument should not be "/dev/foo" nor "/dev/dm-1".
Signed-off-by: morimoto-cybozu <kenji_morimoto@cybozu.co.jp>
Satoru Takeuchi
Blaine Gardner
Sébastien Han
Santosh Pillai
parth-gr
Blaine Gardner
Denis Egorenko
Mateusz Gozdek
subhamkrai
subhamkrai
Travis Nielsen
morimoto-cybozu