forked from rook/rook
Sébastien Han
bd58790c31
When the CephCluster is configured with Multus and multiple networks are used to deploy Ceph some commands are failing to be executed from the Operator. These commands, in particular, `radosgw-admin` ones need access to the "ceph public network" to talk to OSDs. Unfortunately, the Rook-Ceph Operator does not have the network annotations and thus doesn't have the networks available and cannot reach OSDs. So the commands end up hanging and eventually time out. Applying the annotations to the Operator pod is possible but will result in restarting the operator too and this should be avoided at all costs. Also, applying the annotations beforehand is not possible since the Multus declaration is in the CephCluster specification. So we would have no idea what to do. So the current approach runs a new sidecar container in the mgr pod to act as a proxy for "some" ceph commands, only the `radosgw-admin` ones for multi-site setup. This is a small container with admin access running idle waiting for commands to be executed. In a sense, it is similar to the toolbox but we didn't want to clearly expose it, so running as a sidecar is quite nice. Proxying command is obviously not always recommended since we add an extra hop in the network path. Now each request has to go from the operator pod to the API server to the remote pod to Ceph. Previously, the command only goes from the operator to Ceph. It's worth noting that external mode is not impacted since no rgw pod is configured. This scenario is flexible and allows us to scale pretty well since any CephCluster with Multus will see its mgr sidecar deployed and can then talk to Ceph. We are not limited. Signed-off-by: Sébastien Han <seb@redhat.com>
81 lines
2.7 KiB
Go
81 lines
2.7 KiB
Go
/*
|
|
Copyright 2016 The Rook Authors. All rights reserved.
|
|
|
|
Licensed under the Apache License, Version 2.0 (the "License");
|
|
you may not use this file except in compliance with the License.
|
|
You may obtain a copy of the License at
|
|
|
|
http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
Unless required by applicable law or agreed to in writing, software
|
|
distributed under the License is distributed on an "AS IS" BASIS,
|
|
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
See the License for the specific language governing permissions and
|
|
limitations under the License.
|
|
*/
|
|
|
|
package clusterd
|
|
|
|
import (
|
|
"github.com/coreos/pkg/capnslog"
|
|
netclient "github.com/k8snetworkplumbingwg/network-attachment-definition-client/pkg/client/clientset/versioned/typed/k8s.cni.cncf.io/v1"
|
|
rookclient "github.com/rook/rook/pkg/client/clientset/versioned"
|
|
"github.com/rook/rook/pkg/util/exec"
|
|
"github.com/rook/rook/pkg/util/sys"
|
|
"github.com/tevino/abool"
|
|
apiextensionsclient "k8s.io/apiextensions-apiserver/pkg/client/clientset/clientset"
|
|
"k8s.io/client-go/dynamic"
|
|
"k8s.io/client-go/kubernetes"
|
|
"k8s.io/client-go/rest"
|
|
"sigs.k8s.io/controller-runtime/pkg/client"
|
|
)
|
|
|
|
// Context for loading or applying the configuration state of a service.
|
|
type Context struct {
|
|
|
|
// The kubernetes config used for this context
|
|
KubeConfig *rest.Config
|
|
|
|
// Clientset is a connection to the core kubernetes API
|
|
Clientset kubernetes.Interface
|
|
|
|
// DynamicClientset is a dynamic connection to the Kubernetes API
|
|
DynamicClientset dynamic.Interface
|
|
|
|
// Represents the Client provided by the controller-runtime package to interact with Kubernetes objects
|
|
Client client.Client
|
|
|
|
// APIExtensionClientset is a connection to the API Extension kubernetes API
|
|
APIExtensionClientset apiextensionsclient.Interface
|
|
|
|
// RookClientset is a typed connection to the rook API
|
|
RookClientset rookclient.Interface
|
|
|
|
// The implementation of executing a console command
|
|
Executor exec.Executor
|
|
|
|
// The implementation of executing remotely a console command to a given pod
|
|
RemoteExecutor exec.RemotePodCommandExecutor
|
|
|
|
// The root configuration directory used by services
|
|
ConfigDir string
|
|
|
|
// A value indicating the desired logging/tracing level
|
|
LogLevel capnslog.LogLevel
|
|
|
|
// The full path to a config file that can be used to override generated settings
|
|
ConfigFileOverride string
|
|
|
|
// Information about the network for this machine and its cluster
|
|
NetworkInfo NetworkInfo
|
|
|
|
// NetworkClient is a connection to the CNI plugin API
|
|
NetworkClient netclient.K8sCniCncfIoV1Interface
|
|
|
|
// The local devices detected on the node
|
|
Devices []*sys.LocalDisk
|
|
|
|
// RequestCancelOrchestration manages the orchestration and its possible cancellation
|
|
RequestCancelOrchestration *abool.AtomicBool
|
|
}
|