This commit is a large refactor on how the operator starts, stops and
how it starts various sub-components such as the ceph-csi driver. It
also refines the way we cancel orchestrations. We don't use breakpoints
anymore but send our self a SIGUP to reload our controller runtime
manager.
The reload will happen under different circonstances like:
* a new adminission controller secret is created/deleted/changed
* a CephCluster CR is edited
As mentioned earlier, the csi driver now has its own controller, just
like flex. It reacts to change in the operator config map for particular
ROOK_CSI_ fields.
A second new controller for the operator's general config has been
created, it manages:
* the logging level
* the ceph CLI command timeout
* the discovery daemon
The operator reacts much more rapidly to cancellation events by stopping
the manager's context and reloading it.
Signed-off-by: Sébastien Han <seb@redhat.com>
Recently lib-bucket-provisioner add support for update() API.
Include that on the obc implementation since it can be used to
update quota for OBC.
Fixes: #7146
Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
The go-ceph library has removed the `Debug` field from the API type in
https://github.com/ceph/go-ceph/pull/543. Since the HTTP Client can be
mutated we now have our own client to dump requests and responses when
the operator log level is DEBUG.
Signed-off-by: Sébastien Han <seb@redhat.com>
The lib-bucket-provisioner was missing the cephCluster spec so the check
in RunAdminCommandNoMultisite would fail.
This is similar to ad54ed8aac just a
different place.
Signed-off-by: Sébastien Han <seb@redhat.com>
The Rados Gateway Admin OPS API has changed its behavior from Nautilus
to Pacific. Calling user create on an existing user wil generate
additional keys to the user on Nautilus. Where in Pacific it will report
an error with UserAlreadyExists.
So to handle both scenarios, let's first get the user, and if the user
does not exist (NoSuchUser) we then create it.
Signed-off-by: Sébastien Han <seb@redhat.com>
Currently `setObjectStorePort()` queries port from RGW service and
picks up the first port from the list, other places of the code
prefers for TLS, if it is enabled so for OBC's we end up having
`https` endpoint with non-secure port.
Fixes: #8141
Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
The debug for adminOps client can be enabled by setting `Debug` flag.
In this PR, it is enabled for OBC and cephobjectstore not healthchecker.
Also add similar changes while calling `NewS3Agent` in the bucket
provisioner code path.
Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
For the TLS communication for AdminOps Api, httpclient is required and
filled with TLS certs, currently it is set to nil pointer in
`buckethealthchecker` and `cephobjectstoreuser`.
Thanks @Krast76 finding the issue even proposing the fix.
Fixes: #8132
Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
We have been having many issues with external mode with Ceph version
mismatching. The operator would have a Ceph version different than the
external cluster. The `radosgw-admin` was used to interact with S3
users, even a small version delta would cause the command to coredump.
After checking with the rgw core team it appears Rook was misusing the
CLI and the admin ops API should be used instead.
So this patch is the first introduction of go-ceph in Rook to consume
the rgw admin ops API instead of the `radosgw-admin` CLI, **only** for
user management in this initial commit.
Later we can do more such as bucket operation, zone management etc.
Closes: https://github.com/rook/rook/issues/7924
Signed-off-by: Sébastien Han <seb@redhat.com>
OBC operations will fail if RGW only enabled HTTPS. With this change, OBC operations
will use needed TLS certs to succeed the operation.
Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
In case ssl enabled for RGW, bucket healthcheck won't work since access is denied from the server.
In that case configure S3 agent using "InsecureSkipVerify: true" option.
Fixes: 7288
Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
Update to the latest lib bucket provisioner code.
Fixes issue 6650
Modifies CRD for objectbucketclaims to fix an additional bug where an
ObjectBucket's 'ClaimRef' is lost due to the CRD validation being
specified incorrectly.
Changes OBC deletion/cleanup to delete the bucket before the user. A
user cannot be deleted without an unsafe purge option if the user has
buckets associated to it.
Does not reintroduce bug 6767 from previous fix for 6650
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
During the 1.3 cycle, we had implemented support for OBC external mode
through an Endpoint in the StorageClass parameter. In 1.4, this is not
the case anymore as we directly use the CephObjectStore external mode.
However, we must maintain backward compatibility with the cluster using
the Endpoint and not fail the upgrade.
Thus checking for a CephObjectStore is invalid and should not always be
assumed, now if we detect an Endpoint we return a simple context which
fixes the upgrade.
Signed-off-by: Sébastien Han <seb@redhat.com>
User unlink is performed just before deleting OBC resources.
Since user is internal to OBC and next to remove to delete user,
unlinking user is not necessary.
Fixes: 6300
Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
this commit will enable one more linter ineffassign
in golangci-lint.
This linter throws an error when variable is assigned and never used.
`golangci-lint run --disable-all -E ineffassign` is used detects ineffassign
errors only.
Signed-off-by: subhamkrai <subhamkumarrai03@gmail.com>
By addding ".svc.cluster.local" to the endpoint of the object store, the
OBC requests won't go through a proxy if any is configured.
Signed-off-by: Sébastien Han <seb@redhat.com>
There is incorrect usage of log functions in the object directory, changing those instances
Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
- Add realm/zone group/zone to Object Context,
so that any call to runAdminCommand has the
same realm, zone group, and zone as the object
store that the call is going to.
- Also modify the delete code for the object-store
using multisite to remove the endpoints of an
object store from a zone instead of the normal
clean-up
- Added more debug logging all around the object-store
code related to multisite
- files generated by rerun of `make codegen`
- change back the edit on the Copyright in object.go
Signed-off-by: Ali Maredia <amaredia@redhat.com>
To provide more context for executing commands in a ceph cluster,
the full clusterInfo is now passed to the ceph execution commands.
All information about the cluster will now be available throughout
all the areas of the operator. The namespace, ceph credentials,
mon endpoints, and other info is a core part of that cluster info.
Arguments passed through the controllers are also simplified for
mons, mgr, osds, and other daemons where the parameters had
become too complex.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The output of the bucket policy change should not be logged in case
there is sensitive information in the output.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
We have introduced a new goroutine to check the state of the rgw
endpoint. It will run every minute and perform operations on a bucket.
The success or failure will be reported as part of the status field of
the CephObjectStore CR.
A good status will look like:
status:
endpointStatus:
lastChanged: "2020-06-25T13:47:45Z"
lastChecked: "2020-06-25T13:48:46Z"
phase: Connected
A failed status:
status:
endpointStatus:
details: |-
error creating bucket "rook-ceph-internal-s3-bucket-checker": RequestError: send request failed
caused by: Put http://rook-ceph-rgw-my-store.rook-ceph:8080/rook-ceph-internal-s3-bucket-checker: dial tcp 10.108.189.148:8080: connect: connection refused
health: ERROR
This check works for both converged and external modes. Note that the
CephObjectStore CRD has a new field called "externalRgwEndpoints" which
allows you to define a list of IP addresses pointing to rgws.
Closes: https://github.com/rook/rook/issues/5692
Signed-off-by: Sébastien Han <seb@redhat.com>
When obc creation fails in Grant() or Provision() the resources such as rgw user or bucket
created while provisioning need to remove.
Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
Some external consumers of Rook-Ceph are using the
lib-bucket-provisioner operator which creates different CRDs for both
objectbuckets.objectbucket.io and objectbucketclaims.objectbucket.io.
They have the `additionalConfig` declared in their spec. Rook-Ceph does
not and thus ignores it.
For the OB creation to work, the `additionalConfig` must be acknowledged
by the operator and populate or initialize it if empty.
This is preventing the following error:
controller.go:190] error syncing 'default/ceph-retain-bucket-converged': error creating OB "": ObjectBucket.objectbucket.io "obc-default-ceph-retain-bucket-converged" is invalid: spec.endpoint.additionalConfig: Invalid value: "null": spec.endpoint.additionalConfig in body must be of type object: "null", requeuing
Signed-off-by: Sébastien Han <seb@redhat.com>
When obc is deleted, the attached ceph user will be also removed
This will cause obc creation failure for the same bucket since user
was missing. With this patch the user will be retained revoking his permissions
Also removed 'GetBucketPolicy' permission from Allowed action. The same set of
permission is used for denying the owner/old user of OBC. So that in the next
obc request for the bucket, owner can set polciy for the new user
Closes: https://github.com/rook/rook/issues/5510
Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
This PR is for rebasing #4683 with master. With new controller runtime changes most the changes with #4683 got redundant.
Only valid change is waiting for all the object buckets to be cleaned up before removing the finalizer.
Co-authored-by: Elise Gafford <egafford@redhat.com>
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
When a user or a bucket does not exist anymore, let's stop the reconcile
loop instead of waiting forever.
Also, unlink the user before deleting it otherwise the deletion fails.
Signed-off-by: Sébastien Han <seb@redhat.com>
Sébastien Han
Jiffin Tony Thottan
Blaine Gardner
Blaine Gardner
Arun Kumar Mohan
Travis Nielsen
subhamkrai
Ali Maredia
Santosh PillaiandElise Gafford