Commit Graph
566 Commits
Author SHA1 Message Date
giovanism cc62702d39 ceph: Use rook's NetworkSpec with type embedding
ceph: Use the new NetworkSpec api

ceph: Change all tests to use the NetworkSpec

ceph: Remove TypeMeta field from NetworkSpec

ceph: Change IsHost network func signature

Signed-off-by: giovanism <giovanism@outlook.co.id>
2019-08-26 16:17:03 +07:00
rohan47 7307575637 Added portable and failure-domain label to the OSD deployment
Signed-off-by: rohan47 <rohgupta@redhat.com>
2019-08-23 01:49:14 +05:30
Sébastien Han b3f58863e9 Merge pull request #3600 from mvollman/bugfix-osdsPerDevice
ceph: fix osdsPerDevice config
2019-08-22 20:24:01 +02:00
Travis Nielsen 1d28489c4a Merge pull request #3633 from ashishranjan738/fixsds
fix(storageClassDeviceSet): fixes resource limits and placements not getting applied to osds
2019-08-20 17:38:47 -06:00
Michael Vollman 33711b7d36 ceph: fix osdsPerDevice config
Fix variable scoping issue that is ignoring osdsPerDevice config
setting.  Ensure that in all scenarios both the global and per OSD
osdsPerDevice setting are honored.  When both are set the more specific
setting will be used. (#3599)

Signed-off-by: Michael Vollman <michael.b.vollman@gmail.com>
2019-08-20 07:09:12 -04:00
Odin Ugedal 7bd1ff1049 ceph: remove duplicated code
Signed-off-by: Odin Ugedal <odin@ugedal.com>
2019-08-18 11:10:28 +02:00
Odin Ugedal 0e73bbe2f3 ceph: set osd mem target correctly
Previously, the value was set to commonArgs, resulting in dead code,
since common arg isn't used after the append.

Signed-off-by: Odin Ugedal <odin@ugedal.com>
2019-08-16 17:29:38 +02:00
Ashish Ranjan a8827f3652 fix(storageClassDeviceSet): fixes resource limits and placements for osds
This commit fixes resource limits and placements not getting applied to osd pods provisioned using storageClassDeviceSets

Signed-off-by: Ashish Ranjan <aranjan@redhat.com>
2019-08-16 02:44:37 +05:30
rohan47andAshish Ranjan d2f52aebe5 Adds support for storageClassDeviceSet in rook-ceph operator
- Added code to support StorageClassDeviceSet spec provided in the cluster-on-pvc.yaml
- The code reads the StorageClassDeviceSet spec and creates pvc based on the ‘count’ field for each device set.
- OSD prepare job is started for each PVC which activates the ceph-volume on each PVC
- Finally OSD is started on each of the PVC device.

Co-authored-by: rohan47 <rohgupta@redhat.com>
Co-authored-by: Ashish Ranjan <aranjan@redhat.com>
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
2019-08-12 09:24:13 -06:00
Michael Vollman f3ef680298 ceph: Add support for per OSD metdataDevices
Add ceph-volume support for per OSD metdataDevices

Signed-off-by: Michael Vollman <michael.b.vollman@gmail.com>
2019-08-02 07:41:45 -04:00
Blaine Gardner 55a726fda9 ceph: osd use copy-binaries top level cmd
Replace usage of the custom osd copybins command with the new top-level
copy-binaries command introduced with cmd-reporter.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2019-07-29 09:23:45 -06:00
Blaine Gardner d774fec858 Merge pull request #3470 from rohantmp/fixTopologyAware
Fix: topologyAware does not pick up failure domains.
2019-07-22 10:19:28 -06:00
xieyanker 7672582e0a Correct typo about skipVolumeForDirectory's code comment
Signed-off-by: xieyanker <xjsisnice@gmail.com>
2019-07-22 19:05:14 +08:00
Rohan CJ 747619180c Fix: toplogyAware does not pick up failure domains.
Properly initialize c.validStorage, and ensure that it is used
for provision config from then on.

Signed-off-by: Rohan CJ <rohantmp@gmail.com>
2019-07-19 18:38:16 +05:30
Sébastien Han 9120791e24 ceph: chown ceph directories
At the beginning of every container startup we now change the ownership
of every known ceph directories to ceph:ceph.

This will allow daemons to start properly as well as allowing daemon to
writes their logs on files.

This is needed because ceph daemon now run as 'ceph' instead of 'root'

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-07-19 11:13:42 +02:00
Sébastien Han 900abbc967 ceph: run ceph processes with the 'ceph' user
Ceph daemons will run as the 'ceph' user instead of the 'root' user.
Existing data store (osd and mon) owned by root will continue to run as
'root'.
ALl CLI calls also inherate from this and will use them when fetching a key
for instance. If we run on a version of ceph that does not know these
  flags they will simply get ignored.

Consider the following:

[leseb@tarox~/go/src/github.com/rook/rook][use-ceph-user !] kubectl get pods -n rook-ceph
NAME                                     READY     STATUS      RESTARTS   AGE
rook-ceph-mds-myfs-a-67cff7df69-fq4l5    1/1       Running     0          3m
rook-ceph-mds-myfs-b-85bd5f59f5-d8hdn    1/1       Running     0          3m
rook-ceph-mgr-a-7d88b9457b-jzln6         1/1       Running     2          4m
rook-ceph-mon-a-64bf697496-r94tc         1/1       Running     0          4m
rook-ceph-osd-0-c5fdcd74c-9kvb4          1/1       Running     0          3m
rook-ceph-osd-prepare-minikube-shz9l     0/2       Completed   0          3m
rook-ceph-rbd-mirror-a-5cbff8469-xv64c   1/1       Running     0          3m
rook-ceph-rgw-my-store-f7645b44-vlbp4    1/1       Running     0          3m

MDS daemon:
ceph         1  0.2  0.1 449508 29200 ?        Ssl  15:10   0:00 ceph-mds --fsid=bf359e2c-a1df-4a80-9137-8134b41a5459 --keyring=/etc/ceph/keyring-store/keyring --log-to-stderr=true --err-to-stderr=true --mon-cluster-log-to-stderr=true --log-stderr-prefix=debug  --mon-host=10.110.204.228:6789 --mon-initial-members=a --name=mds.myfs-a --setuser=ceph --setgroup=ceph --foreground --mds-standby-for-fscid=1 --mds-standby-replay=true

MGR daemon:
ceph         1  0.8  0.6 867176 104732 ?       Ssl  15:10   0:02 ceph-mgr --fsid=bf359e2c-a1df-4a80-9137-8134b41a5459 --keyring=/etc/ceph/keyring-store/keyring --log-to-stderr=true --err-to-stderr=true --mon-cluster-log-to-stderr=true --log-stderr-prefix=debug  --mon-host=10.110.204.228:6789 --mon-initial-members=a --name=mgr.a --setuser=ceph --setgroup=ceph --foreground

MON daemon:
ceph         1     0  0 15:10 ?        00:00:02 ceph-mon --fsid=bf359e2c-a1df-4a80-9137-8134b41a5459 --keyring=/etc/ceph/keyring-store/keyring --log-to-stderr=true --err-to-stderr=true --mon-cluster-log-to-stderr=true --log-stderr-prefix=debug  --mon-host=10.110.204.228:6789 --mon-initial-members=a --name=mon.a --setuser=ceph --setgroup=ceph --foreground --public-addr=10.110.204.228 --public-bind-addr=172.17.0.6 --setuser-match-path=/var/lib/ceph/mon/ceph-a

OSD daemon:
ceph     30985  0.9  0.5 885736 81236 ?        Ssl  15:10   0:04      |   \_ ceph-osd --foreground --id 0 --conf /var/lib/rook/osd0/rook-ceph.config --osd-data /var/lib/rook/osd0 --setuser-match-path /var/lib/rook/osd0 --setuser ceph --setgro
up ceph --keyring /var/lib/rook/osd0/keyring --cluster rook-ceph --osd-uuid 98b4527d-f7a8-44f0-ac47-f8b55ff5ee05 --osd-journal=/var/lib/rook/osd0/journal

RBD mirror daemon:
ceph         1  0.0  0.1 914500 21848 ?        Ssl  15:10   0:00 rbd-mirror --setuser ceph --setgroup ceph --foreground -n client.rbd-mirror.a --conf /etc/ceph/ceph.conf --keyring /etc/ceph/keyring

RGW daemon:
ceph         1  0.2  0.3 5179452 52244 ?       Ssl  15:11   0:01 radosgw --setuser ceph --setgroup ceph --foreground --name=client.radosgw.gateway --rgw-mime-types-file=/var/lib/rook/rgw/mime.types

Even though Ceph processes are running as Ceph, the container itself runs as root.
We haven't reached the situation where we can runAsUser != "root" and enable SecurityContext to RunAsNonRoot, but that's our ultimate goal.

Resolves: https://github.com/rook/rook/issues/2664
Signed-off-by: Sébastien Han <seb@redhat.com>
2019-07-19 11:13:42 +02:00
Sébastien Han 05d8d23f18 ceph: upgrade: pass daemonType and Name directly
We do not determine the daemon type and name by string parsing anymore
but passing that info directly in the update call, it's more reliable.
Also removing a bunch of unused functions.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-07-17 00:58:31 +02:00
Sébastien Han 7eb7e8a192 ceph: upgrade: always retrieve cluster version
Do not consider clusterInfo Ceph version since it's the one picked by
the spec image.
The CephVersion provided here needs to be the current running version
of Ceph for when it is later passed to OkToStop and checked with
IsAtLeast. If you are upgrading from a pre 14.2.1 version to a
version 14.2.1 or newer, IsAtLeast will incorrectly return true and
then fail trying to run mon ok-to-stop on a mon pod running a pre 14.2.1 version.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-07-17 00:39:18 +02:00
Sébastien Han f317cfdd1c ceph: osd: remove leftover info logs
These were used on a previous patch as a debug, let's remove them. They
are not intended to be displayed.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-07-17 00:37:43 +02:00
Sébastien Han 9824bd2ff9 ceph: improve upgrade procedure
When a cluster is updated with a different image version, this triggers
a serialized restart of all the pods. Prior to this commit, no safety
check were performed and rook was hoping for the best outcome.

Now before doing restarting a daemon we check it can be restarted. Once
it's restarted we also check we can pursue with the rest of the
platform. For instance, with monitors we check that they are in quorum,
for OSD we check that PGs are clean and for MDS we make sure they are
 all active.

Fixes: https://github.com/rook/rook/issues/2889
Signed-off-by: Sébastien Han <seb@redhat.com>
2019-07-17 00:37:43 +02:00
rohan47 d78631841e Updated condition check to avoid operator crash
Empty deployment list made the operator to crash as the condition check
was checking if the value at the 0th location is nill wich fails in case
of empty list. Updated it to check length of list not equals 0.

Signed-off-by: rohan47 <rohgupta@redhat.com>
2019-07-11 14:56:10 +05:30
Sébastien Han 6d62005472 ceph: osd: fix cli arg
Somehow during startup the arg got misinterpreted by ceph-osd and was
failing with:

2019-06-19 15:23:09.098967 I | exec: Running command: ceph-osd --foreground --id 0 --osd-uuid feceb7d7-3148-4a0d-af7a-753ba2f21a92 --conf /var/lib/rook/osd0/rook-ceph.config --cluster ceph --default-log-to-file false --ms-learn-addr-from-peer false
2019-06-19 15:23:09.135921 I | unrecognized arg false
failed to start osd. Failed to complete '': exit status 1.

Now we concatenate the whole flag to avoid that.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-07-04 16:19:57 +02:00
travisn 8226d577f6 remove obsolete check for setting the resource ownerreferences
The check for whether to set the OwnerReferences is no longer
necessary. When passing the correct value to the apiVersion,
the OwnerReferences can be successfully set for OpenShift.

Signed-off-by: travisn <tnielsen@redhat.com>
2019-07-01 06:55:07 -06:00
Travis Nielsen 1d35f26302 Merge pull request #3313 from leseb/osd-sdn
ceph: osd: fix startup on sdn
2019-06-18 16:13:52 -06:00
Sébastien Han b2f2ff6b9d ceph: osd: fix startup on sdn
This commit adds a new flag to the osd startup so that on msgr2 (default
on Nautilus and above) the osd is able to find the IP address in the
container to bind to.

This requires this Ceph patch https://github.com/ceph/ceph/pull/28589
and is already present in Octopus.

Closes: https://github.com/rook/rook/issues/3140
Signed-off-by: Sébastien Han <seb@redhat.com>
2019-06-18 10:19:19 +02:00
rohan47 a420db69fe osd: Clean the osds that are out and safe-to-destroy
- When an osd is marked out, and it is safe to be destroyed then delete the osds deployment.
- Removed osdGracePeriod
- Updated unit tests to test osd marked out action

Signed-off-by: rohan47 <rohgupta@redhat.com>
2019-06-14 22:50:33 +05:30
Sébastien Han cba9a359a0 ceph: upgrade apply osd nautilus flag
When OSDs are running on Nautilus we always disable old osd features and
aplpy the onces for Nautilus as described in the upgrade doc.
During an upgrade or the next time an orchestration will be called the
command will be applied. The command is idempotent so we can run it each
time.
This can be backported for 1.0.3

Closes: https://github.com/rook/rook/issues/2960
Signed-off-by: Sébastien Han <seb@redhat.com>
2019-06-13 22:15:15 +02:00
rohan47 a380ca7dc1 fixed ceph pg dump pgs_brief json deserialization
Signed-off-by: rohan47 <rohgupta@redhat.com>
2019-06-06 21:54:37 +05:30
Noah Watkins 71c88ae9e9 ceph: log osd recovery at info level
for the osd recovery case (DOWN->UP) log at a matching log level as the
message indicating the OSD went down.

fixes: #2904

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
2019-06-04 13:57:49 -07:00
Jose A. Rivera f3a1512983 Read node labels for provider topology
Signed-off-by: Jose A. Rivera <jarrpa@redhat.com>
2019-05-14 13:18:00 -05:00
Travis Nielsen 221560bf24 Merge pull request #3090 from kshlm/delete-active-job-if-requested
ceph: Delete and recreate rook-ceph-detect-version job
2019-05-01 10:16:18 -06:00
travisn 77c7eb52f9 ceph: query the latest list of nodes for osd orchestration
The list of nodes to be queried for configuring OSDs
should include all of the K8s nodes before the placement
or other filters are applied. The operator was sometimes
missing configuring OSDs on nodes if the discover pod
hadn't finished running on a node. Now the osd prepare
job will be triggered on a node even if the discover pod
hasn't yet run there.

Signed-off-by: travisn <tnielsen@redhat.com>
2019-04-30 17:07:12 -06:00
Kaushal M 5e8375e921 ceph: Delete and recreate rook-ceph-detect-version job
The ceph controller will delete and recreate the detec-version job, when
an already existing job is found. This helps avoid cluster creation from
getting hung up on waiting for an already existing/stale job to finish.

Fixes #3081.

Signed-off-by: Kaushal M <kshlmster@gmail.com>
2019-04-30 18:19:56 +05:30
travisn b6568b07b4 ceph: stop disabling scrubbing during osd orchestration
Signed-off-by: travisn <tnielsen@redhat.com>
2019-04-26 09:35:50 -06:00
Rohan CJ fb4d23345e Revert "Merge pull request #2986 from rohantmp/mainMon"
This reverts commit b7a564e929, reversing
changes made to 6996571b5c.

Reverting: Linking ceph noout to a node taint.
The taint is applied for reasons other than maintenance as well.
This also needs a way to respect user-set noout.
Need to revisit this with a better design.

Signed-off-by: Rohan CJ <rohantmp@gmail.com>
2019-04-24 22:38:38 +05:30
Blaine Gardner f4e2be50a6 ceph spec: add ceph-version label to controllers
Add a 'ceph-version' label to application controllers in the same manner
as the prior 'rook-version' label to help with upgrades.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2019-04-18 15:09:33 -06:00
travisn bc87b440fb update to the k8s 1.14 client libraries
Signed-off-by: travisn <tnielsen@redhat.com>
2019-04-18 07:51:41 -06:00
Travis Nielsen b7a564e929 Merge pull request #2986 from rohantmp/mainMon
Ceph: Set noout on cluster when storage nodes are in maintenance
2019-04-17 09:56:50 -06:00
Travis Nielsen 6996571b5c Merge pull request #3005 from leseb/osd-mem-tune
ceph: osd, do not auto tune osd-memory-target
2019-04-17 09:56:03 -06:00
Rohan CJ 894f4bffbd - Set noout on cluster when storage nodes are in maintenance (unschedulable)
- Add noout functions to osd client
- export the functions DiscoverStorageNodes,GetAllStorageNodes
  from pkg/operator/ceph/cluster/osd
- Update PendingReleaseNotes.md

Signed-off-by: Rohan CJ <rohantmp@gmail.com>
2019-04-17 19:31:55 +05:30
Alexander Trost ea03048ef1 Merge pull request #2831 from galexrt/feature_2308
crds: Set annotations pods, deployments and so on
2019-04-17 10:27:14 +02:00
Sébastien Han f0b0f340f8 ceph: osd, do not auto tune osd-memory-target
As per https://github.com/ceph/ceph/pull/26856 Ceph tunes the value of
osd-memory-target when running the OSD process so there is no need to
enforce it.

We also set a safety factor to that value which has been calculated by
the Ceph eng/perf team. Without this factor, the OSD will get OOM killed
since it is consuming memory for other purposes too.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-04-17 00:14:04 +02:00
Alexander Trost 0c6eb7aa3d crds: Set annotations pods, deployments and so on
Allow the user to set annotations in the CRDs of the following operators which will then be set on
the resulting Pods (Deployments, StatefulSets and so on).
Operators:
* Cassandra
* Ceph
* CockroachDB
* EdgeFS
* Minio
* NFS

The Annotations related structures have been added to the
rook.io/v1alpha2 pkg.

Signed-off-by: Alexander Trost <galexrt@googlemail.com>
2019-04-16 20:47:05 +02:00
Noah Watkins c25e262c75 ceph/osd: mount /run/udev for lvs speedup
When Ceph OSD spins up using ceph-volume, `ceph-volume lvm activate
--no-systemd --bluestore...` runs `lvs` which can cause significant
delays when /run/udev is not mounted from the host.

Before:

[root@rook-ceph-osd-0-858f49b5-md4t9 /]# time lvs
  WARNING: Failed to connect to lvmetad. Falling back to device scanning.
  WARNING: Device /dev/sda not initialized in udev database even after waiting 10000000 microseconds.
...
  LV                                            VG                                        Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  osd-data-729c1fb6-440f-434c-be49-f199fa59f1c0 ceph-3bf7823b-6db3-42b6-bf8f-c36301e85238 -wi-ao---- <10.00g

real    1m30.545s

After:

[root@rook-ceph-osd-0-d8c4698b8-5vqnk /]# time lvs
  WARNING: Failed to connect to lvmetad. Falling back to device scanning.
  LV                                            VG                                        Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  osd-data-6c7249cc-cbd1-4f3b-9027-03725886bb30 ceph-9808319f-8567-4930-803a-41f70e537c1f -wi-ao---- <10.00g

real    0m0.043s

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
2019-04-16 11:09:13 -07:00
Sébastien Han 4438672e47 ceph: expose ceph version to osd pod
We must expose the ceph version to the osd pod so that we can generate
the proper ceph configuration file. If we don't do this all the version
check will assume they run the opposite of the desired version and the
formatting of the `ceph.conf` will be wrong.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-04-15 15:57:23 +02:00
Sébastien Han b8874d7f3f ceph: allow logging control
We now run all the daemon with a new option from Nautilus 14.2.1 which
allows us to tell to a daemon to not log on file. However, we can decide
to activate logging by editing the configuration flag 'log_to_file' via
the centralized config store like this for a particular daemon:

ceph config set mon.a log_to_file true

This is useful when a daemon keeps crashing and we want to collect log
files on the system.

Fixes: https://github.com/rook/rook/issues/2881
Signed-off-by: Sébastien Han <seb@redhat.com>
2019-04-15 10:52:03 +02:00
Sébastien Han d4c48bb293 ceph: expose /var/log/ceph to the host
It's quite convinient to expose /var/log/ceph so that we can decide to
activate logs locally on the machine and see what's going on.
This is only a placeholder when a daemon is stuck crashlooping and we
want to allow administrator to gather log files.
We still do not log on file but this can be activated via a config
option passed to the centralized config option store.

For some daemons, which typically do not store any data (rgw, rbd-mirror
and mds) we had to propagate dataDirHostPath from the cluster spec to
each creation call so that the bindmount can happen.

Fixes: https://github.com/rook/rook/issues/2881
Signed-off-by: Sébastien Han <seb@redhat.com>
2019-04-15 10:47:09 +02:00
Sébastien Han 4627b66636 ceph: expose clusterInfo to the osd code
When instanciating an osd, we now pass clusterInfo which basically
represents a collection of information about a particular Ceph cluster. Rook uses information
// about the cluster to configure daemons to connect to the desired cluster.

From clusterInfo, we use the CephVersion which helps us putting
conditions for our code execution.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-04-15 10:47:09 +02:00
Blaine Gardner 685a8df08d ceph: add 'rook-version' label to controllers
Add a `rook-version` label to all controller resources used by Ceph:
deployments, daemonsets, and jobs. Labels are added to controller
resources only and not to the pod templates within because changing pod
templates causes updates due to the label change. The controller
resources themselves are not updated with the label addition and
therefore don't cause an unnecessary upgrade.

Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
2019-04-12 13:56:31 -06:00
Alexander Trost 8bc26d9096 k8sclient: Update all operators to use apps/v1
All usages of k8s go client are now also using the versioned `AppsV1() `
call for the client.

Updated MySQL and Wordpress, and Kube Registy examples to use apps/v1
Deployments.

Signed-off-by: Alexander Trost <galexrt@googlemail.com>
2019-04-12 09:25:42 +02:00