This touches a lot where a typo could cause issues for someone, so I have self-
reviewed it many times line-by-line already.
There are a few types of changes:
- All resources get tied to their own YAML document separator (`---`)
before their definition, reducing the risk of not separating resources
correctly when rendering multiple into a file. I think the changes
will fix edge case bugs for people with mutiple items listed in
`cephFileSystems` and similar configuration.
- `toYaml ...` is made into `... | toYaml` consistently.
- `default A B` is made into `B | default A` consistently.
- Consistent whitespace chomping, so instead of `{{ end }}` you would
find `{{- end }}` typically. The system is that you should chomp left
almost all the time, and almost all the time _not_ chomp right. You
would chomp right in the beginning of a file or helper function in
order to clear some initial whitespace, otherwise not.
- Multiline helm template comments `{{- /* ... */ }}` get their content
indented 2 spaces.
- List items are consistently rendered with two spaces of indentation.
- `kind` is put before `apiVersion` consistently, to follow a practice
observed partially in the repo.
- Calls to the helm `template` function have been replaced with
`include` to be consistent and to stick with modern practices.
- Use of `indent` have been replaced with `nindent #` and a left
whitespace chomp for a consistent easy to follow practice that gets
good result.
- Several `if` statements like `if X -> render X` have been replaced
with `with X -> render .` to avoid repetition. The `with` statement
won't render if the provided context, making it function also like an
`if` statement.
- Fixed a bug with `if .Values.revisionHistoryLimit`, which should
really be `if not (typeIs "<nil>" .Values.revisionHistoryLimit)` to
handle situations when its set to `0`.
Signed-off-by: Erik Sundell <erik@sundellopensource.se>
Reduce CRD size by removing some descriptions for things that
(a) are long, (b) are repeated, and (c) are commong kubernetes types.
Remove the `Status` field from VolumeClaimTemplates (a.k.a. PVCs) in
types.go to remove all unnecessary status-related fields.
Use `yq` to go through generated `crds.yaml` to strip out descriptions
for well-known k8s types that are long and repeated where we can't
remove them from types.go. This is mostly placement descriptions and
volumeSource descriptions.
The overall `crds.yaml` file is reduced to less than 60% of its original
size.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
Signed-off-by: Redouane Kachach <rkachach@redhat.com>
Co-authored-by: Blaine Gardner <blaine.gardner@ibm.com>
Long CRD descriptions were pushing the size of the CRDs manifest
over 1MB, which causes issues when creating the CRDs. To shorten
the manifest, the descriptions are now limited to 100 chars.
This still gives some ability for dev tools to show a brief
description if needed. The generated CRD docs still contain
the full descriptions for reference.
Signed-off-by: travisn <tnielsen@redhat.com>
In the past we had few cases where the new crd
was missed adding in metadata-common.yaml file.
So, now we'll check if crds.yaml and metadata-common.yaml
crd name are matching.
Signed-off-by: subhamkrai <srai@redhat.com>
Currently, if there is any change in the API we need to separately
run the make generate-crd-docs but with this change
make crds will also update the generated docs
Also improved the file locations
Closes: https://github.com/rook/rook/issues/12310
Signed-off-by: parth-gr <paarora@redhat.com>
removing the line added by gen-crd-api-reference-docs to avoid commit
specification.md with only the reference to the git commit
Signed-off-by: Javier <sjavierlopez@gmail.com>
The crds generation for the helm crds is intermittently hitting a build
error due to multiple builds generating the crds at the same time.
We avoid this race condition by first writing the crds to a temp file
then move the file to the helm crd resources file.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The volume replication crds were being added to the generation directory
and causing subsequent builds to pick up those crds in the generation
of crds.yaml and the helm generation. To keep the volume replication
crds out of the generation, the directory needs to be cleaned
with each run.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The volume replication crds are not owned by rook, but if installing
with the csv, ownership must be specified for the CRDs. Thus, we limit
ownership to the csv for the volume replication crds.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Simply applying `LC_ALL=C` will work for the current shell only. For all
the shells and sub-shell to use it, we must set it globally.
Making it the first thing in the Make entrypoint now so the CRD and RBAC
sorting works on all the platforms.
Signed-off-by: Sébastien Han <seb@redhat.com>
Linux doesn't guarantee file ordering, so when dealing with files in
order, we can use 'sort' to make sure the order is deterministic. For
yq's usage of files, we need to keep file names in an intermediate
variable, and this should be an array to support file paths with spaces.
'readarray' is not available on MacOS's built-in bash, so a longer
'while read' loop is needed for the array creation.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
Use yq instead of Python for parsing RBAC from the Helm chart. We need
to use yq v4.14.1 or higher to fix yq's handling of the yaml header
markers ('---'). Update the Makefile's yq version to v4, which also
requires updating the script to update the CRDs. This was quite easy.
It is very difficult, however, to change the version of yq used by the
CSV generating/parsing scripts, which already used their own yq
download. Continue using yq v3 for this.
In order to make sure the scripts are using the right version of yq, add
basic validation to them to verify they are running v3 or v4 as required
for their operation.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
The volume replication CRDs are an external component, not owned by Rook.
Therefore, they should be installed as any other independent component
in case the admin will install other consumers of the volumereplication CRDs
in the future in addition to Rook and the CSI driver.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
In Rook v1.8 the min version of K8s supported is updated to 1.16.
Users running on older versions of K8s are recommended to update
to 1.16 or newer before updating to Rook v1.8.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The flex driver has been fully deprecated and thus removed from Rook.
Before upgrading to v1.8, users will need to convert existing flex volumes
from flex to csi volumes.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
deepcopy-gen appends a header starting with `// +build` which makes
gofmt to complain in Go 1.17 so until this is fixed in
https://github.com/kubernetes/gengo/pull/210 let's skip those files for
formatting.
Signed-off-by: Sébastien Han <seb@redhat.com>
to use prerelease of kubernetes it requires
to have `-0` added in kubernetes version check
ex:
```{{- if semverCompare ">=1.16.0-0" .Capabilities.KubeVersion.GitVersion }}```
Closes: https://github.com/rook/rook/issues/8548
Signed-off-by: subhamkrai <srai@redhat.com>
If a user were to `make build` Rook, and after the Ceph CSV was
generated and before make had finished, the Rook repo would be left with
changes caused by the build modifying files temporarily in-place.
All CSV template generation in the Ceph makefile is now self-contained
and can be configured to operate outside of the main repo so that files
are not modified in-place.
In-place modification of files was also incorrect. On Mac some files
with `-e` trailing were left, and in-place backups were also present in
the form `<filename>''`. Therefore, also fix the generation to fix
in-place modifications.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
The CRDs v1 requires the full schema for all settings, so we now
generate the CRDs for nfs for full fidelity of all settings.
Co-authored-by: Nicolaj Græsholt <figaw@hotmail.com>
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The CRDs v1 requires the full schema for all settings, so we now
generate the CRDs for cassandra for full fidelity of all the
settings.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The upgrade of the csv is failing to the latest schema when
the crushDeviceClass is specified and when the volumeClaimTemplate
annotations do not preserve the unknown fields. To fix the upgrade,
we preserve the fields even though it should not be necessary
from the strict schema.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Allow metadata on VolumeClaimTemplates in StorageClassDeviceSets to be
persisted. The latest changes to controller-gen allows us to embed
objectmeta for embedded types.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
We generate vendor directory in building process. It doesn't reduce
build time. In addition, we have encountered some vendoring-related
problems. So let's stop vendoring.
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
The POSIX spec for sed requires the flag -i to have a string specified.
`sed -i'' -e '<expression>'` is portable
The POSIX spec for sed also requires the insert "i" option to have
newlines. The following is portable:
```
sed -i'' -e '<line-num>i\
text'
```
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
The schema generation was not generating the metadata properties
under the VolumeClaimTemplate type, resulting in the properties being
ignored by the api server. The preserve unknown fields tag was
not working recursively on the volumeClaimTemplates since the subtypes
were defined. Now we post-process the schema so we can preserve
the unknown fields for the volume claim templates metadata.
We now use 'yq' to hot-fix the CRDs, the only 'downside' is that 'yq'
will not preserve the existing formatting, this is a known problem:
https://github.com/mikefarah/yq/issues/465
Signed-off-by: Sébastien Han <seb@redhat.com>
This reverts commit a9fb9d20bb. During the
CSV generation we strip the description lines from the CRD spec, however
the 'sed' command relies on the 'descrition' section to operate. Next
commit will fix this.
Signed-off-by: Sébastien Han <seb@redhat.com>
With the help of https://github.com/rook/rook/pull/7403/, rook will soon
integrate the volume replication controller as a side-car. Thus Rook
needs to inject the new CRDs.
Signed-off-by: Sébastien Han <seb@redhat.com>
The schema generation was not generating the metadata properties
under the VolumeClaimTemplate type, resulting in the properties being
ignored by the api server. The preserve unknown fields tag was
not working recursively on the volumeClaimTemplates since the subtypes
were defined. Now we post-process the schema so we can preserve
the unknown fields for the volume claim templates metadata.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
We don't want to include those CRDs anymore, as they get imported from
another operator.
This trims further down the size of our CSV generation.
Additionally, we always clean the csv directory before proceeding.
Signed-off-by: Sébastien Han <seb@redhat.com>
The CSV file size cannot be bigger than 1MB so let's remove all the
description fields make it lighter.
With that change the CRDs size went from 692K to 372K so the size was
reduced by almost 50%.
Signed-off-by: Sébastien Han <seb@redhat.com>
On mac the make crds command was failing on the sed commands.
Now we remove the dependency on sed and just append the files
to generate the helm resources
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Let's not use any controllen-gen version we might find on the system,
always pull the version we need.
Same goes for yq.
Signed-off-by: Sébastien Han <seb@redhat.com>
The helm chart was not installing the CRDs on pre-1.16 k8s due
to a regression in the conditionals for the chart that was introduced
with 7103.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Finally! We can now stop editing manually our crds definition. Simply
run `make crds-gen`. These two files:
* `cluster/examples/kubernetes/ceph/crds.yaml`
* `cluster/charts/rook-ceph/templates/resources.yaml`
will be autogenerated for us.
We rely on the controller-gen tool, it reads our API definitions from
`pkg/apis/` and produces the CRD files accordingly.
It uses "markers" to add extra yaml fields to the CRD, for instance we
have a lot fields with:
```
nullable: true
x-kubernetes-preserve-unknown-fields: true
```
The corresponding API type markers are:
```
// +kubebuilder:pruning:PreserveUnknownFields
// +nullable
```
For more API convention see
https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#optional-vs-required
and for the markers see: https://book.kubebuilder.io/reference/markers/crd-validation.html
Signed-off-by: Sébastien Han <seb@redhat.com>
Erik Sundell
Allen Conlon
subhamkrai
Redouane KachachandBlaine Gardner
travisn
parth-gr
Javier
Khris Richardson
yati1998
Sébastien Han
Satoru Takeuchi