Third Copilot review round:
- Makefile.am: run the test-data stamp recipe body under set -e. A
failed symlink mid-loop previously did not fail the compound command
(only the last command's status counted), so a partially-populated
build tree could be stamped complete. Now any failed setup command
aborts the recipe and the stamp is not created.
- parallel-make-check.py: fail-fast sent SIGTERM only, so a test that
traps or ignores SIGTERM could keep the job alive until the workflow
timeout. abort_others() now polls the swept processes and SIGKILLs
whatever is still alive after a 10 s grace period, and the
post-registration race-window kill escalates the same way (bounded
wait, then SIGKILL). Verified with a config running
"trap '' TERM; sleep 300": the run completes in ~10 s with the
stubborn config reported as aborted and no surviving processes.
Address the Copilot review:
- parallel-make-check.py: validate "configure" (list of strings) and
cflags/ldflags (strings) so a malformed entry fails the load instead
of exploding a string into per-character configure arguments; print
a single line for passing configs instead of dumping their full
make-check.log into the CI log (failure dumps unchanged; the logs
remain in build-<name>/ for the failure artifacts).
- Makefile.am: use rm -rf for the certs/input/quit setup and distclean
cleanup. A --private-dir run replaces the certs symlink with a
private directory copy that rm -f cannot remove (verified: make
distclean in a build dir with a privatized certs/ now succeeds and
removes it).
- psk.yml, disable-pk-algs.yml: normalize the single-dash tokens
(-disable-rsa, -disable-ecc, -disable-aescbc, -enable-cryptonly)
carried verbatim from the old matrices to the canonical double-dash
form. No coverage change: configure honors single-dash spellings
(verified -disable-rsa sets NO_RSA with no unrecognized-option
warning), so these were always in effect; both touched configs
re-validated end-to-end.
The --cc default stays "ccache gcc": ccache resolves the compiler
through its own masquerade symlinks (verified: no recursion and normal
cache hits with /usr/lib/ccache prepended to PATH), and the explicit
CC= also covers jobs that use ccache without the PATH masquerade.
Replace the one-runner-per-configuration matrices across the
make-check workflow family with a generic pooled runner,
.github/scripts/parallel-make-check.py. Each workflow keeps its
configuration list as JSON next to the invocation; one runner (or a
small fixed set of shards, balanced by measured per-config minutes)
builds every config in its own out-of-tree (VPATH) build directory off
a single checkout/autogen, on a pool of one-per-CPU worker threads,
longest first. Concurrent checks are isolated with bubblewrap network
namespaces, compilations are cached with ccache, the first failure
aborts the rest (fail-fast, with --no-fail-fast to run everything),
and per-config timings plus pool efficiency land in the step summary.
Failure logs upload as artifacts. smoke-test.yml is likewise reworked
into a single pooled job that runs its nine configs on one runner.
Converted workflows (runner jobs per full pass):
os-check.yml 101 -> 8 (92 Ubuntu configs -> 4 shards;
the macOS matrix, the user-settings jobs and
the standalone
macos-apple-native-cert-validation.yml fold
into one macOS runner; Windows unchanged)
pq-all.yml 21 -> 2 shards
disable-pk-algs.yml 15 -> 1
wolfCrypt-Wconversion.yml 11 -> 1
trackmemory.yml 7 -> 1
cryptocb-only.yml 8 -> 1 (incl. the two new SHA512 entries)
multi-compiler.yml 6 -> 1
smallStackSize.yml 6 -> 1
multi-arch.yml 6 -> 1
async.yml 5 -> 1
psk.yml 5 -> 1
no-malloc.yml 3 -> 1
wolfsm.yml 3 -> 1
opensslcoexist.yml 2 -> 1
Measured against current upstream passing runs (job execution time,
queue excluded): ~200 runner jobs / ~374 runner-minutes per full pass
become 23 jobs / ~168 runner-minutes, with more coverage than before.
multi-arch's old matrix combined an "include" list of four
architectures with an "opts" axis; GitHub's include-merge rules made
each arch entry overwrite the previous one, so only the armel
combinations actually ran. The pooled list restores the intended
aarch64/armhf/riscv64 coverage (23 combinations; riscv64 x sp-math is
omitted as invalid - configure rejects sp-math without SP, and
--enable-riscv-asm, unlike --enable-sp-asm, does not bring SP in).
Out-of-tree build fixes this depends on:
- Makefile.am: symlink the read-only test data (certs/, tests/ config
files, sniffer captures and helpers, examples/crypto_policies,
input, quit) into the build tree via a BUILT_SOURCES stamp, removed
again in distclean-local. ChangeToWolfRoot() and the script tests
resolve everything relative to the working directory, so out-of-tree
make check and make distcheck now pass.
- scripts/multi-msg-record.py: locate the client binary from the build
tree working directory rather than the script's source directory.
- configure.ac + wolfssl/include.am: run
support/gen-debug-trace-error-codes.sh from $srcdir; it reads the
error-code headers from the source tree and generates into the build
tree.
- tests/swdev: a WOLFBUILD variable points the sub-make at the build
tree for the configure-generated headers (wolfssl/options.h,
wolfssl/version.h); the in-tree-only guards are dropped.
Portions of PR #10649 are incorporated: the cross-platform
ccache-setup composite action, repository_owner gates on check-headers
and check-source-text, the docs-only paths-ignore on os-check, and the
libspdm timeout bumps.
linuxkm/Makefile:
* don't use `readarray -d` -- it's a recent bashism;
* rework libwolfssl-user-build/src/.libs/libwolfssl.so recipe to better isolate sub-build settings.
* add support for HOSTCC and HOSTCFLAGS in libwolfssl.so build.
* deploy $(QFLAG) --no-print-directory --no-silent in several submakes for neatness and resilience.
* tweak $(LIBWOLFSSL_NAME).ko.signed recipe to add a "skipping" message and some consistency checking.
linuxkm/README.md: update FIPS DRBG /proc/crypto content to show seed source.
linuxkm/linuxkm_memory.c: fixes for format character portability in a RELOC_DEBUG_PRINTF() in wc_reloc_normalize_text).
linuxkm/linuxkm_wc_port.h: pull in linux/moduleparam.h, and if WC_LINUXKM_SUPPORT_DUMP_TO_FILE, pull in linux/fs.h and linux/uaccess.h.
linuxkm/module_hooks.c: implement WC_LINUXKM_SUPPORT_DUMP_TO_FILE: dump_to_file() and module args text_dump_path=... and rodata_dump_path=...
linuxkm/patches/7.0/WOLFSSL_LINUXKM_HAVE_GET_RANDOM_CALLBACKS-7v0.patch: add to accommodate patch-breaking change in Linux 7dff99b354.
Add:
* linuxkm/linuxkm-fips-hash.c
* linuxkm/linuxkm-fips-hash-wrapper.sh
* linuxkm/linuxkm_memory.h
Move from linuxkm/module_hooks.c to linuxkm/linuxkm_memory.c:
* reloc_layouts[]
* find_reloc_tab_offset()
* the body of wc_linuxkm_normalize_relocations() as wc_reloc_normalize_text()
* most of updateFipsHash() as wc_fips_generate_hash()
Move from linuxkm/linuxkm_wc_port.h to linuxkm/linuxkm_memory.h:
* struct wc_linuxkm_pie_reloc_tab_ent
* enum wc_reloc_dest_segment
* enum wc_reloc_type
linuxkm/Makefile:
* Update GENERATE_RELOC_TAB recipe to populate new fields in struct wc_reloc_table_ent.
* Add targets:
* libwolfssl-user-build/src/.libs/libwolfssl.so
* linuxkm-fips-hash
* module-with-matching-fips-hash
* module-with-matching-fips-hash-no-sign
* Add support for alternate target module name, via LIBWOLFSSL_NAME make variable.
linuxkm/linuxkm_wc_port.h and linuxkm/module_hooks.c:
* Fixes to make linuxkm-pie work with CONFIG_KASAN.
* Implement WC_LINUXKM_STACK_DEBUG:
* wc_linuxkm_stack_bottom()
* wc_linuxkm_stack_top()
* wc_linuxkm_stack_current()
* wc_linuxkm_stack_left()
* wc_linuxkm_stack_hwm_prepare()
* wc_linuxkm_stack_hwm_measure_rel()
* wc_linuxkm_stack_hwm_measure_total()
wolfssl/wolfcrypt/settings.h:
* When WOLFSSL_KERNEL_MODE, make sure WOLFSSL_GENERAL_ALIGNMENT is at least SIZEOF_LONG.
* When WOLFCRYPT_FIPS_CORE_DYNAMIC_HASH_VALUE, make sure WOLFSSL_BASE16 is defined.
configure.ac and wolfcrypt/benchmark/benchmark.c: Disable scrypt when KERNEL_MODE_DEFAULTS, due to excessive memory requirements.
* Minor fixes to the CMakeLists.txt
* Add more options to the CMake infrastructure already present in the
autoconf infrastructure
* An autoconf build now also generates and installs files required to
consume the installed wolfssl library via CMake.
* Added test for autoconf-CMake interworking
Work is mostly done by Codex and Curser.
* add module-update-fips-hash rule, for in-place FIPS hash update without rebuild;
* improve PIE sequence in module build rule to double-check stability of the relocation table after final rebuild;
Makefile.am: add a module-update-fips-hash passthrough target.
wolfssl/wolfcrypt/settings.h: in WOLFSSL_LINUXKM section, #ifdef LINUXKM_LKCAPI_REGISTER, #define WOLFSSL_TEST_SUBROUTINE to nothing, and #define WC_TEST_EXPORT_SUBTESTS.
linuxkm/lkcapi_glue.c:
* add check_skcipher_driver_masking() and check_aead_driver_masking(),
* use _masking() checks in all linuxkm_test_*().
* add !WOLFSSL_AESGCM_STREAM implementation of linuxkm_test_aesgcm().
* add implementations of linuxkm_test_aesctr(), linuxkm_test_aesofb(), and linuxkm_test_aesecb()
* remove incomplete+disabled AES-CCM shim implementation.
linuxkm/module_hooks.c: pull in wolfcrypt/test/test.h if LINUXKM_LKCAPI_REGISTER.
linuxkm/Makefile: build wolfcrypt/test/test.o if ENABLED_LINUXKM_LKCAPI_REGISTER.
Makefile.am: add ENABLED_LINUXKM_LKCAPI_REGISTER to exports in BUILD_LINUXKM section.
configure.ac: add AC_SUBST([ENABLED_LINUXKM_LKCAPI_REGISTER]); in ENABLED_LINUXKM_DEFAULTS set up, remove `-DWOLFSSL_TEST_SUBROUTINE=static` from AM_CFLAGS adds; fix whitespace.
.wolfssl_known_macro_extras: add WC_WANT_FLAG_DONT_USE_AESNI.
wolfcrypt/test/test.c: add `|| defined(WC_TEST_EXPORT_SUBTESTS)` to outermost gate, add wc_test_ prefix to render_error_message() and export it,
wolfcrypt/test/test.h: add prototype for wc_test_render_error_message(), and #ifdef WC_TEST_EXPORT_SUBTESTS, add prototypes for all the subtests.
also, additional backward-compatibility measures around cp and clean recipe in linuxkm/Makefile.
also, in sp_int.c, tweak DECL_DYN_SP_INT_ARRAY() to use an explicit XMEMSET() to clear n[], to avoid unshimmable implicit memset() from gcc on aarch64.
* Redirect the AesEncrypt_C call to device
* Fix function declarations
* Force CC=nvcc with CUDA
* Don't let C++ mangle function names
* Add larger parallelization
* Add in memory copy to device
* `nvcc` does not support '-Wall' nor '-Wno-unused'
* Add in README.md
* Clean up script to output color coded data
* Fix Asymmetric cipher comparisons
* Add in standard output parsing in addition to the CSV
* Add option to output results in a CSV
---------
Co-authored-by: Andras Fekete <andras@wolfssl.com>
add support for EXTRA_CPPFLAGS, EXTRA_CCASFLAGS, and EXTRA_LDFLAGS;
fix typo in setup for CFLAG_VISIBILITY;
lightly refactor handling of CPPFLAGS/AM_CPPFLAGS in handlers for --with-liboqs, --with-wnr, and --with-cavium;
refactor+enhance options.h generation to handle -U directives.
* First test
* Don't forget to run autogen.sh!
* Add tools needed by automake
* Try additional platforms
* Add in qemu for other platforms
* No real support for arm containers
* Fix indentation
* Simplify container build with a testing script
* Simpler names for actions
* No need to distribute OpenWRT test files
* Better list to put ignore files onto
* Create an 'ignore_files' list after all
* Add in some documentation of how OpenWRT tests work
* Fix up naming of OpenWrt
Co-authored-by: Andras Fekete <andras@wolfssl.com>
Makefile.am: clean .build_params file;
ecc.c: fix misplaced gat #endif in wc_ecc_shared_secret_gen_sync();
move AM_CFLAGS+=-include /.build_params to before AC_SUBST([]AM_CFLAGS);
fix new unused-label defect in wc_ecc_shared_secret_gen_sync();
fix integer.[ch] mp_exch() to return int not void (sp_exch() and TFM mp_exch() can both fail on allocations);
fix NO_INLINE ForceZero() prototype;
ecc.c: add missing if (err == MP_OKAY) in build_lut();
wolfcrypt/test/test.c: revert "rename hkdf_test to wc_hkdf_test to eliminate namespace collision", restoring unconditional static qualifier, to fix crash at return from main() on Xilinx Zynq ARM test;
ecc.c: refactor build_lut() flow control to fix uninited variable scenario found by scan-build;
WOLFCRYPT_ONLY and OPENSSL_EXTRA: fix gating to allow successful build with --enable-all-crypto, and add configure error if crypt-only and opensslall are combined.
refactor the HAVE_WC_INTROSPECTION mechanism to pass build params via $output_objdir/.build_params rather than abusing autotools config.h to pass them;
add support for EXTRA_CFLAGS on the make command line;
in FIPS builds, exclude pkcallbacks from --enable-all;
linuxkm: move test.o out of PIE container (uses function pointers as operands).
* Remove NTRU and OQS
* Keep the DTLS serialization format backwards compatible.
* Remove n from mygetopt_long() call.
* Fix over-zealous deletion.
* Resolve problems found by @SparkiDev
* Adds `--enable-linuxkm-pie` and associated infrastructure, to support FIPS mode in the Linux kernel module.
* Adds `tests/api.c` missing (void) arglist to `test_SSL_CIPHER_get_xxx()`.
This the second pass at this after seeing how fips is added to tree in later phases.
This allow autoreconf to be directly called which allows the Makefile to rebuild when seeing that changes have been ( having an autogen.sh is older convention which left to history in the way autotools are invoked )
This fixes "make distcheck" and "make maintainer-clean" which are required by most distributions packaging systems.
The files previously touched by autogen.sh are now properly placed into autoconf.
The include files files are generated by configure. ( There is a note placed in configure.ac as to why and reference to the automake documention for this ). Append to file was done on purpose, touch cannot be in configure safetly. Normally autoheader would be used for this but since the include files are created out of tree, care has to be taken to not overwrite those file.
For the source files, they were moved into the coresponding automake file. It is safe to use touch in automake. Since files can optionally copied from elsewhere, they have to be listed in BUILT_SOURCES. They are written srcdir in order to allow make to do VPATH builds ( which is configure by make distcheck ).
To show fips files are preserved without having the actual fips files, a C style comment can be echoed into the files.
There are a few current, but outstanding issues.
1) config.h needs to be fixed configure.ac to use autoheader in order to allow configure to know to rebuilt depencies on its changes. ( Out of scope for this patch. )
2) verion.h checked into the tree and it is a built file. A make maintainer-clean followed by "git status --ignored" will confirm this. ( Out of scope for this patch )
3) autogen.sh has not been updated to reflect fixes. I believe that for this patch, it should be left alone and checked for regression in Jenkins by itself.
4) There is an out of date .spec file for building RPM which should be updated now that distcheck is working.
5) maintainer-clean should have rule added to remove build-aux testdriver.
This has been tested on current Ubuntu testing, OSX, Fedora 34, and Debian 10.
Additionaly "make distcheck" should be added to regression testing, along with "make maintainer-check".
Other improvement possibilities:
A possible future improvement is to let autoconf handle build with optional out of dist files.
Modify fips configure.ac check to allow for an injection of comments into blank fips files in order to prove distribution of fips/non-fips builds.
Update git rules to use 'make maintainer-clean', 'autoreconf -if', 'make distcheck'.
Need to check if `unit.test` was run from make process and set
different path to run unit test executable.
Writing files in the dist is not allowed during distcheck so write
files to subdirectory used build during distmake
- Begin adding options to enable/disable different features.
- Increase minimum CMake version to 3.2.
- Support installation of the built files.
- Add checks for necessary include files, functions etc.
- Generate options.h and config.h.
- Use GNUInstallDirs to support installation, which is designed to be somewhat
cross-platform.
- Export wolfssl CMake target during installation, so others using CMake can
link against wolfssl easily.
- Disallow in-source builds.
- Place the generation of BUILD_* flags (controlled with AM_CONDITIONALs
in configure.ac) in a separate function in functions.cmake,
generate_build_flags.
- Implement the logic to conditionally add source files from
src/include.am in a function in functions.cmake, generate_lib_src_list.
- Exclude tls_bench from Windows. Doesn't compile with MSVC. WIP.
- Update INSTALL with latest CMake build instructions.
- Add a cmake/include.am to ensure CMake files get added to the distribution.
Juliusz Sosinowicz
Sean Parkinson
David Garske
HIDEKI MIYAZAKI
Daniel Pouzzner
jordan
Tobias Frauenschläger
gojimmypi
András Fekete
Eric Blankenhorn
Lealem Amedie
Andras Fekete
John Bland
John Bland
Chris Conlon
Anthony Hu
Maxime Vincent
TakayukiMatsuo
Brian Aker
Elms
Hayden Roche