Using CMake's `string(JSON` to parse JSON array leads to quadratic
running time in number of tests, see https://gitlab.kitware.com/cmake/cmake/-/work_items/27985
This leads to _terrible_ runtime for `catch_discover_tests` when called
on binaries with lot of tests (1k+). To get reasonable runtimes, we have
to avoid using `string(JSON` to parse out the individual test objects
from the array with all tests.
This commit replaces the sane approach of using real JSON parser with
a set of terrible hacks, where we use CMake's string APIs to split the
JSON array on what looks like object boundary (`}<ws>*,<ws>*{`), and then
checking whether the resulting thing can be parsed as JSON object. If not,
we append the next piece and check again. And again, and again, until we
get a proper JSON object.
This is all around a hilariously terrible idea, however:
1) It works in practice for all tested inputs.
2) It improves the time it takes to run `catch_discover_tests` on binary
with 1k tests from 4.2s to 1.1s and 2k tests from 16s to 3.9s.
Repeatedly calling `string(APPEND` on the same destination string
leads to quadratic running time. We avoid this by flushing the CTest
script into a file periodically, currently every 50kB of text.
This can cause _slight_ slowdown just around the flush boundary (each
file write is quite expensive, so if we flush just before the last test
is written, it hurts), but it avoids terrible performance for large
test suites.
The performance is roughly equal at 500 tests to discover, and clearly
wins at more; for 1/2/4/8 k tests, the improvements are 0.3/1.6/6.4/27 s.
The previous approach was for `add_command` to behave as append function
via concatenating the command string internally and then saving it into
`PARENT_SCOPE`. Because this in practice ended up meaning concatenating
a copy of the string inside the function and then overwriting the string
outside the function, the performance was lacking.
The new approach is for `prepare_command` to only escape & return the
command from single call, and the caller is responsible for concatenating
the result. Since the actual concatenation no longer crosses function
scope boundaries, the performance is much better, even though the scaling
is still quadratic.
The new approach only takes 1/4-1/5 of the time, saving ~1s at 1k tests,
4.5s at 2k tests and 19s at 4k tests.
When using catch_discover_tests() with DISCOVERY_MODE PRE_TEST and a
multi-config generator (e.g. Ninja Multi-Config), if a test target has
zero discoverable tests (e.g. all tests tagged with [.]), ctest fails:
CMake Error: include could not find requested file:
.../test-hidden-b12d07c_tests-Release.cmake
The early return added in #2962 (76f70b14) correctly prevented a JSON
parsing crash for zero tests, but skipped writing the ctest file. The
PRE_TEST include script unconditionally includes this file, so the
missing file causes a hard error that aborts all test discovery.
Write an empty file before returning early so the include always
succeeds.
Classes will automatically inherit the virtual-ness of their base
class destructors. If the base class already has a virtual
destructor and the derived class needs default destructor semantics
then the derived class can omit defining the destructor in favor of
the compiler automatically defining it.
This has an additional benefit of reenabling move semantics. The
presence of a user-specified destructor automatically disables move
operations.
Introducing a new DISCOVERY_MODE mode option, which provides greater
control over when catch_discover_tests perforsm test discovery.
It has two supported modes:
* POST_BUILD: The default behavior, which adds a POST_BUILD command
to perform test discovery after the test has been built as was
always done so far.
* PRE_TEST: New mode, which delays test discovery until test execution.
The generated include file generates the appropriate CTest files at
runtime and regenerates the CTest files when the executable is
updated.
This mode can be used in build-environments that don't allow for
executing the linked binaries at build-time (like in a
cross-compilation environment).
DISCOVERY_MODE can be controlled in two ways:
1. Setting the DISCOVERY_MODE when calling catch_discover_tests.
2. Setting the global CMAKE_CATCH_DISCOVER_TESTS_DISCOVERY_MODE prior
to calling gtest_discover_tests.
Closes#2493
Move test discovery logic into new catch_discover_tests_impl method
and make CatchAddTests aware of whether it is being launched in
CMake's script mode.
When launched in script mode, catch_discover_tests_impl is called
passing arguments obtained from the definitions passed into the call
to cmake. This preserves the existing behavior assumed by Catch.cmake.
Looking ahead, it also allows CatchAddTests to be included in
generated files and call catch_discover_tests_impl to perform test
discovery at test runtime with the new PRE_TEST discovery mode
introduced later.
Martin Hořeňovský
Phil Nash
Manuel Knörle
Nikolay Baklicharov
Sung, Po Han
Chris Thrasher
Holger Kaelberer
Chien-Yu Lin
Andy Phillips
Cristian Le
Holger KaelbererandHolger Kaelberer
Ilya Arzhannikov
Robin Christ
Arne Mertz
Bob Miller