Using CMake's `string(JSON` to parse JSON array leads to quadratic
running time in number of tests, see https://gitlab.kitware.com/cmake/cmake/-/work_items/27985
This leads to _terrible_ runtime for `catch_discover_tests` when called
on binaries with lot of tests (1k+). To get reasonable runtimes, we have
to avoid using `string(JSON` to parse out the individual test objects
from the array with all tests.
This commit replaces the sane approach of using real JSON parser with
a set of terrible hacks, where we use CMake's string APIs to split the
JSON array on what looks like object boundary (`}<ws>*,<ws>*{`), and then
checking whether the resulting thing can be parsed as JSON object. If not,
we append the next piece and check again. And again, and again, until we
get a proper JSON object.
This is all around a hilariously terrible idea, however:
1) It works in practice for all tested inputs.
2) It improves the time it takes to run `catch_discover_tests` on binary
with 1k tests from 4.2s to 1.1s and 2k tests from 16s to 3.9s.
The previous approach was for `add_command` to behave as append function
via concatenating the command string internally and then saving it into
`PARENT_SCOPE`. Because this in practice ended up meaning concatenating
a copy of the string inside the function and then overwriting the string
outside the function, the performance was lacking.
The new approach is for `prepare_command` to only escape & return the
command from single call, and the caller is responsible for concatenating
the result. Since the actual concatenation no longer crosses function
scope boundaries, the performance is much better, even though the scaling
is still quadratic.
The new approach only takes 1/4-1/5 of the time, saving ~1s at 1k tests,
4.5s at 2k tests and 19s at 4k tests.
To make this all work, I had to remove the stringification cache
from matchers. In theory, this can cause performance penalty in
cases where single matcher instance is stringified multiple times,
but in practice this does not happen much, and the difference is
surprisingly small anyway, because the performance of stringification
is already horrible and full of allocating strings just to throw
them away.
The matcher combinators need P2738 from C++26 to be `constexpr`.
Closes#3091
Not being able to filter generators to specific element has been regularly
causing problems. It was possible to use a dynamic section to run tests
for specific element in a generator, at least if the element had a nice
string representation, but the test case would still run once per element
in the generator.
With this change, it is possible to have the generator return only one
specific element, and do so based on the index, rather than the string
representation of the element. This enables simple debugging of tests
that fail for specific generator element.
Specifically, this commit makes the `-c`/`--section` parameter
strictly ordered and hierarchical, unlike how it behaved before,
which was a huge mess -- see #3038 for details.
Closes#3038
Adding unreachable to `FAIL` and `SKIP` made Clang become very good
at figuring out that tests of `FAIL` and `SKIP` will never return
and thus could be marked as [[noreturn]].
To avoid introducing lot of warning suppression noise into the
test files, we just disable it instead.
* Utility for extended mult n x n bits -> 2n bits
* Utility to adapt output from URBG to target (unsigned) integral
type
* Utility to reorder signed values into unsigned type while keeping
the order.
* AssertionEnd does not reset the assertion info yet. That is done after populateReaction. And reset assertion info would also reset the result disposition to normal, so that any uncaught exception would be reported as failure
* Approving test output changes due to added unit tests
* Unit tests to throw std::runtime_error instead of std::exception
* Add a unit test to test incomplete assertion handler
---------
Co-authored-by: Ross <ross.tang@gfo-x.com>
Also split out helpers for testing matcher ranges (types whose
begin/end/empty/etc require ADL lookup, types whose iteration
uses iterator + sentinel pair, etc) into their own file.
* Add new SKIP macro for skipping tests at runtime
This adds a new `SKIP` macro for dynamically skipping tests at runtime.
The "skipped" status of a test case is treated as a first-class citizen,
like "succeeded" or "failed", and is reported with a new color on the
console.
* Don't show "skipped assertions" in console/compact reporters
Also extend skip tests to cover a few more use cases.
* Return exit code 4 if all test cases are skipped
* Use LightGrey for the skip colour
This isn't great, but is better than the deep blue that was borderline
invisible on dark backgrounds. The fix is to redo the colouring
a bit, including introducing light-blue that is actually visible.
* Add support for explicit skips in all reporters
* --allow-running-no-tests also allows all tests to be skipped
* Add docs for SKIP macro, deprecate IEventListener::skipTest
Co-authored-by: Martin Hořeňovský <martin.horenovsky@gmail.com>
This has two effects:
1) This significantly improves the performance of ApprovalTests
run in WSL, because running the massively multi-reporter
test would cause lots of small and parallel writes across the
WSL-Windows FS boundary, completely mudering performance.
2) ApprovalTests can be run from multiple build dirs in parallel,
as long as they do not fail. However, if they do fail,
the multiple runs will still step on each other toes when
writing the unapproved files for user.
This is primarily done to support new `std::*_ordering` types,
but the refactoring also supports any other type with this
property.
The compilation overhead is surprisingly low. Testing it with
clang on a Linux machine, compiling our SelfTest project takes
only 2-3% longer with these changes than it takes otherwise.
Closes#2555
They also got slapped with the `[approvals]` tag in the process,
because we have too many approval tests and want less of them,
and these particular tests don't bring much value.
Related to #2090
This changes the compact reporter's summary of test run totals to use
the same format as the console reporter. This means that while output is
no longer on a single line (two instead), it now includes totals for
`failedButOk` test cases and assertions, which were previously missing.
There is an increasing number of places where Catch2 wants to parse
strings into numbers, but being stuck in C++14 world, we do not
have good stdlib facilities to do this (`strtoul` and `stoul`
are both bad).
Not all reporters use a format that supports this, so TeamCity
and Automake reporters still do not report it. The console
reporter now reports it even on successful runs, where before
it only reported the rng seed in the header, which was showed
either for failed run, or for run with `-s`.
CLoses#2065
This means that the CLI interface now uses the new key-value oriented
reporter spec, the common reporter base creates the colour implementation
based on the reporter-specific configuration, and it also stores the
custom configuration options for each reporter instance.
Closes#339 as it allows per-reporter forcing of ansi colour codes.
Test case hashing includes tags and class name
As the hasher involves more code now, it was split out into its own file
and it got its own set of tests.
Closes#2304
This includes always compiling the ANSI and None colour
implementations, as they don't need to touch any platform
specific APIs, and removing their respective compile-time
configuration options.
Because the Win32 colour implementation requires Win32-specific
APIs, it is still hidden behind a compile-time toggle,
`CATCH_CONFIG_COLOUR_WIN32` (renamed from `..._COLOUR_WINDOWS`).
The commandline options for colours were also changed. The
option now uses different name, and allows to select between
different implementations, rather than changing whether
the compiled-in colour implementation is used through
"yes/no/default" options.