Using CMake's `string(JSON` to parse JSON array leads to quadratic
running time in number of tests, see https://gitlab.kitware.com/cmake/cmake/-/work_items/27985
This leads to _terrible_ runtime for `catch_discover_tests` when called
on binaries with lot of tests (1k+). To get reasonable runtimes, we have
to avoid using `string(JSON` to parse out the individual test objects
from the array with all tests.
This commit replaces the sane approach of using real JSON parser with
a set of terrible hacks, where we use CMake's string APIs to split the
JSON array on what looks like object boundary (`}<ws>*,<ws>*{`), and then
checking whether the resulting thing can be parsed as JSON object. If not,
we append the next piece and check again. And again, and again, until we
get a proper JSON object.
This is all around a hilariously terrible idea, however:
1) It works in practice for all tested inputs.
2) It improves the time it takes to run `catch_discover_tests` on binary
with 1k tests from 4.2s to 1.1s and 2k tests from 16s to 3.9s.
The previous approach was for `add_command` to behave as append function
via concatenating the command string internally and then saving it into
`PARENT_SCOPE`. Because this in practice ended up meaning concatenating
a copy of the string inside the function and then overwriting the string
outside the function, the performance was lacking.
The new approach is for `prepare_command` to only escape & return the
command from single call, and the caller is responsible for concatenating
the result. Since the actual concatenation no longer crosses function
scope boundaries, the performance is much better, even though the scaling
is still quadratic.
The new approach only takes 1/4-1/5 of the time, saving ~1s at 1k tests,
4.5s at 2k tests and 19s at 4k tests.
Not being able to filter generators to specific element has been regularly
causing problems. It was possible to use a dynamic section to run tests
for specific element in a generator, at least if the element had a nice
string representation, but the test case would still run once per element
in the generator.
With this change, it is possible to have the generator return only one
specific element, and do so based on the index, rather than the string
representation of the element. This enables simple debugging of tests
that fail for specific generator element.
Specifically, this commit makes the `-c`/`--section` parameter
strictly ordered and hierarchical, unlike how it behaved before,
which was a huge mess -- see #3038 for details.
Closes#3038
This tells Catch2 to create an empty file at specified path before
the tests start, and delete it after the tests finish. This allows
callers to catch cases where the test binary silently exits before
finishing (e.g. via call to `exit(0)` inside the code under test),
by looking whether the file still exists.
Closes#3020
We still want to build VS 2017 through AppVeyor, and those images
have CMake 3.16.2 installed. We could install newer CMake as part
of the build, but since we don't use newer CMake features yet, this
is simpler.
The main reason for this is to be able to distinguish between
different errors (or "errors") based on the return code. Before
this change, it was impossible to use the exit code to figure out
whether a test binary failed because all tests were skipped or
because exactly 4 assertions have failed.
This meant that using `catch_discover_tests` and telling it to
check for exit code == 4 to determine skipped tests could lead to
false negatives.
Previously it could be just plain reporter name, e.g. `xml`, but
it could not specify other reporter options. This change is not
particularly useful for the built-in reporters, as it mostly comes
in handy for combining specific custom reporter with custom arguments,
and the built-in reporters do not have those.
This changes the compact reporter's summary of test run totals to use
the same format as the console reporter. This means that while output is
no longer on a single line (two instead), it now includes totals for
`failedButOk` test cases and assertions, which were previously missing.
Not all reporters use a format that supports this, so TeamCity
and Automake reporters still do not report it. The console
reporter now reports it even on successful runs, where before
it only reported the rng seed in the header, which was showed
either for failed run, or for run with `-s`.
CLoses#2065
When the added Bazel configuration flag is enabled,
a default JUnit reporter will be added if the XML
envrioment variable is defined.
Fix include paths for generated config header.
Enable Bazel config by default when building with
Bazel.
Co-authored-by: Martin Hořeňovský <martin.horenovsky@gmail.com>
This greatly simplifies running Catch2 tests in single binary
in parallel from external test runners. Instead of having to
shard the tests by tags/test names, an external test runner
can now just ask for test shard 2 (out of X), and execute that
in single process, without having to know what tests are actually
in the shard.
Note that sharding also applies to test listing, and happens after
tests were ordered according to the `--order` feature.
This means that e.g. for `TEST_CASE` with two sibling `SECTION`s
the event will fire twice, because the `TEST_CASE` will be entered
twice.
Closes#2107 (the event mentioned there already exists, but this
is its counterpart that we also want to provide to users)
A test runner already has a --durations option to print durations.
However, this isn't entirely satisfactory.
When there are many tests, this produces output spam which makes it hard
to find the test failure output. Nevertheless, it is helpful to be
informed of tests which are unusually slow.
Therefore, introduce a new option --min-duration that causes all
durations above a certain threshold to be printed. This allows slow
tests to be visible without mentioning every test.
Martin Hořeňovský
Eisuke Kawashima
Chris Thrasher
Cristian Le
Robin Christ
Lars Toenning
Philip Salzmann
Lukasz OkraszewskiandMartin Hořeňovský
Ben Dunkin
John Bytheway