There were 2 separate bugs, one recently added and one ancient
(since roughly the first version of the registration script).
1) The recent refactoring of how the JSON output from Catch2 is
parsed caused shadowing between the variable containing the
per-test JSON fragments and the accumulator of test names for
<target>_TESTS variable. This led to the variable containing
both the names and the JSON fragments, and thus being completely
wrong.
2) The test name accumulation has never accounted for characters
that need escaping to be present in a CMake list. This means
that e.g. test names with semicolon in them would end up with
two elements in the test list.
Both of these are now fixed, at the cost of extra complexity and
my sanity as I had to learn more about CMake escaping rules.
(CMake escaping rules are dumb)
* Quiet verbosity provides just the test names
* Normal verbosity adds tags
* High verbosity add the source location of the tests
Listing tests with the quiet verbosity results in about 1/4 of the
previous output, which leads to measurably faster execution of
`catch_discover_tests` when it does not need tags for labels.
(If it does need tags, the output is about 1/2 of previous).
Using CMake's `string(JSON` to parse JSON array leads to quadratic
running time in number of tests, see https://gitlab.kitware.com/cmake/cmake/-/work_items/27985
This leads to _terrible_ runtime for `catch_discover_tests` when called
on binaries with lot of tests (1k+). To get reasonable runtimes, we have
to avoid using `string(JSON` to parse out the individual test objects
from the array with all tests.
This commit replaces the sane approach of using real JSON parser with
a set of terrible hacks, where we use CMake's string APIs to split the
JSON array on what looks like object boundary (`}<ws>*,<ws>*{`), and then
checking whether the resulting thing can be parsed as JSON object. If not,
we append the next piece and check again. And again, and again, until we
get a proper JSON object.
This is all around a hilariously terrible idea, however:
1) It works in practice for all tested inputs.
2) It improves the time it takes to run `catch_discover_tests` on binary
with 1k tests from 4.2s to 1.1s and 2k tests from 16s to 3.9s.
The previous approach was for `add_command` to behave as append function
via concatenating the command string internally and then saving it into
`PARENT_SCOPE`. Because this in practice ended up meaning concatenating
a copy of the string inside the function and then overwriting the string
outside the function, the performance was lacking.
The new approach is for `prepare_command` to only escape & return the
command from single call, and the caller is responsible for concatenating
the result. Since the actual concatenation no longer crosses function
scope boundaries, the performance is much better, even though the scaling
is still quadratic.
The new approach only takes 1/4-1/5 of the time, saving ~1s at 1k tests,
4.5s at 2k tests and 19s at 4k tests.
As the types and values sent into the writer are determined by Catch2,
I do not expect to actually need this support, but it is better to have
it and not be surprised in the future.
The forward iteration logic already bounds-check for m_it != m_string->end(),
do the same for the backward iteration. The issue with the assert is
that the assert() might not be compiled in, and it is happening after
the dereference, so it was too late.
To make this all work, I had to remove the stringification cache
from matchers. In theory, this can cause performance penalty in
cases where single matcher instance is stringified multiple times,
but in practice this does not happen much, and the difference is
surprisingly small anyway, because the performance of stringification
is already horrible and full of allocating strings just to throw
them away.
The matcher combinators need P2738 from C++26 to be `constexpr`.
Closes#3091
The old internals reached into the global hub to stash the allocation(s)
for enum value -> string value there, then kept around a potentially
invalid (in case the hub was cleaned up) reference into it, going through
whole bunch of virtual dispatch in the process.
The new internals just store the data in a static variable inside the
`StringMaker` specialization. This avoids potential lifetime issues,
avoids all virtual dispatch and (almost) reduces the include bloat in
the main header path.
The reason for (almost) there is that for full include correctness,
`EnumInfo` needs `<utility>` include for `std::pair`. However, this brings
in things like `std::relops`, because the std headers in C++ are dumb.
As this was not included before, and instead we relied on `std::pair`
existing in an internal stdlib header that we transitively included, the
full include size ends up bigger than before.
For line-wrapping bytes were counted instead of codepoints. This
resulted in line-breaks being inserted at the wrong position and also
breaking UTF-8 characters.
Fixes#1022.
Previously the mapping was eager on construction and calls to
`next()`. This was fine when generators were always exhausted fully,
but with the new generator filtering, we might not want to call
the map function eagerly; rather we want to call it only once,
after the generator is moved to the target element.
Not being able to filter generators to specific element has been regularly
causing problems. It was possible to use a dynamic section to run tests
for specific element in a generator, at least if the element had a nice
string representation, but the test case would still run once per element
in the generator.
With this change, it is possible to have the generator return only one
specific element, and do so based on the index, rather than the string
representation of the element. This enables simple debugging of tests
that fail for specific generator element.
* Add validation for --benchmark-samples to prevent crash with zero samples
* Add automated test for benchmark samples validation
* Update baselines for benchmark-samples validation
* Add missing test for benchmark samples validation
---------
Co-authored-by: Chan Aung <chan@Thinkpad.localdomain>
The original implementation of `UNSCOPED_X` message macros used a
clever hack to make the original implementation simpler: construct
an instance of `ScopedMessage` to manage its lifetime, but store
it in a vector, so its lifetime is not actually scope-based, and
we can manage it through the vector instance.
This hack made it so that the lifetime of the vector that manages
the fake `ScopedMessage`s must be outlived by the vector with the
actual messages. Originally this wasn't a problem, because they both
lived inside the run context instance. However, since then these
vectors became globals and thread-local. When this happened, it
still wasn't a problem; the two globals were declared in the right
order, so they were destroyed in the right order as well.
Then, in f80956a43a, these globals
were turned into magic static globals to improve their behaviour
in MSVC's Debug build mode. This caused their lifetimes to be
runtime-dependent; if a specific test thread added its first scoped
message before it added first unscoped message, the lifetimes
would be correct. If it instead added first unscoped message
before adding first scoped message, then there **might** be
invalid reads during thread destruction.
The fix is simple: do things properly and manage the lifetime of
messages in `UNSCOPED_X` explicitly. Then we don't have to deal
with the destruction of fake `ScopedMessage`s while the thread is
being destroyed, and the lifetime of the two vectors is no longer
tied together.
I also threw them both into a new type, to encapsulate some of the
unscoped message logic.
Specifically, this commit makes the `-c`/`--section` parameter
strictly ordered and hierarchical, unlike how it behaved before,
which was a huge mess -- see #3038 for details.
Closes#3038
Because it requires full rebuild of the base library, it adds about
20% to the build time of the test suite. Between the fact that the
option is deprecated, and that Bazel has added the `BAZEL_TEST`
env var _years_ ago, nobody should be using it, and the chance
of breakage is tiny, the test is not worth its compile-time cost.
This commit causes small (~5%) slowdown when processing strings
that consist mostly of escaped characters, but improves throughput
by about 100% for strings that consist mostly of characters that
do not need escaping.
This cherry-picks the source changes from PR made by @pkleymonov-qnx,
without including the build path changes that are irrelevant unless
we can add a QNX target to our CI.
Close#2953
This tells Catch2 to create an empty file at specified path before
the tests start, and delete it after the tests finish. This allows
callers to catch cases where the test binary silently exits before
finishing (e.g. via call to `exit(0)` inside the code under test),
by looking whether the file still exists.
Closes#3020
Forbid deducing reference types for m_predicate in FilterGenerator to prevent dangling references.
This is needed for out-of-line predicates to work correctly instead of undefined behavior or crashes.
---------
Co-authored-by: Tek Mate <mate.tek@evosoft.com>
Martin Hořeňovský
Dirk Müller
Jan Niklas Hasse
dragon-archer
qerasedandChan Aung
SzapiandTek Mate