The underlying benchmarking machinery is templated, so that it can use
different clock implementations (either for testing, or to support user
with platform-specific clocks). However, 99.99% of all users interact
with it using the `BENCHMARK` macro, which means that they use it with
`std::chrono::steady_clock`.
By adding outlined implementation of benchmarking helpers specialized
onto `std::chrono::steady_clock`, we save some amount of time per every
TU that uses benchmarks.
The goal is to do the equivalent of clobbering memory, which forces
the compiler to keep side effects that are external to the benchmarked
function, while allowing it to optimize inside the benchmarked function.
E.g. this cannot be optimized away:
```cpp
BENCHMARK("foo") {
global_count += 1;
};
```
while this can:
```cpp
BENCHMARK("bar") {
size_t local_count = 0;
for (size_t i = 0; i < 100; ++i) {
local_count += 1;
}
};
```
* fix: work around clang 20/21 + libc++ compile failure in TEMPLATE_PRODUCT_TEST_CASE with differing arities
Fixes#3115
* docs: reference llvm/llvm-project#130778 in the clang 20/21 workaround comment
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Previously, the `catch_discover_tests` would prepare the entire CTest
command (e.g. `add_test(...)` or `set_tests_properties(...)`) first, and
then escape it when finished. However, this caused lot of the command args
to be escaped over and over again (e.g. executable name or Catch2's reporter
args), for no reason, as they were always the same, and thus their escaping
was always the same.
Until recently, the performance overhead didn't matter as there were many
spots which had quadratic runtime in number of tests. However, the recent
refactorings fixed these, and this commit now improves the throughput by
10-20%.
Also extended the benchmarked COUNTS in `benchmark_discovery.py`,
because the performance is now good enough that it is reasonable
to benchmark 16k tests.
This was another place where the script triggered the quadratic runtime
from calling `string(APPEND` (or `list(APPEND`) repeatedly. As in
60c8b87, we avoid this by flushing the test list into a file every
50kB of text input.
As the string with the test list never grew quite as much as the
string that contains all the test definitions, this only provides
significant savings for high number of tests. It starts being
properly measurable around 4k tests at ~100ms, but grows to ~4s at
32k tests.
There were 2 separate bugs, one recently added and one ancient
(since roughly the first version of the registration script).
1) The recent refactoring of how the JSON output from Catch2 is
parsed caused shadowing between the variable containing the
per-test JSON fragments and the accumulator of test names for
<target>_TESTS variable. This led to the variable containing
both the names and the JSON fragments, and thus being completely
wrong.
2) The test name accumulation has never accounted for characters
that need escaping to be present in a CMake list. This means
that e.g. test names with semicolon in them would end up with
two elements in the test list.
Both of these are now fixed, at the cost of extra complexity and
my sanity as I had to learn more about CMake escaping rules.
(CMake escaping rules are dumb)
* Quiet verbosity provides just the test names
* Normal verbosity adds tags
* High verbosity add the source location of the tests
Listing tests with the quiet verbosity results in about 1/4 of the
previous output, which leads to measurably faster execution of
`catch_discover_tests` when it does not need tags for labels.
(If it does need tags, the output is about 1/2 of previous).
Using CMake's `string(JSON` to parse JSON array leads to quadratic
running time in number of tests, see https://gitlab.kitware.com/cmake/cmake/-/work_items/27985
This leads to _terrible_ runtime for `catch_discover_tests` when called
on binaries with lot of tests (1k+). To get reasonable runtimes, we have
to avoid using `string(JSON` to parse out the individual test objects
from the array with all tests.
This commit replaces the sane approach of using real JSON parser with
a set of terrible hacks, where we use CMake's string APIs to split the
JSON array on what looks like object boundary (`}<ws>*,<ws>*{`), and then
checking whether the resulting thing can be parsed as JSON object. If not,
we append the next piece and check again. And again, and again, until we
get a proper JSON object.
This is all around a hilariously terrible idea, however:
1) It works in practice for all tested inputs.
2) It improves the time it takes to run `catch_discover_tests` on binary
with 1k tests from 4.2s to 1.1s and 2k tests from 16s to 3.9s.
Repeatedly calling `string(APPEND` on the same destination string
leads to quadratic running time. We avoid this by flushing the CTest
script into a file periodically, currently every 50kB of text.
This can cause _slight_ slowdown just around the flush boundary (each
file write is quite expensive, so if we flush just before the last test
is written, it hurts), but it avoids terrible performance for large
test suites.
The performance is roughly equal at 500 tests to discover, and clearly
wins at more; for 1/2/4/8 k tests, the improvements are 0.3/1.6/6.4/27 s.
The previous approach was for `add_command` to behave as append function
via concatenating the command string internally and then saving it into
`PARENT_SCOPE`. Because this in practice ended up meaning concatenating
a copy of the string inside the function and then overwriting the string
outside the function, the performance was lacking.
The new approach is for `prepare_command` to only escape & return the
command from single call, and the caller is responsible for concatenating
the result. Since the actual concatenation no longer crosses function
scope boundaries, the performance is much better, even though the scaling
is still quadratic.
The new approach only takes 1/4-1/5 of the time, saving ~1s at 1k tests,
4.5s at 2k tests and 19s at 4k tests.
The "Could not jump to Nth element: not enough elements" message thrown
by throw_generator_exception() had "Coud" misspelled in all three places
it appears (catch_generators.hpp, catch_generators_adapters.hpp, and
catch_interfaces_generatortracker.cpp).
This provides nice speedup when writing strings that don't need
escaping, at about 6% in Debug build and ~40% in Release build.
If the strings do need escaping the speedup is much smaller, at
~1% in Debug and ~6% in Release build, as the cost of actually
escaping the strings dwarves the cost of checking.
As the types and values sent into the writer are determined by Catch2,
I do not expect to actually need this support, but it is better to have
it and not be surprised in the future.
The forward iteration logic already bounds-check for m_it != m_string->end(),
do the same for the backward iteration. The issue with the assert is
that the assert() might not be compiled in, and it is happening after
the dereference, so it was too late.
To make this all work, I had to remove the stringification cache
from matchers. In theory, this can cause performance penalty in
cases where single matcher instance is stringified multiple times,
but in practice this does not happen much, and the difference is
surprisingly small anyway, because the performance of stringification
is already horrible and full of allocating strings just to throw
them away.
The matcher combinators need P2738 from C++26 to be `constexpr`.
Closes#3091
P3168 turned `std::optional` into a range type, so the partial specialization
of `StringMaker` for `std::optional<T>` conflicted with the partial
specialization for range types. Ideally we will fix this in the future
to support user-provided partial specializations for range-like types,
but for now we just disable the partial specialization for `std::optional<T>`
if P3168 is implemented.
Martin Hořeňovský
Matt Van HornandMatt Van Horn
Matt Rasa
dvir arad
Phil Nash
Manuel Knörle
Nikolay Baklicharov
Dominic Koepke
offa
Dirk Müller