This folder holds benchmarks for Catch2. It should not be built for the small(er) test sets, as it is only relevant for running (mainly runtime) benchmarks.

Below you will find some practical examples using hyperfine to determine the performance of various scenarios. They assume two parallel checkouts, one named Catch2-old and the other Catch2-new. You will need to change the paths to work on your own machine.

Runtime benchmarks

The runtime benchmarks currently consist of few different TEST_CASEs, each with a simple loop over different assertion type. They are compiled into two binaries, one for assertion slow path and one for assertion fast path.

Assumptions and notes about real-world usage

The basic assertion macro, REQUIRE (and CHECK) is the most common one, by far. Thus, it is the most important one to run quickly. It is probably followed by the matcher macro, REQUIRE_THAT.

The most common result of an assertion is that it passes. Even if it is stringified and reported, that is most likely due to some listener/reporter that wants the string representation, not because it failed.

The performance of both Debug and Release builds are important; users will run tests in both. LTO runtime performance is not too important, but compile time perf is.

Examples

Compare performance of REQUIRE in slow path, debug build

hyperfine --warmup 2 --shell none --parameter-list version old,new '/home/xarn/benches/Catch2-{version}/build-debug/benchmarks/AssertionsSlowPath -o /dev/null "REQUIRE"'

Compare performance of REQUIRE_THAT in fast path, release build

hyperfine --warmup 2 --shell none --parameter-list version old,new '/home/xarn/benches/Catch2-{version}/build-release/benchmarks/AssertionsFastPath -o /dev/null "REQUIRE_THAT"'

Compare performance of REQUIRE with stringification enabled, release build

hyperfine --warmup 2 --shell none --parameter-list version old,new '/home/xarn/benches/Catch2-{version}/build-release/benchmarks/AssertionsFastPath -s -o /dev/null "REQUIRE"'

Note that we redirect the output to /dev/null to reduce the overhead of the actual output printing, to see just the impact of stringification.

TODO:

Start empty binary (set up cost base)
Section tracking

Compilation benchmarks

As tests are often iterated upon and relinked, the compilation cost of Catch2 is also important.

Examples

Compare overhead of including catch_test_macros.hpp

hyperfine --warmup 2 --shell none --parameter-list version old,new '/usr/bin/c++  -I/home/xarn/benches/Catch2-{version}/src/catch2/.. -I/home/xarn/benches/Catch2-{version}/build-debug/generated-includes -g -o /dev/null -c /home/xarn/benches/Catch2-{version}/benchmarks/only_include.cpp'

Compare build time of Catch2's SelfTest test suite, Debug build

hyperfine --warmup 2 --parameter-list version old,vas --prepare 'find ~/benches/Catch2-{version}/tests/SelfTest -type f -name "*.cpp" -exec touch {} +' 'ninja -j 1 -C ~/benches/Catch2-{version}/build-debug'

TODO:

Link-only recipe

Misc. benchmarks

`catch_discover_tests`

The first JSON-based implementation of catch_discover_tests turned out to have quadratic complexity in number of tests, which meant that registering say 1k test cases took ~4s , and it became unusably slow with larger test suites.

To prevent backsliding, and enable future optimizations, there is now a benchmarking script for catch_discover_tests in the discover_tests directory.

discover_tests/benchmark_discovery.py runs the real catch_discover_tests implementation (through CMake script-mode) on a synthesized JSON listing (generated from discover_tests/listing_template.json) through an executor shim (discover_tests/copy_shim.cmake). This means it can run even without real test binary.

If the JSON output from Catch2 changes, the synthesized JSON listing can be updated by taking a real listing from SelfTest binary, and pruning it down to keep only ~10 entries.

Examples

Benchmark the current catch_discover_tests, with and without tags-as-labels

./benchmarks/discover_tests/benchmark_discovery.py --script ./extras/CatchAddTests.cmake

Note that 8k test cases is unrealistically high, but still useful to see scaling. Benchmark for single test case is useful to provide estimate of the flat overhead from using catch_discover_tests at all.