diff --git a/benchmarks/readme.md b/benchmarks/readme.md new file mode 100644 index 00000000..706f7936 --- /dev/null +++ b/benchmarks/readme.md @@ -0,0 +1,67 @@ + + +# Benchmarks of Catch2 + +This folder holds benchmarks for Catch2. It should not be built for the +small(er) test sets, as it is only relevant for running (mainly runtime) +benchmarks. + +Below you will find some practical examples using +[hyperfine](https://github.com/sharkdp/hyperfine) to determine the +performance of various scenarios. They assume two parallel checkouts, one +named `Catch2-old` and the other `Catch2-new`. You will need to change +the paths to work on your own machine. + + +## Runtime benchmarks + +The runtime benchmarks currently consist of few different `TEST_CASE`s, +each with a simple loop over different assertion type. They are compiled +into two binaries, one for assertion slow path and one for assertion fast +path. + + +### Assumptions and notes about real-world usage + +The basic assertion macro, `REQUIRE` (and `CHECK`) is the most common one, +by far. Thus, it is the most important one to run quickly. It is probably +followed by the matcher macro, `REQUIRE_THAT`. + +The most common result of an assertion is that it passes. Even if it is +stringified and reported, that is most likely due to some listener/reporter +that wants the string representation, not because it failed. + +The performance of both Debug and Release builds are important; users +will run tests in both. LTO runtime performance is not too important, +but compile time perf is. + + +### Examples + +**Compare performance of `REQUIRE` in slow path, debug build** +```text +hyperfine --warmup 2 --shell none --parameter-list version old,new '/home/xarn/benches/Catch2-{version}/build-debug/benchmarks/AssertionsSlowPath -o /dev/null "REQUIRE"' +``` + +**Compare performance of `REQUIRE_THAT` in fast path, release build** +```text +hyperfine --warmup 2 --shell none --parameter-list version old,new '/home/xarn/benches/Catch2-{version}/build-release/benchmarks/AssertionsFastPath -o /dev/null "REQUIRE_THAT"' +``` + +**Compare performance of `REQUIRE` with stringification enabled, release build** +``` +hyperfine --warmup 2 --shell none --parameter-list version old,new '/home/xarn/benches/Catch2-{version}/build-release/benchmarks/AssertionsFastPath -s -o /dev/null "REQUIRE"' +``` +_Note that we redirect the output to `/dev/null` to reduce the overhead of the actual output printing, to see just the impact of stringification._ + + +## Compilation benchmarks + +As tests are often iterated upon and relinked, the compilation cost of +Catch2 is also important. + +TODO: + * Single file vs test suite build + * Practical examples + * Include-only overhead + * Link-only example