Avoid quadratic JSON array parse behaviour in catch_discover_tests

Using CMake's `string(JSON` to parse JSON array leads to quadratic
running time in number of tests, see https://gitlab.kitware.com/cmake/cmake/-/work_items/27985

This leads to _terrible_ runtime for `catch_discover_tests` when called
on binaries with lot of tests (1k+). To get reasonable runtimes, we have
to avoid using `string(JSON` to parse out the individual test objects
from the array with all tests.

This commit replaces the sane approach of using real JSON parser with
a set of terrible hacks, where we use CMake's string APIs to split the
JSON array on what looks like object boundary (`}<ws>*,<ws>*{`), and then
checking whether the resulting thing can be parsed as JSON object. If not,
we append the next piece and check again. And again, and again, until we
get a proper JSON object.

This is all around a hilariously terrible idea, however:

1) It works in practice for all tested inputs.
2) It improves the time it takes to run `catch_discover_tests` on binary
   with 1k tests from 4.2s to 1.1s and 2k tests from 16s to 3.9s.
This commit is contained in:
Martin Hořeňovský committed 2026-07-26 19:51:58 +02:00
1 parent 60c8b87829
commit 1079da4c5f
4 files changed
+361 -15

No files matched your search

@@ -41,3 +41,24 @@ public:
TEST_CASE_METHOD(TestCaseFixture, "A test case as method", "[tagstagstags]") {}
TEST_CASE( "Newlines\nAnd\rOther\n\tWhitespace", "[whitespace-going-wild]" ) {}
// Some JSON-like and JSON-adjacent characters and substrings in the test names/tags
// This serves to test that the parse-json-via-string-splitting hack in
// catch_discover_tests works properly.
TEST_CASE( "We split on variants of },{ in name" ) {}
TEST_CASE( "Then }\t, { we }\t,\t{ concatenate } , { them } ,{back},{" ) {}
TEST_CASE( "}},{{" ) {}
TEST_CASE( "Arrays [{},{}] wheee", "[also},{tags]" ) {}
TEST_CASE( "Let's add semicolon into the mix ;},{;},{};,{}" ) {}
TEST_CASE( "[", "[unmatched-square-bracket]" ) {}
TEST_CASE( "]", "[unmatched-square-bracket]" ) {}
// Some CMake-like special strings ($ as dereference) strings in test names.
// This serves to test that the names of test cases are not evaluated
// inside the catch_discover_tests.
TEST_CASE( "Plain ${NOT_A_VAR} variable" ) {}
TEST_CASE( "Env variable access $ENV{HOME}" ) {}
TEST_CASE( "Cache check $CACHE{FOO}" ) {}
TEST_CASE( "Also some generator exprs $<CONFIG> in $<1:yes> name" ) {}
TEST_CASE( "Mess of bare $ $$ $$$ and unterminated $} ${ exprs" ) {}
TEST_CASE( "$ENV{X};[weird]{},{ mix $<0:no> ${VAR} };,{ }},{{" ) {}