Compare commits

...

23 Commits

Author SHA1 Message Date
Martin Hořeňovský 8b08d4d795 v3.15.3 2026-07-26 22:19:31 +02:00
Martin Hořeňovský 1079da4c5f Avoid quadratic JSON array parse behaviour in catch_discover_tests
Using CMake's `string(JSON` to parse JSON array leads to quadratic
running time in number of tests, see https://gitlab.kitware.com/cmake/cmake/-/work_items/27985

This leads to _terrible_ runtime for `catch_discover_tests` when called
on binaries with lot of tests (1k+). To get reasonable runtimes, we have
to avoid using `string(JSON` to parse out the individual test objects
from the array with all tests.

This commit replaces the sane approach of using real JSON parser with
a set of terrible hacks, where we use CMake's string APIs to split the
JSON array on what looks like object boundary (`}<ws>*,<ws>*{`), and then
checking whether the resulting thing can be parsed as JSON object. If not,
we append the next piece and check again. And again, and again, until we
get a proper JSON object.

This is all around a hilariously terrible idea, however:

1) It works in practice for all tested inputs.
2) It improves the time it takes to run `catch_discover_tests` on binary
   with 1k tests from 4.2s to 1.1s and 2k tests from 16s to 3.9s.
2026-07-26 19:51:58 +02:00
Martin Hořeňovský 60c8b87829 Avoid n^2 behaviour when appending script commands in catch_discover_tests
Repeatedly calling `string(APPEND` on the same destination string
leads to quadratic running time. We avoid this by flushing the CTest
script into a file periodically, currently every 50kB of text.

This can cause _slight_ slowdown just around the flush boundary (each
file write is quite expensive, so if we flush just before the last test
is written, it hurts), but it avoids terrible performance for large
test suites.

The performance is roughly equal at 500 tests to discover, and clearly
wins at more; for 1/2/4/8 k tests, the improvements are 0.3/1.6/6.4/27 s.
2026-07-26 15:50:33 +02:00
Martin Hořeňovský 64a551e2e7 Optimize adding test commands in catch_discover_tests
The previous approach was for `add_command` to behave as append function
via concatenating the command string internally and then saving it into
`PARENT_SCOPE`. Because this in practice ended up meaning concatenating
a copy of the string inside the function and then overwriting the string
outside the function, the performance was lacking.

The new approach is for `prepare_command` to only escape & return the
command from single call, and the caller is responsible for concatenating
the result. Since the actual concatenation no longer crosses function
scope boundaries, the performance is much better, even though the scaling
is still quadratic.

The new approach only takes 1/4-1/5 of the time, saving ~1s at 1k tests,
4.5s at 2k tests and 19s at 4k tests.
2026-07-26 15:50:31 +02:00
Martin Hořeňovský 2b971368dd Determinize the test registration order in catch_discover_tests 2026-07-26 15:23:39 +02:00
Martin Hořeňovský f5db82a5c4 Add Python script for benchmarking catch_discover_tests 2026-07-24 23:53:50 +02:00
Martin Hořeňovský dcbb2d5d84 Fix bad ToC link in docs/command-line.md 2026-07-24 23:52:44 +02:00
dvir arad 1a9625a6e7 Fix typo in generator exception message ("Coud" -> "Could")
The "Could not jump to Nth element: not enough elements" message thrown
by throw_generator_exception() had "Coud" misspelled in all three places
it appears (catch_generators.hpp, catch_generators_adapters.hpp, and
catch_interfaces_generatortracker.cpp).
2026-07-24 17:12:50 +02:00
Martin Hořeňovský bb8873ccd7 JSONWriter uses LUT for faster checking if a char needs escaping
This provides nice speedup when writing strings that don't need
escaping, at about 6% in Debug build and ~40% in Release build.
If the strings do need escaping the speedup is much smaller, at
~1% in Debug and ~6% in Release build, as the cost of actually
escaping the strings dwarves the cost of checking.
2026-07-23 19:10:14 +02:00
Martin Hořeňovský 46bdf1a04c Optimize writing of non-string values to JSON
In practice, the majority of values the JSON writer handles are
strings, but this makes the code more obvious and can still provide
a small speed-up.
2026-07-23 14:34:43 +02:00
Martin Hořeňovský c267251e70 Better support for infs and NaNs in JSON writer
As the types and values sent into the writer are determined by Catch2,
I do not expect to actually need this support, but it is better to have
it and not be surprised in the future.
2026-07-23 14:26:26 +02:00
Martin Hořeňovský 0cc833ea95 Make JSONWriter's number handling locale-independent
Closes #3176
2026-07-23 00:10:28 +02:00
Martin Hořeňovský 8494e2dce4 Add TEST_CASE with newlines & stuff to the catch_discover_tests's tests 2026-07-22 21:11:58 +02:00
Phil Nash eb431764b4 Document that DISCOVERY_MODE PRE_TEST should be used with catch_discover_tests when used with XCode.
This addresses https://github.com/catchorg/Catch2/issues/2411 by making the "workaround" the official approach and is consistent with how gtest and cmake deal with the same situation.
2026-07-22 20:50:17 +02:00
Manuel Knörle ae5d271da2 Remove debug message for environment variable check 2026-07-15 11:19:22 +02:00
Nikolay Baklicharov a15f718c82 Optimize JSON parsing in catch_discover_tests() 2026-07-08 10:29:59 +02:00
Martin Hořeňovský 191fa38c9b v3.15.2 2026-07-07 20:44:33 +02:00
Martin Hořeňovský dd94b9a780 Fix --warn InfiniteGenerators firing even if the generator was filtered 2026-07-07 14:09:11 +02:00
Martin Hořeňovský 9f7c9f6872 Add extra test binary & test for CATCH_CONFIG_FAST_COMPILE
Closes #3100
2026-07-06 11:03:47 +02:00
Martin Hořeňovský 919385f704 Fix some test binaries in ExtraTests not having warnings enabled 2026-07-06 10:22:55 +02:00
Martin Hořeňovský 15d52830ee Fix -Wunused-parameter warning with exceptions disabled
Closes #3114
2026-07-05 12:08:06 +02:00
Martin Hořeňovský 9ec44dd62b catch_discover_tests uses tempfile to retrieve JSON from the binary
This allows it to deal with badly behaved code, where 3rd party
dependencies write into stdout during global construction.

Closes #3162
Closes #3166
2026-07-04 16:36:05 +02:00
Martin Hořeňovský 675f9eaeb1 Add LLM policy to contributing.md 2026-06-14 20:25:57 +02:00
52 changed files with 2071 additions and 159 deletions
+1 -1
View File
@@ -35,7 +35,7 @@ if(CMAKE_BINARY_DIR STREQUAL CMAKE_CURRENT_SOURCE_DIR)
endif()
project(Catch2
VERSION 3.15.1 # CML version placeholder, don't delete
VERSION 3.15.3 # CML version placeholder, don't delete
LANGUAGES CXX
HOMEPAGE_URL "https://github.com/catchorg/Catch2"
DESCRIPTION "A modern, C++-native, unit test framework."
@@ -0,0 +1,200 @@
#!/usr/bin/env python3
# Copyright Catch2 Authors
# Distributed under the Boost Software License, Version 1.0.
# (See accompanying file LICENSE.txt or copy at
# https://www.boost.org/LICENSE_1_0.txt)
# SPDX-License-Identifier: BSL-1.0
import argparse
import json
import os
import re
import statistics
import subprocess
import sys
import tempfile
import time
HERE = os.path.dirname(os.path.abspath(__file__))
TEMPLATE = os.path.join(HERE, "listing_template.json")
SHIM = os.path.join(HERE, "copy_shim.cmake")
COUNTS = [1, 10, 500, 1000, 2000, 4000, 8000]
#COUNTS = [1, 10, 100]
REPEATS = 5
def load_template(path: str) -> dict:
with open(path, encoding="utf-8") as f:
catch_out = json.load(f)
if 'listings' not in catch_out or 'tests' not in catch_out['listings']:
sys.exit(f"Template '{path}' does not contain the expected 'listings.tests' data")
return catch_out
def synthesize_listing(template_doc: dict, count: int, file) -> str:
"""Writes JSON listing with exactly `count` test cases into `file`.
The template entries are cycled through, and each name is made unique by
appending an index, so that every registered CTest test has a distinct name.
"""
doc_copy = template_doc.copy()
template_tests = template_doc['listings']['tests']
num_original = len(template_tests)
out_tests = []
for i in range(count):
entry = template_tests[i % num_original].copy()
entry["name"] = f"{entry['name']} #{i // num_original}"
out_tests.append(entry)
doc_copy['listings']['tests'] = out_tests
# indent=2 matches Catch2's pretty printing
json.dump(doc_copy, file, indent=2)
file.write('\n')
def build_command(args, listing_path: str, ctest_file: str, add_tags: bool) -> list[str]:
# The executor needs to be a CMake list, so it has to be separated by ';'
executor = ";".join(['cmake', f"-DBENCH_LISTING={listing_path}", "-P", SHIM, "--"])
return [
'cmake',
"-DTEST_TARGET=benchmark",
f"-DTEST_EXECUTABLE={listing_path}",
f"-DTEST_EXECUTOR={executor}",
f"-DTEST_WORKING_DIR={os.path.dirname(ctest_file)}",
f"-DCTEST_FILE={ctest_file}",
f"-DADD_TAGS_AS_LABELS={'TRUE' if add_tags else 'FALSE'}",
"-P",
args.script,
]
def run_once(cmd: list[str]) -> float:
start = time.monotonic()
result = subprocess.run(cmd, capture_output=True, text=True)
elapsed = time.monotonic() - start
if result.returncode != 0:
sys.stderr.write("Benchmark invocation failed:\n")
sys.stderr.write(f" cmd: {' '.join(cmd)}\n")
sys.stderr.write(f" stdout: {result.stdout}\n")
sys.stderr.write(f" stderr: {result.stderr}\n")
sys.exit(1)
return elapsed
def verify_output(ctest_file: str, count: int, add_tags: bool):
"""Sanity-check the generated CTest script and return its add_test count."""
with open(ctest_file, encoding="utf-8") as f:
content = f.read()
add_test_lines = sum(1 for line in content.splitlines() if line.startswith("add_test("))
if add_test_lines != count:
sys.exit(
f"Sanity check failed: expected {count} add_test() lines, "
f"found {add_test_lines} in {ctest_file}"
)
if add_tags and count > 0 and "LABELS" not in content:
sys.exit(
f"Sanity check failed: ADD_TAGS_AS_LABELS was on but no LABELS found "
f"in {ctest_file}"
)
def cmake_version() -> str:
stdout = subprocess.run(
['cmake', "--version"], capture_output=True, text=True, check=True
).stdout
match = re.match("cmake version (.+)", stdout)
return match.group(1)
def print_table(counts, tag_modes, results):
timing_by_ct = {(count, tags): median_time for (count, tags, median_time) in results}
headers = ["N"]
if False in tag_modes:
headers.append("tags:off")
if True in tag_modes:
headers.append("tags:on")
def fmt_cell(count, tags_on):
median_time = timing_by_ct[(count, tags_on)]
return f"{median_time:8.1f} ms"
# Header
row = f"{headers[0]:>6}"
if False in tag_modes:
row += f" {headers[1]:>14}"
if True in tag_modes:
row += f" {headers[2]:>14}"
print()
print("-" * len(row))
print(row)
print("-" * len(row))
for count in counts:
row = f"{count:>6}"
if False in tag_modes:
row += f" {fmt_cell(count, False):>14}"
if True in tag_modes:
row += f" {fmt_cell(count, True):>14}"
print(row)
def main():
parser = argparse.ArgumentParser(
description="Benchmark the CMake-side cost of catch_discover_tests."
)
parser.add_argument(
"--script",
required=True,
help="Path to the CatchAddTests.cmake to benchmark.",
)
args = parser.parse_args()
if not os.path.exists(args.script):
sys.exit(f"Script to benchmark '{args.script}' does not exist")
if not os.path.exists(SHIM):
sys.exit(f"Copy-shim '{SHIM}' does not exist")
tag_modes = [False, True]
template_doc = load_template(TEMPLATE)
print(f"Benchmarking: {os.path.abspath(args.script)}")
print(f"CMake version: {cmake_version()}")
print(f"Ns: {COUNTS}")
print(f"Repeats: {REPEATS}")
print()
results = [] # (count, tags_on, median_ms)
with tempfile.TemporaryDirectory() as workdir:
os.makedirs(workdir, exist_ok=True)
for count in COUNTS:
listing_path = os.path.join(workdir, f"listing-{count}.json")
with open(listing_path, "w", encoding="utf-8") as f:
synthesize_listing(template_doc, count, f)
for add_tags in tag_modes:
mode = "on" if add_tags else "off"
ctest_file = os.path.join(workdir, f"ctest-{count}-{mode}.cmake")
cmd = build_command(args, listing_path, ctest_file, add_tags)
# Untimed warmup run (populates any filesystem/OS caches).
run_once(cmd)
verify_output(ctest_file, count, add_tags)
timings = [run_once(cmd) for _ in range(REPEATS)]
median_ms = statistics.median(timings) * 1000.0
results.append((count, add_tags, median_ms))
print(f'N = {count} done in ~{len(tag_modes) * sum(timings):.2f} s')
print_table(COUNTS, tag_modes, results)
if __name__ == "__main__":
main()
+40
View File
@@ -0,0 +1,40 @@
# SPDX-License-Identifier: BSL-1.0
# Fake Catch2 binary used by benchmark_discovery.py.
#
# catch_discover_tests_impl discovers tests by running
#
# ${TEST_EXECUTOR} "${TEST_EXECUTABLE}" ... --list-tests --reporter json --out <file>
#
# and then reading the JSON listing back from <file>. This script fakes
# being Catch2 binary by taking the fake JSON listing prepared by
# `benchmark_discovery.py` and copying it over to the file path specified
# after the `--out` argument.
#
# The fake JSON listing is passed as BENCH_LISTING CMake argument.
if(NOT DEFINED BENCH_LISTING)
message(FATAL_ERROR "copy_shim: BENCH_LISTING is not set")
endif()
if(NOT EXISTS "${BENCH_LISTING}")
message(FATAL_ERROR "copy_shim: BENCH_LISTING '${BENCH_LISTING}' does not exist")
endif()
# FIXME: better handling of bad --out
# Find the "--out <file>" argument pair among the forwarded arguments.
set(_out_file "")
math(EXPR _last "${CMAKE_ARGC} - 1")
foreach(_i RANGE 0 ${_last})
if("${CMAKE_ARGV${_i}}" STREQUAL "--out")
math(EXPR _next "${_i} + 1")
set(_out_file "${CMAKE_ARGV${_next}}")
break()
endif()
endforeach()
if(_out_file STREQUAL "")
message(FATAL_ERROR "copy_shim: could not find '--out <file>' in arguments")
endif()
file(COPY_FILE "${BENCH_LISTING}" "${_out_file}")
@@ -0,0 +1,134 @@
{
"version": 1,
"metadata": {
"name": "benchmark-template",
"catch2-version": "3.15.2"
},
"listings": {
"tests": [
{
"name": "Comparing function pointers",
"class-name": "",
"tags": [
"function pointer",
"Tricky"
],
"source-location": {
"filename": "/mnt/c/ubuntu/Catch2/tests/SelfTest/UsageTests/Tricky.tests.cpp",
"line": 266
}
},
{
"name": "Testing checked-if 4",
"class-name": "",
"tags": [
"!shouldfail",
"checked-if"
],
"source-location": {
"filename": "/mnt/c/ubuntu/Catch2/tests/SelfTest/UsageTests/Misc.tests.cpp",
"line": 216
}
},
{
"name": "count_equidistant_floats - double",
"class-name": "",
"tags": [
"approvals",
"distance",
"floating-point"
],
"source-location": {
"filename": "/mnt/c/ubuntu/Catch2/tests/SelfTest/IntrospectiveTests/FloatingPoint.tests.cpp",
"line": 103
}
},
{
"name": "Usage of AllTrue range matcher",
"class-name": "",
"tags": [
"matchers",
"quantifiers",
"templated"
],
"source-location": {
"filename": "/mnt/c/ubuntu/Catch2/tests/SelfTest/UsageTests/MatchersRanges.tests.cpp",
"line": 372
}
},
{
"name": "Exception matchers that succeed",
"class-name": "",
"tags": [
"!throws",
"exceptions",
"matchers"
],
"source-location": {
"filename": "/mnt/c/ubuntu/Catch2/tests/SelfTest/UsageTests/Matchers.tests.cpp",
"line": 428
}
},
{
"name": "A TEMPLATE_TEST_CASE_METHOD based test run that succeeds - float",
"class-name": "Template_Fixture",
"tags": [
"class",
"template"
],
"source-location": {
"filename": "/mnt/c/ubuntu/Catch2/tests/SelfTest/UsageTests/Class.tests.cpp",
"line": 82
}
},
{
"name": "Approximate PI",
"class-name": "",
"tags": [
"Approx",
"PI"
],
"source-location": {
"filename": "/mnt/c/ubuntu/Catch2/tests/SelfTest/UsageTests/Approx.tests.cpp",
"line": 134
}
},
{
"name": "TextFlow::Column respects width setting",
"class-name": "",
"tags": [
"approvals",
"column",
"TextFlow"
],
"source-location": {
"filename": "/mnt/c/ubuntu/Catch2/tests/SelfTest/IntrospectiveTests/TextFlow.tests.cpp",
"line": 38
}
},
{
"name": "Generators internals",
"class-name": "",
"tags": [
"generators",
"internals"
],
"source-location": {
"filename": "/mnt/c/ubuntu/Catch2/tests/SelfTest/IntrospectiveTests/GeneratorsImpl.tests.cpp",
"line": 24
}
},
{
"name": "Mayfail test case with nested sections",
"class-name": "",
"tags": [
"!mayfail"
],
"source-location": {
"filename": "/mnt/c/ubuntu/Catch2/tests/SelfTest/UsageTests/Condition.tests.cpp",
"line": 83
}
}
]
}
}
+37 -1
View File
@@ -56,9 +56,9 @@ _Note that we redirect the output to `/dev/null` to reduce the overhead of the a
TODO:
* Start empty binary (set up cost base)
* Start binary with X (100/1k/10k) tests (test registration cost)
* Section tracking
## Compilation benchmarks
As tests are often iterated upon and relinked, the compilation cost of
@@ -79,3 +79,39 @@ hyperfine --warmup 2 --parameter-list version old,vas --prepare 'find ~/benches/
TODO:
* Link-only recipe
## Misc. benchmarks
### `catch_discover_tests`
The first JSON-based implementation of `catch_discover_tests` turned out
to have quadratic complexity in number of tests, which meant that registering
say 1k test cases took ~4s , and it became unusably slow with larger
test suites.
To prevent backsliding, and enable future optimizations, there is now
a benchmarking script for `catch_discover_tests` in the `discover_tests`
directory.
`discover_tests/benchmark_discovery.py` runs the real `catch_discover_tests`
implementation (through CMake script-mode) on a synthesized JSON listing
(generated from `discover_tests/listing_template.json`) through an executor
shim (`discover_tests/copy_shim.cmake`). This means it can run even without
real test binary.
If the JSON output from Catch2 changes, the synthesized JSON listing
can be updated by taking a real listing from `SelfTest` binary, and pruning
it down to keep only ~10 entries.
#### Examples
**Benchmark the current `catch_discover_tests`, with and without tags-as-labels**
```text
./benchmarks/discover_tests/benchmark_discovery.py --script ./extras/CatchAddTests.cmake
```
Note that 8k test cases is unrealistically high, but still useful to see scaling.
Benchmark for single test case is useful to provide estimate of the flat
overhead from using `catch_discover_tests` at all.
+11
View File
@@ -212,6 +212,17 @@ execution (useful e.g. in cross-compilation environments).
calling ``catch_discover_tests``. This provides a mechanism for globally
selecting a preferred test discovery behavior.
_Note that on Apple Silicon with the Xcode generator you must use `PRE_TEST`,
e.g. `catch_discover_tests(tests DISCOVERY_MODE PRE_TEST)`. With the default
`POST_BUILD` mode the build fails with `Result: Subprocess killed`, because
macOS on Apple Silicon refuses to run unsigned binaries and Xcode code-signs
the test executable only **after** the post-build script that `POST_BUILD`
mode uses to run it for test discovery. `PRE_TEST` avoids this by delaying
discovery until test time, when the executable is already signed. The same
limitation affects CMake's `gtest_discover_tests`; see
[Catch2 #2411](https://github.com/catchorg/Catch2/issues/2411) and
[CMake #21845](https://gitlab.kitware.com/cmake/cmake/-/issues/21845)._
* `SKIP_IS_FAILURE`
Skipped tests will be marked as failed instead.
+1 -1
View File
@@ -26,7 +26,7 @@
[Disable statistical analysis of collected benchmark samples](#disable-statistical-analysis-of-collected-benchmark-samples)<br>
[Specify the amount of time in milliseconds spent on warming up each test](#specify-the-amount-of-time-in-milliseconds-spent-on-warming-up-each-test)<br>
[Usage](#usage)<br>
[Specify the section to run](#specify-the-section-to-run)<br>
[Specify the section/generator element to run](#specify-the-sectiongenerator-element-to-run)<br>
[Filenames as tags](#filenames-as-tags)<br>
[Override output colouring](#override-output-colouring)<br>
[Test Sharding](#test-sharding)<br>
+15
View File
@@ -7,6 +7,8 @@
[Writing documentation](#writing-documentation)<br>
[Writing code](#writing-code)<br>
[CoC](#coc)<br>
[Using LLMs when contributing](#using-llms-when-contributing)<br>
So you want to contribute something to Catch2? That's great! Whether it's
a bug fix, a new feature, support for additional compilers - or just
@@ -332,6 +334,19 @@ When adding new `CATCH_CONFIG` option, there are multiple places to edit:
This project has a [CoC](../CODE_OF_CONDUCT.md). Please adhere to it
while contributing to Catch2.
## Using LLMs when contributing
I do not care whether you used LLM for your contribution. What I care
about is the quality of the contribution, not whether you made it through
prompting LLM, letting GAs run wild, dictated it into the computer, wrote
it using your nose or used butterflies to flip the correct bits.
The flipside of this is that I am also not going to iterate your LLM for
you. If a PR looks LLM generated and does not pass the muster to be merged
as-is, I am going to close it.
-----------
_This documentation will always be in-progress as new information comes
+34
View File
@@ -2,6 +2,8 @@
# Release notes
**Contents**<br>
[3.15.3](#3153)<br>
[3.15.2](#3152)<br>
[3.15.1](#3151)<br>
[3.15.0](#3150)<br>
[3.14.0](#3140)<br>
@@ -76,6 +78,38 @@
[Even Older versions](#even-older-versions)<br>
## 3.15.3
### Fixes
* The JSON reporter's number handling is locale-independent. (#3176)
* Removed leftover debug message from `catch_discover_tests` (#3177)
* Fixed typo in the "Could not jump to the Nth element" exception message (#3181)
### Improvements
* `catch_discover_tests` registers tests in deterministic (alphabetical) order.
* `catch_discover_tests` has been rewritten to be massively faster.
* Preparing the actual CTest script is significantly faster.
* Parsing the JSON test array is significantly faster.
* Running without `ADD_TAGS_AS_LABELS` set is 10-15% faster (as opposed to being the same speed) (#3169)
* The new implementation is about 4x-5x faster, so registering 500 tests now takes ~350ms (down from 1.1s).
* JSON writing is faster
* Small improvement in writing non-string values
* ~6-40% improvement in writing string values that do not need escaping
* ~1-6% improvement in writing string values that do need escaping
* Better support for infs and NaNs in JSON output
## 3.15.2
### Fixes
* Fixed `--warn InfiniteGenerators` triggering even if the generator was limited to specific element via path filtering.
* Fixed `-Wunused-parameter` triggering in `-fnoexceptions` builds.
### Improvements
* `catch_discover_tests` can handle cases where the binary prints out non-Catch2 output due to global constructors (#3162)
## 3.15.1
### Fixes
+8 -2
View File
@@ -44,7 +44,7 @@ same as the Catch name; see also ``TEST_PREFIX`` and ``TEST_SUFFIX``.
``catch_discover_tests`` sets up a post-build command on the test executable
that generates the list of tests by parsing the output from running the test
with the ``--list-test-names-only`` argument. This ensures that the full
with the ``--list-tests --reporter json`` argument. This ensures that the full
list of tests is obtained. Since test discovery occurs at build time, it is
not necessary to re-run CMake when the list of tests changes.
However, it requires that :prop_tgt:`CROSSCOMPILING_EMULATOR` is properly set
@@ -67,7 +67,7 @@ same as the Catch name; see also ``TEST_PREFIX`` and ``TEST_SUFFIX``.
``TEST_SPEC arg1...``
Specifies test cases, wildcarded test cases, tags and tag expressions to
pass to the Catch executable with the ``--list-test-names-only`` argument.
pass to the Catch executable when listing the tests.
``EXTRA_ARGS arg1...``
Any extra arguments to pass on the command line to each test case.
@@ -146,6 +146,12 @@ same as the Catch name; see also ``TEST_PREFIX`` and ``TEST_SUFFIX``.
calling ``catch_discover_tests``. This provides a mechanism for globally selecting
a preferred test discovery behavior without having to modify each call site.
On Apple Silicon with the Xcode generator you must use ``PRE_TEST``. With the
default ``POST_BUILD`` mode the build fails with ``Result: Subprocess killed``,
because macOS on Apple Silicon refuses to run unsigned binaries and Xcode
code-signs the test executable only after the post-build script that
``POST_BUILD`` mode uses to run it for test discovery. See Catch2 issue #2411.
``SKIP_IS_FAILURE``
Disables skipped test detection.
+229 -20
View File
@@ -1,7 +1,146 @@
# Distributed under the OSI-approved BSD 3-Clause License. See accompanying
# file Copyright.txt or https://cmake.org/licensing for details.
function(add_command NAME)
# Because natively using CMake's JSON processing for arrays leads to quadratic
# running times, we do terrible hack and split JSON array by object boundary
# + commas and try to reconstruct valid JSON objects. During this, we need
# to replace CMake characters that could be in the test name/tags with
# placeholder, so it doesn't affect CMake's processing of the strings/lists
# we create during the parsing.
#
# We use 0x01, 0x02, 0x03, and 0x04 as placeholder bytes, as those cannot
# exist in JSON unescaped.
#
# 0x01 <=> ';' (CMake list separator)
# 0x02 == element boundary marker used while splitting the tests array
# 0x03 <-> '[' (opens a CMake bracket-argument context)
# 0x04 <-> ']' (closes a CMake bracket-argument context)
#
string(ASCII 1 _SemicolonEscape)
string(ASCII 2 _BoundaryEscape)
string(ASCII 3 _OpenBracketEscape)
string(ASCII 4 _CloseBracketEscape)
# Placeholder bytes in the listing would break our parsing hack, so
# we check they don't exist. They shouldn't exist in valid JSON, but
# the reporter might not be escaping them properly.
function(validate_input_noescapes listing_var)
foreach(byte "${_SemicolonEscape}" "${_BoundaryEscape}" "${_OpenBracketEscape}" "${_CloseBracketEscape}")
string(FIND "${${listing_var}}" "${byte}" found)
if(NOT found EQUAL -1)
message(FATAL_ERROR
"The test listing contains raw control byte (0x01-0x04) which should not "
"be there. This means either bad escaping in JSON reporter, or corrupted file. "
)
endif()
endforeach()
endfunction()
# Replaces relevant characters with their placeholders, see the top of this file.
# Modifies argument `var` in place.
function(magic_escape_chars var)
set(value "${${var}}")
string(REPLACE ";" "${_SemicolonEscape}" value "${value}")
string(REPLACE "[" "${_OpenBracketEscape}" value "${value}")
string(REPLACE "]" "${_CloseBracketEscape}" value "${value}")
set(${var} "${value}" PARENT_SCOPE)
endfunction()
# Turns placeholders back into original characters, see the top of this file.
# Modifies argument `var` in place.
function(magic_unescape_chars var)
set(value "${${var}}")
string(REPLACE "${_SemicolonEscape}" ";" value "${value}")
string(REPLACE "${_OpenBracketEscape}" "[" value "${value}")
string(REPLACE "${_CloseBracketEscape}" "]" value "${value}")
set(${var} "${value}" PARENT_SCOPE)
endfunction()
# Abuses knowledge of Catch2's JSON reporter output for listing tests to
# split JSON array of the test listings into a CMake list of strings,
# with each element being the JSON string of one array entry.
#
# This avoids the terrible quadratic running time of using CMake's JSON
# support to parse the JSON reporter output "properly", where the whole
# JSON array of tests is parsed again for every element. Instead, we can
# use the CMake's API to only parse the individual test's objects, which
# are usually small and only have to be reparsed fixed number of times
# (once for test names, once for labels).
#
# We process the string representing the JSON array by splitting it on
# `}<ws>*,<ws>*{` and then checking for each chunk whether it is a valid
# JSON object representing Catch2's test. If not (e.g. because we split
# on the presence of `}<ws>*,<ws>*{` inside a test name), then we append
# the next chunk to the current one and check again. And again, until
# we get back to a valid JSON.
#
# Note that to support passing the object strings back from the function,
# they will still contain the placeholders and need to be unescaped before
# further processing (e.g. sending them into CMake's JSON parsing API).
function(split_json_array json_array_var out_var)
# We have to pass the input by var name to avoid CMake processing
# the input as an arg.
set(json_in "${${json_array_var}}")
# Strip the array brackets at the start and end of the JSON array.
# Must happen before we escape the other [] instances below from the
# actual array data.
string(REGEX REPLACE "^[ \t\r\n]*\\[" "" json_in "${json_in}")
string(REGEX REPLACE "\\][ \t\r\n]*$" "" json_in "${json_in}")
magic_escape_chars(json_in)
# We need to keep the whitespace around comma around, so that if we
# split inside the test object, we can reconstruct it losslessly.
string(REGEX REPLACE "(}[ \t\r\n]*)[,]([ \t\r\n]*{)" "\\1${_BoundaryEscape}\\2" json_in "${json_in}")
# We escaped all list separators above, so now we can turn the JSON
# string into a CMake list of fragments in single pass.
string(REPLACE "${_BoundaryEscape}" ";" fragments "${json_in}")
# And now we have to reconstruct the actual JSON structure from fragments.
set(array_elements "")
set(accumulator "")
foreach(next_fragment IN LISTS fragments)
if(accumulator)
set(accumulator "${accumulator},${next_fragment}")
else()
set(accumulator "${next_fragment}")
endif()
# Because the fragments (might) contain invalid JSON characters due
# to escaping, we have to unescape it before checking if we can parse it.
set(maybe_json "${accumulator}")
magic_unescape_chars(maybe_json)
string(JSON unused ERROR_VARIABLE err GET "${maybe_json}" "name")
if(err STREQUAL "NOTFOUND")
list(APPEND array_elements "${accumulator}")
set(accumulator "")
endif()
endforeach()
set(${out_var} "${array_elements}" PARENT_SCOPE)
endfunction()
# TBD: Further possible optimization is that most arguments for per-test
# `prepare_command` call are constant across one invocation of
# `catch_discover_tests`, and thus need checking and escaping only
# once, instead of for each test.
# This would provide nice speed-up of the actual command preparation,
# but it will make the script much harder to read, and it is utterly
# dwarfed by the quadratic scaling of parsing JSON arrays in CMake.
# Prepare command with escaped (bracketed) arguments and return it via `_Command` out variable.
#
# To avoid quadratic performance when concatenating all commands together,
# the actual concatenation must be done by the caller, by appending it
# into a string of all other commands.
function(prepare_command NAME)
set(_args "")
# use ARGV* instead of ARGN, because ARGN splits arrays into multiple arguments
math(EXPR _last_arg ${ARGC}-1)
@@ -13,11 +152,45 @@ function(add_command NAME)
set(_args "${_args} ${_arg}")
endif()
endforeach()
set(script "${script}${NAME}(${_args})\n" PARENT_SCOPE)
set(_Command "${NAME}(${_args})\n" PARENT_SCOPE)
endfunction()
# Generates random filename in the temp folder.
# Temp folder is retrieved by checking env vars from various platforms.
function(make_temp_file_path OUT_VARIABLE FALLBACK_PATH)
set(TEMP_DIR "")
set(ENV_VARS
# From XDG base dir specification
XDG_RUNTIME_DIR
# From POSIX standard
TMPDIR
# From Windows
TMP
TEMP
)
foreach(var ${ENV_VARS})
if(DEFINED ENV{${var}} AND NOT "$ENV{${var}}" STREQUAL "")
set(TEMP_DIR "$ENV{${var}}")
break()
endif()
endforeach()
# If all checks fail, we use the fallback path
if(TEMP_DIR STREQUAL "")
set(TEMP_DIR "${FALLBACK_PATH}")
endif()
file(TO_CMAKE_PATH "${TEMP_DIR}" TEMP_DIR)
# Generate the random file name
string(RANDOM LENGTH 8 RAND_ID)
set(FINAL_TEMP_PATH "${TEMP_DIR}/Catch2-test-listing.${RAND_ID}.json")
set(${OUT_VARIABLE} "${FINAL_TEMP_PATH}" PARENT_SCOPE)
endfunction()
function(catch_discover_tests_impl)
cmake_parse_arguments(
""
""
@@ -26,6 +199,12 @@ function(catch_discover_tests_impl)
${ARGN}
)
# We periodically append to the output file below, so we have to ensure
# that it is empty at the start, or we get duplicated test scripts.
file(REMOVE "${_CTEST_FILE}")
# Size (in Bytes) at which the intermediate `script` var is dumped to file.
set(_WriteToFileThreshold 50000)
set(add_tags "${_ADD_TAGS_AS_LABELS}")
set(prefix "${_TEST_PREFIX}")
set(suffix "${_TEST_SUFFIX}")
@@ -72,8 +251,14 @@ function(catch_discover_tests_impl)
set(ENV{DYLD_FRAMEWORK_PATH} "${paths}")
endif()
make_temp_file_path(listing_output_path "${_TEST_WORKING_DIR}")
execute_process(
COMMAND ${_TEST_EXECUTOR} "${_TEST_EXECUTABLE}" ${spec} --list-tests --reporter json
COMMAND ${_TEST_EXECUTOR} "${_TEST_EXECUTABLE}" ${spec}
--list-tests
--reporter json
--out "${listing_output_path}"
--order lex # Make sure the output order, and thus test registration order, is consistent across runs.
OUTPUT_VARIABLE listing_output
RESULT_VARIABLE result
WORKING_DIRECTORY "${_TEST_WORKING_DIR}"
@@ -86,6 +271,11 @@ function(catch_discover_tests_impl)
)
endif()
# Read the JSON output back from the output file and validate it.
file(READ ${listing_output_path} listing_output)
file(REMOVE ${listing_output_path})
validate_input_noescapes(listing_output)
# Prepare reporter
if(reporter)
set(reporter_arg "--reporter ${reporter}")
@@ -140,22 +330,34 @@ function(catch_discover_tests_impl)
message(FATAL_ERROR "Unsupported catch output version: '${version}'")
endif()
# Speed-up reparsing by cutting away unneeded parts of JSON.
# Extract just the JSON array with tests and then split them into
# individual objects.
string(JSON test_listing GET "${listing_output}" "listings" "tests")
string(JSON num_tests LENGTH "${test_listing}")
split_json_array(test_listing tests)
# Exit early if no tests are detected
if(num_tests STREQUAL "0")
if(NOT tests)
file(WRITE "${_CTEST_FILE}" "")
return()
endif()
# CMake's foreach-RANGE is inclusive, so we have to subtract 1
math(EXPR num_tests "${num_tests} - 1")
# Each element in the tests is JSON-string representing one test object.
# We have to parse it and then turn it into CTest script commands.
foreach(single_test IN LISTS tests)
# Because appending to the same string in CMake has quadratic runtime,
# we flush the script into the file periodically to avoid the worst case.
string(LENGTH "${script}" script_len)
if (script_len GREATER _WriteToFileThreshold)
file(APPEND "${_CTEST_FILE}" "${script}")
set(script "")
endif()
foreach(idx RANGE ${num_tests})
string(JSON single_test GET ${test_listing} ${idx})
string(JSON test_tags GET "${single_test}" "tags")
# The elements are still escaped and contain JSON-invalid characters,
# they have to be unescaped before parsing them as JSON.
magic_unescape_chars(single_test)
if(add_tags)
string(JSON test_tags GET "${single_test}" "tags")
endif()
string(JSON plain_name GET "${single_test}" "name")
# Escape characters in test case names that would be parsed by Catch2
@@ -171,7 +373,7 @@ function(catch_discover_tests_impl)
endif()
# ...and add to script
add_command(add_test
prepare_command(add_test
"${prefix}${plain_name}${suffix}"
${_TEST_EXECUTOR}
"${_TEST_EXECUTABLE}"
@@ -180,12 +382,14 @@ function(catch_discover_tests_impl)
"${reporter_arg}"
"${output_dir_arg}"
)
add_command(set_tests_properties
string(APPEND script "${_Command}")
prepare_command(set_tests_properties
"${prefix}${plain_name}${suffix}"
PROPERTIES
WORKING_DIRECTORY "${_TEST_WORKING_DIR}"
${properties}
)
string(APPEND script "${_Command}")
if(add_tags)
string(JSON num_tags LENGTH "${test_tags}")
@@ -205,19 +409,21 @@ function(catch_discover_tests_impl)
list(APPEND tag_list "${a_tag}")
endforeach()
add_command(set_tests_properties
prepare_command(set_tests_properties
"${prefix}${plain_name}${suffix}"
PROPERTIES
LABELS "${tag_list}"
)
string(APPEND script "${_Command}")
endif()
endif(add_tags)
if(environment_modifications)
add_command(set_tests_properties
prepare_command(set_tests_properties
"${prefix}${plain_name}${suffix}"
PROPERTIES
ENVIRONMENT_MODIFICATION "${environment_modifications}")
string(APPEND script "${_Command}")
endif()
list(APPEND tests "${prefix}${plain_name}${suffix}")
@@ -225,13 +431,16 @@ function(catch_discover_tests_impl)
# Create a list of all discovered tests, which users may use to e.g. set
# properties on the tests
add_command(set ${_TEST_LIST} ${tests})
prepare_command(set ${_TEST_LIST} ${tests})
string(APPEND script "${_Command}")
# Write CTest script
file(WRITE "${_CTEST_FILE}" "${script}")
# Write any script leftovers we have
file(APPEND "${_CTEST_FILE}" "${script}")
endfunction()
if(CMAKE_SCRIPT_MODE_FILE)
# To enable `include`ing this file in the unit test scripts, we only run
# the impl if an actual `TEST_EXECUTABLE` is provided.
if(CMAKE_SCRIPT_MODE_FILE AND DEFINED TEST_EXECUTABLE)
catch_discover_tests_impl(
TEST_EXECUTABLE ${TEST_EXECUTABLE}
TEST_EXECUTOR ${TEST_EXECUTOR}
+108 -34
View File
@@ -6,8 +6,8 @@
// SPDX-License-Identifier: BSL-1.0
// Catch v3.15.1
// Generated: 2026-06-14 10:51:56.053498
// Catch v3.15.3
// Generated: 2026-07-26 22:17:52.418168
// ----------------------------------------------------------
// This file is an amalgamation of multiple different files.
// You probably shouldn't edit it directly.
@@ -2394,7 +2394,7 @@ namespace Catch {
}
Version const& libraryVersion() {
static Version version( 3, 15, 1, "", 0 );
static Version version( 3, 15, 3, "", 0 );
return version;
}
@@ -2541,7 +2541,7 @@ namespace Catch {
bool isValid = next();
if ( !isValid ) {
Detail::throw_generator_exception(
"Coud not jump to Nth element: not enough elements" );
"Could not jump to Nth element: not enough elements" );
}
}
}
@@ -4678,12 +4678,28 @@ namespace Detail {
#include <cmath>
#include <locale>
namespace Catch {
namespace {
static bool needsEscape( char c ) {
return c == '"' || c == '\\' || c == '\b' || c == '\f' ||
c == '\n' || c == '\r' || c == '\t';
struct EscapeLUT {
bool escape[256] = {};
constexpr EscapeLUT() {
escape[static_cast<unsigned char>( '"' )] = true;
escape[static_cast<unsigned char>( '\\' )] = true;
escape[static_cast<unsigned char>( '\b' )] = true;
escape[static_cast<unsigned char>( '\f' )] = true;
escape[static_cast<unsigned char>( '\n' )] = true;
escape[static_cast<unsigned char>( '\r' )] = true;
escape[static_cast<unsigned char>( '\t' )] = true;
}
};
static constexpr EscapeLUT escapeLUT{};
static constexpr bool needsEscape( char c ) {
return escapeLUT.escape[static_cast<unsigned char>( c )];
}
static Catch::StringRef makeEscapeStringRef( char c ) {
@@ -4795,7 +4811,10 @@ namespace Catch {
JsonValueWriter::JsonValueWriter( std::ostream& os,
std::uint64_t indent_level ):
m_os{ os }, m_indent_level{ indent_level } {}
m_os{ os }, m_indent_level{ indent_level } {
// We use C locale so that writing of numerical values is locale-independent.
m_sstream.imbue( std::locale::classic() );
}
JsonObjectWriter JsonValueWriter::writeObject() && {
return JsonObjectWriter{ m_os, m_indent_level };
@@ -4809,26 +4828,55 @@ namespace Catch {
writeImpl( value, true );
}
void JsonValueWriter::write( float value ) && {
writeFloatingPoint( value );
}
void JsonValueWriter::write( double value ) && {
writeFloatingPoint( value );
}
void JsonValueWriter::write( bool value ) && {
writeImpl( value ? "true"_sr : "false"_sr, false );
}
template <typename T>
void JsonValueWriter::writeFloatingPoint( T value ) {
if ( Catch::isnan( value ) ) {
writeImpl( "NaN"_sr, false );
} else if ( std::isinf( value ) ) {
writeImpl( value < 0 ? "-Infinity"_sr : "Infinity"_sr, false );
} else {
m_sstream << value;
writeImpl( m_sstream.str(), false );
}
}
void JsonValueWriter::writeImpl( Catch::StringRef value, bool quote ) {
if ( quote ) { m_os << '"'; }
size_t current_start = 0;
for ( size_t i = 0; i < value.size(); ++i ) {
if ( needsEscape( value[i] ) ) {
if ( current_start < i ) {
m_os << value.substr( current_start, i - current_start );
// Escaping only makes sense for actual strings, which are passed
// with `quote == true`. Non-quoted values are things like bools
// and numbers, which cannot create inputs that need escaping.
if ( !quote ) {
m_os << value;
} else {
m_os << '"';
size_t current_start = 0;
for ( size_t i = 0; i < value.size(); ++i ) {
if ( needsEscape( value[i] ) ) {
if ( current_start < i ) {
m_os
<< value.substr( current_start, i - current_start );
}
m_os << makeEscapeStringRef( value[i] );
current_start = i + 1;
}
m_os << makeEscapeStringRef( value[i] );
current_start = i + 1;
}
if ( current_start < value.size() ) {
m_os << value.substr( current_start,
value.size() - current_start );
}
m_os << '"';
}
if ( current_start < value.size() ) {
m_os << value.substr( current_start, value.size() - current_start );
}
if ( quote ) { m_os << '"'; }
}
} // namespace Catch
@@ -6016,6 +6064,8 @@ namespace Catch {
auto getGenerator() const -> GeneratorBasePtr const& override {
return m_generator;
}
bool isFilteredImpl() const override { return m_isFiltered; }
};
} // namespace
}
@@ -6360,17 +6410,6 @@ namespace Catch {
SourceLineInfo lineInfo,
Generators::GeneratorBasePtr&& generator ) {
// TBD: Do we want to avoid the warning if the generator is filtered?
if ( m_config->warnAboutInfiniteGenerators() &&
!generator->isFinite() ) {
// We want the semantics of `FAIL()`, but we inline it
// to avoid issues with conditionally prefixed macros
INTERNAL_CATCH_MSG( "FAIL",
Catch::ResultWas::ExplicitFailure,
Catch::ResultDisposition::Normal,
"GENERATE() would run infinitely" );
}
auto nameAndLoc = TestCaseTracking::NameAndLocation( static_cast<std::string>( generatorName ), lineInfo );
auto& currentTracker = m_trackerContext.currentTracker();
assert(
@@ -6383,11 +6422,24 @@ namespace Catch {
m_trackerContext,
&currentTracker,
CATCH_MOVE( generator ) );
auto ret = newTracker.get();
// The warning shouldn't fire if the generator is infinite, **but** filtered down.
if ( m_config->warnAboutInfiniteGenerators() &&
!newTracker->m_generator->isFinite() &&
!newTracker->isFiltered() ) {
// We want the semantics of `FAIL()`, but we inline it
// to avoid issues with conditionally prefixed macros
INTERNAL_CATCH_MSG( "FAIL",
Catch::ResultWas::ExplicitFailure,
Catch::ResultDisposition::Normal,
"GENERATE() would run infinitely" );
}
auto returnPtr = newTracker.get();
currentTracker.addChild( CATCH_MOVE( newTracker ) );
ret->open();
return ret;
returnPtr->open();
return returnPtr;
}
bool RunContext::testForMissingAssertions(Counts& assertions) {
@@ -7470,6 +7522,28 @@ namespace TestCaseTracking {
m_ctx.setCurrentTracker( this );
}
bool SectionTracker::isFilteredImpl() const {
// TBD: This is currently _very_ similar to the block in `isComplete`.
// Is this neccessarily that way, or just accident of current semantics?
const size_t filterIndex =
m_newStyleFilters ? m_allTrackerDepth : m_sectionOnlyDepth;
if ( filterIndex < m_filterRef->size() ) {
// 1) New style filter must explicitly target section
if ( m_newStyleFilters && ( *m_filterRef )[filterIndex].type !=
PathFilter::For::Section ) {
return true;
}
// 2) Both style filters must match the trimmed name exactly
if ( m_trimmed_name !=
StringRef( ( *m_filterRef )[filterIndex].filter ) ) {
return true;
}
}
return false;
}
SectionTracker::SectionTracker( NameAndLocation&& nameAndLocation, TrackerContext& ctx, ITracker* parent )
: TrackerBase( CATCH_MOVE(nameAndLocation), ctx, parent ),
m_trimmed_name(trim(StringRef(ITracker::nameAndLocation().name)))
+34 -7
View File
@@ -6,8 +6,8 @@
// SPDX-License-Identifier: BSL-1.0
// Catch v3.15.1
// Generated: 2026-06-14 10:51:55.600632
// Catch v3.15.3
// Generated: 2026-07-26 22:17:52.004020
// ----------------------------------------------------------
// This file is an amalgamation of multiple different files.
// You probably shouldn't edit it directly.
@@ -7484,6 +7484,8 @@ namespace Catch {
return m_translateFunction( ex );
}
#else
(void)it;
(void)itEnd;
return "You should never get here!";
#endif
}
@@ -7570,7 +7572,7 @@ namespace Catch {
#define CATCH_VERSION_MAJOR 3
#define CATCH_VERSION_MINOR 15
#define CATCH_VERSION_PATCH 1
#define CATCH_VERSION_PATCH 3
#endif // CATCH_VERSION_MACROS_HPP_INCLUDED
@@ -7826,7 +7828,7 @@ namespace Generators {
void skipToNthElementImpl( std::size_t n ) override {
if ( n >= m_values.size() ) {
Detail::throw_generator_exception(
"Coud not jump to Nth element: not enough elements" );
"Could not jump to Nth element: not enough elements" );
}
m_idx = n;
}
@@ -7910,8 +7912,7 @@ namespace Generators {
bool isFinite() const override {
for (auto const& gen : m_generators) {
if (!gen.isFinite()) { return false;
}
if (!gen.isFinite()) { return false; }
}
return true;
}
@@ -8010,7 +8011,7 @@ namespace Generators {
void skipToNthElementImpl( std::size_t n ) override {
if ( n >= m_target ) {
Detail::throw_generator_exception(
"Coud not jump to Nth element: not enough elements" );
"Could not jump to Nth element: not enough elements" );
}
m_generator.skipToNthElement( n );
@@ -10308,11 +10309,19 @@ namespace Catch {
writeImpl( value, !std::is_arithmetic<T>::value );
}
void write( StringRef value ) &&;
void write( float value ) &&;
void write( double value ) &&;
void write( bool value ) &&;
private:
void writeImpl( StringRef value, bool quote );
// Helper to deal with non-finite floating point values, which
// are not standard JSON, but we use JS/Python/etc. approach of
// emitting `NaN`, `Infinity`, `-Infinity` as number(like).
template <typename T>
void writeFloatingPoint( T value );
// Without this SFINAE, this overload is a better match
// for `std::string`, `char const*`, `char const[N]` args.
// While it would still work, it would cause code bloat
@@ -10657,6 +10666,8 @@ namespace TestCaseTracking {
using Children = std::vector<ITrackerPtr>;
virtual bool isFilteredImpl() const = 0;
protected:
enum CycleState {
NotStarted,
@@ -10754,6 +10765,20 @@ namespace TestCaseTracking {
* for internal debug checks.
*/
virtual bool isGeneratorTracker() const;
/**
* Returns true if the concrete tracker instance has a filter that applies to it.
*/
bool isFiltered() const {
// Fast path: are there even filters for tracker in this position?
const size_t filter_depth =
m_newStyleFilters ? m_allTrackerDepth : m_sectionOnlyDepth;
if ( m_filterRef->size() <= filter_depth ) { return false; }
// Slow path: If there are filters, ask the concrete tracker.
// This handles things like match-all filters for that tracker.
return isFilteredImpl();
}
};
class TrackerContext {
@@ -10809,6 +10834,8 @@ namespace TestCaseTracking {
// to not own the name, the name still has to outlive the `ITracker` parent, so
// this should still be safe.
StringRef m_trimmed_name;
bool isFilteredImpl() const override;
public:
SectionTracker( NameAndLocation&& nameAndLocation, TrackerContext& ctx, ITracker* parent );
+1 -1
View File
@@ -8,7 +8,7 @@
project(
'catch2',
'cpp',
version: '3.15.1', # CML version placeholder, don't delete
version: '3.15.3', # CML version placeholder, don't delete
license: 'BSL-1.0',
meson_version: '>=0.54.1',
)
+2
View File
@@ -41,6 +41,8 @@ namespace Catch {
return m_translateFunction( ex );
}
#else
(void)it;
(void)itEnd;
return "You should never get here!";
#endif
}
+1 -1
View File
@@ -36,7 +36,7 @@ namespace Catch {
}
Version const& libraryVersion() {
static Version version( 3, 15, 1, "", 0 );
static Version version( 3, 15, 3, "", 0 );
return version;
}
+1 -1
View File
@@ -10,6 +10,6 @@
#define CATCH_VERSION_MAJOR 3
#define CATCH_VERSION_MINOR 15
#define CATCH_VERSION_PATCH 1
#define CATCH_VERSION_PATCH 3
#endif // CATCH_VERSION_MACROS_HPP_INCLUDED
+2 -3
View File
@@ -95,7 +95,7 @@ namespace Generators {
void skipToNthElementImpl( std::size_t n ) override {
if ( n >= m_values.size() ) {
Detail::throw_generator_exception(
"Coud not jump to Nth element: not enough elements" );
"Could not jump to Nth element: not enough elements" );
}
m_idx = n;
}
@@ -179,8 +179,7 @@ namespace Generators {
bool isFinite() const override {
for (auto const& gen : m_generators) {
if (!gen.isFinite()) { return false;
}
if (!gen.isFinite()) { return false; }
}
return true;
}
@@ -27,7 +27,7 @@ namespace Generators {
void skipToNthElementImpl( std::size_t n ) override {
if ( n >= m_target ) {
Detail::throw_generator_exception(
"Coud not jump to Nth element: not enough elements" );
"Could not jump to Nth element: not enough elements" );
}
m_generator.skipToNthElement( n );
@@ -28,7 +28,7 @@ namespace Catch {
bool isValid = next();
if ( !isValid ) {
Detail::throw_generator_exception(
"Coud not jump to Nth element: not enough elements" );
"Could not jump to Nth element: not enough elements" );
}
}
}
+65 -16
View File
@@ -7,14 +7,31 @@
// SPDX-License-Identifier: BSL-1.0
#include <catch2/internal/catch_enforce.hpp>
#include <catch2/internal/catch_jsonwriter.hpp>
#include <catch2/internal/catch_polyfills.hpp>
#include <catch2/internal/catch_unreachable.hpp>
#include <cmath>
#include <locale>
namespace Catch {
namespace {
static bool needsEscape( char c ) {
return c == '"' || c == '\\' || c == '\b' || c == '\f' ||
c == '\n' || c == '\r' || c == '\t';
struct EscapeLUT {
bool escape[256] = {};
constexpr EscapeLUT() {
escape[static_cast<unsigned char>( '"' )] = true;
escape[static_cast<unsigned char>( '\\' )] = true;
escape[static_cast<unsigned char>( '\b' )] = true;
escape[static_cast<unsigned char>( '\f' )] = true;
escape[static_cast<unsigned char>( '\n' )] = true;
escape[static_cast<unsigned char>( '\r' )] = true;
escape[static_cast<unsigned char>( '\t' )] = true;
}
};
static constexpr EscapeLUT escapeLUT{};
static constexpr bool needsEscape( char c ) {
return escapeLUT.escape[static_cast<unsigned char>( c )];
}
static Catch::StringRef makeEscapeStringRef( char c ) {
@@ -126,7 +143,10 @@ namespace Catch {
JsonValueWriter::JsonValueWriter( std::ostream& os,
std::uint64_t indent_level ):
m_os{ os }, m_indent_level{ indent_level } {}
m_os{ os }, m_indent_level{ indent_level } {
// We use C locale so that writing of numerical values is locale-independent.
m_sstream.imbue( std::locale::classic() );
}
JsonObjectWriter JsonValueWriter::writeObject() && {
return JsonObjectWriter{ m_os, m_indent_level };
@@ -140,26 +160,55 @@ namespace Catch {
writeImpl( value, true );
}
void JsonValueWriter::write( float value ) && {
writeFloatingPoint( value );
}
void JsonValueWriter::write( double value ) && {
writeFloatingPoint( value );
}
void JsonValueWriter::write( bool value ) && {
writeImpl( value ? "true"_sr : "false"_sr, false );
}
template <typename T>
void JsonValueWriter::writeFloatingPoint( T value ) {
if ( Catch::isnan( value ) ) {
writeImpl( "NaN"_sr, false );
} else if ( std::isinf( value ) ) {
writeImpl( value < 0 ? "-Infinity"_sr : "Infinity"_sr, false );
} else {
m_sstream << value;
writeImpl( m_sstream.str(), false );
}
}
void JsonValueWriter::writeImpl( Catch::StringRef value, bool quote ) {
if ( quote ) { m_os << '"'; }
size_t current_start = 0;
for ( size_t i = 0; i < value.size(); ++i ) {
if ( needsEscape( value[i] ) ) {
if ( current_start < i ) {
m_os << value.substr( current_start, i - current_start );
// Escaping only makes sense for actual strings, which are passed
// with `quote == true`. Non-quoted values are things like bools
// and numbers, which cannot create inputs that need escaping.
if ( !quote ) {
m_os << value;
} else {
m_os << '"';
size_t current_start = 0;
for ( size_t i = 0; i < value.size(); ++i ) {
if ( needsEscape( value[i] ) ) {
if ( current_start < i ) {
m_os
<< value.substr( current_start, i - current_start );
}
m_os << makeEscapeStringRef( value[i] );
current_start = i + 1;
}
m_os << makeEscapeStringRef( value[i] );
current_start = i + 1;
}
if ( current_start < value.size() ) {
m_os << value.substr( current_start,
value.size() - current_start );
}
m_os << '"';
}
if ( current_start < value.size() ) {
m_os << value.substr( current_start, value.size() - current_start );
}
if ( quote ) { m_os << '"'; }
}
} // namespace Catch
+8
View File
@@ -39,11 +39,19 @@ namespace Catch {
writeImpl( value, !std::is_arithmetic<T>::value );
}
void write( StringRef value ) &&;
void write( float value ) &&;
void write( double value ) &&;
void write( bool value ) &&;
private:
void writeImpl( StringRef value, bool quote );
// Helper to deal with non-finite floating point values, which
// are not standard JSON, but we use JS/Python/etc. approach of
// emitting `NaN`, `Infinity`, `-Infinity` as number(like).
template <typename T>
void writeFloatingPoint( T value );
// Without this SFINAE, this overload is a better match
// for `std::string`, `char const*`, `char const[N]` args.
// While it would still work, it would cause code bloat
+18 -14
View File
@@ -196,6 +196,8 @@ namespace Catch {
auto getGenerator() const -> GeneratorBasePtr const& override {
return m_generator;
}
bool isFilteredImpl() const override { return m_isFiltered; }
};
} // namespace
}
@@ -540,17 +542,6 @@ namespace Catch {
SourceLineInfo lineInfo,
Generators::GeneratorBasePtr&& generator ) {
// TBD: Do we want to avoid the warning if the generator is filtered?
if ( m_config->warnAboutInfiniteGenerators() &&
!generator->isFinite() ) {
// We want the semantics of `FAIL()`, but we inline it
// to avoid issues with conditionally prefixed macros
INTERNAL_CATCH_MSG( "FAIL",
Catch::ResultWas::ExplicitFailure,
Catch::ResultDisposition::Normal,
"GENERATE() would run infinitely" );
}
auto nameAndLoc = TestCaseTracking::NameAndLocation( static_cast<std::string>( generatorName ), lineInfo );
auto& currentTracker = m_trackerContext.currentTracker();
assert(
@@ -563,11 +554,24 @@ namespace Catch {
m_trackerContext,
&currentTracker,
CATCH_MOVE( generator ) );
auto ret = newTracker.get();
// The warning shouldn't fire if the generator is infinite, **but** filtered down.
if ( m_config->warnAboutInfiniteGenerators() &&
!newTracker->m_generator->isFinite() &&
!newTracker->isFiltered() ) {
// We want the semantics of `FAIL()`, but we inline it
// to avoid issues with conditionally prefixed macros
INTERNAL_CATCH_MSG( "FAIL",
Catch::ResultWas::ExplicitFailure,
Catch::ResultDisposition::Normal,
"GENERATE() would run infinitely" );
}
auto returnPtr = newTracker.get();
currentTracker.addChild( CATCH_MOVE( newTracker ) );
ret->open();
return ret;
returnPtr->open();
return returnPtr;
}
bool RunContext::testForMissingAssertions(Counts& assertions) {
@@ -167,6 +167,28 @@ namespace TestCaseTracking {
m_ctx.setCurrentTracker( this );
}
bool SectionTracker::isFilteredImpl() const {
// TBD: This is currently _very_ similar to the block in `isComplete`.
// Is this neccessarily that way, or just accident of current semantics?
const size_t filterIndex =
m_newStyleFilters ? m_allTrackerDepth : m_sectionOnlyDepth;
if ( filterIndex < m_filterRef->size() ) {
// 1) New style filter must explicitly target section
if ( m_newStyleFilters && ( *m_filterRef )[filterIndex].type !=
PathFilter::For::Section ) {
return true;
}
// 2) Both style filters must match the trimmed name exactly
if ( m_trimmed_name !=
StringRef( ( *m_filterRef )[filterIndex].filter ) ) {
return true;
}
}
return false;
}
SectionTracker::SectionTracker( NameAndLocation&& nameAndLocation, TrackerContext& ctx, ITracker* parent )
: TrackerBase( CATCH_MOVE(nameAndLocation), ctx, parent ),
m_trimmed_name(trim(StringRef(ITracker::nameAndLocation().name)))
@@ -12,6 +12,7 @@
#include <catch2/internal/catch_source_line_info.hpp>
#include <catch2/internal/catch_unique_ptr.hpp>
#include <catch2/internal/catch_stringref.hpp>
#include <catch2/internal/catch_path_filter.hpp>
#include <string>
#include <vector>
@@ -81,6 +82,8 @@ namespace TestCaseTracking {
using Children = std::vector<ITrackerPtr>;
virtual bool isFilteredImpl() const = 0;
protected:
enum CycleState {
NotStarted,
@@ -178,6 +181,20 @@ namespace TestCaseTracking {
* for internal debug checks.
*/
virtual bool isGeneratorTracker() const;
/**
* Returns true if the concrete tracker instance has a filter that applies to it.
*/
bool isFiltered() const {
// Fast path: are there even filters for tracker in this position?
const size_t filter_depth =
m_newStyleFilters ? m_allTrackerDepth : m_sectionOnlyDepth;
if ( m_filterRef->size() <= filter_depth ) { return false; }
// Slow path: If there are filters, ask the concrete tracker.
// This handles things like match-all filters for that tracker.
return isFilteredImpl();
}
};
class TrackerContext {
@@ -233,6 +250,8 @@ namespace TestCaseTracking {
// to not own the name, the name still has to outlive the `ITracker` parent, so
// this should still be safe.
StringRef m_trimmed_name;
bool isFilteredImpl() const override;
public:
SectionTracker( NameAndLocation&& nameAndLocation, TrackerContext& ctx, ITracker* parent );
+14
View File
@@ -625,6 +625,20 @@ if(CATCH_ENABLE_CMAKE_HELPER_TESTS)
COST 240
LABELS "uses-python"
)
add_test(NAME "CMakeHelper::PrepareCommand"
COMMAND
"${CMAKE_COMMAND}"
"-DCATCH_ADD_TESTS_SCRIPT=${CATCH_DIR}/extras/CatchAddTests.cmake"
-P "${CMAKE_CURRENT_LIST_DIR}/TestScripts/DiscoverTests/TestPrepareCommand.cmake"
)
add_test(NAME "CMakeHelper::DecomposeJsonArray"
COMMAND
"${CMAKE_COMMAND}"
"-DCATCH_ADD_TESTS_SCRIPT=${CATCH_DIR}/extras/CatchAddTests.cmake"
-P "${CMAKE_CURRENT_LIST_DIR}/TestScripts/DiscoverTests/TestDecomposeJsonArray.cmake"
)
endif()
foreach(reporterName # "Automake" - the simple .trs format does not support any kind of comments/metadata
+80 -24
View File
@@ -526,29 +526,18 @@ set_tests_properties(TestSpecs::SkippingAllTestsFails
WILL_FAIL ON
)
set(EXTRA_TEST_BINARIES
AllSkipped
PrefixedMacros
DisabledMacros
DisabledExceptions-DefaultHandler
DisabledExceptions-CustomHandler
FallbackStringifier
DisableStringification
PartialTestCaseEvents
DuplicatedTestCases-SameNameAndTags
DuplicatedTestCases-SameNameDifferentTags
DuplicatedTestCases-DuplicatedTestCaseMethods
NoTests
ListenersGetEventsBeforeReporters
MixingClearedAndUnclearedMessages
# DebugBreakMacros
)
add_executable(FastCompileMacros ${TESTS_DIR}/X07-FastCompileMacros.cpp)
target_link_libraries(FastCompileMacros PRIVATE Catch2_buildall_interface)
target_compile_definitions(FastCompileMacros PRIVATE CATCH_CONFIG_FAST_COMPILE)
# Notice that we are modifying EXTRA_TEST_BINARIES destructively, do not
# use it after this point!
list(FILTER EXTRA_TEST_BINARIES EXCLUDE REGEX "DisabledExceptions.*")
list(APPEND CATCH_TEST_TARGETS ${EXTRA_TEST_BINARIES})
set(CATCH_TEST_TARGETS ${CATCH_TEST_TARGETS} PARENT_SCOPE)
add_test(
NAME CompileConfiguration::FastCompile
COMMAND $<TARGET_FILE:FastCompileMacros>
)
set_tests_properties(CompileConfiguration::FastCompile
PROPERTIES
PASS_REGULAR_EXPRESSION "test cases: 6 \\| 1 passed \\| 1 failed \\| 4 failed as expected\nassertions: 13 \\| 6 passed \\| 2 failed \\| 5 failed as expected"
)
# This sets up a one-off executable that compiles against the amalgamated
# files, and then runs it for a super simple check that the amalgamated
@@ -594,12 +583,79 @@ add_executable(InfiniteGenerators ${TESTS_DIR}/X95-InfiniteGenerators.cpp)
target_link_libraries(InfiniteGenerators PRIVATE Catch2::Catch2WithMain)
add_test(
NAME Warnings::InfiniteGenerators
NAME Warnings::InfiniteGenerators::NoFilterWarns
COMMAND $<TARGET_FILE:InfiniteGenerators> --warn InfiniteGenerators
)
set_tests_properties(Warnings::InfiniteGenerators
set_tests_properties(Warnings::InfiniteGenerators::NoFilterWarns
PROPERTIES
# One test case fails with infinite generator, but the other one runs
PASS_REGULAR_EXPRESSION "test cases: 2 \\| 1 passed \\| 1 failed"
TIMEOUT 5
)
add_test(
NAME Warnings::InfiniteGenerators::MatchAllFilterWarns
COMMAND $<TARGET_FILE:InfiniteGenerators>
--warn InfiniteGenerators
--path-filter g:*
)
set_tests_properties(Warnings::InfiniteGenerators::MatchAllFilterWarns
PROPERTIES
# One test case fails with infinite generator, but the other one runs
PASS_REGULAR_EXPRESSION "test cases: 2 \\| 1 passed \\| 1 failed"
TIMEOUT 5
)
add_test(
NAME Warnings::InfiniteGenerators::MatchOneFilterDoesntWarn
COMMAND $<TARGET_FILE:InfiniteGenerators>
--warn InfiniteGenerators
--path-filter g:1
)
set_tests_properties(Warnings::InfiniteGenerators::MatchOneFilterDoesntWarn
PROPERTIES
# One test case fails with infinite generator, but the other one runs
PASS_REGULAR_EXPRESSION "All tests passed \\(1 assertion in 2 test cases\\)"
TIMEOUT 5
)
add_test(
NAME Warnings::InfiniteGenerators::SectionFilterWarns
COMMAND $<TARGET_FILE:InfiniteGenerators>
--warn InfiniteGenerators
--section FooBarBaz
)
set_tests_properties(Warnings::InfiniteGenerators::SectionFilterWarns
PROPERTIES
# One test case fails with infinite generator, but the other one runs
PASS_REGULAR_EXPRESSION "test cases: 2 \\| 1 passed \\| 1 failed"
TIMEOUT 5
)
set(EXTRA_TEST_BINARIES
AllSkipped
PrefixedMacros
DisabledMacros
DisabledExceptions-DefaultHandler
DisabledExceptions-CustomHandler
FallbackStringifier
DisableStringification
PartialTestCaseEvents
DuplicatedTestCases-SameNameAndTags
DuplicatedTestCases-SameNameDifferentTags
DuplicatedTestCases-DuplicatedTestCaseMethods
NoTests
ListenersGetEventsBeforeReporters
MixingClearedAndUnclearedMessages
FastCompileMacros
InfiniteGenerators
ThreadSafetyTests
# DebugBreakMacros
)
# Notice that we are modifying EXTRA_TEST_BINARIES destructively, do not
# use it after this point!
list(FILTER EXTRA_TEST_BINARIES EXCLUDE REGEX "DisabledExceptions.*")
list(APPEND CATCH_TEST_TARGETS ${EXTRA_TEST_BINARIES})
set(CATCH_TEST_TARGETS ${CATCH_TEST_TARGETS} PARENT_SCOPE)
-1
View File
@@ -3,7 +3,6 @@ yet:
CATCH_CONFIG_COUNTER // Use __COUNTER__ to generate unique names for test cases
CATCH_CONFIG_WINDOWS_SEH // Enable SEH handling on Windows
CATCH_CONFIG_FAST_COMPILE // Sacrifices some (rather minor) features for compilation speed
CATCH_CONFIG_POSIX_SIGNALS // Enable handling POSIX signals
CATCH_CONFIG_WINDOWS_CRTDBG // Enable leak checking using Windows's CRT Debug Heap
CATCH_CONFIG_DEFAULT_REPORTER
@@ -0,0 +1,63 @@
// Copyright Catch2 Authors
// Distributed under the Boost Software License, Version 1.0.
// (See accompanying file LICENSE.txt or copy at
// https://www.boost.org/LICENSE_1_0.txt)
// SPDX-License-Identifier: BSL-1.0
/**\file
* Test that various basic macros work with CATCH_CONFIG_FAST_COMPILE.
*
* Note that the current checking is rather loose. We check that the
* macros compile, and that the test cases (don't) fail as they are
* supposed to.
*/
#include <catch2/catch_test_macros.hpp>
#include <stdexcept>
namespace {
[[noreturn]]
static void throws() {
throw std::runtime_error{ "sup" };
}
static void doesnt_throw() {}
[[noreturn]]
static int throws_i() {
throw std::runtime_error{ "sup" };
}
} // namespace
TEST_CASE( "Passing macros work" ) {
REQUIRE( 1 != 2 );
CHECK( 2 == 2 );
REQUIRE_THROWS( throws() );
REQUIRE_NOTHROW( doesnt_throw() );
}
TEST_CASE( "Failing macros work", "[!shouldfail]" ) {
CHECK( 1 != 2 );
CHECK( 2 == 2 );
CHECK( 3 == 2 );
}
TEST_CASE( "Failing NOTHROW works", "[!shouldfail]" ) {
REQUIRE_NOTHROW( throws() );
}
TEST_CASE( "Failing THROW works", "[!shouldfail]" ) {
REQUIRE_THROWS( doesnt_throw() );
}
TEST_CASE( "Unexpected exception in REQUIRE gets inverted properly",
"[!shouldfail]" ) {
REQUIRE( throws_i() == 1 );
}
TEST_CASE( "Unexpected exception in REQUIRE fails properly" ) {
REQUIRE( throws_i() == 2 );
}
@@ -33,6 +33,7 @@ namespace {
TEST_CASE() {
auto _ = GENERATE( make_infinite_generator() );
(void)_;
}
TEST_CASE() {
@@ -190,6 +190,9 @@ Nor would this
:test-result: PASS Inequality checks that should succeed
:test-result: PASS JsonWriter
:test-result: PASS JsonWriter escapes characters in strings properly
:test-result: PASS JsonWriter serializes non-finite FP using Python spelling - double
:test-result: PASS JsonWriter serializes non-finite FP using Python spelling - float
:test-result: PASS JsonWriter serializes numbers independently of the global locale
:test-result: PASS Lambdas in assertions
:test-result: PASS Less-than inequalities with different epsilons
:test-result: PASS ManuallyRegistered
@@ -188,6 +188,9 @@
:test-result: PASS Inequality checks that should succeed
:test-result: PASS JsonWriter
:test-result: PASS JsonWriter escapes characters in strings properly
:test-result: PASS JsonWriter serializes non-finite FP using Python spelling - double
:test-result: PASS JsonWriter serializes non-finite FP using Python spelling - float
:test-result: PASS JsonWriter serializes numbers independently of the global locale
:test-result: PASS Lambdas in assertions
:test-result: PASS Less-than inequalities with different epsilons
:test-result: PASS ManuallyRegistered
@@ -1248,6 +1248,26 @@ Json.tests.cpp:<line number>: passed: sstream.str() == "\"\\n\"" for: ""\n"" ==
Json.tests.cpp:<line number>: passed: sstream.str() == "\"\\r\"" for: ""\r"" == ""\r""
Json.tests.cpp:<line number>: passed: sstream.str() == "\"\\t\"" for: ""\t"" == ""\t""
Json.tests.cpp:<line number>: passed: sstream.str() == "\"\\\\/\\t\\r\\n\"" for: ""\\/\t\r\n"" == ""\\/\t\r\n""
Json.tests.cpp:<line number>: passed: sstream.str() == "NaN" for: "NaN" == "NaN"
Json.tests.cpp:<line number>: passed: sstream.str() == "Infinity" for: "Infinity" == "Infinity"
Json.tests.cpp:<line number>: passed: sstream.str() == "-Infinity" for: "-Infinity" == "-Infinity"
Json.tests.cpp:<line number>: passed: sstream.str() == "NaN" for: "NaN" == "NaN"
Json.tests.cpp:<line number>: passed: sstream.str() == "Infinity" for: "Infinity" == "Infinity"
Json.tests.cpp:<line number>: passed: sstream.str() == "-Infinity" for: "-Infinity" == "-Infinity"
Json.tests.cpp:<line number>: passed: sstream.str(), ContainsSubstring( "\"double\": 1.5," ) && ContainsSubstring( "\"int\": 1234567," ) && ContainsSubstring( "\"bool-1\": true," ) && ContainsSubstring( "\"bool-2\": false," ) && ContainsSubstring( "\"array\": [\n 2.5,\n 1234567\n ]\n}" ) for: "{
"double": 1.5,
"int": 1234567,
"bool-1": true,
"bool-2": false,
"array": [
2.5,
1234567
]
}" ( contains: ""double": 1.5," and contains: ""int": 1234567," and contains: ""bool-1": true," and contains: ""bool-2": false," and contains: ""array": [
2.5,
1234567
]
}" )
Compilation.tests.cpp:<line number>: passed: []() { return true; }() for: true
Approx.tests.cpp:<line number>: passed: d <= Approx( 1.24 ) for: 1.22999999999999998
<=
@@ -3000,7 +3020,7 @@ InternalBenchmark.tests.cpp:<line number>: passed: med == 18. for: 18.0 == 18.0
InternalBenchmark.tests.cpp:<line number>: passed: q3 == 23. for: 23.0 == 23.0
Misc.tests.cpp:<line number>: passed:
Misc.tests.cpp:<line number>: passed:
test cases: 451 | 331 passed | 96 failed | 6 skipped | 18 failed as expected
assertions: 2416 | 2215 passed | 158 failed | 43 failed as expected
test cases: 454 | 334 passed | 96 failed | 6 skipped | 18 failed as expected
assertions: 2423 | 2222 passed | 158 failed | 43 failed as expected
@@ -1246,6 +1246,26 @@ Json.tests.cpp:<line number>: passed: sstream.str() == "\"\\n\"" for: ""\n"" ==
Json.tests.cpp:<line number>: passed: sstream.str() == "\"\\r\"" for: ""\r"" == ""\r""
Json.tests.cpp:<line number>: passed: sstream.str() == "\"\\t\"" for: ""\t"" == ""\t""
Json.tests.cpp:<line number>: passed: sstream.str() == "\"\\\\/\\t\\r\\n\"" for: ""\\/\t\r\n"" == ""\\/\t\r\n""
Json.tests.cpp:<line number>: passed: sstream.str() == "NaN" for: "NaN" == "NaN"
Json.tests.cpp:<line number>: passed: sstream.str() == "Infinity" for: "Infinity" == "Infinity"
Json.tests.cpp:<line number>: passed: sstream.str() == "-Infinity" for: "-Infinity" == "-Infinity"
Json.tests.cpp:<line number>: passed: sstream.str() == "NaN" for: "NaN" == "NaN"
Json.tests.cpp:<line number>: passed: sstream.str() == "Infinity" for: "Infinity" == "Infinity"
Json.tests.cpp:<line number>: passed: sstream.str() == "-Infinity" for: "-Infinity" == "-Infinity"
Json.tests.cpp:<line number>: passed: sstream.str(), ContainsSubstring( "\"double\": 1.5," ) && ContainsSubstring( "\"int\": 1234567," ) && ContainsSubstring( "\"bool-1\": true," ) && ContainsSubstring( "\"bool-2\": false," ) && ContainsSubstring( "\"array\": [\n 2.5,\n 1234567\n ]\n}" ) for: "{
"double": 1.5,
"int": 1234567,
"bool-1": true,
"bool-2": false,
"array": [
2.5,
1234567
]
}" ( contains: ""double": 1.5," and contains: ""int": 1234567," and contains: ""bool-1": true," and contains: ""bool-2": false," and contains: ""array": [
2.5,
1234567
]
}" )
Compilation.tests.cpp:<line number>: passed: []() { return true; }() for: true
Approx.tests.cpp:<line number>: passed: d <= Approx( 1.24 ) for: 1.22999999999999998
<=
@@ -2989,7 +3009,7 @@ InternalBenchmark.tests.cpp:<line number>: passed: med == 18. for: 18.0 == 18.0
InternalBenchmark.tests.cpp:<line number>: passed: q3 == 23. for: 23.0 == 23.0
Misc.tests.cpp:<line number>: passed:
Misc.tests.cpp:<line number>: passed:
test cases: 451 | 331 passed | 96 failed | 6 skipped | 18 failed as expected
assertions: 2416 | 2215 passed | 158 failed | 43 failed as expected
test cases: 454 | 334 passed | 96 failed | 6 skipped | 18 failed as expected
assertions: 2423 | 2222 passed | 158 failed | 43 failed as expected
@@ -1743,6 +1743,6 @@ due to unexpected exception with message:
Why would you throw a std::string?
===============================================================================
test cases: 451 | 349 passed | 76 failed | 7 skipped | 19 failed as expected
assertions: 2394 | 2215 passed | 136 failed | 43 failed as expected
test cases: 454 | 352 passed | 76 failed | 7 skipped | 19 failed as expected
assertions: 2401 | 2222 passed | 136 failed | 43 failed as expected
@@ -8219,6 +8219,103 @@ Json.tests.cpp:<line number>: PASSED:
with expansion:
""\\/\t\r\n"" == ""\\/\t\r\n""
-------------------------------------------------------------------------------
JsonWriter serializes non-finite FP using Python spelling - double
NaN
-------------------------------------------------------------------------------
Json.tests.cpp:<line number>
...............................................................................
Json.tests.cpp:<line number>: PASSED:
REQUIRE( sstream.str() == "NaN" )
with expansion:
"NaN" == "NaN"
-------------------------------------------------------------------------------
JsonWriter serializes non-finite FP using Python spelling - double
Pos nf
-------------------------------------------------------------------------------
Json.tests.cpp:<line number>
...............................................................................
Json.tests.cpp:<line number>: PASSED:
REQUIRE( sstream.str() == "Infinity" )
with expansion:
"Infinity" == "Infinity"
-------------------------------------------------------------------------------
JsonWriter serializes non-finite FP using Python spelling - double
Neg inf
-------------------------------------------------------------------------------
Json.tests.cpp:<line number>
...............................................................................
Json.tests.cpp:<line number>: PASSED:
REQUIRE( sstream.str() == "-Infinity" )
with expansion:
"-Infinity" == "-Infinity"
-------------------------------------------------------------------------------
JsonWriter serializes non-finite FP using Python spelling - float
NaN
-------------------------------------------------------------------------------
Json.tests.cpp:<line number>
...............................................................................
Json.tests.cpp:<line number>: PASSED:
REQUIRE( sstream.str() == "NaN" )
with expansion:
"NaN" == "NaN"
-------------------------------------------------------------------------------
JsonWriter serializes non-finite FP using Python spelling - float
Pos nf
-------------------------------------------------------------------------------
Json.tests.cpp:<line number>
...............................................................................
Json.tests.cpp:<line number>: PASSED:
REQUIRE( sstream.str() == "Infinity" )
with expansion:
"Infinity" == "Infinity"
-------------------------------------------------------------------------------
JsonWriter serializes non-finite FP using Python spelling - float
Neg inf
-------------------------------------------------------------------------------
Json.tests.cpp:<line number>
...............................................................................
Json.tests.cpp:<line number>: PASSED:
REQUIRE( sstream.str() == "-Infinity" )
with expansion:
"-Infinity" == "-Infinity"
-------------------------------------------------------------------------------
JsonWriter serializes numbers independently of the global locale
-------------------------------------------------------------------------------
Json.tests.cpp:<line number>
...............................................................................
Json.tests.cpp:<line number>: PASSED:
REQUIRE_THAT( sstream.str(), ContainsSubstring( "\"double\": 1.5," ) && ContainsSubstring( "\"int\": 1234567," ) && ContainsSubstring( "\"bool-1\": true," ) && ContainsSubstring( "\"bool-2\": false," ) && ContainsSubstring( "\"array\": [\n 2.5,\n 1234567\n ]\n}" ) )
with expansion:
"{
"double": 1.5,
"int": 1234567,
"bool-1": true,
"bool-2": false,
"array": [
2.5,
1234567
]
}" ( contains: ""double": 1.5," and contains: ""int": 1234567," and contains:
""bool-1": true," and contains: ""bool-2": false," and contains: ""array": [
2.5,
1234567
]
}" )
-------------------------------------------------------------------------------
Lambdas in assertions
-------------------------------------------------------------------------------
@@ -20134,6 +20231,6 @@ Misc.tests.cpp:<line number>
Misc.tests.cpp:<line number>: PASSED:
===============================================================================
test cases: 451 | 331 passed | 96 failed | 6 skipped | 18 failed as expected
assertions: 2416 | 2215 passed | 158 failed | 43 failed as expected
test cases: 454 | 334 passed | 96 failed | 6 skipped | 18 failed as expected
assertions: 2423 | 2222 passed | 158 failed | 43 failed as expected
@@ -8217,6 +8217,103 @@ Json.tests.cpp:<line number>: PASSED:
with expansion:
""\\/\t\r\n"" == ""\\/\t\r\n""
-------------------------------------------------------------------------------
JsonWriter serializes non-finite FP using Python spelling - double
NaN
-------------------------------------------------------------------------------
Json.tests.cpp:<line number>
...............................................................................
Json.tests.cpp:<line number>: PASSED:
REQUIRE( sstream.str() == "NaN" )
with expansion:
"NaN" == "NaN"
-------------------------------------------------------------------------------
JsonWriter serializes non-finite FP using Python spelling - double
Pos nf
-------------------------------------------------------------------------------
Json.tests.cpp:<line number>
...............................................................................
Json.tests.cpp:<line number>: PASSED:
REQUIRE( sstream.str() == "Infinity" )
with expansion:
"Infinity" == "Infinity"
-------------------------------------------------------------------------------
JsonWriter serializes non-finite FP using Python spelling - double
Neg inf
-------------------------------------------------------------------------------
Json.tests.cpp:<line number>
...............................................................................
Json.tests.cpp:<line number>: PASSED:
REQUIRE( sstream.str() == "-Infinity" )
with expansion:
"-Infinity" == "-Infinity"
-------------------------------------------------------------------------------
JsonWriter serializes non-finite FP using Python spelling - float
NaN
-------------------------------------------------------------------------------
Json.tests.cpp:<line number>
...............................................................................
Json.tests.cpp:<line number>: PASSED:
REQUIRE( sstream.str() == "NaN" )
with expansion:
"NaN" == "NaN"
-------------------------------------------------------------------------------
JsonWriter serializes non-finite FP using Python spelling - float
Pos nf
-------------------------------------------------------------------------------
Json.tests.cpp:<line number>
...............................................................................
Json.tests.cpp:<line number>: PASSED:
REQUIRE( sstream.str() == "Infinity" )
with expansion:
"Infinity" == "Infinity"
-------------------------------------------------------------------------------
JsonWriter serializes non-finite FP using Python spelling - float
Neg inf
-------------------------------------------------------------------------------
Json.tests.cpp:<line number>
...............................................................................
Json.tests.cpp:<line number>: PASSED:
REQUIRE( sstream.str() == "-Infinity" )
with expansion:
"-Infinity" == "-Infinity"
-------------------------------------------------------------------------------
JsonWriter serializes numbers independently of the global locale
-------------------------------------------------------------------------------
Json.tests.cpp:<line number>
...............................................................................
Json.tests.cpp:<line number>: PASSED:
REQUIRE_THAT( sstream.str(), ContainsSubstring( "\"double\": 1.5," ) && ContainsSubstring( "\"int\": 1234567," ) && ContainsSubstring( "\"bool-1\": true," ) && ContainsSubstring( "\"bool-2\": false," ) && ContainsSubstring( "\"array\": [\n 2.5,\n 1234567\n ]\n}" ) )
with expansion:
"{
"double": 1.5,
"int": 1234567,
"bool-1": true,
"bool-2": false,
"array": [
2.5,
1234567
]
}" ( contains: ""double": 1.5," and contains: ""int": 1234567," and contains:
""bool-1": true," and contains: ""bool-2": false," and contains: ""array": [
2.5,
1234567
]
}" )
-------------------------------------------------------------------------------
Lambdas in assertions
-------------------------------------------------------------------------------
@@ -20123,6 +20220,6 @@ Misc.tests.cpp:<line number>
Misc.tests.cpp:<line number>: PASSED:
===============================================================================
test cases: 451 | 331 passed | 96 failed | 6 skipped | 18 failed as expected
assertions: 2416 | 2215 passed | 158 failed | 43 failed as expected
test cases: 454 | 334 passed | 96 failed | 6 skipped | 18 failed as expected
assertions: 2423 | 2222 passed | 158 failed | 43 failed as expected
+10 -1
View File
@@ -1,7 +1,7 @@
<?xml version="1.0" encoding="UTF-8"?>
<testsuitesloose text artifact
>
<testsuite name="<exe-name>" errors="17" failures="141" skipped="12" tests="2428" hostname="tbd" time="{duration}" timestamp="{iso8601-timestamp}">
<testsuite name="<exe-name>" errors="17" failures="141" skipped="12" tests="2435" hostname="tbd" time="{duration}" timestamp="{iso8601-timestamp}">
<properties>
<property name="random-seed" value="1"/>
<property name="filters" value="&quot;*&quot; ~[!nonportable] ~[!benchmark] ~[approvals]"/>
@@ -774,6 +774,15 @@ at Condition.tests.cpp:<line number>
<testcase classname="<exe-name>.global" name="JsonWriter escapes characters in strings properly/carriage return in a string is escaped" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="JsonWriter escapes characters in strings properly/tab in a string is escaped" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="JsonWriter escapes characters in strings properly/combination of characters is escaped" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="JsonWriter serializes non-finite FP using Python spelling - double" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="JsonWriter serializes non-finite FP using Python spelling - double/NaN" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="JsonWriter serializes non-finite FP using Python spelling - double/Pos nf" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="JsonWriter serializes non-finite FP using Python spelling - double/Neg inf" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="JsonWriter serializes non-finite FP using Python spelling - float" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="JsonWriter serializes non-finite FP using Python spelling - float/NaN" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="JsonWriter serializes non-finite FP using Python spelling - float/Pos nf" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="JsonWriter serializes non-finite FP using Python spelling - float/Neg inf" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="JsonWriter serializes numbers independently of the global locale" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Lambdas in assertions" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Less-than inequalities with different epsilons" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="ManuallyRegistered" time="{duration}" status="run"/>
@@ -1,6 +1,6 @@
<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
<testsuite name="<exe-name>" errors="17" failures="141" skipped="12" tests="2428" hostname="tbd" time="{duration}" timestamp="{iso8601-timestamp}">
<testsuite name="<exe-name>" errors="17" failures="141" skipped="12" tests="2435" hostname="tbd" time="{duration}" timestamp="{iso8601-timestamp}">
<properties>
<property name="random-seed" value="1"/>
<property name="filters" value="&quot;*&quot; ~[!nonportable] ~[!benchmark] ~[approvals]"/>
@@ -773,6 +773,15 @@ at Condition.tests.cpp:<line number>
<testcase classname="<exe-name>.global" name="JsonWriter escapes characters in strings properly/carriage return in a string is escaped" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="JsonWriter escapes characters in strings properly/tab in a string is escaped" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="JsonWriter escapes characters in strings properly/combination of characters is escaped" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="JsonWriter serializes non-finite FP using Python spelling - double" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="JsonWriter serializes non-finite FP using Python spelling - double/NaN" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="JsonWriter serializes non-finite FP using Python spelling - double/Pos nf" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="JsonWriter serializes non-finite FP using Python spelling - double/Neg inf" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="JsonWriter serializes non-finite FP using Python spelling - float" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="JsonWriter serializes non-finite FP using Python spelling - float/NaN" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="JsonWriter serializes non-finite FP using Python spelling - float/Pos nf" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="JsonWriter serializes non-finite FP using Python spelling - float/Neg inf" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="JsonWriter serializes numbers independently of the global locale" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Lambdas in assertions" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Less-than inequalities with different epsilons" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="ManuallyRegistered" time="{duration}" status="run"/>
@@ -262,6 +262,15 @@ at AssertionHandler.tests.cpp:<line number>
<testCase name="JsonWriter escapes characters in strings properly/carriage return in a string is escaped" duration="{duration}"/>
<testCase name="JsonWriter escapes characters in strings properly/tab in a string is escaped" duration="{duration}"/>
<testCase name="JsonWriter escapes characters in strings properly/combination of characters is escaped" duration="{duration}"/>
<testCase name="JsonWriter serializes non-finite FP using Python spelling - double" duration="{duration}"/>
<testCase name="JsonWriter serializes non-finite FP using Python spelling - double/NaN" duration="{duration}"/>
<testCase name="JsonWriter serializes non-finite FP using Python spelling - double/Pos nf" duration="{duration}"/>
<testCase name="JsonWriter serializes non-finite FP using Python spelling - double/Neg inf" duration="{duration}"/>
<testCase name="JsonWriter serializes non-finite FP using Python spelling - float" duration="{duration}"/>
<testCase name="JsonWriter serializes non-finite FP using Python spelling - float/NaN" duration="{duration}"/>
<testCase name="JsonWriter serializes non-finite FP using Python spelling - float/Pos nf" duration="{duration}"/>
<testCase name="JsonWriter serializes non-finite FP using Python spelling - float/Neg inf" duration="{duration}"/>
<testCase name="JsonWriter serializes numbers independently of the global locale" duration="{duration}"/>
</file>
<file path="tests/<exe-name>/IntrospectiveTests/Parse.tests.cpp">
<testCase name="Parse uints" duration="{duration}"/>
@@ -261,6 +261,15 @@ at AssertionHandler.tests.cpp:<line number>
<testCase name="JsonWriter escapes characters in strings properly/carriage return in a string is escaped" duration="{duration}"/>
<testCase name="JsonWriter escapes characters in strings properly/tab in a string is escaped" duration="{duration}"/>
<testCase name="JsonWriter escapes characters in strings properly/combination of characters is escaped" duration="{duration}"/>
<testCase name="JsonWriter serializes non-finite FP using Python spelling - double" duration="{duration}"/>
<testCase name="JsonWriter serializes non-finite FP using Python spelling - double/NaN" duration="{duration}"/>
<testCase name="JsonWriter serializes non-finite FP using Python spelling - double/Pos nf" duration="{duration}"/>
<testCase name="JsonWriter serializes non-finite FP using Python spelling - double/Neg inf" duration="{duration}"/>
<testCase name="JsonWriter serializes non-finite FP using Python spelling - float" duration="{duration}"/>
<testCase name="JsonWriter serializes non-finite FP using Python spelling - float/NaN" duration="{duration}"/>
<testCase name="JsonWriter serializes non-finite FP using Python spelling - float/Pos nf" duration="{duration}"/>
<testCase name="JsonWriter serializes non-finite FP using Python spelling - float/Neg inf" duration="{duration}"/>
<testCase name="JsonWriter serializes numbers independently of the global locale" duration="{duration}"/>
</file>
<file path="tests/<exe-name>/IntrospectiveTests/Parse.tests.cpp">
<testCase name="Parse uints" duration="{duration}"/>
+15 -1
View File
@@ -2054,6 +2054,20 @@ ok {test-number} - sstream.str() == "\"\\r\"" for: ""\r"" == ""\r""
ok {test-number} - sstream.str() == "\"\\t\"" for: ""\t"" == ""\t""
# JsonWriter escapes characters in strings properly
ok {test-number} - sstream.str() == "\"\\\\/\\t\\r\\n\"" for: ""\\/\t\r\n"" == ""\\/\t\r\n""
# JsonWriter serializes non-finite FP using Python spelling - double
ok {test-number} - sstream.str() == "NaN" for: "NaN" == "NaN"
# JsonWriter serializes non-finite FP using Python spelling - double
ok {test-number} - sstream.str() == "Infinity" for: "Infinity" == "Infinity"
# JsonWriter serializes non-finite FP using Python spelling - double
ok {test-number} - sstream.str() == "-Infinity" for: "-Infinity" == "-Infinity"
# JsonWriter serializes non-finite FP using Python spelling - float
ok {test-number} - sstream.str() == "NaN" for: "NaN" == "NaN"
# JsonWriter serializes non-finite FP using Python spelling - float
ok {test-number} - sstream.str() == "Infinity" for: "Infinity" == "Infinity"
# JsonWriter serializes non-finite FP using Python spelling - float
ok {test-number} - sstream.str() == "-Infinity" for: "-Infinity" == "-Infinity"
# JsonWriter serializes numbers independently of the global locale
ok {test-number} - sstream.str(), ContainsSubstring( "\"double\": 1.5," ) && ContainsSubstring( "\"int\": 1234567," ) && ContainsSubstring( "\"bool-1\": true," ) && ContainsSubstring( "\"bool-2\": false," ) && ContainsSubstring( "\"array\": [\n 2.5,\n 1234567\n ]\n}" ) for: "{ "double": 1.5, "int": 1234567, "bool-1": true, "bool-2": false, "array": [ 2.5, 1234567 ] }" ( contains: ""double": 1.5," and contains: ""int": 1234567," and contains: ""bool-1": true," and contains: ""bool-2": false," and contains: ""array": [ 2.5, 1234567 ] }" )
# Lambdas in assertions
ok {test-number} - []() { return true; }() for: true
# Less-than inequalities with different epsilons
@@ -4851,5 +4865,5 @@ ok {test-number} - q3 == 23. for: 23.0 == 23.0
ok {test-number} -
# xmlentitycheck
ok {test-number} -
1..2428
1..2435
@@ -2052,6 +2052,20 @@ ok {test-number} - sstream.str() == "\"\\r\"" for: ""\r"" == ""\r""
ok {test-number} - sstream.str() == "\"\\t\"" for: ""\t"" == ""\t""
# JsonWriter escapes characters in strings properly
ok {test-number} - sstream.str() == "\"\\\\/\\t\\r\\n\"" for: ""\\/\t\r\n"" == ""\\/\t\r\n""
# JsonWriter serializes non-finite FP using Python spelling - double
ok {test-number} - sstream.str() == "NaN" for: "NaN" == "NaN"
# JsonWriter serializes non-finite FP using Python spelling - double
ok {test-number} - sstream.str() == "Infinity" for: "Infinity" == "Infinity"
# JsonWriter serializes non-finite FP using Python spelling - double
ok {test-number} - sstream.str() == "-Infinity" for: "-Infinity" == "-Infinity"
# JsonWriter serializes non-finite FP using Python spelling - float
ok {test-number} - sstream.str() == "NaN" for: "NaN" == "NaN"
# JsonWriter serializes non-finite FP using Python spelling - float
ok {test-number} - sstream.str() == "Infinity" for: "Infinity" == "Infinity"
# JsonWriter serializes non-finite FP using Python spelling - float
ok {test-number} - sstream.str() == "-Infinity" for: "-Infinity" == "-Infinity"
# JsonWriter serializes numbers independently of the global locale
ok {test-number} - sstream.str(), ContainsSubstring( "\"double\": 1.5," ) && ContainsSubstring( "\"int\": 1234567," ) && ContainsSubstring( "\"bool-1\": true," ) && ContainsSubstring( "\"bool-2\": false," ) && ContainsSubstring( "\"array\": [\n 2.5,\n 1234567\n ]\n}" ) for: "{ "double": 1.5, "int": 1234567, "bool-1": true, "bool-2": false, "array": [ 2.5, 1234567 ] }" ( contains: ""double": 1.5," and contains: ""int": 1234567," and contains: ""bool-1": true," and contains: ""bool-2": false," and contains: ""array": [ 2.5, 1234567 ] }" )
# Lambdas in assertions
ok {test-number} - []() { return true; }() for: true
# Less-than inequalities with different epsilons
@@ -4840,5 +4854,5 @@ ok {test-number} - q3 == 23. for: 23.0 == 23.0
ok {test-number} -
# xmlentitycheck
ok {test-number} -
1..2428
1..2435
@@ -470,6 +470,12 @@
##teamcity[testFinished name='JsonWriter' duration="{duration}"]
##teamcity[testStarted name='JsonWriter escapes characters in strings properly']
##teamcity[testFinished name='JsonWriter escapes characters in strings properly' duration="{duration}"]
##teamcity[testStarted name='JsonWriter serializes non-finite FP using Python spelling - double']
##teamcity[testFinished name='JsonWriter serializes non-finite FP using Python spelling - double' duration="{duration}"]
##teamcity[testStarted name='JsonWriter serializes non-finite FP using Python spelling - float']
##teamcity[testFinished name='JsonWriter serializes non-finite FP using Python spelling - float' duration="{duration}"]
##teamcity[testStarted name='JsonWriter serializes numbers independently of the global locale']
##teamcity[testFinished name='JsonWriter serializes numbers independently of the global locale' duration="{duration}"]
##teamcity[testStarted name='Lambdas in assertions']
##teamcity[testFinished name='Lambdas in assertions' duration="{duration}"]
##teamcity[testStarted name='Less-than inequalities with different epsilons']
@@ -470,6 +470,12 @@
##teamcity[testFinished name='JsonWriter' duration="{duration}"]
##teamcity[testStarted name='JsonWriter escapes characters in strings properly']
##teamcity[testFinished name='JsonWriter escapes characters in strings properly' duration="{duration}"]
##teamcity[testStarted name='JsonWriter serializes non-finite FP using Python spelling - double']
##teamcity[testFinished name='JsonWriter serializes non-finite FP using Python spelling - double' duration="{duration}"]
##teamcity[testStarted name='JsonWriter serializes non-finite FP using Python spelling - float']
##teamcity[testFinished name='JsonWriter serializes non-finite FP using Python spelling - float' duration="{duration}"]
##teamcity[testStarted name='JsonWriter serializes numbers independently of the global locale']
##teamcity[testFinished name='JsonWriter serializes numbers independently of the global locale' duration="{duration}"]
##teamcity[testStarted name='Lambdas in assertions']
##teamcity[testFinished name='Lambdas in assertions' duration="{duration}"]
##teamcity[testStarted name='Less-than inequalities with different epsilons']
+98 -2
View File
@@ -9894,6 +9894,102 @@ Approx( 3.14150000000000018 )
</Section>
<OverallResult success="true" skips="0"/>
</TestCase>
<TestCase name="JsonWriter serializes non-finite FP using Python spelling - double" tags="[floating-point][JsonWriter]" filename="tests/<exe-name>/IntrospectiveTests/Json.tests.cpp" >
<Section name="NaN" filename="tests/<exe-name>/IntrospectiveTests/Json.tests.cpp" >
<Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/Json.tests.cpp" >
<Original>
sstream.str() == "NaN"
</Original>
<Expanded>
"NaN" == "NaN"
</Expanded>
</Expression>
<OverallResults successes="1" failures="0" expectedFailures="0" skipped="false"/>
</Section>
<Section name="Pos nf" filename="tests/<exe-name>/IntrospectiveTests/Json.tests.cpp" >
<Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/Json.tests.cpp" >
<Original>
sstream.str() == "Infinity"
</Original>
<Expanded>
"Infinity" == "Infinity"
</Expanded>
</Expression>
<OverallResults successes="1" failures="0" expectedFailures="0" skipped="false"/>
</Section>
<Section name="Neg inf" filename="tests/<exe-name>/IntrospectiveTests/Json.tests.cpp" >
<Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/Json.tests.cpp" >
<Original>
sstream.str() == "-Infinity"
</Original>
<Expanded>
"-Infinity" == "-Infinity"
</Expanded>
</Expression>
<OverallResults successes="1" failures="0" expectedFailures="0" skipped="false"/>
</Section>
<OverallResult success="true" skips="0"/>
</TestCase>
<TestCase name="JsonWriter serializes non-finite FP using Python spelling - float" tags="[floating-point][JsonWriter]" filename="tests/<exe-name>/IntrospectiveTests/Json.tests.cpp" >
<Section name="NaN" filename="tests/<exe-name>/IntrospectiveTests/Json.tests.cpp" >
<Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/Json.tests.cpp" >
<Original>
sstream.str() == "NaN"
</Original>
<Expanded>
"NaN" == "NaN"
</Expanded>
</Expression>
<OverallResults successes="1" failures="0" expectedFailures="0" skipped="false"/>
</Section>
<Section name="Pos nf" filename="tests/<exe-name>/IntrospectiveTests/Json.tests.cpp" >
<Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/Json.tests.cpp" >
<Original>
sstream.str() == "Infinity"
</Original>
<Expanded>
"Infinity" == "Infinity"
</Expanded>
</Expression>
<OverallResults successes="1" failures="0" expectedFailures="0" skipped="false"/>
</Section>
<Section name="Neg inf" filename="tests/<exe-name>/IntrospectiveTests/Json.tests.cpp" >
<Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/Json.tests.cpp" >
<Original>
sstream.str() == "-Infinity"
</Original>
<Expanded>
"-Infinity" == "-Infinity"
</Expanded>
</Expression>
<OverallResults successes="1" failures="0" expectedFailures="0" skipped="false"/>
</Section>
<OverallResult success="true" skips="0"/>
</TestCase>
<TestCase name="JsonWriter serializes numbers independently of the global locale" tags="[JsonWriter]" filename="tests/<exe-name>/IntrospectiveTests/Json.tests.cpp" >
<Expression success="true" type="REQUIRE_THAT" filename="tests/<exe-name>/IntrospectiveTests/Json.tests.cpp" >
<Original>
sstream.str(), ContainsSubstring( "\"double\": 1.5," ) &amp;&amp; ContainsSubstring( "\"int\": 1234567," ) &amp;&amp; ContainsSubstring( "\"bool-1\": true," ) &amp;&amp; ContainsSubstring( "\"bool-2\": false," ) &amp;&amp; ContainsSubstring( "\"array\": [\n 2.5,\n 1234567\n ]\n}" )
</Original>
<Expanded>
"{
"double": 1.5,
"int": 1234567,
"bool-1": true,
"bool-2": false,
"array": [
2.5,
1234567
]
}" ( contains: ""double": 1.5," and contains: ""int": 1234567," and contains: ""bool-1": true," and contains: ""bool-2": false," and contains: ""array": [
2.5,
1234567
]
}" )
</Expanded>
</Expression>
<OverallResult success="true" skips="0"/>
</TestCase>
<TestCase name="Lambdas in assertions" filename="tests/<exe-name>/UsageTests/Compilation.tests.cpp" >
<Expression success="true" type="REQUIRE" filename="tests/<exe-name>/UsageTests/Compilation.tests.cpp" >
<Original>
@@ -23385,6 +23481,6 @@ Approx( -1.95996398454005449 )
</Section>
<OverallResult success="true" skips="0"/>
</TestCase>
<OverallResults successes="2215" failures="158" expectedFailures="43" skips="12"/>
<OverallResultsCases successes="331" failures="96" expectedFailures="18" skips="6"/>
<OverallResults successes="2222" failures="158" expectedFailures="43" skips="12"/>
<OverallResultsCases successes="334" failures="96" expectedFailures="18" skips="6"/>
</Catch2TestRun>
@@ -9894,6 +9894,102 @@ Approx( 3.14150000000000018 )
</Section>
<OverallResult success="true" skips="0"/>
</TestCase>
<TestCase name="JsonWriter serializes non-finite FP using Python spelling - double" tags="[floating-point][JsonWriter]" filename="tests/<exe-name>/IntrospectiveTests/Json.tests.cpp" >
<Section name="NaN" filename="tests/<exe-name>/IntrospectiveTests/Json.tests.cpp" >
<Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/Json.tests.cpp" >
<Original>
sstream.str() == "NaN"
</Original>
<Expanded>
"NaN" == "NaN"
</Expanded>
</Expression>
<OverallResults successes="1" failures="0" expectedFailures="0" skipped="false"/>
</Section>
<Section name="Pos nf" filename="tests/<exe-name>/IntrospectiveTests/Json.tests.cpp" >
<Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/Json.tests.cpp" >
<Original>
sstream.str() == "Infinity"
</Original>
<Expanded>
"Infinity" == "Infinity"
</Expanded>
</Expression>
<OverallResults successes="1" failures="0" expectedFailures="0" skipped="false"/>
</Section>
<Section name="Neg inf" filename="tests/<exe-name>/IntrospectiveTests/Json.tests.cpp" >
<Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/Json.tests.cpp" >
<Original>
sstream.str() == "-Infinity"
</Original>
<Expanded>
"-Infinity" == "-Infinity"
</Expanded>
</Expression>
<OverallResults successes="1" failures="0" expectedFailures="0" skipped="false"/>
</Section>
<OverallResult success="true" skips="0"/>
</TestCase>
<TestCase name="JsonWriter serializes non-finite FP using Python spelling - float" tags="[floating-point][JsonWriter]" filename="tests/<exe-name>/IntrospectiveTests/Json.tests.cpp" >
<Section name="NaN" filename="tests/<exe-name>/IntrospectiveTests/Json.tests.cpp" >
<Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/Json.tests.cpp" >
<Original>
sstream.str() == "NaN"
</Original>
<Expanded>
"NaN" == "NaN"
</Expanded>
</Expression>
<OverallResults successes="1" failures="0" expectedFailures="0" skipped="false"/>
</Section>
<Section name="Pos nf" filename="tests/<exe-name>/IntrospectiveTests/Json.tests.cpp" >
<Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/Json.tests.cpp" >
<Original>
sstream.str() == "Infinity"
</Original>
<Expanded>
"Infinity" == "Infinity"
</Expanded>
</Expression>
<OverallResults successes="1" failures="0" expectedFailures="0" skipped="false"/>
</Section>
<Section name="Neg inf" filename="tests/<exe-name>/IntrospectiveTests/Json.tests.cpp" >
<Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/Json.tests.cpp" >
<Original>
sstream.str() == "-Infinity"
</Original>
<Expanded>
"-Infinity" == "-Infinity"
</Expanded>
</Expression>
<OverallResults successes="1" failures="0" expectedFailures="0" skipped="false"/>
</Section>
<OverallResult success="true" skips="0"/>
</TestCase>
<TestCase name="JsonWriter serializes numbers independently of the global locale" tags="[JsonWriter]" filename="tests/<exe-name>/IntrospectiveTests/Json.tests.cpp" >
<Expression success="true" type="REQUIRE_THAT" filename="tests/<exe-name>/IntrospectiveTests/Json.tests.cpp" >
<Original>
sstream.str(), ContainsSubstring( "\"double\": 1.5," ) &amp;&amp; ContainsSubstring( "\"int\": 1234567," ) &amp;&amp; ContainsSubstring( "\"bool-1\": true," ) &amp;&amp; ContainsSubstring( "\"bool-2\": false," ) &amp;&amp; ContainsSubstring( "\"array\": [\n 2.5,\n 1234567\n ]\n}" )
</Original>
<Expanded>
"{
"double": 1.5,
"int": 1234567,
"bool-1": true,
"bool-2": false,
"array": [
2.5,
1234567
]
}" ( contains: ""double": 1.5," and contains: ""int": 1234567," and contains: ""bool-1": true," and contains: ""bool-2": false," and contains: ""array": [
2.5,
1234567
]
}" )
</Expanded>
</Expression>
<OverallResult success="true" skips="0"/>
</TestCase>
<TestCase name="Lambdas in assertions" filename="tests/<exe-name>/UsageTests/Compilation.tests.cpp" >
<Expression success="true" type="REQUIRE" filename="tests/<exe-name>/UsageTests/Compilation.tests.cpp" >
<Original>
@@ -23384,6 +23480,6 @@ Approx( -1.95996398454005449 )
</Section>
<OverallResult success="true" skips="0"/>
</TestCase>
<OverallResults successes="2215" failures="158" expectedFailures="43" skips="12"/>
<OverallResultsCases successes="331" failures="96" expectedFailures="18" skips="6"/>
<OverallResults successes="2222" failures="158" expectedFailures="43" skips="12"/>
<OverallResultsCases successes="334" failures="96" expectedFailures="18" skips="6"/>
</Catch2TestRun>
@@ -7,11 +7,14 @@
// SPDX-License-Identifier: BSL-1.0
#include <catch2/catch_test_macros.hpp>
#include <catch2/catch_template_test_macros.hpp>
#include <catch2/benchmark/catch_benchmark.hpp>
#include <catch2/generators/catch_generators.hpp>
#include <catch2/internal/catch_jsonwriter.hpp>
#include <catch2/matchers/catch_matchers_string.hpp>
#include <limits>
#include <locale>
#include <sstream>
namespace {
@@ -20,6 +23,27 @@ namespace {
return os << "custom";
}
// Obviously wrong numpunct if it is actually used by the JSON writer.
class TestNumpunct : public std::numpunct<char> {
char do_decimal_point() const override { return '?'; }
char do_thousands_sep() const override { return '!'; }
std::string do_grouping() const override { return "\1"; }
std::string do_truename() const override { return "real"; }
std::string do_falsename() const override { return "fake"; }
};
class LocaleGuard {
std::locale m_previous_locale;
public:
explicit LocaleGuard( std::locale const& locale ):
m_previous_locale{ std::locale() } {
std::locale::global( locale );
}
~LocaleGuard() { std::locale::global( m_previous_locale ); }
};
TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) {
std::stringstream stream;
@@ -152,6 +176,54 @@ TEST_CASE( "JsonWriter escapes characters in strings properly", "[JsonWriter]" )
}
}
TEST_CASE( "JsonWriter serializes numbers independently of the global locale",
"[JsonWriter]" ) {
using Catch::Matchers::ContainsSubstring;
LocaleGuard locale_guard{ std::locale{ std::locale(), new TestNumpunct } };
std::stringstream sstream;
{
auto writer = Catch::JsonValueWriter{ sstream }.writeObject();
writer.write( "double" ).write( 1.5 );
writer.write( "int" ).write( 1234567 );
writer.write( "bool-1" ).write( true );
writer.write( "bool-2" ).write( false );
writer.write( "array" ).writeArray().write( 2.5 ).write( 1234567 );
}
REQUIRE_THAT( sstream.str(),
ContainsSubstring( "\"double\": 1.5," ) &&
ContainsSubstring( "\"int\": 1234567," ) &&
ContainsSubstring( "\"bool-1\": true," ) &&
ContainsSubstring( "\"bool-2\": false," ) &&
ContainsSubstring(
"\"array\": [\n 2.5,\n 1234567\n ]\n}" ) );
}
TEMPLATE_TEST_CASE( "JsonWriter serializes non-finite FP using Python spelling",
"[JsonWriter][floating-point]",
float,
double ) {
std::stringstream sstream;
SECTION( "NaN" ) {
Catch::JsonValueWriter{ sstream }.write(
std::numeric_limits<TestType>::quiet_NaN() );
REQUIRE( sstream.str() == "NaN" );
}
SECTION( "Pos nf" ) {
Catch::JsonValueWriter{ sstream }.write(
std::numeric_limits<TestType>::infinity() );
REQUIRE( sstream.str() == "Infinity" );
}
SECTION( "Neg inf" ) {
Catch::JsonValueWriter{ sstream }.write(
-std::numeric_limits<TestType>::infinity() );
REQUIRE( sstream.str() == "-Infinity" );
}
}
TEST_CASE( "JsonWriter benchmarks", "[JsonWriter][!benchmark]" ) {
const auto input_length = GENERATE( as<size_t>{}, 10, 100, 10'000 );
std::string test_input( input_length, 'a' );
@@ -0,0 +1,194 @@
# SPDX-License-Identifier: BSL-1.0
# Unit tests for `split_json_array` helper in `extras/CatchAddTests.cmake`.
#
# Yes, we are at the stage where the script helpers need unit tests.
#
# Run as
# cmake -DCATCH_ADD_TESTS_SCRIPT=/path/to/extras/CatchAddTests.cmake \
# -P TestDecomposeJsonArray.cmake
cmake_minimum_required(VERSION 3.19)
if(NOT DEFINED CATCH_ADD_TESTS_SCRIPT)
message(FATAL_ERROR "Missing argument `CATCH_ADD_TESTS_SCRIPT`")
endif()
if(NOT EXISTS "${CATCH_ADD_TESTS_SCRIPT}")
message(FATAL_ERROR "Cannot find CatchAddTests.cmake at '${CATCH_ADD_TESTS_SCRIPT}'")
endif()
# Pull in the helper functions. Without `TEST_EXECUTABLE` being defined,
# `catch_discover_tests_impl` is not called.
include("${CATCH_ADD_TESTS_SCRIPT}")
set(_failures 0)
# Parses out test names from provided listings and returns them through `out_var`.
# Semicolons in test names are escaped as `@SEMI@`.
function(decomposed_names listing_var out_var)
split_json_array(${listing_var} elements)
set(names "")
foreach(element IN LISTS elements)
magic_unescape_chars(element)
string(JSON name ERROR_VARIABLE err GET "${element}" "name")
if(NOT err STREQUAL "NOTFOUND")
set(${out_var} "PARSE-ERROR" PARENT_SCOPE)
return()
endif()
string(REPLACE ";" "@SEMI@" name "${name}")
list(APPEND names "${name}")
endforeach()
set(${out_var} "${names}" PARENT_SCOPE)
endfunction()
# Assert that decomposing the provided listing returns the expected list
# of (test) names.
# Semicolons inside expected names must be escaped as `@SEMI@`.
function(expect_names description listing_var expected_names)
decomposed_names(${listing_var} actual_names)
if(actual_names STREQUAL expected_names)
message(" [PASS] ${description}")
else()
message(" [FAIL] ${description}")
message(" expected: ${expected_names}")
message(" actual: ${actual_names}")
math(EXPR _n "${_failures} + 1")
set(_failures "${_n}" PARENT_SCOPE)
endif()
endfunction()
# Assert that the tags in one decomposed element are preserved exactly.
# Semicolons in expected tags have to be escaped as `@SEMI@`.
function(expect_tags description listing_var element_index expected_tags)
split_json_array(${listing_var} elements)
list(GET elements ${element_index} element)
magic_unescape_chars(element)
string(JSON tags ERROR_VARIABLE err GET "${element}" "tags")
if(NOT err STREQUAL "NOTFOUND")
set(actual_tags "PARSE-ERROR")
else()
string(JSON tag_count LENGTH "${tags}")
set(actual_tags "")
if(tag_count GREATER 0)
math(EXPR last_tag "${tag_count} - 1")
foreach(tag_index RANGE ${last_tag})
string(JSON tag GET "${tags}" ${tag_index})
string(REPLACE ";" "@SEMI@" tag "${tag}")
list(APPEND actual_tags "${tag}")
endforeach()
endif()
endif()
if(actual_tags STREQUAL expected_tags)
message(" [PASS] ${description}")
else()
message(" [FAIL] ${description}")
message(" expected: ${expected_tags}")
message(" actual: ${actual_tags}")
math(EXPR _n "${_failures} + 1")
set(_failures "${_n}" PARENT_SCOPE)
endif()
endfunction()
# Convenience for building a minimal-but-realistic (pretty-printed) listing.
function(make_listing out_var)
set(objects "")
foreach(name IN LISTS ARGN)
# Build each object by hand; the names passed in are already JSON-safe.
string(APPEND objects
" {\n"
" \"class-name\" : \"\",\n"
" \"name\" : \"${name}\",\n"
" \"tags\" : [ \"[tag]\" ]\n"
" },\n")
endforeach()
string(REGEX REPLACE ",\n$" "\n" objects "${objects}")
set(${out_var} "[\n${objects}]" PARENT_SCOPE)
endfunction()
message(STATUS "Running split_json_array correctness tests")
# There are 2 main difficulties in the array decomposition that we need
# to check for:
# 1) Names/tags that contain the expected element boundary (`}<ws>*,<ws>*{`)
# inside them, and thus are split into invalid JSON.
# 2) Names/tags that contain CMake-relevant characters (e.g. semicolon,
# which is list separator) and thus cause issues when processing the
# string splits.
make_listing(listing "")
expect_names("No tests" listing "")
make_listing(listing "plain")
expect_names("Single test" listing "plain")
make_listing(listing "n1" "n2" "n3")
expect_names("Multiple tests" listing "n1;n2;n3")
make_listing(listing "before },{ after" "second")
expect_names("The element boundary in a test name" listing "before },{ after;second")
make_listing(listing "},{" "next")
expect_names("Test name is just the boundary" listing "},{;next")
make_listing(listing "a},{b},{c" "x" "y},{z")
expect_names("Test name has multiple boundaries" listing "a},{b},{c;x;y},{z")
# Listings with semicolons have to be built by hand, or CMake would mess
# them up before decomposition.
set(listing "[{\"name\":\"has;semicolon\",\"tags\":[]},{\"name\":\"and;another;one\",\"tags\":[]}]")
expect_names("Test names with semicolons" listing "has@SEMI@semicolon;and@SEMI@another@SEMI@one")
set(listing "[{\"name\":\"C:\\\\path\\\\file\",\"tags\":[]},{\"name\":\"plain\",\"tags\":[]}]")
expect_names("Test names with backslashes" listing "C:\\path\\file;plain")
set(listing "[{\"name\":\"compact1\",\"tags\":[]},{\"name\":\"compact2\",\"tags\":[]}]")
expect_names("compact json" listing "compact1;compact2")
make_listing(listing "Then } , { we }\t,\t{ concatenate } , { them } ,{back},{" "second")
expect_names("Whitespace around commas in test names survive split"
listing "Then } , { we }\t,\t{ concatenate } , { them } ,{back},{;second")
make_listing(listing "}},{{" "second")
expect_names("Doubled up boundary braces in names" listing "}},{{;second")
# Square brackets and array likes in the test names.
set(listing "[{\"name\":\"Arrays [{},{}] wheee\",\"tags\":[\"also},{tags\",\"tag;with;semicolons\"]},{\"name\":\"n\",\"tags\":[]}]")
expect_names("array-like substring in name" listing "Arrays [{},{}] wheee;n")
expect_tags("boundary-like and semicolon tags are preserved" listing 0
"also},{tags;tag@SEMI@with@SEMI@semicolons")
expect_tags("empty tags are preserved" listing 1 "")
# Listings with square brackets have to be built by hand, or CMake would
# mess them up before decomposition.
set(listing "[{\"name\":\"[\",\"tags\":[]},{\"name\":\"middle\",\"tags\":[]},{\"name\":\"]\",\"tags\":[]}]")
expect_names("unmatched square brackets" listing "[;middle;]")
set(listing "[{\"name\":\"a[b]c\",\"tags\":[]},{\"name\":\"[open\",\"tags\":[]},{\"name\":\"close]\",\"tags\":[]},{\"name\":\"[]\",\"tags\":[]}]")
expect_names("Test names with mess of brackets" listing "a[b]c;[open;close];[]")
# Without careful handling, these could be evaluated as variables.
make_listing(listing
"curly \${NOT_A_VAR}"
"env \$ENV{HOME}"
"cache \$CACHE{FOO}"
"genex \$<CONFIG>"
"bare \$ and \$\$ and \${ unterminated")
expect_names("Test names with dollars and various brackets (CMake vars)" listing
"curly \${NOT_A_VAR};env \$ENV{HOME};cache \$CACHE{FOO};genex \$<CONFIG>;bare \$ and \$\$ and \${ unterminated")
# Listings with semicolons have to be built by hand, or CMake would mess
# them up before decomposition.
# This is just a huge mess of everything to see if anything shakes loose.
set(listing "[{\"name\":\"\$ENV{X};[weird]{},{ mix \$<0:no> \${VAR} };,{ }},{{\",\"tags\":[]},{\"name\":\"after\",\"tags\":[]}]")
expect_names("combined mega-case" listing
"\$ENV{X}@SEMI@[weird]{},{ mix \$<0:no> \${VAR} }@SEMI@,{ }},{{;after")
if(_failures GREATER 0)
message(FATAL_ERROR "${_failures} decomposition test(s) failed")
endif()
message(STATUS "All split_json_array correctness tests passed")
@@ -0,0 +1,61 @@
# SPDX-License-Identifier: BSL-1.0
# Unit tests for `prepare_command` helper in `extras/CatchAddTests.cmake`.
#
# Yes, we are at the stage where the script helpers need unit tests.
#
# Run as
# cmake -DCATCH_ADD_TESTS_SCRIPT=/path/to/extras/CatchAddTests.cmake \
# -P TestPrepareCommand.cmake
cmake_minimum_required(VERSION 3.19)
if(NOT DEFINED CATCH_ADD_TESTS_SCRIPT)
message(FATAL_ERROR "Missing argument `CATCH_ADD_TESTS_SCRIPT`")
endif()
if(NOT EXISTS "${CATCH_ADD_TESTS_SCRIPT}")
message(FATAL_ERROR "Cannot find CatchAddTests.cmake at '${CATCH_ADD_TESTS_SCRIPT}'")
endif()
# Pull in the helper functions. Without `TEST_EXECUTABLE` being defined,
# `catch_discover_tests_impl` is not called.
include("${CATCH_ADD_TESTS_SCRIPT}")
set(_failures 0)
function(expect_equal description actual expected)
if(actual STREQUAL expected)
message(" [PASS] ${description}")
else()
message(" [FAIL] ${description}")
message(" expected: [${expected}]")
message(" actual: [${actual}]")
math(EXPR _n "${_failures} + 1")
set(_failures "${_n}" PARENT_SCOPE)
endif()
endfunction()
prepare_command(add_test SimpleName /path/to/tests)
expect_equal("Simple arg, no quotes" "${_Command}" "add_test( SimpleName /path/to/tests)\n")
prepare_command(add_test "Name with spaces")
expect_equal("Spaces in arg, needs quotes" "${_Command}" "add_test( [==[Name with spaces]==])\n")
prepare_command(set_tests_properties Foo PROPERTIES LABELS "tagA\;tagB\;tagC")
expect_equal("semicolons in argument are kept and quoted"
"${_Command}" "set_tests_properties( Foo PROPERTIES LABELS [==[tagA\;tagB\;tagC]==])\n")
set(_Command "PRE-EXISTING")
prepare_command(set_tests_properties Foo PROPERTIES BAR baz)
expect_equal("_Command var does no accumulate commands"
"${_Command}" "set_tests_properties( Foo PROPERTIES BAR baz)\n")
if(_failures GREATER 0)
message(FATAL_ERROR "${_failures} prepare_command test(s) failed")
else()
message(STATUS "All prepare_command tests passed")
endif()
@@ -12,6 +12,7 @@ import subprocess
import sys
import re
import json
import tempfile
from collections import namedtuple
from typing import List
@@ -72,14 +73,19 @@ def get_test_names(build_path: str) -> List[TestInfo]:
config_path = "Debug" if os.name == 'nt' else ""
full_path = os.path.join(build_path, config_path, 'tests')
cmd = [full_path, '--reporter', 'json', '--list-tests']
result = subprocess.run(cmd,
capture_output = True,
check = True,
text = True)
test_listing = json.loads(result.stdout)
with tempfile.TemporaryDirectory() as tmpdir:
fname = f'{tmpdir}/listing-output.json'
cmd = [full_path,
'--list-tests',
'--reporter', 'json',
'--out', fname
]
result = subprocess.run(cmd,
capture_output = False,
check = True,
text = True)
with open(fname, mode='r', encoding='utf-8') as file:
test_listing = json.load(file)
assert test_listing['version'] == 1
@@ -96,10 +102,18 @@ def get_ctest_listing(build_path):
os.chdir(build_path)
cmd = ['ctest', '-C', 'debug', '--show-only=json-v1']
result = subprocess.run(cmd,
capture_output = True,
check = True,
text = True)
try:
result = subprocess.run(cmd,
capture_output = True,
check = True,
text = True)
except subprocess.CalledProcessError as err:
print('Error when getting output from CTest')
print(f'cmd: {err.cmd}')
print(f'stderr: {err.stderr}')
print(f'stdout: {err.stdout}')
exit(4)
os.chdir(old_path)
return result.stdout
@@ -8,6 +8,24 @@
#include <catch2/catch_test_macros.hpp>
#include <cstdio>
#include <iostream>
namespace {
struct PrintsWhenConstructed {
PrintsWhenConstructed() {
std::cout << "Hello\n";
std::cerr << "Holla\n";
std::fprintf(stdout, "Hullo\n");
std::fprintf(stderr, "Hillo\n");
}
};
static PrintsWhenConstructed instance;
}
TEST_CASE("@Script[C:\\EPM1A]=x;\"SCALA_ZERO:\"", "[script regressions]"){}
TEST_CASE("Some test") {}
TEST_CASE( "Let's have a test case with a long name. Longer. No, even longer. "
@@ -21,3 +39,26 @@ public:
};
TEST_CASE_METHOD(TestCaseFixture, "A test case as method", "[tagstagstags]") {}
TEST_CASE( "Newlines\nAnd\rOther\n\tWhitespace", "[whitespace-going-wild]" ) {}
// Some JSON-like and JSON-adjacent characters and substrings in the test names/tags
// This serves to test that the parse-json-via-string-splitting hack in
// catch_discover_tests works properly.
TEST_CASE( "We split on variants of },{ in name" ) {}
TEST_CASE( "Then }\t, { we }\t,\t{ concatenate } , { them } ,{back},{" ) {}
TEST_CASE( "}},{{" ) {}
TEST_CASE( "Arrays [{},{}] wheee", "[also},{tags]" ) {}
TEST_CASE( "Let's add semicolon into the mix ;},{;},{};,{}" ) {}
TEST_CASE( "[", "[unmatched-square-bracket]" ) {}
TEST_CASE( "]", "[unmatched-square-bracket]" ) {}
// Some CMake-like special strings ($ as dereference) strings in test names.
// This serves to test that the names of test cases are not evaluated
// inside the catch_discover_tests.
TEST_CASE( "Plain ${NOT_A_VAR} variable" ) {}
TEST_CASE( "Env variable access $ENV{HOME}" ) {}
TEST_CASE( "Cache check $CACHE{FOO}" ) {}
TEST_CASE( "Also some generator exprs $<CONFIG> in $<1:yes> name" ) {}
TEST_CASE( "Mess of bare $ $$ $$$ and unterminated $} ${ exprs" ) {}
TEST_CASE( "$ENV{X};[weird]{},{ mix $<0:no> ${VAR} };,{ }},{{" ) {}