Introduction

This is a test suite for C++ refactoring tools. It consists of a set of source files with comments marking different locations in the code where a refactoring could be tested.

Over time, I've tried to add runtime assertions to the code to give an additional level of testing around the refactored area. While these assertions help to verify the validity of a refactoring applied by a tool, the pass or fail of a test case is up to the person invoking the refactoring tool and examining the diffs.

In general, when evaluating results from a refactoring tool, you should treat the tool as skeptically as you would treat any other programmer on your team. If you wouldn't accept a set of diffs from a human, then you shouldn't accept those same diffs from a computer.

The original release of the test suite is described in this blog post.

This test suite is explained in detail in a Utah C++ Programmers video Refactoring Tools Roundup from January, 2024.

How To Contribute

Results (January, 2024)

Summary Results

Results for tools with significant testing.

• CLion 2023.3.2
• clang-tidy 17.0.6
• ReSharper for C++ 2023.3.2
• Visual AssistX 10.9.2508.0
• Visual Studio 2022, 17.7.5

Preliminary Results

Preliminary results for tools with minimal testing.

• Apple Xcode
• Eclipse CDT
• Qt Creator

Unknown Results

• Cevelop - Extensions for Eclipse CDT
• Code::Blocks
• KDevelop - Refactoring

Building

CMake presets are provided for building the code. The default workflow preset configures, builds the code and runs the tests. To rebuild the test suite and perform internal validation, simply build the code using the default preset for configuration and build. To run the CTest related tests, the default preset can be used.

Examples:

cmake --preset default               # configure
cmake --build --preset default       # build
ctest --preset default               # test
cmake --workflow --preset default    # configure, build and test

The refactor-test-suite target compiles all the source code to be refactored. The code contains some runtime checks to validate that the semantics of refactored code has not changed. The test suite executes as a post-build event so that simply building refactor-test-suite validates the semantics as part of the build.

The CTest related tests perform validation of the results files for the tools.

To execute clang-tidy related tests, a compile_commands.json compilation database is required. This can be generated with the ninja configuration preset:

cmake --preset ninja

See the documentation on how to contribute for information about how to execute tests.