c++ - ctest 报告退出状态为 1 的测试成功

Question

我有一个带有单元测试的项目，当我运行 ctest（macos 上的 cmake 版本 3.18.2）时，所有测试都报告成功。但是，如果我自己运行其中一项测试，它的退出状态为 1。据我所知，这不应该发生，那么是什么原因造成的，我该如何解决？

这个问题是一个粗心的小错误，但为了其他遇到这个问题的人的利益，我提供了一个答案，其余问题可以跳过。

不幸的是，我无法用较小的最小工作示例重现此行为。我会尽量提供尽可能多的相关信息，如果我遗漏了什么，请告诉我。下面是单元测试的代码：

#include "alignment_reader.h"

#define CATCH_CONFIG_MAIN
#define CATCH_CONFIG_COLOUR_NONE
#include "catch.h"

#include "string_conversions.h"

#include "exceptions.h"

namespace paste_alignments {

namespace test {

namespace {

SCENARIO("Test correctness of AlignmentReader::FromFile.",
         "[AlignmentReader][FromFile][correctness]") {

  GIVEN("The name of a valid input file.") {
    std::string input_file{"test_data/valid_alignment_file.tsv"};

    WHEN("Constructed with default number of fields and chunk size.") {
      CHECK_NOTHROW(AlignmentReader::FromFile(input_file));
      /*AlignmentReader reader{AlignmentReader::FromFile(input_file)};

      THEN("Field number and chunk size are at default values.") {
        CHECK(reader.NumFields() == 12);
        CHECK(reader.ChunkSize() == 128 * 1000 * 1000);
      }*/
    }
  }
}

} // namespace

} // namespace test

} // namespace paste_alignments

当我取消注释注释部分时，退出代码更改为 2 并且 ctest 仍然报告成功。

这是我运行 ctest 时发生的情况（有和没有注释部分）：

$ ctest
Test project /Users/Jasper/cpp_projects/PasteAlignments/debug
    Start 1: alignment_reader_test
1/1 Test #1: alignment_reader_test ............   Passed    0.16 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) =   0.17 sec

如果我单独运行测试并检查退出状态（注释掉的东西），这是我得到的：

$ ./test/alignment_reader_test 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
alignment_reader_test is a Catch v2.12.2 host application.
Run with -? for options

-------------------------------------------------------------------------------
Scenario: Test correctness of AlignmentReader::FromFile.
      Given: The name of a valid input file.
       When: Constructed with default number of fields and chunk size.
-------------------------------------------------------------------------------
/Users/Jasper/cpp_projects/PasteAlignments/test/alignment_reader_test.cc:64
...............................................................................

/Users/Jasper/cpp_projects/PasteAlignments/test/alignment_reader_test.cc:65: FAILED:
  CHECK_NOTHROW( AlignmentReader::FromFile(input_file) )
due to unexpected exception with message:
  Unable to open file: 'test_data/valid_alignment_file.tsv'.

===============================================================================
test cases: 1 | 1 failed
assertions: 1 | 1 failed

$ echo $?
1

当我包含注释部分时，唯一改变的是两个测试失败并且退出状态是 2 而不是 1。

这里有--verbose标志：

$ ctest --verbose
UpdateCTestConfiguration  from :/Users/Jasper/cpp_projects/PasteAlignments/debug/DartConfiguration.tcl
Parse Config file:/Users/Jasper/cpp_projects/PasteAlignments/debug/DartConfiguration.tcl
UpdateCTestConfiguration  from :/Users/Jasper/cpp_projects/PasteAlignments/debug/DartConfiguration.tcl
Parse Config file:/Users/Jasper/cpp_projects/PasteAlignments/debug/DartConfiguration.tcl
Test project /Users/Jasper/cpp_projects/PasteAlignments/debug
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 1
    Start 1: alignment_reader_test

1: Test command: /Users/Jasper/cpp_projects/PasteAlignments/debug/test/alignment_reader_test
1: Test timeout computed to be: 1500
1: ===============================================================================
1: All tests passed (1 assertion in 1 test case)
1: 
1/1 Test #1: alignment_reader_test ............   Passed    0.36 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) =   0.36 sec

我没有更改 ctest 配置（如此处所建议）。CMakeLists.txt 看起来像这样：

add_executable(alignment_reader_test
        "${PROJECT_SOURCE_DIR}/test/alignment_reader_test.cc"
        "${PROJECT_SOURCE_DIR}/src/alignment_reader.cc"
        "${PROJECT_SOURCE_DIR}/src/alignment_batch.cc"
        "${PROJECT_SOURCE_DIR}/src/scoring_system.cc"
        "${PROJECT_SOURCE_DIR}/src/alignment.cc"
        "${PROJECT_SOURCE_DIR}/src/helpers.cc")
target_include_directories(alignment_reader_test PUBLIC
        "${PROJECT_SOURCE_DIR}/test"
        "${PROJECT_SOURCE_DIR}/include"
        "${PROJECT_SOURCE_DIR}/lib/catch/include")
add_test(NAME alignment_reader_test COMMAND alignment_reader_test)

标签： c++cmakectestexitstatuscatch2

Answer 1

感谢@Tsyvarev，我发现了这个问题。

构建目录结构如下：

build/
|--test/
   |--test_executable
   |--test_data/
      |--test_datafile

当 CMake 运行test_executable时，它会从目录中运行它test。

当我单独运行可执行文件时，我从build目录中运行它。( ./test/test_executable)。

在单元测试代码中，test_datafile被称为test_data/test_datafile. 当测试从build可执行文件运行时（就像我所做的那样），而不是从build/test目录运行时，这无法识别。

因此，当 ctest 运行测试时，它实际上应该成功了。

事实上，如果我先cd进入test，测试的退出代码应该是 0。