strange behavior of cmake ctest for bigger CTEST_PARALLEL_LEVEL

Question

I am new to the SO. I have a simple unit test code where I am doing following operations :

1. calculating square root of the number using mysqrt library .
2. Using the output of square root, adding this result with same number and display the result.

When I am running the code with CTEST_PARALLEL_LEVEL = 1 my all test cases are passing.

But when I do CTEST_PARALLEL_LEVEL = 8 then my test cases are failing some time for some input which is not fixed in every run.

99% the ALL results are passing but 1% its failing.

Error:

mysqrt.o: file not recognized: File truncated

I have deleted the object file explicitly by using rm *.o ,But still this error is coming after few runs .

I am not sure why this error is coming with CTEST_PARALLEL_LEVEL = 8

I am attaching my CMakeList only as some of Stack Overflow expert can understand the issue by checking these 3 CMakeLists.txt files.

NOTE: As per the Stack overflow guidelines I am not attaching my source code of sqrt and addition function to avoid the bigger length of the question .

My folder structure:

SAMPLE_TEST

├── CMakeLists.txt
├── MathFunctions
│   ├── CMakeLists.txt
│   ├── MathFunctions.h
│   └── mysqrt.cpp
└── unit_test
    ├── CMakeLists.txt
    └── step2
        ├── CMakeLists.txt
        ├── execute.cpp
        └── tutorial.cpp

SAMPLE_TEST

CMakeLists.txt

cmake_minimum_required(VERSION 3.1)
project(Tutorial)
ENABLE_TESTING()    
add_subdirectory(MathFunctions)
add_subdirectory(unit_test)

MathFunctions folder

CMakeLists.txt

add_library(MathFunctions mysqrt.cpp)
set(REF_FILES mysqrt.cpp)
add_definitions(-Wall -Wextra -pedantic -std=c++11)
add_custom_target(build_reference_library
      DEPENDS sqrtlib
      COMMENT "Generating sqrtlib")
ADD_LIBRARY(sqrtlib OBJECT ${REF_FILES})

unit_test folder

CMakeLists.txt

set(REF_MATHLIB_DIR ${CMAKE_CURRENT_SOURCE_DIR}/../MathFunctions)

macro(GENERATION file input)
  set(ip_generator ctest_input_${input})

  add_executable(${ip_generator}
    ${file}
    $<TARGET_OBJECTS:sqrtlib>
    )

  target_compile_options(${ip_generator} PUBLIC
    -Wall -Wextra -g -std=c++11 
    -DCTEST_INPUT=${input})


  target_link_libraries(${ip_generator} PUBLIC
    dl pthread
    )

  target_include_directories(${ip_generator} PUBLIC
    ${REF_MATHLIB_DIR}
    )

  set(INPUT_FILE0 ip0_${input}.y)
  set(INPUT_FILE0_TXT ip0_${input}.txt)

  add_custom_command(
    OUTPUT ${INPUT_FILE0}  ${INPUT_FILE0_TXT}
    COMMAND ${ip_generator} > ${INPUT_FILE0_TXT}
    MAIN_DEPENDENCY ${sqrtlib}
    COMMENT "Generating output files of for testcase")
  
  add_custom_target(gen_input_${input}
    DEPENDS ${INPUT_FILE0}
    COMMENT "Generated output files")

endmacro() 

####################

macro(EXECUTE file input)
  get_filename_component(main_base_name ${file} NAME_WE)
  set(main_base_name_mangled ${main_base_name}_${input})
  set(exe_generator ctest_ref_${input})

  add_executable(${exe_generator}
    ${file}
    $<TARGET_OBJECTS:sqrtlib>
    )

  target_compile_options(${exe_generator} PUBLIC
    -Wall -Wextra -g -std=c++11 
    -DCTEST_INPUT=${input})


  target_link_libraries(${exe_generator} PUBLIC
    dl pthread
    )

  target_include_directories(${exe_generator} PUBLIC
    ${REF_MATHLIB_DIR}
    )

  set(INPUT_FILE0 ip0_${input}.y)

  set(EXE_FILE0 exeadd_${input}.y)
  set(EXE_FILE_TXT exeadd_${input}.txt)

  add_custom_command(
    OUTPUT ${EXE_FILE0} ${EXE_FILE_TXT}
    COMMAND ${exe_generator}  > ${EXE_FILE_TXT}
    MAIN_DEPENDENCY ${INPUT_FILE0} ${sqrtlib}
    COMMENT "Generating output files of for testcase")
  
  add_custom_target(gen_execute_${input}
    DEPENDS ${EXE_FILE0}
    COMMENT "Generated output files")

    # add test to simulate
  add_test(NAME ctest_execute_${input}
      COMMAND ${CMAKE_COMMAND} --build ${CMAKE_BINARY_DIR}
                               --target gen_execute_${input})

  #add_dependencies(execute_${main_base_name_mangled}   
  #gen_input)

endmacro() 

#++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++#
# add test directories
#++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++#

set(TEST_DIRECTORIES
    step2
   )

foreach(dir ${TEST_DIRECTORIES})
  add_subdirectory(${dir})
endforeach()

step2 folder

CMakeLists.txt

set(UT_IPGEN_FILES tutorial.cpp)
set(UT_EXECUTE_FILES execute.cpp)

set(input_integer_range 1 4 9 16 25 36 49 64 81 100 121 144 )

foreach(ip_integer ${input_integer_range})
  GENERATION(${UT_IPGEN_FILES} ${ip_integer})
  EXECUTE(${UT_EXECUTE_FILES} ${ip_integer})
endforeach(ip_integer)

Result: 1st Run:

      Start  1: ctest_execute_1
      Start  2: ctest_execute_4
      Start  3: ctest_execute_9
      Start  4: ctest_execute_16
      Start  5: ctest_execute_25
      Start  6: ctest_execute_36
      Start  7: ctest_execute_49
      Start  8: ctest_execute_64
 1/12 Test  #4: ctest_execute_16 .................***Failed    1.14 sec
 2/12 Test  #6: ctest_execute_36 .................   Passed    1.27 sec
 3/12 Test  #7: ctest_execute_49 .................   Passed    1.32 sec
 4/12 Test  #8: ctest_execute_64 .................   Passed    1.32 sec
      Start  9: ctest_execute_81
      Start 10: ctest_execute_100
      Start 11: ctest_execute_121
      Start 12: ctest_execute_144
 5/12 Test  #1: ctest_execute_1 ..................   Passed    1.33 sec
 6/12 Test  #2: ctest_execute_4 ..................   Passed    1.33 sec
 7/12 Test  #3: ctest_execute_9 ..................   Passed    1.33 sec
 8/12 Test  #5: ctest_execute_25 .................   Passed    1.33 sec
 9/12 Test #10: ctest_execute_100 ................   Passed    0.54 sec
10/12 Test #11: ctest_execute_121 ................   Passed    0.55 sec
11/12 Test  #9: ctest_execute_81 .................   Passed    0.55 sec
12/12 Test #12: ctest_execute_144 ................   Passed    0.55 sec
92% tests passed, 1 tests failed out of 12

Total Test time (real) =   1.88 sec

The following tests FAILED:
      4 - ctest_execute_16 (Failed)

2nd Run:

      Start  1: ctest_execute_1
      Start  2: ctest_execute_4
      Start  3: ctest_execute_9
      Start  4: ctest_execute_16
      Start  5: ctest_execute_25
      Start  6: ctest_execute_36
      Start  7: ctest_execute_49
      Start  8: ctest_execute_64
 1/12 Test  #6: ctest_execute_36 .................   Passed    1.31 sec
 2/12 Test  #7: ctest_execute_49 .................   Passed    1.36 sec
 3/12 Test  #8: ctest_execute_64 .................   Passed    1.36 sec
      Start  9: ctest_execute_81
      Start 10: ctest_execute_100
      Start 11: ctest_execute_121
 4/12 Test  #1: ctest_execute_1 ..................   Passed    1.37 sec
 5/12 Test  #2: ctest_execute_4 ..................   Passed    1.37 sec
 6/12 Test  #3: ctest_execute_9 ..................   Passed    1.36 sec
 7/12 Test  #4: ctest_execute_16 .................   Passed    1.36 sec
 8/12 Test  #5: ctest_execute_25 .................   Passed    1.37 sec
      Start 12: ctest_execute_144
 9/12 Test #11: ctest_execute_121 ................   Passed    0.50 sec
10/12 Test #10: ctest_execute_100 ................   Passed    0.51 sec
11/12 Test  #9: ctest_execute_81 .................   Passed    0.51 sec
12/12 Test #12: ctest_execute_144 ................   Passed    0.34 sec

100% tests passed, 0 tests failed out of 12

Total Test time (real) =   2.01 sec

Answer 1

Your tests executing

COMMAND ${CMAKE_COMMAND} --build ${CMAKE_BINARY_DIR} --target ...

which effectively runs make (or whatever build tool you use) in the project's build directory.

But concurrent invocations of make in the same directory are never guarantee to work correctly. This is why you got weird errors when run tests in parallel (with CTEST_PARALLEL_LEVEL variable being set).

E.g. all these tests are trying to create the same object file mysqrt.o, and this creation is definitely not thread-safe.

By running

make sqrtlib

before

ctest

you may be sure that the object file is already created when tests are run, and tests wouldn't attempt to create it again. But you still could get other conflicts in the parallel tests.

---

It depends on what actually you want to check by the testing, but usually a test checks behavior of some program or library, and it doesn't intend to check a compilation(building) of that program. Because of that, compilation(building) commands are performed before the testing.

Usually it is convenient to follow(implement) this workflow for testing:

# Configure the project
cmake <source-directory>
# Build the project.
# It builds both program/library intended, and the tests themselves.
make
# run tests
ctest <params>

In that case a test could have the following definition:

add_test(NAME ctest_execute_${input} COMMAND ${exe_generator})

(Unless you want to check output of the test by some automatic way, no need to explicitely save this output by redirecting into the file. ctest by itself would collect the output of the test, so you may read it if needed).