-1

How it is possible for C compiler validation suites to use any feature of the C standard library if the C standard library is actually under test and may be buggy (which invalidates the validation suite itself)?

For example, the suite may use itself memset, fopen, etc., but memset, fopen, etc. are actually under test and may be buggy.

Confused. Clarification / experience is needed.

UPD. Clarified after examining suite's source code. The suite uses two compilers:

  1. Reference compiler is used to build the suite itself.
  2. IUT (Implementation Under Test) compiler is used to run tests on it.
pmor
  • 5,392
  • 4
  • 17
  • 36
  • 4
    Why not start by asking why these test suites are written in C and compiled by a C compiler? Answer: there is more than one compiler and more than one standard library implementation in the world. (C compiler validation suites are not often written in C, they are more likely to be implemented as shell scripts, but shells are often written in C). – n. m. could be an AI Aug 05 '21 at 12:46
  • 2
    If a test tests `memset()`, I'd expect that it does **not** call the very same function except to test it, but other functions are independent and can be used. The assumption is that all functions with the exception of the function-under-test are correct. – the busybee Aug 05 '21 at 13:04
  • That question could be better answered if a particular example of a test suite is given. For example, I can suppose that the suite which uses `memset` doesn't necessarily test `memset` and assumes it's implementation matches the reference one, because there is other suite which will test `memset` as required. But without the specific example it is all tea leaves guessing. – SergeyA Aug 05 '21 at 13:20
  • Strategically, you could choose to *not* test trivial stuff like memset(). If it *would* behave badly, you'd soon find out, anyway. – wildplasser Aug 05 '21 at 13:25
  • 1
    You use a *different* compiler and library implementation that has already been validated to perform the tests on the *new* compiler and library implementation. – John Bode Aug 05 '21 at 13:35
  • 1
    A good test suite may consist of many stages or layers. First you test **A** against something fundamental. Then, once you know **A** is correct, you can use **A** as part of your test to validate **B**. Then, you can use **A** and **B** to validate **C**, etc. – Steve Summit Aug 05 '21 at 14:08
  • Sometimes it's even important to test your tests. For example, I've written code to deliberately cause **A** to fail, then make sure that my test for a failing **A** will catch the failure, to increase my confidence that when my test of a non-broken **A** *doesn't* fail, that means that **A** is good. – Steve Summit Aug 05 '21 at 14:10
  • pmor, Many reference functions are slow, inefficient, yet more clearly correct **and** have history of acceptance. The function under test is then assessed against the reference one. Still a discrepancy between the 2 does not always point to the function under test failure (could be the ref one) as well as consistent output does not mean both are right. – chux - Reinstate Monica Aug 05 '21 at 15:04
  • 1
    When a function has a bug, sometimes the bug is left in as it is deemed more important to have consistent functionality than fix the bug. [1900 leap day](https://en.wikipedia.org/wiki/Leap_year_problem#:~:text=Microsoft%20Excel%20has%2C%20since%20its,the%20purpose%20of%20backward%20compatibility.) – chux - Reinstate Monica Aug 05 '21 at 15:07
  • 1
    Not a C example, but: I once wrote a program that searched for strings in files (using binary search), and I wrote a test suite for it, which among other things generated random files full of random strings which it then used my program to search for. To make sure it found (or didn't find) the right string, I also searched using GNU grep, which should get the same answer, although maybe slower (since it uses sequential search). One day my test suite found a discrepancy. But my program was fine -- it turned out I'd found a bug in GNU grep. – Steve Summit Aug 05 '21 at 17:46

1 Answers1

1

Yes, testing implementation X using implementation X is not correct (with exceptions, see below). In my particular case the situation is clarified after examining suite's source code. The suite uses two compilers:

  1. Reference compiler is used to build the suite itself.
  2. IUT (Implementation Under Test) compiler is used to run tests on it.

Exceptions. As user Steve Summit mentioned:

A good test suite may consist of many stages or layers. First you test A against something fundamental. Then, once you know A is correct, you can use A as part of your test to validate B. Then, you can use A and B to validate C, etc.

This is viable approach. Hover, since the first (upper) layer cannot be tested (for example the correctness of the == operator), we take as an axiom that the first layer is error free.

Also continue quoting user Steve Summit:

Sometimes it's even important to test your tests.

I'd say that it is always important to test the correctness of the tests themselves.

pmor
  • 5,392
  • 4
  • 17
  • 36