1

My unit tests run fine, but the make test target won't return an error code when the tests fail (echo $? is 0) (it does when there is a lisp error).

This is not good for a CI system.

My main system definition ends with in-order-to:

  :in-order-to ((test-op (test-op "mysystem/test"))))

My test system has a perform one:

  :perform (test-op (o s)
                    (uiop:symbol-call '#:mysestem.test '#:test-all)))

where test-all is a function that runs Fiveam test suites with run! 'test-suite.

The make target:

test:
  # (with proper indentation…)
  $(LISP) --non-interactive \
    --eval "(asdf:load-asd \""$(ASD_PATH)"\")" \
    --eval "(ql:quickload :mysystem/test)" \
    --eval "(asdf:test-system \"mysystem\")"

Note that follownig ASDF's best practices in doing

                    (symbol-call :fiveam #':run! :test-suite)))

errors out with "The function :run! is undefined."

How do we do this ?

Thank you.

Ehvince
  • 17,274
  • 7
  • 58
  • 79
  • `ASDF:TEST-SYSTEM` doesn't kill the image, so it doesn't affect the exit code one way or another. How to exit with an error code is implementation dependent, but uiop has `UIOP:QUIT`. – jkiiski Feb 26 '19 at 18:22
  • That actually makes sense, since we can call it from the repl. So I'm trying things like `--eval "(unless (asdf:test-system \"mysystem\") (uiop:quit 1))"` with no success so far. I have a failing test and still `0`, but the last output is `Didn't run anything...huh?` so maybe fiveam doesn't return `nil` here. – Ehvince Feb 27 '19 at 11:55
  • Now I have a clean output with one failing test, but still `echo $?` = 0. – Ehvince Feb 27 '19 at 11:59
  • 3
    The return value of `ASDF:TEST-SYSTEM` doesn't indicate test success/failure. The manual suggests signalling a condition from the [test-op](https://common-lisp.net/project/asdf/asdf.html#test_002dop). – jkiiski Feb 27 '19 at 13:53

1 Answers1

1

There seem to be typos:

  • '#:mysestem.test ⇒ '#:mysystem.test
  • #':run! ⇒ '#:run!
Svante
  • 50,694
  • 11
  • 78
  • 122
  • The `run` one is a real one. I put it the name of a test suite (as in `:test-suite`) but it doesn't run anything :/ "Didn't run anything...huh?" – Ehvince Feb 27 '19 at 11:36