2

Goal

I want to write a test that runs as part of dune test but does not run as part of dune build (for example, because it is slow). This test should have an expected output file and compare the actual output to it, and dune promote should be able to update the expected output.

First attempt: implicit rules

Following the instructions in the manual, I created the following minimal diff test:

dune-project:

(lang dune 3.3)

dune:

(test (name mytest))

mytest.ml:

let () =
  Printf.printf "this is mytest\n"

mytest.expected:

xxx

Since mytest.expected exists, dune automatically creates a diff test. However, it runs mytest.exe > mytest.output as part of dune build:

$ dune clean
$ dune build --verbose
[...]
Running[4]: (cd _build/default && ./mytest.exe) > _build/default/mytest.output
$ ls _build/default/
mytest.exe*  mytest.ml  mytest.mli  mytest.output

Then when I run dune test, it does the comparison to the output file it already made:

$ dune test --verbose
[...]
Command [1] exited with code 1:
$ /usr/bin/git --no-pager diff --no-index --color=always -u _build/default/mytest.expected _build/default/mytest.output
diff --git a/_build/default/mytest.expected b/_build/default/mytest.output
index d6459e0..9dac0ea 100644
--- a/_build/default/mytest.expected
+++ b/_build/default/mytest.output
@@ -1 +1 @@
-xxx
+this is mytest

My goal is for dune build to not run the test, and only run it with dune test.

Second attempt: explicit rules

I tried explicitly writing my own rules in the dune file, also closely following what is in the manual:

(test
  (name mytest)
)

; ADDED
(rule
  (with-stdout-to tests.output
    (run ./mytest.exe)))

; ADDED
(rule
  (alias runtest)
  (action
    (diff tests.expected tests.output)
  )
)

but the same thing happens: tests.output is created by dune build. (I also created tests.expected.)

Third attempt: more (alias runtest)?

I tried making the generation rule part of runtest:

(test
  (name mytest)
)

(rule
  (alias runtest)                      ; ADDED
  (action                              ; ADDED 'action' wrapper
    (with-stdout-to tests.output
      (run ./mytest.exe)
    )
  )
)

(rule
  (alias runtest)
  (action
    (diff tests.expected tests.output)
  )
)

but this still creates tests.output during dune build.

Fourth attempt: enabled_if

I looked at enabled_if but didn't see any promising condition to test.

Fifth attempt: one rule

I tried putting all of the logic into one (alias runtest) rule:

(test
  (name mytest)
)

; COMBINED
(rule
  (alias runtest)
  (action
    (progn
      (with-stdout-to tests.output
        (run ./mytest.exe)
      )
      (diff tests.expected tests.output)
    )
  )
)

but not only does the test still run during dune build (why?), but now dune promote fails if run afterward:

$ dune promote
Skipping promotion of _build/default/tests.output to tests.expected as the
  file is missing.

A clue: tests that have output files always run during the build

After further experimentation, I found that this dune file:

(rule
  (alias runtest)
  (action
    (bash "echo test with no saved output")
  )
)

(rule
  (alias runtest)
  (action
    (with-outputs-to outputs.txt
      (bash "echo test that saves output to a file")
    )
  )
)

will run the first test as part of dune test but the second as part of dune build. So my working hypothesis is any rule that has any output file will run with dune build. That in turn makes it impossible to run a diff test as part of dune test (and not dune build).

Scott McPeak
  • 8,803
  • 2
  • 40
  • 79

0 Answers0