Situation
Normally, unit tests like ExUnit should be self-contained with input, function call and desired output, so that the test can run on any system and always tests correctly regardless of environment.
On the other side, if your application does syscalls, for example with Elixir's System.cmd/3 or Erlang's :os.cmd/1 and works with the results, your tests may get different results because of reasons like different/updated binaries, changed circumstances, different operating systems and so on.
Of course, it is good that tests fail in these cases, so that your coverage of real life situations increases. When developing, however, you would want to first get your functions to do the right thing, and only then to do the thing right. If the outside world changes, it is difficult or even impossible to always run the tests predictably.
Additionally, you may want to test for conditions that rarely or almost never happen, but your system calls do not give you that information, because it is very rare to happen indeed. You would need to somehow mock the output of the syscall and separate it from the inner logic of your program.
Example
To keep it simple (the same principle applies in more complicated situations), consider reading the boot time of the system and responding depending on the cleaned result:
def what_time do
time =
:os.cmd('who -b | cut -d\' \' -f14') # Returns something like '13:50\n'
|> to_string
|> String.trim("\n")
|> String.split(":")
|> List.to_tuple
case time do
{"12", "00"} -> {:ok, "It's High Noon!"}
_ -> {:error, "meh"}
end
end
This function can only be tested correctly if you reboot your system at the specific time, which of course is unreasonable. But as the format of the output is roughly known, you could create a list of test values like ['16:04', '23:59', '12:00', "12:00", 2, "xyz", '1.0"] and test the parsing part without the syscall, then compare it to your expected results as usual.
Naive approach
But how is this done? The syscall is the first thing in the function, so if you take it out into a separate function, you could test the syscall, but that does not help you much, because the syscall itself is the problem:
def what_time do
time = get_time
|> to_string
[...]
end
def get_time do
:os.cmd('who -b | cut -d\' \' -f14') # Returns something like '13:50\n'
end
Slightly better...
If you add another helper method that just parses the string/charlist, you can achieve what you want, while making the syscall itself private:
def what_time do
what_time_helper(get_time())
end
def what_time_helper(time) do
time =
time
|> to_string
[...]
end
end
defp get_time do
:os.cmd('who -b | cut -d\' \' -f14') # Returns something like '13:50\n'
end
Now you can call the helper test function in the ExUnit case and the normal program can call the normal function.
... but not good?
While this last idea works in practice, it strikes me as not very elegant. I can see the following downsides:
- Each funtion needs to be split into private syscall, public helper and public normal method, increasing the amount of functions threefold. The resulting code is longer and more difficult to read because of the needless partitioning.
- The helper method needs to be public to be tested, but it should not be exposed to the public. As a result, additional documentation has to be written, the API reference gets longer and the method must do more checks to assure safe operation (whereas before, only values that were produced by the syscall itself could happen).
- Although the small main function only calls the other one with a predefined set, it cannot be included in the test coverage. This complaint is a bit of a nitpick, but I imagine it gets problematic if one uses automatic testing tools that display test coverage in lines of code or number of functions.
Questions
So, my questions would be:
- How to correctly handle such cases in testing, e. g. with ExUnit?
- How to separate syscalls from inner logic and reduce the amount of boilerplate functions?
- Are there any tools or general methods how this is normally done in functional programming?
