(How) do you run "functional" tests on your services automatically?

Question

Consider a simple FreeBSD machine running an SMTP server - how can I automatically verify that it's doing the right thing (e.g. accept incoming SMTP connections for certain recipients, and drop the mail in some Maildir)?

We already use server monitoring software (in this cas, Nagios) and of course we do manual tests, but I was wondering: is there some common way to run automatic functional tests on server services?

An example of what I'm thinking of: when reconfiguring our server (which usually happens inside a VM for testing purposes), I'm considering to

1. Pipe a few manually crafted SMTP sessions into netcat, which connects to the SMTP port of our server and then
2. Run some sort of 'verfication' script on the server which makes sure that expected assertions hold (such as: new files with expected contents appeared on the server, log entries were created, etc.).

I imagine being able to execute a series of such tests for other services (e.g. testing that backups can be created, testing that the IMAP server accepts connections for certain users and that it lists the right mail) might be useful for testing configuration changes, or for verifying that restored system backups work as expected.

Answer 1

This kind of tests can be more or less easily written as a Nagios test (how easy depends on the service in question). Remember, a Nagios test is a simple thing, at least the interface to Nagios, it's just a program or shell script returning a well-defined answer (an return code and some informational text).

Also, there are many modules already available for all kind of services, check the Nagios exchange. A solution for your mail example is this.

Answer 2

You could use expect to interact with the service e.g.

#!/usr/bin/expect -f
set timeout 1
spawn telnet localhost 25
expect "220 *"
send -- "helo localhost\n"
expect -- "250*Hello\ localhost*"
send -- "mail from: root@localhost\n"
expect -- "250\ OK"
send -- "rcpt to: root@localhost\n"
expect -- "250\ Accepted"
send -- "data\n"
expect -- "354*"
send -- "functional test\n.\n"
expect -- "250\ OK*"
send -- "quit\n"
expect "221*closing\ connection"

call the above like so

/path/to/expect.script | grep -q "250 OK"
if [ $? = 0 ]
then
    echo "Message queued successfully"
else
    echo "Message Failed to queue"

You can then go on to see if the message was delivered correctly into the mailspool or Maildir or whatever. You can change localhost for the remote host of your choice althought hat then complicates the delivery checking but you could always use expect to log in - it's a very powerful tool.

Answer 3

I use NAGIOS.

If I'm interested in testing an individual function, which forms a step in a chain that provides some business service, I test that: to test an SMTP server, I'll use the SMTP plugin that connects to port 25 and looks for a suitable banner; ditto an IMAP server; and so on.

If I'm interested in testing an end-to-end sequence, the success or failure of the whole chain, I'll check for its successful completion. To test a chain like you describe, I might well have a cron job that sends mails containing a certain test string every 30 minutes to a certain address, and I'll have a NAGIOS plugin that looks in the recipient mailbox for this string, and checks that it's being updated regularly.

Usually, I'll do both, so if the end-to-end test goes into ALERT state, I expect to see that one or more of the intermediate steps has also gone into alert - maybe the SMTP receiver is down, or the network link between the two has failed, or the disc has filled up on the receiver - and that will give me a first clue in investigating the end-to-end failure. Often, I'll group the individual step tests and the end-to-end test into a single servicegroup, for a fast and easy view of failed components.

But I believe that individual NAGIOS plugins should always be small and lightweight. So note how the end-to-end test above tests only the final success or failure of the chain; it does nothing to test the steps of the chain. The more complexity there is in an individual plugin, the more like it is to fail in unexpected ways, and the more likely I then am to get false positives (or worse, false negatives) from my monitoring system.