8

Is there a way to write a python doctest string to test a script intended to be launched from the command line (terminal) that doesn't pollute the documentation examples with os.popen calls? 

#!/usr/bin/env python
# filename: add
"""
Example:
>>> import os
>>> os.popen('add -n 1 2').read().strip()
'3'
"""

if __name__ == '__main__':
    from argparse import ArgumentParser
    p = ArgumentParser(description=__doc__.strip())
    p.add_argument('-n',type = int, nargs   = 2, default = 0,help  = 'Numbers to add.')
    p.add_argument('--test',action = 'store_true',help  = 'Test script.')
    a = p.parse_args()
    if a.test:
        import doctest
        doctest.testmod()
    if a.n and len(a.n)==2:
        print a.n[0]+a.n[1]

Running doctest.testmod() without using popen just causes a test failure because the script is run within a python shell instead of a bash (or DOS) shell.

The advanced python course at LLNL suggests putting scripts in files that are separate from .py modules. But then the doctest strings only test the module, without the arg parsing. And my os.popen() approach pollutes the Examples documentation. Is there a better way?

hobs
  • 18,473
  • 10
  • 83
  • 106
  • 1
    Am I missing something or could this by solved by adding a `main` function? Do the argument parsing in the `if __main__` block, then call `main(parsed_args)` – Daenyth Apr 02 '12 at 19:02
  • Unfortunately that wouldn't change anything. A main function isn't special. Breaking some of the stuff in if __main__ out into a separate function doesn't change the behavior of the doctest at all. You still can't run it like a shell command, as the script is intended to be used (and documented). – hobs Jul 12 '12 at 14:15

3 Answers3

3

Just found something looking like the answer you want: shell-doctest.

dmitry_romanov
  • 5,146
  • 1
  • 33
  • 36
  • Awesome. Just what I was looking for. Now there's a much better chance that python will start to make inroads in the shell scripting world and I don't have to clutter my doctext with shell-script conversion stuff. – hobs Jul 12 '12 at 14:11
  • 2
    Looks suspiciously Python 2-only. :-( – hBy2Py Jan 11 '18 at 22:58
1

doctest is meant to run python code, so you have to do a conversion somewhere. If you are determined to test the commandline interface directly via doctest, one possibility is to do a regexp substitution to __doc__ before you pass it to argparse, to take out the os.popen wrapper:

clean = re.sub(r"^>>> os\.popen\('(.*)'\).*", r"% \1", __doc__)
p = ArgumentParser(description=clean, ...)

(Of course there are all sorts of nicer ways to do that, depending on what you consider "nice").

That'll clean it up for the end user. If you also want it to look cleaner in the source, you can go the other way: Put commandline examples in the docstring and don't use doctest.testmodule(). Run your docstring through doctest.script_from_examples and post-process it to insert the os calls. (Then you'll have to embed it into something so you can test it with run_docstring_examples.) doctest doesn't care if the input is valid python, so you can do the following:

>>> print doctest.script_from_examples("""
Here is a commandline example I want converted:
>>> add -n 3 4
7
""")
# Here is a commandline example I want converted:
add -n 3 4
# Expected:
## 7

This will still expose the python prompt >>> in the help. If this bothers you, you may just have to process the string in both directions.

alexis
  • 48,685
  • 16
  • 101
  • 161
  • That is a nice way to hide the docstring that tests the argument parser, but it doesn't provide any examples (with expected output) when the user runs `add --help` from the OS shell. Your use of `sys.argv` seems rougly equivalent to the `os.popen` in my code, and it looks equally ugly when the docstring is used for documentation and 'help' rather than for doctest testing. – hobs Apr 02 '12 at 16:49
  • Ok, if you really want commandline-oriented doc and tests, see the new answer. – alexis Apr 02 '12 at 18:36
  • Wow. Quite tricky. Thanks for getting me out of a bind. I understand now why you didn't propose all this complexity in your first answer. Maybe doctest or argparse will incorporate some shell testing features in the future. `$$$` instead of `>>>` anyone? – hobs Apr 04 '12 at 00:49
0

You can also load the docstring yourself and execute the command, like in this test.

import sys

module = sys.modules[__name__]
docstring = module.__doc__
# search in docstring for certain regex, and check that the following line(s) matches a pattern.
bryant1410
  • 5,540
  • 4
  • 39
  • 40
  • That sounds a lot like creating your own doctest module. – hobs Aug 22 '19 at 01:25
  • Yeah, it is. You can make it call `parse_args` so it's much faster than spawning processes. – bryant1410 Aug 22 '19 at 20:46
  • Yeah, that would speed things up. But you'd add a lot of complexity and potential bugs to your testing infrastructure... which is really scary for me. – hobs Aug 27 '19 at 18:10
  • 1
    I'm not sure it's a lot. It wasn't for me at least. I did it here: https://github.com/allenai/allennlp/pull/3185/files#diff-1798bc8ccd6e79010bafe393669f6b49 – bryant1410 Aug 27 '19 at 18:15
  • I'm impressed! A single regex is a lot for me. Regex's are finicky and prone to unexpected incorrect behavior. It took you some trial and error to get that one to work right for your example. I'd be surprised if it worked reliably in all your projects and docstrings, much less everyone else's. – hobs Aug 29 '19 at 03:45
  • It doesn't really took me a lot of trial and error. It was quite simple. I just did 2 different PRs cause the first one spawns processes, which is slow, and the second one calls `argparse` methods. I believe that if you always follow a simple format, such as "$ program-name [args]" and the output goes on in the following line and then there's a blank space or something (some sentinel maybe), it's pretty much straightforward. I do agree regex can be tricky in some contexts (e.g., parsing CSV files) but here it seems straightforward for one project. – bryant1410 Aug 29 '19 at 19:15
  • I admit the only insidious examples I can come up with are unlikely: `try my\n$ 1000 program` or `we signed\n$ million contract` – hobs Sep 03 '19 at 03:49
  • In my case, I put that it should start with "$". – bryant1410 Sep 04 '19 at 13:00
  • My failing test case examples do all start with "$". The preceding text is on the line above. – hobs Sep 05 '19 at 18:48
  • I mean the line starts with it. – bryant1410 Sep 06 '19 at 19:07
  • I get what you mean now. You can do it only there the previous line is blank maybe. – bryant1410 Sep 06 '19 at 19:17