4

Is it possible to parse module-level docstrings with the AST?

I am working on a python documenter here and visiting the module tokens and grabbing the documentation does not yield the module-level docstring. So far, I've had to resort to importing the module and grabbing its __doc__ or using inspect to grab the documentation.

I looked into the pydoc module source for clues as to how other documenters parse docstrings, and discovered that pydoc ends up having to do basically the same thing as my documenter in order to grab the module-level strings.

Am I missing something? Is the only way to parse module-level docstrings through actually importing the module, or is it possible to parse the docstrings out of the AST directly?

mvanveen
  • 9,754
  • 8
  • 33
  • 42

1 Answers1

5

Maybe I miss-understand the question, but can't you just do this (python 2.7.1)?

test file:

"""
DOC STRING!!
"""

def hello():
    'doc string'
    print 'hello'

hello()

Interactive session:

>>> M = ast.parse(''.join(open('test.py')))
>>> ast.get_docstring(M)
'DOC STRING!!'

You can also walk through the ast, looking for the slot the doc string would be in.

>>> M._fields
('body',)
>>> M.body
[<_ast.Expr object at 0x10e5ac710>, <_ast.FunctionDef object at 0x10e5ac790>, <_ast.Expr object at 0x10e5ac910>]
>>> # doc would be in the first slot
>>> M.body[0]._fields
('value',)
>>> M.body[0].value
<_ast.Str object at 0x10e5ac750>
>>> # it contains a string object, so maybe it's the doc string
>>> M.body[0].value._fields
('s',)
>>> M.body[0].value.s
'\nDOC STRING!!\n'
mdaoust
  • 6,242
  • 3
  • 28
  • 29
  • Oh cool! If I understand correctly, is this supposed to be a tool to `eval` examples in the `__doc__` vs what the module itself produces? – yurisich Feb 03 '12 at 03:11
  • That's it! I've been using the `NodeVisitor` class and `ast.get_docstring(node)`. Running it on a module wasn't working. I thought other documenters like `pydoc` might be doing something different, but it ends up using `inspect`. Thanks a lot! – mvanveen Feb 03 '12 at 09:45
  • @Droogans : I think the module you're looking for is `doctest` (`python -m doctest -v mymodule.py`). This is more about getting the doc string without importing the module (importing the module runs it, and that could have side effects), or parsing the text yourself (there's a build in python parser, use it). – mdaoust Feb 03 '12 at 13:20