Parsing Python Code From Within Python?

Question

We have an older C++ tool that generates some python code automatically. I tried to slog through the C++ source tool, today and pretty much wanted to shoot my self. The thing is what i want to do, is clean up the source created by the tool and link the classes to our internal documentation system via adding sphinx tags.

Now what i am wondering is there some sort of wonderful tool for parsing python code within python?

There is alot of stuff like this:

foo._methods_.append()

Snip 500 lines

foo._methods_.append()

ANy suggestions?

Basically i have a functional but insanely messy code structure, i basically want to extract certain chunks, move them to their own files. And cleanup all the miscellanous things that get created.

I looked quickyl at both parser and AST but i cannot find any real examples of it being used.

score 17 · Accepted Answer · edited May 23 '17 at 12:07

17

You may tokenize python code to parse individual tokens using tokenize module. e.g. Script to remove Python comments/docstrings

or you can use the parser module

or use ast module

edited May 23 '17 at 12:07

Community

1
1

answered Dec 30 '09 at 05:35

Anurag Uniyal

85,954
40
175
219

This helped me understand how python is interpreted. It gives me the vocabulary I need to research it further. Thanks – knowingpark Jun 07 '18 at 12:19

score 3 · Answer 2 · answered Dec 30 '09 at 05:54

Depending on your needs, you may also want to check out the 2to3 library. It was written to automatically facilitate the conversion of Python 2.x apps to Python 3.0, so its main use case is taking one Python source file, performing some transformations on it, and then spitting out the result source file.

One benefit that lib2to3 has over the ast module is that ast does not preserve whitespace and comments, whereas lib2to3 does. If you're already dealing with autogenerated code this might not be a problem for you.

score 1 · Answer 3 · answered Dec 30 '09 at 23:18

Try one of the Python doc-generating utilities to see if this might help you with your overall problem. I've used epydoc with great success.

Or if you can get your boss to part with $200, buy a copy of Enterprise Architect from SparxSystems. It will reverse-engineer your Python code, generate a class diagram, and allow you to click on the class diagram and see the underlying code. Works for many languages besides Python, and an excellent design and documentation utility. (There is a $99 version, but this does not include the code import capability.)

Parsing Python Code From Within Python?

3 Answers3

Linked