Best Practices for Text Generation in Python

Question

I'm writing a python script that generates another python script based off an external file. A small section of my code can be seen below. I haven't been exposed to many examples of these kinds of scripts, so I was wondering what the best practices were.

As seen in the last two lines of the code example, the techniques that I'm using can be unwieldy at times.

SIG_DICT_NAME = "sig_dict"

SIG_LEN_KEYWORD = "len"
SIG_BUS_IND_KEYWORD = "ind"
SIG_EP_ADDR_KEYWORD = "ep_addr"

KEYWORD_DEC = "{} = \"{}\""

SIG_LEN_KEYWORD_DEC = KEYWORD_DEC.format(SIG_LEN_KEYWORD, SIG_LEN_KEYWORD)
SIG_BUS_IND_KEYWORD_DEC = KEYWORD_DEC.format(SIG_BUS_IND_KEYWORD, 
                                             SIG_BUS_IND_KEYWORD)
SIG_EP_ADDR_KEYWORD_DEC = KEYWORD_DEC.format(SIG_EP_ADDR_KEYWORD, 
                                             SIG_EP_ADDR_KEYWORD)

SIG_DICT_DEC = "{} = dict()"
SIG_DICT_BODY_LINE = "{}[{}.{}] = {{{}:{}, {}:{}, {}:{}}}"

#line1 = SIG_DICT_DEC.format(SIG_DICT_NAME)
#line2 = SIG_DICT_BODY.format(SIG_DICT_NAME, x, y, z...)

I am very curious to know - what problem are you trying to solve by doing this, ie, what is the code you are trying to generate? — Hugh Bothwell, Feb 07 '15 at 18:32
the best practice is to use more structured less magical methods such as [functions, classes, `hasattr`/`getattr`, metaclasses](https://github.com/lihaoyi/macropy#levels-of-magic) *before* you think about text generation. — jfs, Feb 07 '15 at 18:41
@HughBothwell The external file is a list of digital signals that exist on some bus at a specific index. This file adheres to some constraints (such as maximum index or maximum bus address), but the names of the signals, their busses, and their index can change. I want to be able to generate Python that can parse the external file and create a dictionary mapping a signal name to its properties(bus, index, anything else in the future) and write functions for each signal to set or read the signal. — jwp36, Feb 07 '15 at 18:41
There probably is a much cleaner way than dynamically generating a Python script. — Dr. Jan-Philip Gehrcke, Feb 07 '15 at 18:46
Would it be possible to see a small example input-file and the related desired output? — Hugh Bothwell, Feb 07 '15 at 18:56
@HughBothwell I'm afraid I can't upload the input-file. In the end, another person will be using the generated python to set the value of a signal without needing to worry about its bus or its index. An example of that function call might be: set_sig(sig_name, value). — jwp36, Feb 07 '15 at 19:13
Instead of building the file line-by-line, could you use a template file and the [Template class](https://docs.python.org/2/library/string.html#string.Template)? This approach is commonly used in web development (for generating HTML content) and generally makes the templates much more readable. — Ted Pudlik, Feb 07 '15 at 22:22

score 1 · Answer 1 · answered Feb 07 '15 at 19:03

You don't really see examples of this kind of thing because your solution might be a wee bit over-engineered ;)

I'm guessing that you're trying to collect some "state of things", and then you want to run a script to process that "state of things". Rather than writing a meta-script, what is typically far more convenient is to write a script that will do the processing (say, process.py), and another script that will do the collecting of the "state of things" (say, collect.py).

Then you can take the results from collect.py and throw them at process.py and write out todays_results.txt or some such: collect.py -> process.py -> 20150207_results.txt

If needed, you can write intermediate files to disk with something like:

with open('todays_progress.txt') as f_out:
    for thing, state in states_of_things.iteritems():
        f.write('{}<^_^>{}\n'.format(state, thing))

Then you can parse it back in later with something like:

with open('todays_progress.txt') as f_in:
    lines = f_in.read().splitlines()
things, states = [x, y for x, y in lines.split('<^_^>')]
states_of_things = dict(zip(things, states))

More complicated data structures than a flat dict? Well, this is Python. There's probably more than one module for that! Off the top of my head I would suggest json if plaintext will do, or pickle if you need some more detailed structures. Two warnings with pickle: custom objects don't always get reinstantiated well, and it's vulnerable to code injection attacks, so only use it if your entire workflow is trusted.

Hope this helps!

score 0 · Answer 2 · answered Feb 07 '15 at 18:54

You seem to be translating keyword-by-keyword.

It would almost certainly be better to read each "sentence" into a representative Python class; you could then run the simulation directly, or have each class write itself to an "output sentence".

Done correctly, this should be much easier to write and debug and produce more idiomatic output.

Best Practices for Text Generation in Python

2 Answers2