templated configuration in python or perl?

Question

I'm trying to implement a templated configuration file. I'd prefer python, but I'd take an answer in perl too. I've used perl for my example.

I've searched a bit and found - python single configuration file - ConfigObj - python configuration file generator - ePerl but I could not from those solve my problem.

I'm trying generate a configuration file mostly in the INI format (with not even sections):

# Comments
VAR1 = value1
EDITOR = vi

and I need that generated from a template where I'm embedding a scripting language inside the text:

# Config:
MYPWD = <:   `pwd`  :>

The text in between the '<:' and ':>' would be in the scripting language (python or perl). As with a template, its stdout is captured and inserted in the resulting text. The templating used in the example is basically eperl, but I'd prefer python if available.

and finally, the defined variables should be reusable:

# Config:
CODE_HOME = /some/path
CODE_BIN = <:=$CODE_HOME:>/bin

Here's the test source file that I read in:

# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
# platform.cfg
# This one variable
VAR =value
# this is a templated variable. The langage is perl, but could be python.
HELLO= <: print 'World' :>
# This is a multi-line code which should resolve to a single line value.
LONGER = <:
 if (1) {
    print "abc ";
 }
 $pwd = `/bin/pwd`;
 chomp($pwd);
 print $pwd;
:>
# Another one to test the carriage returns.
MULTIPLE = /<: print "proj" :>/<: print "tahiti":>/<: 
print "pd/1/";
$system = `grep -w VAR platform.cfg | egrep -v 'print|platform.cfg' | cut -d = -f 2-`;
chomp($system);
print $system;
:>
# variables dependent from the previous variable definition
VAR1 = <: print $VAR :>1
# variables dependent from the previous variable definition
VAR2 = <: print $VAR1 :>2
# variables dependent from the previous variable definition
VAR3 = <: print $VAR2 :>3
# variables dependent from the previous variable definition
VAR4 = <: print $VAR3 :>4
# BTW, multi-line comments are significant
# and should be preserved as the documentation for the
# variable just below:
VAR5 = <: print $VAR4 :>5
VAR6 = <: print $VAR5 :>6
# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

And I'm looking to get this result out of the script. I could not figure how to have the variables defined in the config file be part of the interpreter?

# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
# platform.cfg
# This one variable
VAR =value
# this is a templated variable. The langage is perl, but could be python.
HELLO= World
# This is a multi-line code which should resolve to a single line value.
LONGER = abc /src/byop/CODE
# Another one to test the carriage returns.
MULTIPLE = /proj/tahiti/pd/1/value
# variables dependent from the previous variable definition
VAR1 = value1
# variables dependent from the previous variable definition
VAR2 = value12
# variables dependent from the previous variable definition
VAR3 = value123
# variables dependent from the previous variable definition
VAR4 = value1234
# BTW, multi-line comments are significant
# and should be preserved as the documentation for the
# variable just below:
VAR5 = value12345
VAR6 = value123456
# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Thanks for your suggestions.

This appears to be a very specific and idiosyncratic template dialect. Do you know what format this is expected to be? — SingleNegationElimination, Jul 28 '11 at 00:26
@TokenMacGuy: thanks for the comment. I've edited my question to better describe that I merge a config and a template. So to try to answer your question: the format of the incoming file is a config file with brackets: '<:' and ':>'. Anything in between those separators is in the scripting langage (python, or perl, pick one), with the added constraint that the variables previously defined in the template should be accessible to the scripting langage. (Is that more clear?) — GED, Jul 28 '11 at 00:58
So this is a template language that you are making up on the spot? — SingleNegationElimination, Jul 28 '11 at 01:00
@TokenMacGuy: I'm not making it up on the spot. I used the [eperl](http://marginalhacks.com/Hacks/ePerl/) syntax. I'd be glad to use another templating syntax if it works for generating my config file. [Chomp](http://perldoc.perl.org/functions/chomp.html) is a perl instruction that removes trailing newlines. — GED, Jul 28 '11 at 01:11

score 4 · Answer 1 · answered Jul 28 '11 at 02:21

4

I developed Template::MasonLite for exactly this purpose. I was generating a whole lot of config files (Apache configs in my case) and I needed the script to run on multiple hosts without extra dependencies. So I developed a cut-down version of HTML::Mason that could be embedded directly in my script (an extra ~70 lines). The Mason templating syntax uses Perl for any logic (conditionals, loops etc) rather than making up a whole new language.

The Template::MasonLite implementation is not on CPAN (only at the link above) because I didn't want to clutter CPAN with yet another templating engine and also it wasn't clear what name the module should have.

At the time I developed it I was using cfengine which lacked any sane form of templating. I've since migrated to puppet which includes a templating language. I still use Template::MasonLite for generating presentation slides.

answered Jul 28 '11 at 02:21

Grant McLean

6,898
1
21
37

Lightweight templating systems existing on CPAN: [Text::Template](http://p3rl.org/Text::Template) (configurable delimiters, Perl syntax), [Template::Tiny](http://p3rl.org/Template::Tiny) (`[%` `%]` delimiters, Template-Toolkit syntax) – daxim Jul 28 '11 at 09:34
@Grant: I don't see any support for feeding assignments back into the template engine, as in the OP. – bukzor Jul 28 '11 at 17:32
@Grant: Thanks, I tried it, but I could not find a way for reusing a configuration variable (aka: VAR2 = ) – GED Aug 01 '11 at 16:51
When I needed to do that sort of thing, I calculate a number of values (e.g.: `my $var1 = ...`) in a `<%perl>` section at the top of the template and then interpolate the results in the template (e.g.: `VAR1 = <% $var1 %>`). – Grant McLean Aug 01 '11 at 22:54

score 1 · Accepted Answer · answered Jul 28 '11 at 02:15

If you don't mind using a different syntax there are several template libraries you could use, mako is similar in spirit, Jinaj2 is also pretty nice. Go with a tried and tested library! If you really want to implement your own template library this might give you a start:

import re, StringIO, sys

def exec_block(block, variables):
    """Captures output of exec'd code block"""
    code = compile(block.strip(), '<string>', 'exec')
    _stdout, result = sys.stdout, StringIO.StringIO()
    sys.stdout = sys.__stdout__ = result
    exec(code, variables)
    sys.stdout = sys.__stdout__ = _stdout
    return result.getvalue()

def format_template(template):
    """Replaces code blocks with {0} for string formating later"""
    def sub_blocks(matchobj):
        """re.sub function, adds match to blocks and replaces with {0}"""
        blocks.append(matchobj.group(0)[2:-2].strip())
        return '{0}'

    blocks = []
    template = re.sub(r'<:.+?:>', sub_blocks, template, flags=re.DOTALL).splitlines()
    blocks.reverse()
    return blocks, template

def render_template(blocks, template):
    """renders template, execs each code block and stores variables as we go"""
    composed, variables = [], {}
    for line in template:
        if '{0}' in line:
            replacement = exec_block(blocks.pop(), variables).strip()
            line = line.format(replacement)
        if not line.startswith('#') and '=' in line:
            k, v = [x.strip() for x in line.split('=')]
            variables[k] = v
        composed.append(line)
    return '\n'.join(composed)

if __name__ == '__main__':
    import sys
    with open(sys.argv[1]) as f:
        blocks, template = format_template(f.read())
        print rend_template(blocks, template)

Which basically works like the above, except uses Python for the code blocks. Only supports one block per assignment, which actually seems like the best approach to me. You could feed it a configuration file like:

VAR = value
LONGER = <:
    print 'something'
:>
VAR1 = <: print VAR :>1
# comment
VAR2 = <: print VAR1 :>2
VAR3 = <: print VAR2 :>3
VAR4 = <: print VAR3 :>4

And it would exec each block render the variables out for you:

VAR = value
LONGER = something
VAR1 = value1
# comment
VAR2 = value12
VAR3 = value123
VAR4 = value1234

templated configuration in python or perl?

2 Answers2