Handle nested fields with conversion types in string with string.Formatter

Question

Update 2

Alright, my answer to this question is not a complete solution to what I originally wanted but it's ok for simpler things like filename templating (what I originally intended to use this for). I have yet to come up with a solution for recursive templating. It might not matter to me though as I have reevaluated what I really need. Though it's possible I'll need bigger guns in the future, but then I'll probably just choose another more advanced templating engine instead of reinventing the tire.

Update

Ok I realize now string.Template probably is the better way to do this. I'll answer my own question when I have a working example.

I want to accomplish formatting strings by grouping keys and arbitrary text together in a nesting manner, like so

# conversions (!):
# u = upper case
# l = lower case
# c = capital case
# t = title case

fmt = RecursiveNamespaceFormatter(globals())

greeting = 'hello'
person = 'foreName surName'
world = 'WORLD'

sample = 'WELL {greeting!u} {super {person!t}, {tHiS iS tHe {world!t}!l}!c}!'

print(fmt.format(sample))

# output: WELL HELLO Super Forename Surname, this is the World!

I've subclassed string.Formatter to populate the nested fields which I retrieve with regex, and it works fine, except for the fields with a conversion type which doesn't get converted.

import re
from string import Formatter

class RecursiveNamespaceFormatter(Formatter):
   def __init__(self, namespace={}):
       Formatter.__init__(self)
       self.namespace = namespace

   def vformat(self, format_string, *args, **kwargs):
       def func(i):
           i = i.group().strip('{}')
           return self.get_value(i,(),{})
       format_string = re.sub('\{(?:[^}{]*)\}', func, format_string)
       try:
           return super().vformat(format_string, args, kwargs)
       except ValueError:
           return self.vformat(format_string)

   def get_value(self, key, args, kwds):
       if isinstance(key, str):
           try:
               # Check explicitly passed arguments first
               return kwds[key]
           except KeyError:
               return self.namespace.get(key, key) # return key if not found (e.g. key == "this is the World")
       else:
           super().get_value(key, args, kwds)

   def convert_field(self, value, conversion):
       if conversion == "u":
           return str(value).upper()
       elif conversion == "l":
           return str(value).lower()
       elif conversion == "c":
           return str(value).capitalize()
       elif conversion == "t":
           return str(value).title()
       # Do the default conversion or raise error if no matching conversion found
       return super().convert_field(value, conversion)

# output: WELL hello!u super foreName surName!t, tHiS iS tHe WORLD!t!l!c!

What am I missing? Is there a better way to do this?

You mean like `this {is a \{string!c\}!u}`? I guess i could to that but it's not ideal. — fivethous, May 22 '22 at 18:11

fivethous · Answer 1 · 2022-05-26T19:53:00.287

Recursion is a complicated thing with this, especially with the limitations of python's re module. Before I tackled on with string.Template, I experimented with looping through the string and stacking all relevant indexes, to order each nested field in hierarchy. Maybe a combination of the two could work, I'm not sure.

Here's however a working, non-recursive example:

from string import Template, _sentinel_dict

class MyTemplate(Template):
    delimiter = '$'
    pattern = '\$(?:(?P<escaped>\$)|\{(?P<braced>[\w]+)(?:\.(?P<braced_func>\w+)\(\))*\}|(?P<named>(?:[\w]+))(?:\.(?P<named_func>\w+)\(\))*|(?P<invalid>))'

    def substitute(self, mapping=_sentinel_dict, **kws):
        if mapping is _sentinel_dict:
            mapping = kws
        elif kws:
            mapping = _ChainMap(kws, mapping)

        def convert(mo):

            named = mapping.get(mo.group('named'), mapping.get(mo.group('braced')))
            func = mo.group('named_func') or mo.group('braced_func') # i.e. $var.func() or ${var.func()}

            if named is not None:

                if func is not None:
                    # if named doesn't contain func, convert it to str and try again.
                    callable_named = getattr(named, func, getattr(str(named), func, None))
                    if callable_named:
                        return str(callable_named())
                return str(named)

            if mo.group('escaped') is not None:
                return self.delimiter
            if mo.group('invalid') is not None:
                self._invalid(mo)
            if named is not None:
                raise ValueError('Unrecognized named group in pattern',
                                 self.pattern)

        return self.pattern.sub(convert, self.template)

sample1 = 'WELL $greeting.upper() super$person.title(), tHiS iS tHe $world.title().lower().capitalize()!'
S = MyTemplate(sample1)
print(S.substitute(**{'greeting': 'hello', 'person': 'foreName surName', 'world': 'world'}))

# output: WELL HELLO super Forename Surname, tHiS iS tHe World!

sample2 = 'testing${äää.capitalize()}.upper()ing $NOT_DECLARED.upper() $greeting '
sample2 += '$NOT_DECLARED_EITHER ASDF$world.upper().lower()ASDF'
S = MyTemplate(sample2)
print(S.substitute(**{
        'some_var': 'some_value',
        'äää': 'TEST',
        'greeting': 'talofa',
        'person': 'foreName surName',
        'world': 'världen'
}))

# output: testingTest.upper()ing  talofa  ASDFvärldenASDF

sample3 = 'a=$a.upper() b=$b.bit_length() c=$c.bit_length() d=$d.upper()'
S = MyTemplate(sample3)
print(S.substitute(**{'a':1, 'b':'two', 'c': 3, 'd': 'four'}))

# output: a=1 b=two c=2 d=FOUR

As you can see, $var and ${var} works as expected, but the fields can also handle type methods. If the method is not found, it converts the value to str and checks again.

The methods can't take any arguments though. It also only catches the last method so chaining doesn't work either, which I believe is because re do not allow multiple groups to use the same name (the regex module does however).

With some tweaking of the regex pattern and some extra logic in convert both these things should be easily fixed.

MyTemplate.substitute works like MyTemplate.safe_substitute by not throwing exceptions on missing keys or fields.

Handle nested fields with conversion types in string with string.Formatter

Update 2

Update

1 Answers1