How to parse a domain-specific language on top of Python? What architecture with IPython?

Question

I want to construct a domain-specific language as a superset of Python. Cryptic commands like

f7:10y=x^2

designed to minimize typing shall be parsed into plain Python

for k in range(7,10):
    f[k].set_y( expr='x^2' )

before being executed. Probably, the command-line interface shall be IPython.

What would be an appropriate architecture: Shall I implement the cryptic-to-plain-Python translation in the IPython command-line shell or in its kernel daemon? Are there helpful libraries / tutorials / examples?

Or more generically: Are there examples how to add complex syntactic sugar to Python?

score 1 · Answer 1 · answered Dec 03 '14 at 15:27

1

With PyPy this is fairly easy to do. You can just add some things to the grammar file and you're done.

Here's an example: https://bitbucket.org/adonohue/units/src/bb1b20dd739f73fe495723d24cd266b67549f5c9/unitPython/patches?at=default

It allows the units library to do things like these:

print(2cm / 0.5 s)
-> 4.0 cm / s

answered Dec 03 '14 at 15:27

Wolph

78,177
11
137
148

Thank you, @Wolph. Looks good. Now let me wait for part 2 of my question, whether somebody can advise me on IPython integration. Would PyPy go into the shell or the kernel? – Joachim W Dec 03 '14 at 15:36
Although I have not tested this, I would assume you can simply install IPython within the custom Pypy install to make this work. – Wolph Dec 03 '14 at 15:39
Now I see that PyPy is not on top of CPython, but a replacement thereof, which would bar other degrees of freedom (e.g. migrating one day to Cython). Hence not exactly the solution I was hoping for. – Joachim W Dec 03 '14 at 15:49
https://bitbucket.org/adonohue/units/ is no longer available. Does anyone know where it might live now, or of a similar project? – Mike Slinn Mar 03 '21 at 22:00

score 1 · Accepted Answer · edited Jun 20 '20 at 09:12

A solution that will work everywhere is not really an option here. What you are trying to achieve is modifying the core of the Python parser.

The options are:

Use a preparser. Within IPython there are hooks available for this: http://ipython.org/ipython-doc/dev/config/inputtransforms.html

Here's an example for your use case:

import re

range_search = re.compile('''
    (?P<variable>.+)
    (?P<start>\d+):(?P<stop>\d+)
    (?P<k>[^=]+)=(?P<v>.+)
''', re.VERBOSE)

range_replace = '''for k in range(\g<start>, \g<stop>):
    \g<variable>[k].set_\g<k>(expr='\g<v>')'''

print range_search.sub(range_replace, 'f7:10y=x^2')

@StatelessInputTransformer.wrap
def inline_loop(line):
    return range_search.sub(range_replace, line)

Recompile Python with your modifications
Use a language which is more flexible in this regard like ruby

The last two options are excluded in my application context. I definitely meant to ask for an appropriate preparser, and how to integrate it with IPython. — Joachim W, Dec 03 '14 at 16:59

How to parse a domain-specific language on top of Python? What architecture with IPython?

2 Answers2