Are there any non-Lisp dialects that allow for syntactic abstraction?

Question

As Rich Hickey says, the secret sauce of Lisp languages is the ability to directly manipulate the Abstract Syntax Tree through macros. Can this be achieved in any non-Lisp dialect languages?

Answer 1

Being able to "directly manipulate the abstract syntax tree" by itself is nothing new, though it's something that very few languages have. For example, many languages these days have some kind of an eval function -- but it should be obvious that that's not manipulating the abstract syntax tree, instead, it is a manipulation of the concrete syntax -- the direct source code. Incidentally, the mentioned functionality in D falls under the same category, as is CPP: both deal with raw source text.

To give an example of a language that does have that feature (but not something that would be considered macros proper), see OCaml. It has a syntactic extension system, CamlP4, which is essentially a compiler extension toolkit, and it revolves around the OCaml abstract syntax as its most important purpose. But this is still not what makes the corresponding feature in Lisps so great.

The important feature of Lisps is that the extensions that you get using macros are part of the language in the same way that any other syntactic form is. To put this differently, when you use something like if in a Lisp, there is no difference in functionality whether it's implemented as a macro or as a primitive form. (Actually there is a minor difference: in some cases it's important to know the set of primitive forms that don't expand further.) More specifically, a Lisp library can provide plain bindings and macros, which means that libraries can extend the language in a much more interesting way than the usual boring extensions you get in most languages, capable of adding only plain bindings (functions and values).

Now, viewed in this light, something like the D facility is very similar in nature. But the fact that it deals with raw text rather than ASTs limit its utility. If you look at the example on that page,

mixin(GenStruct!("Foo", "bar"));

you can see how this doesn't look like part of the language -- to make it more like Lisp, you'd use it in a natural way:

GenStruct(Foo, bar);

with no need for a mixin keyword that marks where a macro is used, no need for that !, and the identifiers being specified as identifiers rather than strings. Even better, the definition should be expressed more naturally, something like (inventing some bad syntax here):

template expression GenStruct(identifier Name, identifier M1) {
    return [[struct $Name$ { int $M1$; }; ]]
}

One important thing to note here is that since D is a statically typed language, ASTs have crept into this mental exercise in an explicit way -- as the identifier and expression types (I'm assuming here that template marks this as a macro definition, but it still needs a return type).

In Lisp, you're essentially getting something very close to this functionality, rather than the poor string solution. But you get even more -- Lisp intentionally puns over the basic list type, and unifies the ASTs with the runtime language in a very simple way: the AST is made of symbols and lists and other basic literals (numbers, strings, booleans), and those are all part of the runtime language. In fact, for those literals, Lisp takes another step forward, and uses the literals as their own syntax -- for example, the number 123 (a value that exists at runtime) is represented by a syntax which is also the number 123 (but now it's a value that exists at compile-time). The bottom line of this is that macro-related code in Lisp tends to be far easier to deal with than what other languages call "macro"s. Imagine, for example, making the D example code create N int fields in a struct (where N is a new input to the macro) -- that would require using some function to translate a string into a number.

Answer 2

Lisp

The reasons LISP is "special" are...

The built-in functionality is very economical:

• The only built-in data structures are atoms, or lists
• The syntax is implemented in terms of the list data structure
• There are very few "system functions"

It supports functions in such a way that new function definitions are indistinguishable from built-in functions:

• The calling syntax is identical
• Evaluation of arguments can be fully controlled

It supports macros in such a way that arbitrary Lisp code can always be defined in terms of a domain-specific language:

• The calling syntax is just like custom function-call syntax, which is just like built-in function-call syntax
• Evaluation of arguments is completely controllable
• Arbitrary Lisp code-generation is possible
• Macros are evaluated at runtime, so the macro's implementation can call existing code while generating new code

With the above features, you can:

• Re-implement Lisp-within-Lisp, in very little code
• Add any existing programming idioms in a way that is indistinguishable from built-in features

E.g. you can easily implement systems for namespaces, any data structure, classes, polymorphism, and multiple-dispatch on top of Lisp, and such features will work like they were built into Lisp.

Other languages

But it all depends on your definition. Some levels of "syntactic abstraction" are supported in other languages in quite varied ways. Some of these ways are more powerful than others, and nearly match Lisp's flexibility.

Some examples:

• Boo (syntactic macros, and other features)
  http://boo.codehaus.org/
  Boo Wikipedia Entry (for the feature list)

In Boo, you can use syntactic macros to define new DSLs that will automatically be handled by the compiler. With this, you can implement any language feature on top of existing features. The limitation compared to Lisp is that these are evaluated at compile time, so run-time code generation isn't directly supported.

• Javascript/Lua (dynamic evalution, prototypes, and . notation for functions inside arrays)
  http://en.wikipedia.org/wiki/JavaScript#Prototype-based

In Javascript, the data structures are generic and flexible (everything is either a built-in type, or an associative array). It also supports invoking functions directly from associative arrays. With this, you can implement several language features on top of existing features, such as classes and namespaces.

Because Javascript is a dynamic language (names of function calls are evaluated at runtime), and because it exposes built-in features within the context of data structures, it is fully "reflective" and fully mutable.

Because of this, you can replace or shim the existing system functionality with your own functionality. This is often quite useful in shimming in your own runtime debugging features, or for sand-boxing (by un-defining system calls you don't want isolated code to access).

Lua is quite similar to Javascript in most of these ways.

• C/C++ Pre-processor macros
  http://en.wikipedia.org/wiki/C_preprocessor

The C++ pre-processor allows you to define your own DSL with a somewhat similar syntax to existing function calls. It does not let you control evaluation (which is the source of a lot of bugs, and why most people say C/C++ macros are "Evil"), but it does support a somewhat limited form of code generation.

The code generation support in C/C++ macros is limited because macros are evaluated before your code is compiled, and can't be controlled via C code. It is nearly completely limited to textual substitution. This greatly limits the type of code that can be generated.

• C++ Templates
  http://en.wikipedia.org/wiki/Template_metaprogramming

The C++ template feature is quite powerful (WRT to C/C++ macros) for syntactical additions to the language. It can turn a lot of runtime code evaluation into compile-time code evalution, and can do static assertions on your existing code. It can reference existing C++ code, in a limited way.

But template meta-programming (TMP) is very unwieldy because it has a terrible syntax, is a very strictly limited subset of C++, has quite limited code generation ability, and can't be evaluated at runtime. C++ templates also arguably output the most difficult error messages you will ever encounter in programming :)

Note that this hasn't kept template meta-programming from being an active area of research in many communities. See the boost project, of which a good portion is devoted to TMP-support libraries, and TMP-implemented libraries.

• Duck-Typing in Python (and many other languages)
  http://en.wikipedia.org/wiki/Duck_typing

Duck typing can allow you to define a syntax on objects that lets you substitute implementations at runtime. This is similar to how Javascript defines functions on associative arrays.

I can't say for Python (since I don't know it very well), but duck typing is often more limited than Javascript's dynamic features because of a lack of reflectivity, mutability, and exposure of system functionality through reflectable/mutable interfaces. For example, C#'s duck typing is limited in all these ways.

Answer 3

For a sake of completeness, in addition to the already mentioned languages and preprocessors:

• Template Haskell
• Katahdin
• MetaOCaml
• MetaLua (unless you account it as a Lisp dialect)
• Nemerle
• Converge

Answer 4

d

I'm not sure if you'd call it "syntactic abstraction" per se, but it certainly can do much of what Lisp can do: The mixin keyword lets you convert a string into code (in a much better manner than C macros), which, when combined with templates (which are much better than those in C++) you can do pretty much anything you want.

Answer 5

Prolog would be such a language. There are many Prolog dialects. One idea is that their basic building block is a term (similar to an s-expression encoding a function). There are parsers that provide macro facilities for that.

Answer 6

I would say Tcl qualifies -- well, depending on whether you consider Tcl a Lisp or not.

The standard grouping characters { } are actually just a string literal (with no variable interpolation), and there's an eval, so you can easily define your own control flow or looping syntax (and people often do).