Flat evaluation of Lisp s-expressions

Question

I'm trying to figure out how I could implement Lisp evaluation non-recursive. My C based evaluator is Minimal Lisp file l1.c. However several functions there recurse into eval again: eval, apply, evargs, evlist and also the Lisp Ops defineFunc, whileFunc, setqFunc, ifFunc...

I'm trying to figure out an evaluation that is flat. Some possible ways I could come up with:

Transforming to byte code and execute in VM
Implement a flat Forth evaluator and implement Lisp evaluation in Forth, this is kind of what lf.f does.
Another possibility might be to join all recursinge functions in l1.c into one big switch loop. Local variables would be joined into a heap-based struct, calls to recursing subfunctions would be implemented by a heap-based return-stack.

My question is: Are there algorithms/papers/implementations that do flat evaluation in different ways. I'm searching for an implementation that don't transform into byte-code but something similar to the recursion-less "depth-first traversal" using a pushdown stack. I'd like to operate on the original s-expression.

Answer: when implementing the evaluator in c you need to implement the whole thing in a flat loop, implement the return stack and stackframes by hand, model the control flow using goto and switch(). Here is an example: flat .

in general, you answered your question already: have a pushdown stack (on heap) and run a while loop, pushing stuff-to-be-evaluated onto that stack, in a correct order, together with their relevant information, maybe defining your own opcodes to guide you on what to do with the popped entity. Have you read SICP or "Lisp in Small Pieces" or "EOPL" or "Lisp 1.5 manual" or McCarthy's original paper or "the Lambda papers" or the "FUNARG problem" paper? Have you decided on Lisp-1 vs Lisp-2? Environment stack tree vs shallow binding? Dynamic or lexical scoping? — Will Ness, Oct 23 '12 at 14:02
Thanks for the comment. I didnt know there is a name for this kind of problem: "FUNARG problem". Thanks for the many references, now I have some guidlines. The rational is well: The stack in a c-based evaluator is created by the c-compiler. To get rid of the stack you'd need to rewrite the c-compiler to be heap based (which is not what I can do) or write controlflow by hand. Nobody has thought about that maybe, a c-compiler that doesnt use a stack. I'll post my heap-based c-evaluator implementation when it's done. The evaluator is based on Ian Piumarta's lysp.c. Lisp-1, tree, lexical. — Konrad Eisele, Oct 26 '12 at 09:11
Here is the Lisp-evaluator written in c, rewritten as a flat loop: [flat evaluator](https://github.com/eiselekd/MinimalLisp/blob/master/flat1.c) . (The whole thing is an experiment, I'm not intending to write something complete like Scheme, it is explained in [MinimalLisp](https://github.com/eiselekd/MinimalLisp) ). _"It you dont understand Unix you are doomed to reimplement it"_ can be rewritten to: _"If you dont (re)implement Unix yourself you're doomed to not understand it"_ ... Maybe that's the main reason why I wrote this. — Konrad Eisele, Oct 28 '12 at 14:45
btw you can post your own answer and accept it, if no other answer was satisfactory. :) — Will Ness, Oct 28 '12 at 17:36

score 4 · Answer 1 · answered Oct 21 '12 at 07:46

A very important aspect of Lisp, and in fact an important aspect of many functional languages that followed, is that it is compositional. This means that the meaning of an expression is defined using the meanings of its subexpressions -- or, in other words, the definition of evaluation is something that is inherently recursive. In non-functional languages there are some differences as in expressions vs statements, but even there expressions are not limited in some way, so recursion is baked into the definition too. Probably the only cases where the recursiveness of the language's definition is not as apparent, are assembly languages. (Though even there a definition of meaning would, of course, require induction.)

So a fight with some recursion definition of eval is something that you will lose. If you go with compilation to machine code, that code will be recursive (and the generating code would be recursive too). If you do the evaluation via a Forth evaluator, then that evaluator would still be recursive. Even if you go with the CPS suggestion in the other answer, you merely end up having yet another encoding of the stack.

So the bottom line is that the best you can get to is some encoding of the stack that doesn't use the machine stack directly -- no substantial difference, but you usually lose performance (since CPUs handle the stack very efficiently, and an encoding of it on the heap is going to be slower).

I'm not trying to change the semantics of Lisp. Only the underlying evaluator that I try to programm in C [l1.c](https://github.com/eiselekd/MinimalLisp/blob/master/l1.c) I would like to not use the C-compiler generated stack but be heap-based. The biggest benefits you get if you are throwing away the c-compiler implicit stack: Threading gets trivial. Simplicity is more important than performance. I was able to evaluate s-expressions in Forth, the c-compiler generated stack there never grows bigger than a few sub calls, of course you have the FORTH (heap based) stacks that grow. — Konrad Eisele, Oct 21 '12 at 16:11
If your only goal is to avoid the C stack, then your question is not phrased properly. More specifically, it's a general question that is applicable for any situation where you want to avoid it, and a Lisp evaluator is just one example. This also applies to the way to actually implement this: methods like a trampoline loop or something similar are the standard way to do this in all of these situations. CPS, for example, is just a natural way that trades the C call stack for heap-allocated closures that still represent the same stack. — Eli Barzilay, Oct 22 '12 at 02:28

score 3 · Answer 2 · answered Oct 21 '12 at 07:37

3

See this topic: Continuation Passing Style

answered Oct 21 '12 at 07:37

Rainer Joswig

136,269
10
221
346

Continuation Style in C is maybe a "while(op) { op=op(ctx); }" type of execution loop. I'd need to think about how I can rewrite (split up) the [l1.c](https://github.com/eiselekd/MinimalLisp/blob/master/l1.c) 's eval() and apply() to fit in... – Konrad Eisele Oct 21 '12 at 16:01
I thought more about the above "Continuation Passing Style" however it is a transformation of the Lisp expression that should be evaluated. I am searching for such a transformation for the underlying evaluator written in C. Thanks though. – Konrad Eisele Oct 21 '12 at 20:03

score 3 · Accepted Answer · answered Oct 29 '12 at 09:06

When implementing a Lisp-evaluator in C, the C-compiler uses the stack to generate control-flow of subroutine calls. To implement a stack-less evaluator in C you need to write the control-flow by hand using goto and switch():

v *
evargs(ctx *cctx, v *l, v *env)
{
    struct v *r = 0;
    if (l) {
        r = eval(cctx, car(l),env);
        r =  mkCons(r,evargs(cctx, cdr(l),env));
    }
    return r;
}

gets

case EVARGS_0:
    S_SET(0,0);                         /* EVARGS_0: r = 0; */ 
    if (!(v=S(2)))                      /* if (l) */
        goto ret;
    RCALL_EVAL(EVARGS_1, car(v));       /* r = < eval(cctx, car(l),env); > */
    break;    
case EVARGS_1:
    S_SET(3,S(1));                      /* EVARGS_1: < r = ... > */
    RCALL_EVARGS(EVARGS_2, cdr(S(2)));  /*  r =  mkCons(r, < evargs(cctx, cdr(l),env) > ); */
    break;
case EVARGS_2:
    S_SET(0,mkCons(S(3),S(1)));         /* EVARGS_2: < r =  mkCons(r,  evargs(cctx, cdr(l),env)  ); > */
    goto ret;

score 0 · Answer 4 · answered Oct 21 '12 at 08:06

I think the key insight you may be missing is that in a lisp interpreter, a minimal set of functions are implemented as primitives which are non recursive. The exact set of primitives varies, but includes cons, car, cdr and a "most primitive" version of apply that executes one of these real functions rather than the interpreted version of itself.

You should look up John McCarthy's original papers, and/or John Allen's Lisp 1.5

score 0 · Answer 5 · answered Oct 21 '12 at 12:59

I remember having a book on HOPE programming language. It is a functional language similar to ML. It had a thorough conceptual description of its compiler, and thoughts about functional language compilers in general. One observation it made was an argument about Y-combinator. The author suggested that one possible way of dealing with recursive functions would be an implementation of Y-combinator and transformation of each recursive function into a non-recursive one that can be made to recur using Y-combinator.

This would be a similar trick you have in if special form, which provides (and usually suffices) for all kinds of lazy evaluations one can require in a language. In a similar way you could restrict all functions from being recursive, but introduces a special Y function that would allow recursion to commence.

Thanks, I'll try to understand this one. – Konrad Eisele Oct 21 '12 at 16:17 — Konrad Eisele, Oct 21 '12 at 16:17

Flat evaluation of Lisp s-expressions

5 Answers5