how do I pass a list to a common lisp macro?

Question

I am trying to compare the performance of a function and a macro.

EDIT: Why do I want to compare the two? Paul Graham wrote in his ON LISP book that macros can be used to make a system more efficient because a lot of the computation can be done at compile time. so in the example below (length args) is dealt with at compile time in the macro case and at run time in the function case. So, I just wanted how much faster did (avg2 super-list) get computed relative to (avg super-list).

Here is the function and the macro:

(defun avg (args)
   (/ (apply #'+ args) (length args)))

(defmacro avg2 (args)
  `(/ (+ ,@args) ,(length args)))

I have looked at this question How to pass a list to macro in common lisp? and a few other ones but they do not help because their solutions do not work; for example, in one of the questions a user answered by saying to do this:

(avg2 (2 3 4 5))

instead of this:

(avg2 '(2 3 4))

This works but I want a list containg 100,000 items:

(defvar super-list (loop for i from 1 to 100000 collect i))

But this doesnt work.

So, how can I pass super-list to avg2?

Answer 1

First of all, it simply makes no sense to 'compare the performance of a function and a macro'. It only makes sense to compare the performance of the expansion of a macro with a function. So that's what I'll do.

Secondly, it only makes sense to compare the performance of a function with the expansion of a macro if that macro is equivalent to the function. In other words the only places this comparison is useful is where the macro is being used as a hacky way of inlining a function. It doesn't make sense to compare the performance of something which a function can't express, like if or and say. So we must rule out all the interesting uses of macros.

Thirdly it makes no sense to compare the performance of things which are broken: it is very easy to make programs which do not work be as fast as you like. So I'll successively modify both your function and macro so they're not broken.

Fourthly it makes no sense to compare the performance of things which use algorithms which are gratuitously terrible, so I'll modify both your function and your macro to use better algrorithms.

Finally it makes no sense to compare the performance of things without using the tools the language provides to encourage good performance, so I will do that as the last step.

---

So let's address the third point above: let's see how avg (and therefore avg2) is broken.

Here's the broken definition of avg from the question:

(defun avg (args)
   (/ (apply #'+ args) (length args)))

So let's try it:

>  (let ((l (make-list 1000000 :initial-element 0)))
     (avg l))

Error: Last argument to apply is too long: 1000000

Oh dear, as other people have pointed out. So probably I need instead to make avg at least work. As other people have, again, pointed out, the way to do this is reduce:

(defun avg (args)
  (/ (reduce #'+ args) (length args)))

And now a call to avg works, at least. avg is now non-buggy.

We need to make avg2 non-buggy as well. Well, first of all the (+ ,@args) thing is a non-starter: args is a symbol at macroexpansion time, not a list. So we could try this (apply #'+ ,args) (the expansion of the macro is now starting to look a bit like the body of the function, which is unsurprising!). So given

(defmacro avg2 (args)
  `(/ (apply #'+ ,args) (length ,args)))

We get

> (let ((l (make-list 1000000 :initial-element 0)))
    (avg2 l))

Error: Last argument to apply is too long: 1000000

OK, unsurprising again. let's fix it to use reduce again:

(defmacro avg2 (args)
  `(/ (reduce #'+ ,args) (length ,args)))

So now it 'works'. Except it doesn't: it's not safe. Look at this:

> (macroexpand-1 '(avg2 (make-list 1000000 :initial-element 0)))
(/ (reduce #'+ (make-list 1000000 :initial-element 0))
   (length (make-list 1000000 :initial-element 0)))
t

That definitely is not right: it will be enormously slow but also it will just be buggy. We need to fix the multiple-evaluation problem.

(defmacro avg2 (args)
  `(let ((r ,args))
     (/ (reduce #'+ r) (length r))))

This is safe in all sane cases. So this is now a reasonably safe 70s-style what-I-really-want-is-an-inline-function macro.

So, let's write a test-harness both for avg and avg2. You will need to recompile av2 each time you change avg2 and in fact you'll need to recompile av1 for a change we're going to make to avg as well. Also make sure everything is compiled!

(defun av0 (l)
  l)

(defun av1 (l)
  (avg l))

(defun av2 (l)
  (avg2 l))

(defun test-avg-avg2 (nelements niters)
  ;; Return time per call in seconds per iteration per element
  (let* ((l (make-list nelements :initial-element 0))
         (lo (let ((start (get-internal-real-time)))
               (dotimes (i niters (- (get-internal-real-time) start))
                 (av0 l)))))
    (values
     (let ((start (get-internal-real-time)))
       (dotimes (i niters (float (/ (- (get-internal-real-time) start lo)
                                    internal-time-units-per-second
                                    nelements niters)))
         (av1 l)))
     (let ((start (get-internal-real-time)))
       (dotimes (i niters (float (/ (- (get-internal-real-time) start lo)
                                    internal-time-units-per-second
                                    nelements niters)))
         (av2 l))))))

So now we can test various combinations.

OK, so now the fouth point: both avg and avg2 use awful algorithms: they traverse the list twice. Well we can fix this:

(defun avg (args)
  (loop for i in args
        for c upfrom 0
        summing i into s
        finally (return (/ s c))))

and similarly

(defmacro avg2 (args)
  `(loop for i in ,args
         for c upfrom 0
         summing i into s
         finally (return (/ s c))))

These changes made a performance difference of about a factor of 4 for me.

OK so now the final point: we should use the tools the language gives us. As has been clear throughout this whole exercise only make sense if you're using a macro as a poor-person's inline function, as people had to do in the 1970s.

But it's not the 1970s any more: we have inline functions.

So:

(declaim (inline avg))
(defun avg (args)
  (loop for i in args
        for c upfrom 0
        summing i into s
        finally (return (/ s c))))

And now you will have to make sure you recompile avg and then av1. And when I look at av1 and av2 I can now see that they are the same code: the entire purpose of avg2 has now gone.

Indeed we can do even better than this:

(define-compiler-macro avg (&whole form l &environment e)
  ;; I can't imagine what other constant forms there might be in this
  ;; context, but, well, let's be safe
  (if (and (constantp l e)
           (listp l)
           (eql (first l) 'quote))
      (avg (second l))
    form))

Now we have something which:

• has the semantics of a function, so, say (funcall #'avg ...) will work;
• isn't broken;
• uses a non-terrible algorithm;
• will be inlined on any competent implementation of the language (which I bet is 'all implementations' now) when it can be;
• will detect (some?) cases where it can be compiled completely away and replaced by a compile-time constant.

Answer 2

Since the value of super-list is known, one can do all computation at macro expansion time:

(eval-when (:execute :compile-toplevel :load-toplevel)
  (defvar super-list (loop for i from 1 to 100000 collect i)))

(defmacro avg2 (args)
  (setf args (eval args))
  (/ (reduce #'+ args) (length args)))

(defun test ()
  (avg2 super-list))

Trying the compiled code:

CL-USER 10 > (time (test))
Timing the evaluation of (TEST)

User time    =        0.000
System time  =        0.000
Elapsed time =        0.000
Allocation   = 0 bytes
0 Page faults
100001/2

Thus the runtime is near zero.

The generated code is just a number, the result number:

CL-USER 11 > (macroexpand '(avg2 super-list))
100001/2

Thus for known input this macro call in compiled code has a constant runtime of near zero.

Answer 3

I don't think you really want a list of 100,000 items. That would have terrible performance with all that cons'ing. You should consider a vector instead, e.g.

(avg2 #(2 3 4))

You didn't mention why it didn't work; if the function never returns, it's likely a memory issue from such a large list, or attempting to apply on such a large function argument list; there are implementation defined limits on how many arguments you can pass to a function.

Try reduce on a super-vector instead:

(reduce #'+ super-vector)