Scheme: local macro (let-syntax) inside global macro (define-syntax)

Question

I'm trying to write a set of Scheme macros to simplify building simple graph structures using a declarative style, but as a beginning Schemer I'm struggling with macro hygiene.

I have defined two record types :graph and :node:

(define-record-type :graph 
  (make-graph nodes links) 
  graph? 
  (nodes graph-nodes) 
  (links node-links))

(define-record-type :node
  (make-node name pins)
  node?
  (name node-name)
  (pins node-pins))

A pin in my example is just a string (pin name), a link simply a cons-pair of endpoint, and and endpoint a cons-pair of a node name and a pin-name.

What I would like to be able to do is to define macro's graph, node and link that allow my to create graphs in a declarative style, like this:

(define g (graph
  (node "v0"
    (pin "out"))

  (node "v1"
    (pin "out"))

  (node "add"
    (pin "lhs")
    (pin "rhs")
    (pin "out"))

  (link (("v0" . "out") . ("add" . "lhs"))
  (link (("v1" . "out") . ("add" . "rhs"))))

The result of this expression would be a value of type :graph, where (graph-nodes g) would evaluate to a list of 3 :node values, (graph-links g) would evaluate to ((("v0"."out") . "add" . "lhs")) (("v1" . "out") . ("add" . "rhs"))), and each of the 3 nodes would be :node values with the pins they were defined with.

An additional requirement would be that only the graph macro would be defined at global scope, I would very much like to avoid having to define global macro's for anything I can pass as the body of the graph invocation. These macro's make only sense in the context of an expansion of the graph macro, and I'd like to maintain the principle of 'smallest possible scope' for them, and also not litter the global scope with generic names like node or pin.

Obviously it would be relatively simple to simply create the graph directly using make-graph and passing the nodes and links directly as lists. I would prefer to use macro's though, because it allows a more declarative style to define the graph: you don't need to remember the order of the arguments to pass in, you could mix the order of node/links declarations, extend the graph record with more additional fields without breaking the graph definition code, trivially transform the macro call to/from other data types (XML, JSN), etc.

I tried to solve this problem using nested macro's: a global graph macro, which has a local node macro.

For simplicity, I will leave out defining links or pins in this example, to extend the example for this, the let-syntax would have an additional macro link, and the macro for node would itself have another nested macro called pin (so let-syntax for pin inside let-syntax for node inside define-syntax for graph):

(define-syntax graph
  (syntax-rules ()
    ((graph . body)

      (let ((nodes '())
            (links '()))
        
        (let-syntax ((node
          (syntax-rules ()
            ((node name))
            (make-node name '()))))
         
        (begin . body))

      (make-graph nodes links)))))

(graph (node "v0") (node "v1"))

Unfortunately, this does not work, as node will be unbound where body is inserted in the expansion of the graph macro. I think I understand why: since Scheme macro's are hygienic, any variables (pattern variables or locals) will be expanded/renamed to macro-local variables, so the node macro defined by let-syntax will not actually be called node anymore after expansion, so the node in the body I passed into the graph macro will not see it.

A very similar question has been asked here, and while I sort of understand the explanation there, I am not certain how to interpret the proposed solution/workaround.

If I read correctly, both solutions in the other question involve a) explicitly passing/matching references to any functions I would like to call from the body passed into graph (so in my case, node, link, pin), or b) defining node, link, pin as global macro's.

Solution a) is not attractive as it defeats the purpose of the macro (which is to define graphs in declarative style, with minimal syntax), and solution b) feels like it defeats the whole point of having hygienic macro's, because now I need to have macro's like node, link and pin in the global scope, which are very likely to collide with code that wants to define things (local variables, symbols) with the same name.

Considering people have been able to do the craziest things creating DSL's using Scheme macro's, I'm almost certain that what I want to do should be possible some way or other. I've not been able to figure out yet, which makes me think I'm missing some fundamental part of the puzzle and/or have reasoned myself into some dark corner of Scheme (nested/recursive macro's) I should not have to be in because there are much simpler solutions to achieve what I want.

Edit: Based on Shawn's answer, I came up with the following:

(define-record-type :graph
  (make-graph nodes links)
  graph?
  (nodes graph-nodes)
  (links graph-links))

(define-record-type :node
  (make-node name pins)
  node?
  (name node-name)
  (pins node-pins))

(define-record-type :link
  (make-link from to)
  link?
  (from link-from)
  (to link-to))

(define-record-type :pin
  (make-pin name)
  pin?
  (name pin-name))

(define-syntax graph
  (lambda (x)
    (define node-def? (lambda (def) (eq? (car def) 'node)))
    (define link-def? (lambda (def) (eq? (car def) 'link)))
    (define pin-def? (lambda (def) (eq? (car def) 'pin)))

    (define (parse-pin-definition pin-def)
      #`(make-pin (quote #,(cdr pin-def))))

    (define (parse-node-definition node-def)
      (let ((pins (map parse-pin-definition (filter pin-def? (cddr node-def)))))
        #`(make-node (quote #,(cadr node-def)) (list #,@pins))))

    (define (parse-link-definition link-def)
      #`(make-link (quote #,(cadr link-def)) (quote #,(cddr link-def))))

    (define (parse-graph-definition graph-def)
      (let ((nodes '())
            (links '()))
        (map (lambda (def)
          (cond 
            ((node-def? def) 
             (set! nodes (cons (parse-node-definition def) nodes))))
          (cond 
            ((link-def? def)
             (set! links (cons (parse-link-definition def) links)))))
          graph-def)
        (values nodes links)))

    (syntax-case x ()
      ((graph . graph-def)
        (let-values (((nodes links) (parse-graph-definition (syntax->datum #'graph-def))))
          #`(make-graph (list #,@nodes) (list #,@links)))))))

(display (graph 
  (node "v0"
    (pin "out"))
  (node "v1"
    (pin "out")) 
  (node "add"
    (pin "lhs")
    (pin "rhs")
    (pin "sum"))
  (link ("v0" . "out") ("add" . "lhs"))
  (link ("v1" . "out") ("add" . "rhs"))))

This seems to work well and is pretty readable. It lacks any kind of error checking/handling but it illustrates well the concept of using syntax-case for this as recommended by Shawn :)

Answer 1

I'm not sure if you can do what you want with syntax-rules - if you can it's probably something long and complicated and tedious to come up with, but by using syntax-case to validate the structure and types of the arguments to a macro, it becomes fairly easy to do in a single macro.

Note I changed the syntax of the link clauses from

(link ((from-node . from-pin) . (to-node . to-pin)))
; Same as (link ((from-node . from-pin) to-node . to-pin))

to

(link (from-node . from-pin) (to-node . to-pin))

which is simpler and easier to work with. A list of 2-element lists of pairs is also stored in the generated graph record. You might also consider using symbols instead of strings for all those identifiers, which is more idiomatic in lisp family languages.

;;; Tested with guile --r7rs
;;; Should be portable with some tweaking to anything with records and
;;; syntax-case macros, let-values and SRFI-1

(import
  (scheme base)
  (srfi 1))

(define-record-type :graph
  (make-graph nodes links)
  graph?
  (nodes graph-nodes)
  (links node-links)) ; Should this be graph-links instead?

(define-record-type :node
  (make-node name pins)
  node?
  (name node-name)
  (pins node-pins))

(define-syntax graph
  (lambda (stx)
    ;; Split up a list into the leading node and trailing link forms
    ;; and convert the nodes to make-node syntax expressions
    (define (parse-args args)
      (let loop ((args args)
                 (nodes '()))
        (cond
         ((null? args) ; No links
          (values (reverse nodes) '()))
         ;; Check the first element of the list for a valid (node name (pin
         ;; name) ...) form
         ((and (list? (car args))
               (>= (length (car args)) 2)
               (eq? (first (car args)) 'node)
               (string? (second (car args)))
               (every (lambda (pin)
                        (and (list? pin)
                             (= (length pin) 2)
                             (eq? (first pin) 'pin)
                             (string? (second pin))))
                      (cddar args)))
          ;; And if so convert it to a make-node expression
          (loop (cdr args)
                (cons #`(make-node #,(second (car args))
                                   (quote #,(map second (cddar args))))
                      nodes)))
         ;; Check to see if all remaining elements are valid link forms and
         ;; return them if so.
         ((every (lambda (arg)
                   (and (list? arg)
                        (= (length arg) 3)
                        (eq? (first arg) 'link)
                        (pair? (second arg))
                        (string? (car (second arg)))
                        (string? (cdr (second arg)))
                        (pair? (third arg))
                        (string? (car (third arg)))
                        (string? (cdr (third arg)))))
                 args)
          (values (reverse nodes) (map cdr args)))
         (else
          (error "malformed graph element")))))
    (syntax-case stx ()
      ((graph . args)
       (let-values (((nodes links) (parse-args (syntax->datum #'args))))
         #`(make-graph
            (list #,@nodes)
            (quote #,links)))))))

(define g (graph
  (node "v0"
    (pin "out"))

  (node "v1"
    (pin "out"))

  (node "add"
    (pin "lhs")
    (pin "rhs")
    (pin "out"))

  (link ("v0" . "out") ("add" . "lhs"))
  (link ("v1" . "out") ("add" . "rhs"))))

(write (node-name (first (graph-nodes g)))) ; "v0"
(newline)
(write (node-pins (first (graph-nodes g)))) ; ("out")
(newline)
(write (node-links g)) ; ((("v0" . "out") ("add" . "lhs")) (("v1" . "out") ("add" . "rhs")))
(newline)

Answer 2

This seems to be a case where you really don't need macros at all. In particular I think you have, as you suspect, reasoned yourself into some dark corner of Scheme you should not have to be in because there are much simpler solutions to achieve what you want.

Given your record definitions you can define the following two simple procedures to make nodes and pins:

(define (node name . pins)
  (make-node name pins))

(define (pin name)
  name)

Now the link definition does need to be a macro unless you're happy with adding some quotes. Here is a macro for link which is not quite the same syntax as your link descriptions are (I've just got rid of the dotted lists):

(define-syntax link
  ;; link is just a fancy quote
  (syntax-rules ()
    ((_ (sn sp) (tn tp))
     '(link (sn sp) (tn tp)))))

(define (link-description? thing)
  (and (pair? thing)
       (eq? (car thing) 'link)))

(define (link-description-link d)
  (cdr d))

link-description? and link-description-link is going to be used below, and both of these are because I was too lazy to define a record type.

Now here's graph:

(define (graph . things)
  (let next ((tail things)
             (nodes '())
             (link-descriptions '()))
    (cond
      ((null? tail)
       ;; In real life this should turn link descriptions into actual links
       ;; between pins perhaps?
       (make-graph (reverse nodes) (reverse link-descriptions)))
      ((node? (car tail))
       (next (cdr tail)
             (cons (car tail) nodes)
             link-descriptions))
      ((link-description? (car tail))
       (next (cdr tail)
             nodes
             (cons (link-description-link (car tail))
                   link-descriptions)))
      (else
       (error "oops")))))

And now

(define g
  (graph
   (node
    "v0"
    (pin "out"))
   (node
    "v1"
    (pin "out"))
   (node
    "add"
    (pin "lhs")
    (pin "rhs")
    (pin "out"))
   (link ("v0" "out") ("add" "lhs"))
   (link ("v1" "out") ("add" "rhs"))))

I think that, if you did want to do it with a macro to get (perhaps) better diagnostics, then the right approach would be, rather than torturing yourself by trying to make a macro whose expansion is a bunch of local macros work, to use the module/library system to hide parts of the implementation you want to be hidden.

I don't have a good enough R7RS environment to test, but in Racket you could do something like this, for instance:

(module graph racket
  (provide graph)
  ;; record definitions omitted

  (define (compile-graph d)
      ;; In real life compile the graph by walking over the description
      d)
    
    (define-syntax graph
      (syntax-rules ()
        ((_ form ...)
         (expand-graph (form ...) ()))))

    (define-syntax expand-graph
      (syntax-rules (node pin link)
        ((_ () (description ...))
         (compile-graph '(description ...)))
        ((_ ((node name (pin pn) ...) more ...) (description ...))
         (expand-graph (more ...) (description ... (node name pn ...))))
        ((_ ((link (sn sp) (tn tp)) more ...) (description ...))
         (expand-graph (more ...) (description ... (link sn sp tn tp))))
        ((_ (anything ...) (description ...))
         (error 'graph "what is ~s?" '(graph anything ...))))))
        
(require 'graph)

This just works by having (graph ...) rewrite to the private expand-graph macro which does the work of peeling off elements from the graph description until it's reached the end, and then compiling the graph from the resulting description. Obviously you would probably place the module in its own file in real life.

Now

> (graph (node 1 (pin 2)) (link (1 2) (1 2)))
'((node 1 2) (link 1 2 1 2))