Recursive descent parser in Clojure

Question

I'm creating some expert system with Clojure and I need to develop Recursive descent parser for reading rules from a text file and creating clojure functions from it. I have written function which checks if text file is ok with my grammar and it gives me list of strings with element such as function names, numbers, names of facts for my system, arithmetic and logical operators. That is how my grammar look like:

 RULE := EXPR >> FACT 
 EXPR := ( WSK OpA NUM ) || ( FACT ) || ( EXPR OpL EXPR ) || (WSK OpA WSK)

 OpL := AND || OR 
 OpA := > || < || == 
 WSK := [A-Z]+ 
 FACT := [a-z]+ 
 NUM := [0-9]+\.?[0-9]*

And that's my function for checking grammar:

(defn wyr
  "new expression"
  [przetworzone doPrzetworzenia]
  (cond
   (empty? doPrzetworzenia) przetworzone
   (empty? przetworzone) (if (empty? (acceptLP (first doPrzetworzenia)))
               "error-poczatek";todo - error
               (wyr (cons (first doPrzetworzenia) przetworzone) (rest doPrzetworzenia)))
   (not (empty? (acceptLP (first przetworzone)))) (if (empty? (acceptFACT (first doPrzetworzenia)))
                            (if (empty? (acceptWSK (first doPrzetworzenia)))
                              (if (empty? (acceptLP (first doPrzetworzenia)))
                            "error-LP";todo - error
                            (wyr (cons (first doPrzetworzenia) przetworzone) (rest doPrzetworzenia)))
                              (wyr (cons (first doPrzetworzenia) przetworzone) (rest doPrzetworzenia)))
                            (wyr (cons (first doPrzetworzenia) przetworzone) (rest doPrzetworzenia)))
   (not (empty? (acceptFACT (first przetworzone)))) (if (empty? (acceptPP (first doPrzetworzenia)))
                              "error-FACT";todo - error
                              (wyr (cons (first doPrzetworzenia) przetworzone) (rest doPrzetworzenia)))
   (not (empty? (acceptWSK (first przetworzone)))) (if (empty? (acceptOpA (first doPrzetworzenia)))
                             (if (empty? (acceptPP (first doPrzetworzenia)))
                               "error-WSK";todo - error
                               (wyr (cons (first doPrzetworzenia) przetworzone) (rest doPrzetworzenia)))
                             (wyr (cons (first doPrzetworzenia) przetworzone) (rest doPrzetworzenia)))
   (not (empty? (acceptOpA (first przetworzone)))) (if (empty? (acceptNUM (first doPrzetworzenia)))
                             (if (empty? (acceptWSK (first doPrzetworzenia)))
                               "error-OpA";todo - error
                               (wyr (cons (first doPrzetworzenia) przetworzone) (rest doPrzetworzenia)))
                             (wyr (cons (first doPrzetworzenia) przetworzone) (rest doPrzetworzenia)))
   (not (empty? (acceptPP (first przetworzone)))) (if (empty? (acceptOpL (first doPrzetworzenia)))
                            (if (empty? (acceptImplication (first doPrzetworzenia)))
                              "error-PP";todo - error
                              (wyr (cons (first doPrzetworzenia) przetworzone) (rest doPrzetworzenia)))
                            (wyr (cons (first doPrzetworzenia) przetworzone) (rest doPrzetworzenia)))
   (not (empty? (acceptOpL (first przetworzone)))) (if (empty? (acceptLP (first doPrzetworzenia)))
                             "error-OpL";todo - error
                             (wyr (cons (first doPrzetworzenia) przetworzone) (rest doPrzetworzenia)))
   (not (empty? (acceptImplication (first przetworzone)))) (if (empty? (acceptFACT (first doPrzetworzenia)))
                                 "error-Implication";todo - error
                                 (wyr (cons (first doPrzetworzenia) przetworzone) (rest doPrzetworzenia)))
   (not (empty? (acceptNUM (first przetworzone)))) (if (empty? (acceptPP (first doPrzetworzenia)))
                             "error-NUM";todo - error
                             (wyr (cons (first doPrzetworzenia) przetworzone) (rest doPrzetworzenia)))
   :else
   "error")
  )

Now I would like to create a clojure function from my list of string which above function gives to me. Do you have any idea how to do that?

UPDATE Here is an example of rule and hard coded version of it:

( ROC > 100 ) >> buy

(fn
  (cond
    (> (ROC) 100) "buy"
    :else
    ()
  )
)

could you include a little more context (pun intended) ;) where are acceptFACT acceptLP etc. defined. — Arthur Ulfeldt, Jun 17 '13 at 21:13
Where does ROC come from in your example (existing function, argument, something else)? It looks like a function, given that you're calling it. Also, does the outer function take any arguments? I'm assuming function and no arguments for now. — deterb, Jun 23 '13 at 12:34
Is your grammar correct? It doesn't support having more than one character for WSK and FACT, and FACT is only lower case in it, while the example you gave uses upper case. Same question goes for BUY - I don't know where it comes from. Since you're not calling it, I'm going to assume variable, though it also looks like a string. — deterb, Jun 23 '13 at 12:43
ok, I updated my update. deterb, ROC is a function without arguments and I'd like to return string "buy" if value returned by function named ROC is greater than 100. I also updated my grammar, so now WSK and FACT can have more than one character. — dzwonu, Jun 24 '13 at 06:46

score 2 · Answer 1 · answered Jun 17 '13 at 21:06

2

Have you tried instaparse?:

It produces parsers from a context free grammer

(ns example.core
  (:require [instaparse.core :as insta])

(def as-and-bs
  (insta/parser
    "S = AB*
     AB = A B
     A = 'a'+
     B = 'b'+"))

answered Jun 17 '13 at 21:06

Arthur Ulfeldt

90,827
27
201
284

Thanks for your response. I'll study it's documentation and try to use in my project. – dzwonu Jun 19 '13 at 06:46
Could you take a look at http://stackoverflow.com/questions/17432282/clojures-require-and-instaparse ?? – dzwonu Jul 02 '13 at 18:59

deterb · Accepted Answer · 2013-06-23T13:11:29.017

2

First, I'd agree with @Arthur concerning the use of Instaparse for generating your grammer.

The next step is to write a function which transform your grammar into Clojure data structures which represent code.

For example, if the grammar parses to

[:S [:EXPR [:WSK "ROC"] [:OpA ">"] [:NUM "100"]] :>> [:FCT "BUY"]] ; "ROC > 100 << BUY"

your function needs to take the appropriate parts and translate them into the corresponding code. The expression would probably be parsed into a (> (ROC) 100) for example, though it's hard to tell without example inputs and expected outputs.

Generating the functions is just like normal Clojure data manipulation. Take the result of the parser, turn it into code. Then use that generated code in a macro. Here's a simplified example for working with just what this example needs to handle.

(defn parse-expr [expr]
  (let [[_ [part1-type part1-val] [part2-type part2-val] [part3-type part3-val]] expr]
    (if (and (= :WSK part1-type) (= :OpA part2-type) (= :NUM part3-type))
      (let [wsk (variable part1-val)
            opa (variable part2-val)
            num (Integer/valueOf part3-val)]
        (list opa (list wsk) num)))))

(defmacro generate-funcs [parse-tree]
  (let [[_ expr _ [_ fact]] parse-tree
        expr (parse-expr expr)
        fact (symbol fact)]
    `(fn [] (if ~expr (str ~fact) ()))))

Try running

(parse-expr [:EXPR [:WSK "B"] [:OpA "<"] [:NUM "1"]])

and

(macroexpand-1 '(generate-funcs [:S [:EXPR [:WSK "B"] [:OpA "<"] [:NUM "1"]] :>> [:FCT "b"]]))

to get a better grasp of what I'm referring to for translating the data into code.

edited Jun 23 '13 at 13:11

answered Jun 18 '13 at 00:08

deterb

3,994
1
28
33

ok but if I can create for example string expression with str function taking appropriate elements of parsed data structure and then eval it? I also would like to have of my rules in some list so then I could check that list if I can run some rules and generate new facts. – dzwonu Jun 19 '13 at 06:50
You don't need to generate and then evaluate the string using `str`. I would instead expect the code to generate something like `(list (symbol "<") (symbol "A") (Integer/valueOf "1"))`, and that could then be part of the content of the Clojure function. A few sample rules and the hard coded versions of those rules would be really helpful, it's hard to tell what you need the generated code to do. – deterb Jun 19 '13 at 19:47
deterb but what do you mean by (variable part1-val) for example? Because I get an error Unable to resolve symbol: variable in this context. – dzwonu Jul 04 '13 at 07:14
Should be `var`, not `variable`, sorry. When you run that, you need to make sure you have a variable with that name which is accessible. You may want to resolve it as a symbol instead; I'm not sure enough about the context in which your code is run to tell you which is more appropriate. – deterb Jul 04 '13 at 18:28
Thanks for explanation. I used symbol instead, but have some problem with expansion of macro. I'd like to pass as an arg of generate-funcs sth like (grammar line) where line is a string and grammar is my declared parser. That's a problem because of quote on generate-funcs macro. – dzwonu Jul 05 '13 at 06:43
`generate-funcs` is supposed to take the parsed tree, not the string. Just plug your grammer and the string into Instaparse and it should produce the structure that you need to pass into generate-funcs (the first code block in my answer). – deterb Jul 05 '13 at 12:29

Recursive descent parser in Clojure

2 Answers2