Parsing an expression with tokens

Question

I want to write a recursive descent parser for the following grammar

term ---> FINAL | FUNCTION_A (term, term) | FUNCTION_B (term, term)

Currently I am struggeling with the FUNCTION part, since I don't know, how to handle cases, where a command contains nested FUNCTIONS, e.g.

FUNCTION_A (FUNCTION_B (FINAL, FINAL), FINAL)

I've read quite a lot of tutorials but none of which really helped me to get through this. As far as I know, one can use StringTokenizer to approach this problem.

How can I use Tokens to identify the first term and the second term of FUNCTION_A for above example?

score 0 · Answer 1 · answered Mar 10 '15 at 15:58

Sorry if the following is known. A recursive-descent parser is just that:

boolean parseTerm() {
    return parseFinal() || parseFunctionA() || parseFunctionB();
}

boolean parseFunctionA() {
    if (!scannedToken("FUNCTION_A")) {
        return false;
    }
    mustScan("(");
    must(parseTerm());
    mustScan(",");
    must(parseTerm());
    mustScane(")");
    return true;
}

A StringTokenizer deals with the entire text as String, and separates the tokens with delimiters. It also makes no distinctions like identifier, number and such.

A Scanner seems more appropriate. It has a hasNext(pattern) test on the entire token. Again, word boundaries of an identifier/name are not immediately available.

boolean scannedToken(String s) {
    Pattern pattern = Pattern.compile("\\b" + Pattern.quote(s) + "\\b");
    if (scanner.hasNext(pattern)) {
        String actual = scanner.next(pattern);
        return true;
    }
    return false;
}

Parsing an expression with tokens

1 Answers1