3

I am trying to build a syntax tree for regular expression. I use the strategy similar to arithmetic expression evaluation (i know that there are ways like recursive descent), that is, use two stack, the OPND stack and the OPTR stack, then to process.

I use different kind of node to represent different kind of RE. For example, the SymbolExpression, the CatExpression, the OrExpression and the StarExpression, all of them are derived from RegularExpression.

So the OPND stack stores the RegularExpression*.

while(c || optr.top()):
    if(!isOp(c):
        opnd.push(c)
        c = getchar();
    else:
        switch(precede(optr.top(), c){
        case Less:
          optr.push(c)
          c = getchar();
        case Equal:
          optr.pop()
          c = getchar();
        case Greater:
          pop from opnd and optr then do operation, 
          then push the result back to opnd
        }

But my primary question is, in typical RE, the cat operator is implicit. a|bc represents a|b.c, (a|b)*abb represents (a|b)*.a.b.b. So when meeting an non-operator, how should i do to determine whether there's a cat operator or not? And how should i deal with the cat operator, to correctly implement the conversion?

Update

Now i've learn that there is a kind of grammar called "operator precedence grammar", its evaluation is similar to arithmetic expression's. It require that the pattern of the grammar cannot have the form of S -> ...AB...(A and B are non-terminal). So i guess that i just cannot directly use this method to parse the regular expression.

Update II

I try to design a LL(1) grammar to parse the basic regular expression. Here's the origin grammar.（\| is the escape character, since | is a special character in grammar's pattern)

E -> E \| T | T
T -> TF | F
F -> P* | P
P -> (E) | i

To remove the left recursive, import new Variable

E -> TE'
E' -> \| TE' | ε
T -> FT'
T' -> FT' | ε
F -> P* | P   
P -> (E) | i

now, for pattern F -> P* | P， import P'

P' -> * | ε
F -> PP'

However, the pattern T' -> FT' | ε has problem. Consider case (a|b):

E => TE' 
  => FT' E'
  => PT' E'
  => (E)T' E'
  => (TE')T'E'
  => (FT'E')T'E'
  => (PT'E')T'E'
  => (iT'E')T'E'
  => (iFT'E')T'E'

Here, our human know that we should substitute the Variable T' with T' -> ε, but program will just call T' -> FT', which is wrong.

So, what's wrong with this grammar? And how should i rewrite it to make it suitable for the recursive descendent method.

user8510613
  • 1,242
  • 9
  • 27
  • What operators is your regex supporting? I don't see why you can't just say is concatenation if the character is a non-operator. –  Nov 08 '18 at 18:07
  • @JETM consider case like "a|b", how can i directly push concat operator while meeting 'b'? – user8510613 Nov 09 '18 at 04:25

2 Answers2

2

1. LL(1) grammar

I don't see any problem with your LL(1) grammar. You are parsing the string

(a|b)

and you have gotten to this point:

(a   T'E')T'E'   |b)

The lookahead symbol is | and you have two possible productions:

T' ⇒ FT'
T' ⇒ ε

FIRST(F) is {(, i}, so the first production is clearly incorrect, both for the human and the LL(1) parser. (A parser without lookahead couldn't make the decision, but parsers without lookahead are almost useless for practical parsing.)

2. Operator precedence parsing

You are technically correct. Your original grammar is not an operator grammar. However, it is normal to augment operator precedence parsers with a small state machine (otherwise algebraic expressions including unary minus, for example, cannot be correctly parsed), and once you have done that it is clear where the implicit concatenation operator must go.

The state machine is logically equivalent to preprocessing the input to insert an explicit concatenation operator where necessary -- that is, between a and b whenever a is in {), *, i} and b is in {), i}.

You should take note that your original grammar does not really handle regular expressions unless you augment it with an explicit ε primitive to represent the empty string. Otherwise, you have no way to express optional choices, usually represented in regular expressions as an implicit operand (such as (a|), also often written as a?). However, the state machine is easily capable of detecting implicit operands as well because there is no conflict in practice between implicit concatenation and implicit epsilon.

Community
  • 1
  • 1
rici
  • 234,347
  • 28
  • 237
  • 341
  • Thanks for your answer. How to "merge" the insertion phase in the expression's evaluation? – user8510613 Nov 13 '18 at 07:56
  • You just insert the implicit operator on the operator stack (or the implicit operand on the operand stack) as though you had read it, and then do the same state transition as you woukd have done with an operator (or operand). Very straight-forward. – rici Nov 13 '18 at 08:09
  • Could you explain it more specificly? Take ```* i``` for example, in this case, i cannot just directly push a cat operator into the operator stack, since the * operation must be evaluated first. But if i set the current ```c``` as operator cat, how to process the next step of loop, like the ```prev```/ ```getchar```? – user8510613 Nov 13 '18 at 13:28
  • @user8510613: try making `put(stack, operator)` a function (which does whatever is needed to put the operator on the stack) and it will probably reveal the simplicity of the algorithm. – rici Nov 13 '18 at 15:07
  • for example, when meeting an ```i``` after the ```*``` operator, does it means that i have to do all the evaluation until the optr_stack.top precede is less than current operator in the sub-procudure ```put(stack, operator)```? Is it possible that do not mess up the origin procedure? – user8510613 Nov 13 '18 at 15:14
  • There are six cases that have an implicit cat operator(according to your answer): 1. ```)(``` 2. ```)i``` 3.```*(``` 4.```*i``` 5.```i(``` 6.```ii```. How to design strategy for these different cases(if a general one and don't mess up the origin procedure is possible)? – user8510613 Nov 13 '18 at 15:15
  • @user8510613: yes, twice. – rici Nov 13 '18 at 15:15
  • @user8510613: basically, but you can do it with a two-state state machine, which is a lot more scalable. – rici Nov 13 '18 at 15:17
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/183569/discussion-between-user8510613-and-rici). – user8510613 Nov 13 '18 at 15:20
0

I think just keeping track of the previous character should be enough. So if we have

(a|b)*abb
      ^--- we are here
c = a
pc = *

We know * is unary, so 'a' cannot be its operand. So we must have concatentation. Similarly at the next step

(a|b)*abb
       ^--- we are here
c = b
pc = a

a isn't an operator, b isn't an operator, so our hidden operator is between them. One more:

(a|b)*abb
   ^--- we are here
c = b
pc = |

| is a binary operator expecting a right-hand operand, so we do not concatenate.

The full solution probably involves building a table for each possible pc, which sounds painful, but it should give you enough context to get through.

If you don't want to mess up your loop, you could do a preprocessing pass where you insert your own concatenation character using similar logic. Can't tell you if that's better or worse, but it's an idea.

  • If my corollary about "operator precedence grammar" is right, i think it's hard to parse the basic regex by this strategy. Maybe the preprocessing is needed. – user8510613 Nov 10 '18 at 08:12