token_get_all and mathematical operators

Question

I payed around with token_get_all() once again and came across something "special":

Given the following line of PHP code:

<?php $var = 3 * 2 + 5;

when I use token_get_all() on it I get an array of tokens:

array(15) {
  [0]=>
  array(3) {
    [0]=>
    int(376)
    [1]=>
    string(6) "<?php "
    [2]=>
    int(1)
  }
  [1]=>
  array(3) {
    [0]=>
    int(312)
    [1]=>
    string(4) "$var"
    [2]=>
    int(1)
  }
  [2]=>
  array(3) {
    [0]=>
    int(379)
    [1]=>
    string(1) " "
    [2]=>
    int(1)
  }
  [3]=>
  string(1) "="
  [4]=>
  array(3) {
    [0]=>
    int(379)
    [1]=>
    string(1) " "
    [2]=>
    int(1)
  }
  [5]=>
  array(3) {
    [0]=>
    int(308)
    [1]=>
    string(1) "3"
    [2]=>
    int(1)
  }
  [6]=>
  array(3) {
    [0]=>
    int(379)
    [1]=>
    string(1) " "
    [2]=>
    int(1)
  }
  [7]=>
  string(1) "*"
  [8]=>
  array(3) {
    [0]=>
    int(379)
    [1]=>
    string(1) " "
    [2]=>
    int(1)
  }
  [9]=>
  array(3) {
    [0]=>
    int(308)
    [1]=>
    string(1) "2"
    [2]=>
    int(1)
  }
  [10]=>
  array(3) {
    [0]=>
    int(379)
    [1]=>
    string(1) " "
    [2]=>
    int(1)
  }
  [11]=>
  string(1) "+"
  [12]=>
  array(3) {
    [0]=>
    int(379)
    [1]=>
    string(1) " "
    [2]=>
    int(1)
  }
  [13]=>
  array(3) {
    [0]=>
    int(308)
    [1]=>
    string(1) "5"
    [2]=>
    int(1)
  }
  [14]=>
  string(1) ";"
}

Notice that the mathematical operators (=, *, +) and the semicolon (;) are no tokens but just strings. I expected to get something like T_MATH_ADDITION for +, etc.

Why are those "instructions" above are not handled as tokens?

Answer 1

Since they are single chars, they are already terminal symbols. No need to make a token out of it. You can find the list of available parser tokens here: http://php.net/manual/en/tokens.php

Have a look at a (pseudo) grammar:

# Using a token
product := T_NUMBER T_MULT_OPERATOR T_NUMBER

# Using the plain char
product := T_NUMBER '*' T_NUMBER

What looks better? ;)

I suggest to dig into Flex and Bison and write a little parser on your own. Things will get more clear then. Start here: http://web.iitd.ac.in/~sumeet/flex__bison.pdf (Wow, they made the book public available!)