7

I'm building a site to learn basic programming, I'm going to use a pseudolanguage in which users can submit their code and I need to interpret it. However I'm not sure how to build a tokenizer in PHP.

Having a snippet such as this one:

a = 1
b = 2
c = a - b

if(a > b) {
    buy(a)
    } else {
    buy(b)
    }

How would I go about separating this code into tokens?

--

This is what I'm trying now:

$tokens = array();

// First token (define string)
$token  = strtok($botCode, '=');
$tokens[] = $token;

// Loop
while($token) {
    $token  = strtok('=');
    $tokens[] = $token;
}

However I haven't been able to figure out how to use strtok with a list of regular expresions... I could do something similar to strtok but that accepts arrays as needles with substr and strrpos but it seems to me that it should be possible to do it with strtok as it's designed just for this. Any info or pointing in the right direction will be thanked

lisovaccaro
  • 32,502
  • 98
  • 258
  • 410
  • 1
    you can get http://php.net/manual/en/book.tokenizer.php – NullPoiиteя Feb 21 '13 at 03:58
  • 1
    Usually a list of token names to characters or a regex list. As introduction http://compilers.iecc.com/crenshaw/ is very suitable. But, no offense, I don't think it's a task *you* should attempt yet. – mario Feb 21 '13 at 04:15
  • none taken. I'm probably not going to complete it but it's just a personal proyect so I don't care. I just read about strtok, I cannot figure out how to use it with a list of values. If you point me in the right direction I'll accept the answer – lisovaccaro Feb 21 '13 at 04:28

1 Answers1

5

Do not wait some magic from strtok. It is similar to preg_split.

I think that you want to build your own lexer. So you could use article Writing a simple lexer in PHP or something else.

sectus
  • 15,605
  • 5
  • 55
  • 97
  • thanks, I just implemented that, I'll edit your answer to add the code because it was simply fantastic. Copy-paste and running in minutes – lisovaccaro Feb 22 '13 at 04:21