2

I need to tokenize a string based on a few rules in Ruby.

Basically, I need to group the file into sections that fit either a variable name, keyword, integer, or operation.

So, for example, the file:

x:=5;if x > 5 then x:=3; else x:=6; end

would tokenize to:

variable name (x)
assignment operator (:=)
integer (5)
keyword (if)
keyword (;)
variable name (x)
relation operator (>) 
integer (5)
keyword (then)
variable name (x)
assignment operator (:=)
integer (3)
keyword (;)
keyword (else)
variable name (x)
assignment operator (:=)
integer (6)
keyword (;)
keyword (end)

I don't really understand how to parse this string that way. Can anyone point me in the right direction?

Once I know how to tokenize the string, I can then use that to make actual tokens out of them and parse them. But first I need to understand how to digest the string.

Thanks!

Tetramputechture
  • 2,911
  • 2
  • 33
  • 48

1 Answers1

1

First, search for all known keywords and operators. With them identified, only variables and integers should remain.

You can determine whether the identifier is an integer by attempting to convert it into one. !!Integer(identifier) rescue false

If this returns false, you must assume that it is a variable.

Sekalf Nroc
  • 457
  • 3
  • 7