Tokenizing is the act of splitting a string into discrete elements called tokens.
Tokenizing is the act of splitting a stream of text into discrete elements called tokens using a delimiter present in the stream. These tokens can then be processed further, for example to search for a value or assign to an array for looping.
Example (VBA):
Dim tokens As Variant
Dim sampleString As String
Dim i As Long
sampleString = "The quick brown fox jumps over the lazy dog."
' tokenize string based on space delimiter
tokens = Split(sampleString, " ")
' list tokens
For i = LBound(tokens) To UBound(tokens)
MsgBox tokens(i)
Next i
