Regex to match tokens in a string using string.gmatch

Question

I need a regex to use in string.gmatch that matches sequences of alphanumeric characters and non alphanumeric characters (quotes, brackets, colons and the like) as separated, single, matches, so basically:

str = [[
    function test(arg1, arg2) {
        dosomething(0x12f, "String");
    }
]]

for token in str:gmatch(regex) do
    print(token)
end

Should print:

function
test
(
arg1
,
arg2
)
{
dosomething
(
0x121f
,
"
String
"
)
;
}

How can I achieve this? In standard regex I've found that ([a-zA-Z0-9]+)|([\{\}\(\)\";,]) works for me but I'm not sure on how to translate this to Lua's regex.

score 1 · Answer 1 · answered Jun 20 '16 at 14:59

You need a workaround involving a temporary char that is not used in your code. E.g., use a § to insert it after the alphanumeric and non-alphanumeric characters:

str = str:gsub("%s*(%w+)%s*", "%1§") -- Trim chunks of 1+ alphanumeric characters and add a temp char after them
str = str:gsub("(%W)%s*", "%1§")     -- Right trim the non-alphanumeric char one by one and add the temp char after each
for token in str:gmatch("[^§]+") do  -- Match chunks of chars other than the temp char
    print(token)
end

See this Lua demo

Note that %w in Lua is an equivalent of JS [a-zA-Z0-9], as it does not match an underscore, _.

score 1 · Accepted Answer · answered Jun 20 '16 at 16:03

1

local str = [[
    function test(arg1, arg2) {
        dosomething(0x12f, "String");
    }
]]

for p, w in str:gmatch"(%p?)(%w*)" do
   if p ~= "" then print(p) end
   if w ~= "" then print(w) end
end

answered Jun 20 '16 at 16:03

Egor Skriptunoff

23,359
2
34
64

What does `%p` match? – user6245072 Jun 20 '16 at 16:48
`%w` - alphanumeric, `%p` - punctuation (punctuation is all ASCII7 except alphanumeric, space and control characters) – Egor Skriptunoff Jun 20 '16 at 16:51
This is really useful - I could for example add all the tokens to a table and then process them - but how could I preserve in a unique token strings delimited by the character `"`? – user6245072 Jun 20 '16 at 19:34
@user6245072 - It would be more complex program. It should correctly handle comments, escaped quotes inside string literals, etc. – Egor Skriptunoff Jun 20 '16 at 22:26
@user6245072, if handling of quoted fragments is needed, then something like this may be used: http://stackoverflow.com/questions/28664139/lua-split-string-into-words-unless-quoted/28664691#28664691 – Paul Kulchenko Jun 21 '16 at 00:49
@Egor Skriptunoff Oh yeah comments... how do I break a string in lines? – user6245072 Jun 21 '16 at 07:00

Regex to match tokens in a string using string.gmatch

2 Answers2