There are regex dialects that support recursions, namely Perl 5.10, PCRE 4.0, Ruby 2.0. You can match your string with a regex engine that supports recursion as @Vladimir Trifonov pointed out in his answer.
If your regex engine does not support recursion you can match without recursion by building a longer regex that supports up to a max level of nesting, here using your input string as an example:
- max 1 level:
/(\\\([^\)]*\))/g
- match:
["\(def\(ghj)", "\(qrs)", "\(xyz\(abc\(def)", "\(stu)"]
- max 2 levels:
/(\\\((?:[^\\\)]*\\\([^\)]*\))?[^\)]*\))/g
- match:
["\(def\(ghj)klm)", "\(qrs)", "\(xyz\(abc\(def)ghi)", "\(stu)"]
- max 3 levels:
/(\\\((?:[^\\\)]*\\\((?:[^\\\)]*\\\([^\)]*\))?[^\)]*\))?[^\)]*\))/g
- match:
["\(def\(ghj)klm)", "\(qrs)", "\(xyz\(abc\(def)ghi)jkl)", "\(stu)"]
- etc.
In other words, you can increase the max level by nesting (?:[^\\\)]*\\\(
... [^\)]*\))?
patterns; they are made optional with ?
.
EDIT:
In case you need to support unlimited nesting and your regex engine does not support recursion, you can use a series of regexes. See JavaScript example at https://stackoverflow.com/a/66558399/7475450 and Perl example at https://twiki.org/cgi-bin/view/Blog/BlogEntry201109x3