How to match multiple tokens using regular expressions?

Question

I need using regular expressions to match special key and values. There is a special condition that I do not know how to do.

The string likes abcd/abcd. I need match all single words before /. So I write (.)*/, and then I found it only match 1 token (d). What's more, even it matches all I need, I still do not know how many tokens matched.

So what should the correct regular expressions be? The real condition is much more complex than the example, so if it can be achieved by regular expressions, I do not want write a tokenizer.

[`(.*?)\/`](https://regex101.com/r/hB1gD9/2) OR [`^([^\/]*)`](https://regex101.com/r/hB1gD9/3) — Tushar, Mar 21 '16 at 07:42
@Tushar My fault. I make mistake, it only matches `abcd` at once. I need match `a` `b` `c` `d`. — zzy, Mar 21 '16 at 07:46
@WiktorStribiżew Match the special string multi times using `()` is my purpose. — zzy, Mar 21 '16 at 07:48
If it is .NET, you can use your `(.)*` and grab the values from the CaptureCollection. Same can be done in Python with PyPi regex module. And with Boost, too. — Wiktor Stribiżew, Mar 21 '16 at 09:14
@WiktorStribiżew Could you show me reference of boost about CaptureCollection? I think that's what I need. — zzy, Mar 21 '16 at 10:08
Check http://www.boost.org/doc/libs/1_33_1/libs/regex/doc/captures.html — Wiktor Stribiżew, Mar 21 '16 at 10:12
@WiktorStribiżew That's it! Plz post your answer and I will accept it. — zzy, Mar 21 '16 at 10:43

score 2 · Answer 1 · answered Mar 21 '16 at 07:47

2

Why your regex doesn't work

The regex (.)*/ matches any character, zero or more times, followed by a /.

The * quantifier is greedy, it'll try to match whatever it can. Given the string "abcd/abcd", the regex engine matches "abcd/abcd", then it fails to match "/", so it backtracks until it reaches the "d", and finally make one step and matches the "/". The () is a group, and you're catching only the last character.

How to fix it

[^\/]*

This matches anything that's not a "/" (note that it's escaped), and is exactly what you want. If you want to catch the matched regex, you should change it to ([^\/]*), and now the first group contains your regex.

Note that there could be many solutions depending on the language you're using, not necessarily regex ones.

answered Mar 21 '16 at 07:47

Maroun

94,125
30
188
241

Thanks for clear answer, but I do not want to match all the four characters. I need got four groups `a`, `b`, `c`, `d`. – zzy Mar 21 '16 at 07:56
`([^\/]*)` will match the whole sequence before the "/". – Maroun Mar 21 '16 at 07:58
Yes, so what I need is match single char four times and gives me four groups. – zzy Mar 21 '16 at 08:00
@zzy `(.)(.)(.)(.)\/` – Maroun Mar 21 '16 at 08:01
@zzy Then the best solution would be catch the whole group and iterate on it manually, depending on the programming language you're using. – Maroun Mar 21 '16 at 08:04

Wiktor Stribiżew · Accepted Answer · 2018-05-05T17:31:33.143

The Boost library that you are using provides a way to capture repeated groups into a stack provided you compiled the library with BOOST_REGEX_MATCH_EXTRA flag set, otherwise what won't have a member named captures. When you use boost::regex_search or boost::regex_match, pass the boost::match_extra flag, and you will capture all vlaues with your (.)* (matching and capturing any character but a newline, zero or more occurrences) into a stack that is acessible via the captures member of the sub_match object.

Here is a demo method from the official Boost site:

#include <boost/regex.hpp>
#include <iostream>


void print_captures(const std::string& regx, const std::string& text)
{
   boost::regex e(regx);
   boost::smatch what;
   std::cout << "Expression:  \"" << regx << "\"\n";
   std::cout << "Text:        \"" << text << "\"\n";
   if(boost::regex_match(text, what, e, boost::match_extra))
   {
      unsigned i, j;
      std::cout << "** Match found **\n   Sub-Expressions:\n";
      for(i = 0; i < what.size(); ++i)
         std::cout << "      $" << i << " = \"" << what[i] << "\"\n";
      std::cout << "   Captures:\n";
      for(i = 0; i < what.size(); ++i)
      {
         std::cout << "      $" << i << " = {";
         for(j = 0; j < what.captures(i).size(); ++j)
         {
            if(j)
               std::cout << ", ";
            else
               std::cout << " ";
            std::cout << "\"" << what.captures(i)[j] << "\"";
         }
         std::cout << " }\n";
      }
   }
   else
   {
      std::cout << "** No Match found **\n";
   }
}

int main(int , char* [])
{
   print_captures("(.*)bar|(.*)bah", "abcbar");
   return 0;
}

Notice: Compile the regex library with `BOOST_REGEX_MATCH_EXTRA` set, otherwise `what` has no member named `captures`. — zzy, Mar 22 '16 at 01:35

How to match multiple tokens using regular expressions?

2 Answers2