0

I am trying to tokenize a string into character using boost

The present boost tokenizer will tokenize based on space

 typedef boost::tokenizer<boost::char_separator<char> >
    tokenizer;
  boost::char_separator<char> sep("");
  tokenizer tokens(str, sep);

I expect output to be j e f but the the actual output is jef

Mike Kinghan
  • 55,740
  • 12
  • 153
  • 182
Jeffin Sam
  • 98
  • 7

1 Answers1

0

This 

""

is string literal with no characters ended by null terminator. Whereas

" "

is string literal which contains one character - space also ended by null terminator. If you want to split str = "j e f" by space you need to write something like this:

  typedef boost::tokenizer<boost::char_separator<char> >
        tokenizer;
  boost::char_separator<char> sep(" ");
  std::string str = "j e f";
  tokenizer tokens(str, sep);
  for (auto i : tokens)
    cout << i << endl;
  // output
  j
  e
  f

As the name char_separator suggests it takes characters, your string "" contains no chars. Splitting is realized by comparing separator character with input string. How do you want to do this comparison when there is no character to do it, i.e. ""?

rafix07
  • 20,001
  • 3
  • 20
  • 33
  • My input is "jef" and not "j e f", my input has no spaces – Jeffin Sam May 17 '19 at 21:06
  • What is unclear in my answer? `""` as delimiter has no characters, so your input cannot be spliited. There is no character which doest the split. If you want to get `j`, `e` and `f` you can just iterate over "jef" and get each character. – rafix07 May 18 '19 at 06:05