2
StringTokenizer tokenizer = new StringTokenizer(s, " ,.:;?![]'");

Is there a way to also retrieve the delimiter, in this case all thee punctuation marks?

For example, "This is a test, and is that a test too?"

I want the result of tokenization also includes the two tokens , and ?

Is that possible?

Mureinik
  • 297,002
  • 52
  • 306
  • 350
user697911
  • 10,043
  • 25
  • 95
  • 169
  • 1
    User the 3 param constructor https://docs.oracle.com/javase/8/docs/api/java/util/StringTokenizer.html#StringTokenizer-java.lang.String-java.lang.String-boolean- – Federico Piazza May 19 '17 at 22:01

1 Answers1

3

StringTokenizer has an overloaded constructor that takes a third boolean argument. Setting it to true will make the tokenzier return the delimiters too:

StringTokenizer tokenizer = new StringTokenizer(s, " ,.:;?![]'", true);
// Here ---------------------------------------------------------^
AJNeufeld
  • 8,526
  • 1
  • 25
  • 44
Mureinik
  • 297,002
  • 52
  • 306
  • 350
  • That's cool! I tested use "\\s" to represent a space, why doesn't it work? – user697911 May 19 '17 at 22:17
  • 1
    @user697911 `StringTokenzer` takes a string of characters to use as delimiters, not a regular expression. Using `"\\s"` as the delimiter argument will result in `'\'` and `'s'` being used as delimiters, not white space. – AJNeufeld May 19 '17 at 22:26