StringTokenizer showing unwanted results

Question

When I ran the following code I found something strange.

The output of below program is token1=AAAAA token2=BBBBB|

However, From my understanding, it should be token1=AAAAA token2=BBBBB|DUMMY

public class TestToken {

    public static void main(final String[] args) {
        final String delim = "DELIM";
        String token1 = "AAAAA";
        String token2 = "BBBBB|DUMMY";
        final String input = token1 + delim + token2;
        final StringTokenizer tokenizer = new StringTokenizer(input, delim);
        final String text1 = tokenizer.nextToken();
        final String text2 = tokenizer.nextToken();
        System.out.println("token1=" + text1);
        System.out.println("token2=" + text2);
        System.out.println();
    }

}

Can some one explain me how to fix this problem and why it is behaving like this ?

@ScaryWombat-- Could you please tell how ? As I am taken String as Delimiter — T-Bag, Jun 19 '18 at 06:59
something to take into consideration (from the StringTokenizer api) "StringTokenizer is a legacy class that is retained for compatibility reasons although its use is discouraged in new code. It is recommended that anyone seeking this functionality use the split method of String or the java.util.regex package instead." — Stultuske, Jun 19 '18 at 07:06
@ShowStopper When one option is depricated, there are various others that JAVA provides. Please provide three other ways to tokenize at https://stackoverflow.com/a/50922619/1262248 — Aman Chhabra, Jun 19 '18 at 07:31
@AmanChhabra-- I know, I am not looking for work arounds here, but for a root cause that is causing this behavior and which is well elaborated in answers . — T-Bag, Jun 19 '18 at 08:22
@ShowStopper StringTokenizer has been marked legacy for several versions of Java now, it shouldn 't be used at all. Part of it being legacy means that when updates to functionality is done, it will be done to the classes that replace this one, just not to this one. — Stultuske, Jun 19 '18 at 08:26

score 2 · Accepted Answer · answered Jun 19 '18 at 06:59

Excerpt from the constructor's documentation:

The characters in the delim argument are the delimiters for separating tokens.

That means that each character is a delimiter, not the whole string. In fact, you have 5 delimiters (the characters D, E, L, I, and M).

You can see the effect with the following code

while (tokenizer.hasMoreTokens())
   System.out.println(tokenizer.nextToken());

which prints out:

AAAAA
BBBBB|
U
Y

score 1 · Answer 2 · answered Jun 19 '18 at 06:59

No your delimiters are D E L I and M

See the javadocs All characters in the delim argument are the delimiters for separating tokens.

delim - the delimiters.

consider

    final String delim = "DELIM";
    String token1 = "AAAAA";
    String token2 = "BBBBB|ZUMMY";
    final String input = token1 + delim + token2;
    final StringTokenizer tokenizer = new StringTokenizer(input, delim);
    final String text1 = tokenizer.nextToken();
    final String text2 = tokenizer.nextToken();
    System.out.println("token1=" + text1);
    System.out.println("token2=" + text2);
    System.out.println();

score 1 · Answer 3 · answered Jun 19 '18 at 07:01

StringTokenizer takes a string where each character is a delimiter. Since D is one of your delimiters, the second token is cut off after the |.

If you want to use multi-character delimiters, you'll have to use a different technique. E.g., split:

String[] parts = Arrays.toString(input.split(delim));

Aman Chhabra · Answer 4 · 2018-06-19T07:26:06.520

There are multiple option to implement this functionality. To start with why it is happening, I think that is well explained by various posts that it is because your delimeter is not "DELIM", instead it is "D","E","L","I","M"

Now what you can use if you want to seperate string based on another string like DELIM

Option 1: Use String split method which will take the delimeter string as argument and will return the array of tokens

String statement = "AAAADELIMBBBB|DUMMY";
String tokens[] = statement.split("DELIM");

Option 2: Using splitAsStream which will take statement as argument and compile will take regex delimiter as argument

Pattern.compile("DELIM").splitAsStream("AAAADELIMBBBB|DUMMY").forEach(System.out::println);

Option 3: Using Stream.of with split as arguement

Stream.of("AAAADELIMBBBB|DUMMY".split("DELIM")).forEach(System.out::println);

Apart from above super cool ways to split, if you are a die hard fan of String Tokenizer and want to implement it using that only, you can also use String Tokenizer with "D" as delimeter and then for each token received, can check for first four character to be "ELIM". If yes, take the remaninng substring as token and concatenate with further receiving tokens and if not append D in start and then append with the current token.

score 0 · Answer 5 · answered Jun 19 '18 at 07:01

From the doc of StringTokenizer

Constructs a string tokenizer for the specified string. The characters in the delim argument are the delimiters for separating tokens. Delimiter characters themselves will not be treated as tokens.

This means that the DELIM is not a delimiter but all characters in it are delimiters (i.e. D, E, L, I, and M).

When you run the following code:

public static void main(final String[] args) {
    final String delim = "DELIM";
    String token1 = "AAAAA";
    String token2 = "BBBBB|DUMMY";
    final String input = token1 + delim + token2;
    final StringTokenizer tokenizer = new StringTokenizer(input, delim);
    while(tokenizer.hasMoreElements()){
        System.out.println("token =" + tokenizer.nextToken());
    }
}

It gives the following output:

token =AAAAA
token =BBBBB|
token =U
token =Y

As you can see that your input got split on D & M (which were present in your input).

score 0 · Answer 6 · answered Jun 19 '18 at 07:02

As the document explains, All characters in the delim argument are the delimiters for separating tokens.

What you need to do instead is to use the split function.

public static void main(final String[] args) {
    final String delim = "DELIM";
    String token1 = "AAAAA";
    String token2 = "BBBBB|DUMMY";
    final String input = token1 + delim + token2;

    final String[] tokens = input.split("DELIM");
    for (String token:tokens) {
        System.out.println(token);
    }

}

StringTokenizer showing unwanted results

6 Answers6