Java How to Convert Token to String?

Question

Cant seem to figure out how to convert a Token (StringTokenizer) into a String.

I have a String[] of keywords, and I read in multiple lines of text from a text file. StringTokenizer is used to chop the sentence up,,, at which point I need to pass each Token (the String representation of the Token) to a function for word analysis.

Is there a simple way to extract the string value from the token?

Any help appreciated guys....

Great help guys thanks again!!

I don't understand, your tokens are already String arent' they? — reef, Mar 07 '11 at 16:36

Shawn · Accepted Answer · 2011-03-07T16:39:48.653

2

Use the method nextToken() to return the token as a string

StringTokenizer st = new StringTokenizer("this is a test");
         while (st.hasMoreTokens()) {
             wordAnalysis(st.nextToken());//sends string to your function
         }

edited Mar 07 '11 at 16:39

answered Mar 07 '11 at 16:34

Shawn

7,235
6
33
45

in the while loop above, I call a function which expects a String. How do I convert the Token to a String do you know? – Donal.Lynch.Msc Mar 07 '11 at 16:36
The nextToken method returns a String: see http://download.oracle.com/javase/1.5.0/docs/api/java/util/StringTokenizer.html#nextToken(). – reef Mar 07 '11 at 16:37

score 1 · Answer 2 · answered Mar 07 '11 at 16:37

nextToken() called on a StringTokenizer, already returns a String. You don't need to convert that String to a String.

A Token is just a small (string) portion of a larger String. Tokens are used in various forms of processing, and a word (the word token) is needed to differentiate the category of items from the specific items.

For example, every word, period, and comma in this string is a token.

Would lend itself to the following tokens (all of which are strings):

For 
example
, 
every 
word
,
period
,
and 
comma 
in 
this 
string 
is 
a 
token
.

score 1 · Answer 3 · answered Mar 07 '11 at 16:38

1

for(int i = 0; i<stringArray.length; i++) {
    StringTokenizer st = new StringTokenizer(stringArray[i]);
    while(st.hasMoreTokens()) {
       callMyProcessingMethod(st.nextToken());
    }
}

answered Mar 07 '11 at 16:38

dmcnelis

2,913
1
19
28

score 1 · Answer 4 · answered Mar 07 '11 at 16:39

1

StringTokenizer is now a legacy class in Java. As of Java 1.4 it is recommended to use the split() method on String which returns a String[].

http://download.oracle.com/javase/1.4.2/docs/api/java/lang/String.html#split(java.lang.String)

answered Mar 07 '11 at 16:39

Jordan Owens

692
4
10

Odd, there's nothing in the API to indicate that's it's deprecated (or legacy). Also split doesn't do the same thing. Token boundaries are easily identified by "next char" sequences, while REGEX's require a full pattern to match. – Edwin Buck Mar 07 '11 at 16:45
Sorry, it's actually the API doc for StringTokenizer that mentions that it is deprecated. http://download.oracle.com/javase/6/docs/api/java/util/StringTokenizer.html – Jordan Owens Mar 07 '11 at 16:57

Java How to Convert Token to String?

4 Answers4