Tokenize method: Split string into array

Question

I've been really struggling with a programming assignment. Basically, we have to write a program that translates a sentence in English into one in Pig Latin. The first method we need is one to tokenize the string, and we are not allowed to use the Split method usually used in Java. I've been trying to do this for the past 2 days with no luck, here is what I have so far:

  public class PigLatin 
    { 
        public static void main(String[] args) 
        { 
              String s = "Hello there my name is John"; 
              Tokenize(s); 
        } 

        public static String[] Tokenize(String english) 
        { 
             String[] tokenized = new String[english.length()]; 
             for (int i = 0; i < english.length(); i++) 
             { 
                   int j= 0; 
                   while (english.charAt(i) != ' ') 
                   { 
                         String m = ""; 
                         m = m + english.charAt(i); 
                         if (english.charAt(i) == ' ') 
                         { 
                              j++; 
                         } 
                         else 
                         { 
                               break; 
                         } 
                    } 
          for (int l = 0; l < tokenized.length; l++) { 
          System.out.print(tokenized[l] + ", "); 
        }
      }
    return tokenized;
    }
}

All this does is print an enormously long array of "null"s. If anyone can offer any input at all, I would reallllyyyy appreciate it!

Thank you in advance Update: We are supposed to assume that there will be no punctuation or extra spaces, so basically whenever there is a space, it's a new word

If you're allowed to use StringTokenizer, that will do the same as split, but you loop through the tokens that it creates. — Gregory Basior, Feb 23 '15 at 03:04
Is there a possibility you misinterpreted the assignment? It sounds like you're supposed to create a `StringTokenizer` object. — Dando18, Feb 23 '15 at 03:06
m looks like it's declared at the wrong level, it should be outside the while loop. also you don't do anything with it once it's populated. — Nathan Hughes, Feb 23 '15 at 03:07
before you proceed, I sincerely suggest you to learn 1. proper indentation 2. java naming convention — Adrian Shum, Feb 23 '15 at 03:08

Elliott Frisch · Answer 1 · 2015-02-23T03:12:51.660

If I understand your question, and what your Tokenize was intended to do; then I would start by writing a function to split the String

static String[] splitOnWhiteSpace(String str) {
    List<String> al = new ArrayList<>();
    StringBuilder sb = new StringBuilder();
    for (char ch : str.toCharArray()) {
        if (Character.isWhitespace(ch)) {
            if (sb.length() > 0) {
                al.add(sb.toString());
                sb.setLength(0);
            }
        } else {
            sb.append(ch);
        }
    }
    if (sb.length() > 0) {
        al.add(sb.toString());
    }
    String[] ret = new String[al.size()];
    return al.toArray(ret);
}

and then print using Arrays.toString(Object[]) like

public static void main(String[] args) {
    String s = "Hello there my name is John";
    String[] words = splitOnWhiteSpace(s);
    System.out.println(Arrays.toString(words));
}

score 0 · Answer 2 · answered Feb 23 '15 at 03:11

If you're allowed to use the StringTokenizer Object (which I think is what the assignment is asking, it would look something like this:

StringTokenizer st = new StringTokenizer("this is a test");
 while (st.hasMoreTokens()) {
     System.out.println(st.nextToken());
 }

which will produce the output:

 this
 is
 a
 test

Taken from here.

The string is split into tokens and stored in a stack. The while loop loops through the tokens, which is where you can apply the pig latin logic.

score 0 · Answer 3 · answered Feb 23 '15 at 03:18

Some hints for you to do the "manual splitting" work.

There is a method String#indexOf(int ch, int fromIndex) to help you to find next occurrence of a character
There is a method String#substring(int beginIndex, int endIndex) to extract certain part of a string.

Here is some pseudo-code that show you how to split it (there are more safety handling that you need, I will leave that to you)

List<String> results = ...;
int startIndex = 0;
int endIndex = 0;

while (startIndex < inputString.length) {
    endIndex = get next index of space after startIndex
    if no space found {
        endIndex = inputString.length
    }
    String result = get substring of inputString from startIndex to endIndex-1
    results.add(result)
    startIndex = endIndex + 1  // move startIndex to next position after space
}

// here, results contains all splitted words

score 0 · Answer 4 · answered Feb 23 '15 at 03:24

             String english = "hello my fellow friend"
             ArrayList tokenized = new ArrayList<String>(); 
             String m = "";
             int j = 0; //index for tokenised array list.
             for (int i = 0; i < english.length(); i++) 
             { 

                   //the condition's position do matter here, if you  
                   //change them, english.charAt(i) will give index      
                   //out of bounds exception
                   while( i < english.length() && english.charAt(i) != ' ') 
                   { 
                         m = m + english.charAt(i); 
                         i++;

                   }
                   //add to array list if there is some string
                   //if its only  ' ', array will be empty so we are OK.
                   if(m.length() > 0 )
                   {
                       tokenized.add(m);
                       j++;
                       m = "";

                   }

             }    
          //print the array list
          for (int l = 0; l < tokenized.size(); l++) { 
          System.out.print(tokenized.get(l) + ", "); 

                        }

This prints, "hello,my,fellow,friend," I used an array list since at the first sight the length of the array is not clear.

Tokenize method: Split string into array

4 Answers4