Adding tokens from .csv file to ArrayList in Java

Question

I've been having trouble creating an ArrayList in Java from tokens that I get from a .csv file. I've been trying to figure it out for hours with no success. Without creating an ArrayList, all my tokens print out without a problem, but when I create an ArrayList to add them to, I run into trouble. For a .csv file consisting of 60+ lines and 9 tokens (strings) for each line, I print out what I have before adding to the ArrayList, but after creating the ArrayList and adding tokens to them, I can only print out the first few tokens before I run into a NoSuchElementException.

This is all my code:

import java.io.BufferedReader;
import java.io.FileReader;
import java.io.FileWriter;
import java.util.StringTokenizer; 
import java.lang.String;
import java.util.ArrayList; 
import java.util.*; //probably redundant
import java.util.Scanner;  

public class TTcompiler18Feb {

   private static ArrayList<String> list;

   public static void main(String[] args) {

    try
    {
      //csv file containing data
      String strFile = "FileWithTokens.csv";

      //create BufferedReader to read csv file
      BufferedReader br = new BufferedReader( new FileReader(strFile));

      String strLine = "";
      StringTokenizer st = null;
      int lineNumber = 0, tokenNumber = 0;

      list = new ArrayList<String>();
      String token;

      //read comma separated file line by line
      while( (strLine = br.readLine()) != null)
      {
        lineNumber++; 
        st = new StringTokenizer(strLine, ",");

        while(st.hasMoreTokens())
        {
          tokenNumber++;       
          System.out.println("Line # " + 
                             lineNumber + ", Token # " + 
                             tokenNumber + ", Token : "+ 
                             st.nextToken()); 
          list.add(st.nextToken());
        }
        //reset token number
        tokenNumber = 0;
      }
      System.out.println("The size of your list is: " + list.size());
    }
    catch(Exception e)
    {
      System.out.println("Exception while reading csv file: " + e);                  
    }
  }
}

I run my code and get the following:

run TTcompiler18Feb

Line # 1, Token # 1, Token : Row1Token1
Line # 1, Token # 2, Token : Row1Token2
Line # 1, Token # 3, Token : Row1Token3
Line # 1, Token # 4, Token : Row1Token4
Line # 1, Token # 5, Token : Row1Token5
Exception while reading csv file: java.util.NoSuchElementException

I've tried adding to my list in other ways besides list.add(st.nextToken());, such using a .toString method, typecasting st.nextToken(), and assigning a variable to st.nextToken(), like so:

token = st.nextToken().toString();  //defined as string type outside loop
list.add(token);

...but none of these approaches seem to work.

The goal of this program is to take all the tokens in the incoming .csv file and neatly put them in an ArrayList to be used later. I'm still pretty new to Java, and am still unclear as to how generics may play a role in solving this problem. As I understand it, StringTokenizer is of type 'string', but I need to use a <String> to enter my tokens into an ArrayList composed of <Strings>, but not sure how to. Any help/hints/feedback are much appreciated!!!

Answer 1

A) You're calling st.nextToken() twice each time you loop. You just lost every other token, and eventually it pukes because on the second call ... there isn't one (If you had 10 items per line in your csv file, it would succeed and you'd have 5 things in your ArrayList)

B) Don't use StringTokenizer in new code. The Javadoc for it explains it's a legacy class and to use String.split() instead.

String[] elements = strLine.split(",");

Done.

Edit to add: if you really need an ArrayList rather than an array, you can then do:

ArrayList<String> list = new ArrayList<String>(Arrays.asList(elements));

Answer 2

You are already doing an st.nextToken in your System.out. So the token won't exist anymore. If you need it in System.out, assign the st.nextToken to a variable first & then use it.

Answer 3

you are jumping two elements ahead each time you loop. Code read token and then use it:

String token = ""; 
token = st.nextToken();

And then:

                            System.out.println("Line # " + 
                                               lineNumber + ", Token # " + 
                                               tokenNumber + ", Token : "+ 
                                               token ); 

                            list.add(token);

Answer 4

the nextToken method is called twice in the loop, you may need to change your code to something like:

while(st.hasMoreTokens())
                    {
                            tokenNumber++;       
                            String tempStr = st.nextToken();
                            System.out.println("Line # " + 
                                               lineNumber + ", Token # " + 
                                               tokenNumber + ", Token : "+ 
                                               tempStr); 

                            list.add(tempStr);
                    }

Answer 5

As mentioned in other answers, your problem is with calling nextToken more than once in a single iteration, which is not how it's intended for use.

Instead of using the clunky old StringTokenizer, why don't you use String.split() ?

String[] tokens = strLine.split(",");

for(int i = 0; i < tokens.length; i++) { 
    System.out.println("Line # " + 
                        lineNumber + ", Token # " + 
                        i+ ", Token : "+ 
                        token[i]);       
    // do you even need a list?
    list.add(token[i]);
}