Java Tokenizer, separating strings

Question

I don't know why I am struggling so badly with this but any help would be much appreciated.

I am creating my own tokenizer that takes in a file with a list of commands, delimiters and values. It then outputs each "token" along with what type it is.

INPUT: AND 3, 4, 5 ; some comments

I need to output:

AND --- command
3 --- value
, --- delimiter
4 --- value
, --- delimiter
5 --- value

I have it working right now to where I am outputting:

AND 3, 4, 5 --- delimiter

but I need to break it down further.

Here is where I am at currently:

ArrayList<Token> tokenize(String[] input) {
    ArrayList<Token> tokens = new ArrayList<Token>();
    for (String str : input) {
        Token token = new Token(str.trim());
        //Check if int
        try{
            Integer.parseInt(str);
            token.type = "number";
        } catch(NumberFormatException e) {

        }
        if (token.type == null) {
            if (commands.contains(str))
                token.type = "command";
             else if (str.contains(",")) {
                token.type = "delimiter";
            } else if (destValues.contains(str))
                token.type = "destination";
            else
                token.type = "unknown";
        }

        if(! token.type.equals("unknown"))
            tokens.add(token);
    }
    return tokens;
}

Only real constraints I have with this assignment is not being able to use StringTokenizer and regex.

Can you use String.split() or you need to tokenize your string by hand? — davidbuzatto, Oct 25 '12 at 21:05
I am able to use split(), however how can I split it to get rid of the space, but keep the comma's? — btjordan23, Oct 25 '12 at 21:07
Is your input already splited in parts? What you want to improve? — davidbuzatto, Oct 25 '12 at 21:07

davidbuzatto · Accepted Answer · 2012-10-25T21:40:29.907

Its seem s that your input is incorrect. Try this to split the input and then use your tokenize method.

import java.util.*;

public class Foo {

    public static void main( String[] args ) {


        String input = "AND 3,    4, 5   ; some comments";
        List<String> parts = new ArrayList<String>();

        // removing comments
        input = input.split( ";" )[0];

        // splits using spaces
        String[] firstPass = input.trim().split( " " );

        for ( String s : firstPass ) {

            // the current part cannot be empty
            if ( !s.trim().isEmpty() ) {

                // splits using comma
                String[] secondPass = s.split( "," );

                for ( String ss : secondPass ) {
                    parts.add( ss.replace( ",", "" ) );
                }

                // verifies if the current part has a comma
                // and if so, inserts it as a part
                if ( s.contains( "," ) ) {
                    parts.add( "," );
                }

            }

        }

        for ( String a : parts ) {
            System.out.println( a );
        }

    }

}

EDIT: As my first anwer worked, here is a complete example with some refactors...

import java.util.*;

public class MyTinyParser {

    private static final String COMMANDS = "AND OR FOO BAR";

    private List<String> extract( String input ) {

        List<String> parts = new ArrayList<String>();

        // removing comments
        input = input.split( ";" )[0];

        // splits using spaces
        String[] firstPass = input.trim().split( " " );

        for ( String s : firstPass ) {

            // the current part cannot be empty
            if ( !s.trim().isEmpty() ) {

                // splits using comma
                String[] secondPass = s.split( "," );

                for ( String ss : secondPass ) {
                    parts.add( ss.replace( ",", "" ) );
                }

                // verifies if the current part has a comma
                // and if so, inserts it as a part
                if ( s.contains( "," ) ) {
                    parts.add( "," );
                }

            }

        }

        return parts;

    }

    public List<Token> tokenize( String input ) {

        List<Token> tokens = new ArrayList<Token>();

        for ( String str : extract( input ) ) {

            Token token = new Token( str );

            // check if int
            try{
                Integer.parseInt( str );
                token.type = "number";
            } catch(NumberFormatException e) {
            }

            if ( token.type == null ) {

                if ( COMMANDS.contains(str)){
                    token.type = "command";
                } else if (str.contains(",")) {
                    token.type = "delimiter";
                } else {
                    token.type = "unknown";
                }

            }

            if( !token.type.equals( "unknown" ) ) {
                tokens.add( token );
            }

        }

        return tokens;

    }

    private class Token {

        String value;
        String type;

        Token( String value ) {
            this.value = value;
        }

        @Override
        public String toString() {
            return String.format( "Token[%s, %s]", value, type );
        }

    }

    public static void main( String[] args ) {

        MyTinyParser mtp = new MyTinyParser();
        List<Token> tokens = mtp.tokenize( "AND 3,    4, 5   ; some comments" );

        for ( Token t : tokens ) {
            System.out.println( t );
        }

    }

}

@btjordan23 You are welcome! I'm improving my example... Wait a moment. — davidbuzatto, Oct 25 '12 at 21:39

score 2 · Answer 2 · answered Oct 25 '12 at 21:37

If you are allowed to use google's api, you can also try something similar to below.

import com.google.common.base.Splitter;

public class Tmp {

    public static void main(String[] args) {
        String str = "AND 3, 4, 5 ; some comments";

        Iterable<String> stringIterable = Splitter.on(' ').trimResults()
                .omitEmptyStrings()
                .split(str);

        for (String str1 : stringIterable) {
            int commaIndex = str1.indexOf(",");
            if (commaIndex > 0) {
                System.out.println(str1.subSequence(0, commaIndex));
                System.out.println(",");
            } else {
                System.out.println(str1);
            }
        }


    }

}

It prints

AND
3
,
4
,
5
;
some
comments

P.S. Not the best code. It could be improved further, people feel free please chime in.

Java Tokenizer, separating strings

2 Answers2