1

How are import statements and method calls tokenized in java's lexical analysis. For example:

import java.util.Scanner

is this seen as import, java, util, Scanner(4 tokens) or import, java.util.Scanner(2 tokens)

In the same line of thought, in:

Scanner input = new Scanner(System.in);
int x = input.nextInt();

is input.nextInt() seen as input, nextInt() (2 tokens) or input.nextInt() (1 token)

Mo H.
  • 1,788
  • 1
  • 18
  • 28

2 Answers2

1

Lexical analysis is described in Chapter 3 of the JLS.

This means in your first example, it would be tokenized as 

keyword: import 
whitespace 
identifier: java 
seperator: . 
identifier: util 
seperator: .
identifier: Scanner
seperator: ;

So neither 2 nor 4, but 8 tokens (since whitespace and seperators are tokens according to the JLS).

Similary, input.nextInt(); is 6 tokens, since both ( and ) are one token each (See JLS § 3.11).

Polygnome
  • 7,639
  • 2
  • 37
  • 57
-2

I believe import would be 1 lexical ananlysis. Apache Antlr is the tool that defines the grammar. Java is the same too.

I would recommend using Apache Antlr by importing the Java grammar. This would give the correct solution.

Name Name
  • 175
  • 1
  • 2
  • 11