5

Problem: Can't get Unicode character to print correctly.

Here is my grammar:

options { k=1; filter=true;
 // Allow any char but \uFFFF (16 bit -1)
charVocabulary='\u0000'..'\uFFFE'; 
}

ANYCHAR :'$'
|    '_' { System.out.println("Found underscore: "+getText()); }
|    'a'..'z' { System.out.println("Found alpha: "+getText()); }
|    '\u0080'..'\ufffe' { System.out.println("Found unicode: "+getText()); }
;

Code snippet of main method invoking the lexer:

public static void main(String[] args) {
SimpleLexer simpleLexer = new SimpleLexer(System.in);
while(true) {
try {
Token t = simpleLexer.nextToken();
System.out.println("Token : "+t);

} catch(Exception e) {}

}
}

For input "ठ", I'm getting the following output :

Found unicode: 
Token : ["à",<5>,line=1,col=7]
Found unicode: 
Token : ["¤",<5>,line=1,col=8]
Found unicode:  
Token : [" ",<5>,line=1,col=9]

It appears that the lexer is treating Unicode char "ठ" as three separate character. My aim is to scan and print "ठ".

jpalecek
  • 47,058
  • 7
  • 102
  • 144
Shikhar Mishra
  • 1,965
  • 2
  • 13
  • 14
  • 1
    Not related to your problem, but I recommend never to "swallow" your exceptions: when things go wrong, you have no idea what happened (and where). Instead of `catch(Exception e) {}` at the very least do: `catch(Exception e) { e.printStackTrace(); }` – Bart Kiers Sep 03 '10 at 06:21
  • You are correct sir :) Lesson for me: read the copied code all the way to the end before using it. – Shikhar Mishra Sep 03 '10 at 20:04

1 Answers1

6

Your problem is not in the ANTLR generated lexer, but in the Java stream you pass to it. The stream reads bytes only (doesn't interpret them in an encoding), and what you see is an UTF-8 sequence.

If its ANTLR 3, you can use the ANTLRInputStream constructor that takes an ancoding as a parameter:

ANTLRInputStream (InputStream input, String encoding) throws IOException
jpalecek
  • 47,058
  • 7
  • 102
  • 144
  • 1
    Thanks, that was it. Also, I realized that I was using antlr.Tool, instead of org.antlr.Tool, and that wasn't generating the Lexer implementation with a constructor that takes ANTLRInputStream. – Shikhar Mishra Sep 03 '10 at 20:01
  • 2
    I know this is a long shot being over 3 years later. However, do you happen to know how to do something like this for C# and ANTLR4? There is no longer a constructor that uses encoding at all. – SomeoneRandom May 05 '14 at 20:23
  • 1
    Added answer here: http://stackoverflow.com/questions/28126507/antlr4-using-non-ascii-characters-in-token-rules/28129510#28129510 – Terence Parr Jan 24 '15 at 19:45
  • He used `charVocabulary`, which is only available in ANTLR 2. Is there a solution that works with that ANTLR version? – Guillaume F. May 26 '20 at 22:03