Antlr4 use TokenStreamRewriter in ParseTreeWalker

Question

Goal :
To replace modify text, which is a rule I defined in .g4 file, that will enter and exit in my listener class in input String
My code :

def textModify(input: String) = {  
    val loadLexer = new DSLSQLLexer(new ANTLRInputStream(input))  
    val tokens = new CommonTokenStream(loadLexer)  
    val rewriter = new TokenStreamRewriter(tokens)    
    val parser = new DSLSQLParser(tokens)  
    parser.statement()  
    ParseTreeWalker.DEFAULT.walk(listener, stat)  
}

But how can I call the rewriter in my enter function, which can only get the context?
Or can I just rewrite it in textModify()? But rewriter.replace() requires index of token which I can't know when I call it.

I thought it might be easier to show the real situation:
I defined a rule like this:

proJob
    : realJob
    ;
realJob
    : job
    ;

job
    : IDENTIFIER
    | quotedIdentifier
    ;

quotedIdentifier
    : BACKQUOTED_IDENTIFIER
    ;
IDENTIFIER
    : (LETTER | DIGIT | '_')+
    ;
BACKQUOTED_IDENTIFIER
    : '`' ( ~'`' | '``' )* '`'
    ;

I want to change the text of token when it hits rule proJob,for example:
The input : I am a teacher., will enter rule proJob function and I want to modify teacher to teacher_blahblahblah, how can I manage that

score 0 · Answer 1 · answered Aug 03 '18 at 18:45

0

To use the rewriter in your listener methods, you should simply make it an instance variable of your listener. Then the methods will be able to access it.

answered Aug 03 '18 at 18:45

sepp2k

363,768
54
674
675

still don't know how to do it – AI Joes Aug 06 '18 at 06:39

score 0 · Answer 2 · answered Mar 16 '22 at 16:25

(this is really just an elaboration of @sepp2k's answer)

A TokenStreamRewriter is independent of your Listener/Visitor.

note: your example code does not show the creation of the listener. The listener would either have the TokenStreamRewriter passed into it, of create a TokenStreamRewriter in it's constructor.

In your example, in your enterProJobContext method override, you would have access to your TokenStreamRewriter because it's a member of your listener. You can then invoke methods on the rewriter to create a set of changes to be applied to the token stream. (NOTE: you are NOT modifying the Context object itself. Take a look at the methods on the TokenStreamRewriter class to see what capability it provides for modifying the token stream;)

Once you're done, you would ask the TokenStreamRewriter to give you a new String that reflects all the changes you asked for.

so:

def textModify(input: String) = {  
    val loadLexer = new DSLSQLLexer(new ANTLRInputStream(input))  
    val tokens = new CommonTokenStream(loadLexer)  
    val rewriter = new TokenStreamRewriter(tokens)    
    val parser = new DSLSQLParser(tokens)  

    val listener = new MyListener(rewriter) // <===

    parser.statement()  
    ParseTreeWalker.DEFAULT.walk(listener, stat)  

    val revisedSource = listener.rewriter.getText() // <==
}

score 0 · Answer 3 · edited Mar 25 '22 at 21:28

Thanks Mike

Following your answer i think a good solution is to create a new constructor in your new listen class and from the main program use the new listener with giving him the parameters you want.

in my main program

VB6MYListener* Mylistener = new VB6MYListener( &tokens , baseFile, &rewriter , Name );

in my new class

class VB6MYListener : public VB6ParserListener {

        public:         string FicName;
                        wstring BaseFile;
                        CommonTokenStream* TOK ; 
                        TokenStreamRewriter* Rewriter ; 
                
                            //setter
        public:             // void SetPFile(string Name) { FicName = Name; }
                            //setter
        public:             //  void SetBFile(wstring baseFile) { BaseFile = baseFile; }

                            // void SetINStream(std::fstream infile) { INStream = infile ; }              

                            // Getter 
        public:             string getPFile() { return FicName; }
                            //Getter  
        public:             wstring getBFile(wstring) { return BaseFile; }

      
             // Fonctions pour la traduction avec le listener
     
        /*
        VB6MYListener( wstring baseFile , CommonTokenStream* tok , string Name)
                      {
                        TOK = tok;                        // Flux de tokens
                        BaseFile = baseFile;              // Fichier entree des fichiers en VB6 
                        FicName = Name;                   // Fichier resultat type .pas correspondant au fichier courant traduit
                        }
        */
        
        VB6MYListener( CommonTokenStream* tok , wstring baseFile, TokenStreamRewriter* rewriter , string Name)
        {
            TOK = tok;                  // Flux de tokens
            BaseFile = baseFile;
            Rewriter = rewriter;              
            FicName = Name;             // Fichier entree des fichiers en VB6    // Fichier resultat type .pas correspondant au fichier courant traduit
        }

so i can have the visibility on the Token stream and i can use the rewriter in my void functions of the VB6ParserBaseListener.h ( my new class )

i hope that example can help anyone else.

Antlr4 use TokenStreamRewriter in ParseTreeWalker

3 Answers3