How to tokenize a text in lucene 3.0.3

Question

I need to translate this code from lucene.net 2.3.2 to 3.0.3.

This code works fine in 2.3.2 but in 3.0.3 the method tokenStream.Next() does not return a token but a boolean. What I need to understand is where to read the token object when next() returns true.

Analyzer analyzer = new StandardAnalyzer();
StringReader stringReader = new StringReader("the house id red");
TokenStream tokenStream = analyzer.TokenStream("", stringReader);
Token token = tokenStream.Next();
while (token != null)
{
 System.Diagnostics.Debug.Write(token.TermText()); 
 token = tokenStream.Next();
}

score 0 · Accepted Answer · answered Mar 21 '16 at 14:13

The TokenStreams in Lucene 3+ can now represent a whole set of values (called attributes) for each position in the stream. So you need to modify your example to be attribute aware...

        var analyzer = new StandardAnalyzer(Lucene.Net.Util.Version.LUCENE_30);
        var stringReader = new StringReader("the house id red");
        var tokenStream = analyzer.TokenStream(string.Empty, stringReader);
        var termAtt = tokenStream.GetAttribute<ITermAttribute>();
        while (tokenStream.IncrementToken())
        {
            Console.WriteLine(termAtt.Term);
        }

How to tokenize a text in lucene 3.0.3

1 Answers1