1

I want to generate ngram characters for a string. Below is the Lucene 4.1 lib I used for it. 

    Reader reader = new StringReader(text);
    NGramTokenizer gramTokenizer = new NGramTokenizer(reader, 3, 5); //catch contiguous sequence of 3, 4 and 5 characters

    CharTermAttribute charTermAttribute = gramTokenizer.addAttribute(CharTermAttribute.class);

    while (gramTokenizer.incrementToken()) {
        String token = charTermAttribute.toString();
        System.out.println(token);}

However, I want to use Lucene 5.0.0 to do it. The NGramTokenizer changes a lot in Lucene 5.0.0 from the previous version, refer to http://lucene.apache.org/core/5_0_0/analyzers-common/index.html?org/apache/lucene/analysis/ngram/NGramTokenizer.html.

Anyone knows how to use Lucene 5.0.0 to do ngrams?

Einar Sundgren
  • 4,325
  • 9
  • 40
  • 59
HappyCoding
  • 5,029
  • 7
  • 31
  • 51

1 Answers1

3

The following code:

  StringReader stringReader = new StringReader("abcd");
  NGramTokenizer tokenizer = new NGramTokenizer(1, 2);
  tokenizer.setReader(stringReader);
  tokenizer.reset();
  CharTermAttribute termAtt = tokenizer.getAttribute(CharTermAttribute.class);
  while (tokenizer.incrementToken()) {
    String token = termAtt.toString();
    System.out.println(token);
  }

will produce:

a
ab
b
bc
c
cd
d
Ivan Mamontov
  • 2,874
  • 1
  • 19
  • 30