In Lucene how can a TokenFilter emit more than one term?

Question

I'm working with Lucene 3.2. How can I use a TokenFilter that doesn't just filter/modify a term, but can also insert other terms into the stream?

For example, I want a filter that take as input "tv42lcd" and insert into the stream the words "tv42lcd", "tv", "42", "lcd".

I'm aware that I could do this by implementing my own Tokenizer. But I rather still use the provided StandardTokenizer.

How would your tokenizer know where to split the terms? e.g. why not "tv42lcd", "tv", "42", "lc", "d" or "tv4", "2l", "cd"? — Eric Petroelje, Jun 17 '11 at 16:01
This is just an example. But in my application I split based on the numeric chars. — Felipe Hummel, Jun 17 '11 at 16:33

score 1 · Accepted Answer · edited May 23 '17 at 10:33

1

You can always mix default with custom: use StandardTokenizer logic where possible, then wrap its output and add custom tokenization on the top. You can achieve this by extending, but it's almost always better to use composition.

edited May 23 '17 at 10:33

Community

1
1

answered Jun 17 '11 at 21:14

mindas

26,463
15
97
154

In Lucene how can a TokenFilter emit more than one term?

1 Answers1