FullTextIndex in NexusDB, How to tokenize the searchstring

Question

We are using NexusDB for a small database. We have a table with a FulltextIndex defined on it. The index is configured with the following options:

• Character separator
• ccPunctuationDash
• ccPunctuationOther

The user enters a search text in an edit box, and then an SQL statement is constructed with the following WHERE clause (%s substituted with the Editbox.text of course):

WHERE CONTAINS(FullIdx, ''%s'')

When the user enters multiple words in the editbox this goes wrong as the two separate words should have been embedded in the WHERE clause like this:

WHERE CONTAINS(FullIdx, 'word1' and 'word2')

So i have to parse the textbox value, scan it for spaces and split the text at those points. That made me wonder if it was possible to parse the search text for every setting of the fulltextindex, using the actual definition of the fulltextindex to create the correct where clause.

So if ccPunctuationDash is enabled in the FulltextIndex definition, than the search text is also split on a '-'.

If you think of it, it is exactly the same process as when the index is created and all strings are tokenized ...

My question: what is the easiest way of tokenizing a searchstring according to the settings of a FUlltextIndex?

Answer 1

The easiest way is... to create an empty #temporary table with a string field, with the same fulltext index settings as your real table. Set the TnxTable.Options to include dsoAddKeyAsVariantField. Load the string to tokenize into the string field, then view the table indexed by the fulltext index. Presto, you get an extra field displayed, which is the sorted tokens. You can now iterate over the table to read the tokens.