poor search performance for certain wildcard queries

Question

I am having performance issues when using wildcard searching for certain letter combinations, and I am not sure what else I need to to to possibly improve it. All of my documents are following an envelope pattern that look something like the following.

<pdbe:person-envelope>
    <person xmlns="http://schemas.abbvienet.com/people-db/model">
        <account>
            <domain/>
            <username/>
        </account>
        <upi/>
        <title/>
        <firstName>
            <preferred/>
            <given/>
        </firstName>
        <middleName/>
        <lastName>
            <preferred/>
            <given/>
        </lastName>
    </person>
    <pdbe:raw/>
</pdbe:person-envelope>

I have a field defined called name, which includes the firstName and lastName paths:

{
  "field-name": "name",
  "field-path": [
    {
      "path": "/pdbe:person-envelope/pdbm:person/pdbm:firstName",
      "weight": 1
    },
    {
      "path": "/pdbe:person-envelope/pdbm:person/pdbm:lastName",
      "weight": 1
    }
  ],
  "trailing-wildcard-searches": true,
  "trailing-wildcard-word-positions": true,
  "three-character-searches": true
}

When I do some queries using search:search, some come back fast, whereas others come back slow. This is with the filtered queries.

search:search("name:ha*",
  <options xmlns="http://marklogic.com/appservices/search">
    <constraint name="name">
      <word>
        <field name="name"/>
      </word>
    </constraint>
    <return-plan>true</return-plan>
  </options>
  )

I can see from the query plan that it is going to filter over all 136547 fragments in the db. But this query works fast.

<search:query-resolution-time>PT0.013205S</search:query-resolution-time>
<search:snippet-resolution-time>PT0.008933S</search:snippet-resolution-time>
<search:total-time>PT0.036542S</search:total-time>

However a search for name:tj* takes a long time, and also filters over all of the 136547 fragments.

<search:query-resolution-time>PT6.168373S</search:query-resolution-time>
<search:snippet-resolution-time>PT0.004935S</search:snippet-resolution-time>
<search:total-time>PT12.327275S</search:total-time>

I have the same indexes on both. Are there any other indexes I should be enabling when I am specifically just doing a search via the field constraint? I have these other indexes enabled on the database itself, in general.

"collection-lexicon": true,
  "triple-index": true,
  "word-searches": true,
  "word-positions": true

I tried doing an unfiltered query, but that did not help as I got a bunch of matches on the whole document, and not the the fields I wanted. I even tried to set the root-fragment to just my person element, but that did not seem to help things.

 "fragment-root": [
    {
      "namespace-uri": "http://schemas.abbvienet.com/people-db/model",
      "localname": "person"
    }
  ]

Thanks for any ideas.

score 3 · Accepted Answer · answered Jun 06 '16 at 04:57

3

Fragment roots are helpful if you want to use a searchable expression for that person element, and mostly if it occurs multiple times in one document. It won't make your current search constrain on that element.

In your case you enabled a number of relevant options, but the wildcard option only works for 4 characters of more. If you want to search on wildcards with less characters, you need to enable the three, two and one character search options.

The search phrases mentioned above both contained two characters with a wildcard. Since you only enabled the three character option, it had to rely on filtering. The fact some run fast, some slow is probably because of caching. If you repeat the same query, MarkLogic will return the result from cache.

For performance testing you would either have to restart MarkLogic regularly to flush caches, or search on (semi) random strings to avoid MarkLogic being able to cache. Or maybe both..

HTH!

answered Jun 06 '16 at 04:57

grtjn

20,254
1
24
35

In addition to what Geert said, there are also cases where, instead of adding the 2 and 3 char wildcard options, you can also consider leaving the settings are-is and introducing a word lexicon on the element/field in question, Then you essentially do your term expansion based on element-word-match. Again, it depends on your final use-case. You could plug in this approach as a custom constraint This can even be tuned with things like the eager option on cts:field-word-match as well as process through each forest directly. Which is more efficient/faster? You have to test and tune. – David Ennis -CleverLlamas.com Jun 06 '16 at 07:32
Thanks for the comments. I actually am using this as a custom constraint. In the search options (for autocompletion) I am overriding the term handling to expand in to searching these specific fields. This is autocomplete to quickly find the right person, as the user is typing.. this could be by first/last name, username, or internal identifier. For some reason I thought I read somewhere that enabling the three-character-searches would also handle the two and one character cases, but I might have misread that. Enabling those indexes solved the problem. – TJ Tang Jun 06 '16 at 11:34
I see where I misread it now.. the documentation says "This index is not needed if you have three character searches and a word lexicon." As @DavidEnnis mentioned, I could add the word lexicon, which I did not. – TJ Tang Jun 06 '16 at 11:40

poor search performance for certain wildcard queries

1 Answers1