XQuery Full text searching over mixed content

Question

Following is the XML structure - (i have given a very small piece of the entire document with limited data. I have a XML DB of 6 GB, with proper FULL TEXT Index.)

<Docs>
 <Doc>
<Chap>No - 1</Chap>
<Desc>
  <Notes>
    <Para t="sn">departmental report</Para>
  </Notes>
  <Notes>
    <Para t="sn">The equiry commission is good.</Para>
  </Notes>
  <Notes>
    <Para t="sn">departmental process</Para>
    <Para t="ln">The enquiry report for the bomb blast is yet to come.<bL/>
      <bL/>The department working on this is quite lazy.</Para>
  </Notes>
</Desc>
</Doc>
<Doc>
<Chap>No - 2</Chap>
<Desc>
  <Notes>
    <Para t="sn">Enquiry Processes Report</Para>
    <Para t="ln">The enquiry process is very simple.<bL/>
      <bL/>With proper guidance anybody can handle the commission easily.<bL/>
      <bL/>
    </Para>
  </Notes>
  <Notes>
    <Para t="sn">Enquiry - Departmental</Para>
  </Notes>
</Desc>
 </Doc>
 <Doc>
<Chap>No - 3</Chap>
<Desc>
  <Notes>
    <Para t="sn">Physics Department</Para>
  </Notes>
  <Notes>
    <Para t="sn">Working process of physics department is quite lengthy</Para>
    <Para t="ln">Even after proper enquiry, I was told nothing.<bL/>
      <bL/>This was like a bomb blast.</Para>
  </Notes>
  <Notes>
    <Para t="sn">Departmental enquiry.</Para>
    <Para t="ln">There should be a departmental enquiry for this wrong process.</Para>
  </Notes>
</Desc>
</Doc>
</Docs>

Now I want all those Chap nodes containing all words "departmental", "enquiry" and "report".

So far, I am unable to get them using various combinations. One of my try is -

for $x in ft:search("Docs", ("departmental enquiry report"), map{'mode':='all words'})/ancestor::*:Para
 return $x/ancestor::Chap

Can any body guide me on this ?

With regards to NiksD's question: I know that the multipath indexes in xDB (https://community.emc.com/docs/DOC-6822) support this scenario with the option 'include descendants texts' and using the standard XQuery full-text syntax (like Jens Erat showed). I assume that there are other XML databases which support it as well though. — Ewout Graswinckel, Feb 27 '14 at 16:56
@EwoutGraswinckel- Thanks for the information. Can you tell me the names of these? — NiksD, Feb 28 '14 at 06:07
A general note: Support for descendant full-text indexing will be included in a future release of BaseX (timeline is not yet available). — Christian Grün, Mar 02 '14 at 23:43
@ChristianGrün- Thanks a lot. I am EAGERLY waiting for the same :) — John, Mar 03 '14 at 04:49

Jens Erat · Answer 1 · 2014-02-27T09:21:04.817

`ft:search`, and why it Will not Solve the Issue

By looking at BaseX' XQuery Full Text Documentation you will realize that the second argument in ft:search should be a sequence of words:

ft:search($db as xs:string, $terms as item()*, $options as item()) as text()*

So, your query should look something like

for $x in ft:search("Docs", ("departmental", "enquiry", "report"), map{'mode':='all words'})/ancestor::*:Para
return $x/ancestor::Chap

Yet this still will not solve your issue, as this function

[re]turns all text nodes from the full-text index of the database $db that contain the specified $terms.

In other words: all of these words would have to occur in a single text node, but they're spread over multiple in your example input (all over a <Doc/> node).

Using Standard XQuery Full Text

I had to guess from the input and words you're searching for that you actually want to search for <Doc/> nodes that contain all these three words.

for $document in doc("Docs")/Docs/Doc
where $document contains text { 'departmental', 'enquiry', 'report' } all words
return $document/Chap

This will retrieve all documents, apply a full text search on it and finally return the document's chapter node.

Be aware

I removed the namespace wildcard, as no namespaces are included in your example document and
to create a full text index (if you didn't do yet) which will highly increase performance.

Christian Grün · Accepted Answer · 2014-02-27T17:10:57.807

1

The full-text index of BaseX references all terms on text node level. This means that all of your words would need to occur in the same text node.

If you want to take advantage of the full-text query and find all words that occur below a certain element, you could try the following query:

let $words := ("departmental enquiry report")
for $doc in db:open("Docs")//Doc[.//text() contains text { $words } any word]
where $doc[string-join(.//text(), ' ') contains text { $words } all words]
return $doc/Chap

The first contains text expression will be rewritten to an index request. It will return all texts that return any of the searched words. The contains text expression in the where clause will filter out all nodes that do not contain all of your query terms. With string-join(.//text(), ' '), all text nodes below the Doc element will be concatenated, and the search will be performed on the joined string.

The folowing, equivalent representation of the query should yield the same results:

let $words := ("departmental enquiry report")
for $x in ft:search("Docs", $words, map { 'mode': 'any word' })/ancestor::*:Doc
where ft:contains(string-join($x//text(), ' '), $words, map { 'mode': 'all words' })
return $x/Chap

edited Feb 27 '14 at 17:10

answered Feb 27 '14 at 09:29

Christian Grün

6,012
18
34

Thanks for the answer. But still I am not getting the exact result. The above query (first one) is working ONLY when, all the 3 words come under `` tag. Say 2 words, out of the 3, comes under one `` and remaining 1 under the other ``, then NO RESULT, i.e; that `Doc/Chap` is not in result. – John Feb 27 '14 at 12:26
Sorry, I should have tried the query. It is now fixed, and I have also revised the text. – Christian Grün Feb 27 '14 at 17:11
PS: The original query with `.` instead of `string-join(.//text(), ' ')` works if you create the database with the "Chop Whitespaces" flag turned off. This way, all whitespaces between the elements will be stored as extra text nodes. As a result, the size of your database will increase, but the words of different elements will all be separated with enough whitespaces. – Christian Grün Feb 27 '14 at 17:14
@ChristianGrün - Thanks for the query. I tried both. But the second query, the one with ft:search, is giving error. "Invalid Key, simple expression expected". – Abhit Feb 28 '14 at 06:23
@Abhit- Its working. Please check the BaseX version. `ft:contains` is available with 7.8. Same issue was with me. – John Feb 28 '14 at 07:07

XQuery Full text searching over mixed content

2 Answers2

ft:search, and why it Will not Solve the Issue

Using Standard XQuery Full Text

`ft:search`, and why it Will not Solve the Issue