Count number of word occurrences working slow BaseX xquery

Question

I want to count occurrences of the words in the XML document, query giving the actual count but it is working slow.

There are only two xml files size (236 KB, 155 KB) and it is taking 17 sec to produce result.

Below is the query:

let $doc := db:open('test','/ieee/test.xml')

let $tokens := $doc//text()/fn:tokenize(fn:normalize-space(.),'\s')
let $stringtoken := for $x at $pos in $tokens[position() = 1 to fn:last()-1]
                    let $y := string-join($tokens[position() = $pos to $pos + 1],' ')
                    return $y
return                  
<results>
        {
          for $result in distinct-values($stringtoken)
          let $count := count($stringtoken[. = $result])
          return
         <term word="{$result}" count="{$count}"></term>
        }
</results>

In the above query let $count := count($stringtoken[. = $result]) is taking too much time.

Any suggestion to improve the performance of the code much appreciated.

By `$tokens[position() = $pos to $pos + 1]`, you are creating combinations of two adjacent tokens, but this is not indicated in the text. Could you further specify this? — Christian Grün, Feb 26 '22 at 17:46
Thank for your response @christian, Yes I am creating combination of two adjacent — Dharmendra Kumar Singh, Feb 26 '22 at 19:01
So Here I would like to give preference to user if user paas parameters 0 then it will find and count one word, where is +1 i am using for combination of two adjacent user can increase the number instead of +1 it could be +2 so it that case three combination — Dharmendra Kumar Singh, Feb 26 '22 at 19:03

Christian Grün · Accepted Answer · 2022-02-26T23:02:16.717

3

The group by statement will speed up your query a lot:

return <results>{
  for $grouped-token in $stringtoken
  group by $token := $grouped-token
  let $count := count($grouped-token)
  return <term word="{ $token }" count="{ $count }"/>
}</results>

edited Feb 26 '22 at 23:02

answered Feb 26 '22 at 17:54

Christian Grün

6,012
18
34

Thank you Christian, it is really too fast, you made my life easy!!! Thanks again. – Dharmendra Kumar Singh Feb 27 '22 at 06:37

Count number of word occurrences working slow BaseX xquery

1 Answers1