3

I have two field (name, modifiedDate) in my index. i want to store new document based on modifiedDate and keep index sorted on modifiedDate
doc #1 is the oldest document and (modifiedDate) is oldest too
doc #n is most recent document and (modifiedDate) is near to now

1) how can i create this index structure that documents physically stored base on (modifiedDate) and keep the structure even after any change happened in the index (optimize, delete, update)

2) the following structure let me search for documents in specific date range. but i don't want to search the entire index and then filter. i want to use the following structure to skip all other documents if it goes beyond the date range

Current lucene behavior

for (1 to docCount)
if (modifiedDate is in date range filter)
calculate the score based on query

Accepted behavior

for (1 to docCount)
if (modifiedDate is greater than upper bound of date range)
break
else
calculate the score based on query

if i have 3,000,000 document and my date range only meets 20 top document, in current lucene behavior i need to check all of the documents, but in accepted behavior I am only scoring top 20 document, and you can guess the huge performance gain

ajreal
  • 46,720
  • 11
  • 89
  • 119
Ehsan
  • 1,662
  • 6
  • 28
  • 49
  • 2
    you can consider split into multiple monthly collection,and perform merge collection search , – ajreal Aug 12 '11 at 05:46
  • this is a good solution that came to my mind before, but managing the index in this form is hard, I prefer to create the index with the above structure to minimize the overhead of fragmented index. If there is a way to create this structure it would be optimal – Ehsan Aug 12 '11 at 08:32
  • See also this really nice overview of Lucene index sorting: http://shaierera.blogspot.com/2013/04/index-sorting-with-lucene.html – Rob Audenaerde Jun 21 '13 at 07:04

2 Answers2

1

The existing answers are fine but Lucene 4.3.0 came out this year with a new "SortingMergePolicy" that allows advanced Lucene users to use the algorithm suggested in the original poster to cancel a search early. See the javadocs

David Smiley
  • 4,102
  • 2
  • 19
  • 18
0

Lucene will index and query efficiently on numeric fields, see NumericRangeQuery. The javadoc I linked to above have notes about the TrieRangeQuery implementation.

You can store modifiedDate as a NumericField which contains the modified date as a long in ms. Then use a QueryWrapperFilter around a NumericRangeFilter to limit your search to the appropriate date range.

This should be very efficient.

sbridges
  • 24,960
  • 4
  • 64
  • 71
  • I've already do all of this optimization. but i need the structure above – Ehsan Aug 12 '11 at 12:58
  • could you post a code example showing what you are doing. With what I described lucene should not be visiting every document in the index for a query – sbridges Aug 12 '11 at 15:19