2

I'm wondering if anyone has any recommendations for a Python full-text search engine similar to mnogosearch. I'm trying to get it to function like Mnogosearch, but not sure how that compares to other options (if there are better options out there).

Haystack, for example, seems like it does a great job at indexing given Django model fields that you specify, but I'm not sure if it can search the entire contents of a website.

Solr/Lucine seem promising, but I'm not too familiar with it. Whoosh seems like it might also be an interesting option, but I'm wondering if the Python implementation will make it slower?

I'm pretty new to search so I'm trying to wrap my head around the different options. Does anyone have any good opinions on which search technologies work well for indexing an entire site?

Thanks for reading. Any comments are much appreciated.

Joe

Joe J
  • 9,985
  • 16
  • 68
  • 100

1 Answers1

2

IMHO Solr is probably the best option, performance-wise and feature-wise and it's based on heavily tested technology. And if you're looking for a pure Python solution Whoosh looks good - I haven't used it in production, just checked it out and looked at the code. It may be slower, but I doubt that it's noticeable for sites that get lighter traffic on the search feature. A nice feature whoosh has is an implementation of BM25F which can take into consideration site structure such as title/header/body/footer etc. and is considered state of the art in generic web search models.

If you're using Django, as far as I know Haystack would let you change the search engine backend much like you are able to change the RDBMS backend.

I'm not sure what exactly you mean by "whole site content". If most of your site's page content isn't generated from model content then using an integrated search index might not be a best option, maybe something like IndexTank or Google site search. But in the opposite case I'd recommend haystack if you are new to either search technology as it will make life easier for you with magic.

Vasil
  • 36,468
  • 26
  • 90
  • 114
  • Thank you Vasil - your post has helped give me some direction on this. It's good to know that Solr seems pretty stable and standard. Sorry to be vague about 'whole site content'. I am looking for something that does scrape pages and integrate them into a site like Google Site Search or IndexTank. Mnogosearch is a PHP-based solution for doing this locally and I was wondering if there existed similar functionality to that in one of the more common tools like Solr/Lucine, Whoosh, or Xapian, as I'm not sure how much momentum Mnogosearch has. – Joe J Apr 18 '11 at 20:52
  • 1
    @Joe J apache nutch integrates with solr and can crawl a site. Although it might be overkill, but a quality solution nevertheless. http://wiki.apache.org/nutch/NutchTutorial – Vasil Apr 18 '11 at 20:58