Crawler/parser for Xapian

Question

I would like to implement a search engine which should crawl a set of web sites, extract specific information from the pages and create full-text index of that specific information.

It seems to me that Xapian could be a good choice for the search engine library.

What are the options for a crawler/parser to integrate with Xapian?

Would Solr be a better choice than Xapian to integrate with open source crawlers/parsers?

score 2 · Answer 1 · answered Dec 19 '08 at 01:29

2

Here's a little comparison between Xapian and Solr.

But if you want to build a crawler, take a look at Nutch. It's extensible with plugins, so you could write a plugin that analyzes the information that you're looking for.

answered Dec 19 '08 at 01:29

Mauricio Scheffer

98,863
23
192
275

score 2 · Answer 2 · edited Feb 05 '12 at 11:16

2

Flax may provide some of what you're looking for.

edited Feb 05 '12 at 11:16

mmmmmm

32,227
27
88
117

answered Apr 20 '09 at 21:33

Rob Young

1,235
11
19

Crawler/parser for Xapian

2 Answers2