0

I am developing an application which will scrape data from websites. The details like page_url, site_url, last_modified will be stored in the database and page_content will be stored in the file. Later on I want to search the content of the file. I am trying to Thinkingsphinx to achieve this. Does any body have an idea how to implement it ?

Pravin
  • 6,592
  • 4
  • 42
  • 51

2 Answers2

1

I don't believe Sphinx / ThinkingSphinx is right for you as they are intended to talk to databases.

Take a look at this thread: http://groups.google.com/group/thinking-sphinx/browse_thread/thread/69cdc1c8e1c096ff/aa4ae8e21c42e67c?lnk=gst&q=files#aa4ae8e21c42e67c

lebreeze
  • 5,094
  • 26
  • 34
  • I'm not aware of any myself. I'm having a quick look around at the moment and will let you know if I find anything. I can say that Solr / Sunspot do not look like options either :) – lebreeze Mar 22 '11 at 09:50
  • what about xmlpipe options under sphinx? I was actually trying to implement that, but not getting how to.. – Pravin Mar 22 '11 at 09:51
  • Take a look at this which is designed to scrape websites and index them http://rdig.rubyforge.org/ – lebreeze Mar 22 '11 at 10:46
0

Could you switch to Solr Full Text search instead? I hear it can even index binary files like PDF or word

http://lucene.apache.org/solr/tutorial.html[enter link description here][1]

Bashar Abdullah
  • 1,545
  • 1
  • 16
  • 27