Questions tagged [websphinx]

I've tried WebSphinx application. I realize if I put wikipedia.org as the starting URL, it will not crawl further. Hence, how to actually crawl the entire Wikipedia? Can anyone gimme some guidelines? Do I need to specifically go and find those URLs…

I am looking to crawl the entire website and save it locally offline. It should have 2 parts: Authentication This needs to be implemented using Java and I need to override HttpsURLConnection logic to add couple lines of authentication (Hadoop) in…

I have downloaded websphinx to do this but i need it to ask me username and password of website and then submit the username and password to the website and once authenticated it should start crawling the internal links and sublinks and save the…

Here is my code for Regex matching which worked for a webpage: public class RegexTestHarness { public static void main(String[] args) { File aFile = new File("/home/darshan/Desktop/test.txt"); FileInputStream inFile = null; …