I have been building a crawler in Python for the last 10 months. This crawler is using threading and Queue to hold all the visited and non-visited links.
I use BeautifulSoup and request to access the urls and pick up page title, meta description, keywords, cms system and many more.
At the moment the crawler checks the first seed url for data and when it's done scraping data it finds new links from the current page in one single thread, and then it keeps repeating itself.
Everything works fine, except when I want to stop the scraping process, it doesn't stop the threads, but just keeps running, I have added a locked variable that counts the amount of pages crawled, but somehow it stops the crawl process but not the links process, when crawl is reached.
Would below setup be better?
- A Link class that has one job.
This class would make use of Threads to find all the links on the webpage, or if a limit is set only return the links requested.
- A Scraper class that has one job.
This class would run through all the links returned from the Link class, and collect the requested data using Threads.
My Issue:
If I create the new setup, then the Link class would send request to the urls and so would the scraper class.
So if my link class should return 100 urls then I would send at minimum of 100 request to the urls and when the scraper class goes though the links found by link task, then it ads 100 more request.
So instead of 100 request I end up sending 200 requests.
- I could pass a soup object from the link class to some array, but then I would end up having a memory problem as I see it.
What is your advice?
