Finding top 5 common words in a website using scrapy framework

Question

I can find out top 5 common words without scrapy by fetching pages content and finding text form page. Then dumping words in dictionary with count of appearing.

But I want to utilise facility of scrapy to do this. But I am not sure where should I keep that dictionary in project to hold word count so spider can send data to common place then to update that dictionary.

How can I use scrapy to find top common words?
Can I use stat collection module of scrapy so after completion of crawling I can print in stats?

If the question is about "how to write stats in scrapy", then I don't see the point of asking about "top common words on a website". About stats, here is some [useful information](https://stackoverflow.com/questions/22951418/how-to-collect-stats-from-within-scrapy-spider-callback) — eLRuLL, Dec 30 '18 at 13:18
This not about writing stats in scrapy, Its about solving this problem with scrapy itself. Using scrapy there can be multiple approach and stat can be one of them that I am not sure. — Alok, Dec 30 '18 at 13:34

dmmfll · Answer 1 · 2018-12-30T14:33:54.333

I have never used scrapy before, but I think I have a solution for counting all the words in the text of the body of HTML.

In a file named words_spider.py put the following code:

from collections import Counter

import scrapy

class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = [
        'http://quotes.toscrape.com/tag/humor/',
    ]

    def parse(self, response):
        for text in response.xpath('//body//*//text()').extract():
            # Eliminate empty strings
            words_ = (item.strip() for item in text.strip().split(' '))
            words = [item for item in words_ if item]
            if any(words):
                yield Counter(words)

Then in another file named scrapy_runner.py put the following code:

import os
import subprocess
import shlex
import json
from functools import reduce
from operator import add
from collections import Counter
from pprint import pprint

FILENAME = 'counters.json'
SCRIPTNAME = 'words_spider.py'

try:
    os.remove(FILENAME)
except FileNotFoundError:
    pass # No file to remove
# Run the spider.
subprocess.check_call(shlex.split(f'scrapy runspider {SCRIPTNAME} -o {FILENAME}'))

with open(FILENAME) as fh:
    # Create counters out of saved JSON file.
    counts = (Counter(item) for item in json.load(fh))

# Add all the counters together.
pprint(reduce(add, counts), indent=4)

Run the script: python scrapy_runner.py:

The output is:

Counter({ 'humor': 12, 'by': 11, '(about)': 10, 'Tags:': 10, 'a': 7, 'you': 7, 'to': 6, 'in': 6, 'and': 6, 'is': 5, 'think': 5, 'the': 4, '“The': 3, 'be': 3, 'must': 3, 'can': 3, 'Quotes': 2, 'it': 2, 'or': 2, 'who': 2, 'books': 2, 'simile': 2, 'thinks': 2, 'sitting': 2, 'make': 2, 'that': 2, 'of': 2, 'beholder': 2, 'time': 2, 'chocolate': 2, 'Charles': 2, 'right': 2, 'it.”': 2, 'people': 2, 'with': 2, 'only': 2, 'I': 2, 'truth': 2, 'Scrape': 1, 'Login': 1, 'Viewing': 1, 'tag:': 1, 'person,': 1, 'gentleman': 1, 'lady,': 1, 'has': 1, 'not': 1, 'pleasure': 1, 'good': 1, 'novel,': 1, 'intolerably': 1, 'stupid.”': 1, 'Jane': 1, 'Austen': 1, 'aliteracy': 1, 'classic': 1, '“A': 1, 'day': 1, 'without': 1, 'sunshine': 1, 'like,': 1, 'know,': 1, 'night.”': 1, 'Steve': 1, 'Martin': 1, 'obvious': 1, '“Anyone': 1, 'church': 1, 'Christian': 1, 'also': 1, 'garage': 1, 'car.”': 1, 'Garrison': 1, 'Keillor': 1, 'religion': 1, '“Beauty': 1, 'eye': 1, 'may': 1, 'necessary': 1, 'from': 1, 'give': 1, 'stupid': 1, 'misinformed': 1, 'black': 1, 'eye.”': 1, 'Jim': 1, 'Henson': 1, '“All': 1, 'need': 1, 'love.': 1, 'But': 1, 'little': 1, 'now': 1, 'then': 1, "doesn't": 1, 'hurt.”': 1, 'M.': 1, 'Schulz': 1, 'food': 1, '“Remember,': 1, "we're": 1, 'madly': 1, 'love,': 1, 'so': 1, "it's": 1, 'all': 1, 'kiss': 1, 'me': 1, 'anytime': 1, 'feel': 1, 'like': 1, 'Suzanne': 1, 'Collins': 1, '“Some': 1, 'never': 1, 'go': 1, 'crazy.': 1, 'What': 1, 'truly': 1, 'horrible': 1, 'lives': 1, 'they': 1, 'lead.”': 1, 'Bukowski': 1, 'trouble': 1, 'having': 1, 'an': 1, 'open': 1, 'mind,': 1, 'course,': 1, 'will': 1, 'insist': 1, 'on': 1, 'coming': 1, 'along': 1, 'trying': 1, 'put': 1, 'things': 1, 'Terry': 1, 'Pratchett': 1, 'open-mind': 1, 'thinking': 1, '“Think': 1, 'left': 1, 'low': 1, 'high.': 1, 'Oh,': 1, 'up': 1, 'if': 1, 'try!”': 1, 'Dr.': 1, 'Seuss': 1, 'philosophy': 1, 'reason': 1, 'talk': 1, 'myself': 1, 'because': 1, 'I’m': 1, 'one': 1, 'whose': 1, 'answers': 1, 'accept.”': 1, 'George': 1, 'Carlin': 1, 'insanity': 1, 'lies': 1, 'lying': 1, 'self-indulgence': 1, 'Next': 1, '→': 1, 'Top': 1, 'Ten': 1, 'tags': 1, 'love': 1, 'inspirational': 1, 'life': 1, 'reading': 1, 'friendship': 1, 'friends': 1, 'by:': 1, 'GoodReads.com': 1, 'Made': 1, '❤': 1, 'Scrapinghub': 1})

Finding top 5 common words in a website using scrapy framework

1 Answers1