super call to DjangoSpider's parse method not working

Question

I am using django-dynamic-scraper in one of my django projects. What I am doing is pretty simple. I am inheriting dynamic_scraper.spiders.DjangoSpider class to have some custom functionality in its parse method. Following is what I did:

from dynamic_scraper.spiders import DjangoSpider

class CustomSpider(DjangoSpider):
    def __init__(self, *args, **kwargs):
        # some custom stuff here
        super(CustomSpider, self).__init__(*args, **kwargs)

    def parse(self, response):
        # Modify response based on some custom
        # set of rules

        super(CustomSpider, self).parse(response)

Now here the super call in the parse method is not being fired. I have made sure that I inherit the correct class and it does have a parse method.

I have tried printing debug statements in DjangoSpider's parse method, but don't see anything in stdout.

If i try printing the same debug statements after the super call, I see the statements in stdout.

Any ideas ?

Answer 1

parse is a generator, not a regular method.

So you have to do:

def parse(self, response):
    # Modify response based on some custom
    # set of rules

    for x in super(CustomSpider, self).parse(response)
        yield x

Because a generator is returned, parse is iterable. You need to yield each item from the super call that was passed your modified response.

Example

class Base(object):
    def gen(self, data):
        for i in data:
            yield i

class Something(Base):
    def gen(self, data):
        data += "world"
        for i in super(Something, self).gen(data):
            yield i

s = Something()
for i in s.gen("hello"):
    print i

Output

h
e
l
l
o
w
o
r
l
d