subclassing beautifulsoup html parser, getting type error

Question

I wrote a little wrapper using beautifulsoup great html parser

recently I tried to improve the code and make all beautifulsoup methods available directly in the wrapper class (instead of through a class property ) and I thought subclassing the beautifulsoup parser would be the best way to achieve this.

Here is the the class:

class ScrapeInputError(Exception):pass
from BeautifulSoup import BeautifulSoup

class Scrape(BeautifulSoup):
    """base class to be subclassed
    basically a subclassed BeautifulSoup wrapper that providers
    basic url fetching with urllib2
    and the basic html parsing with beautifulsoup
    and some basic cleaning of head,scripts etc'"""

    def __init__(self,file):
        self._file = file
        #very basic input validation
        import re
        if not re.search(r"^http://",self._file):
            raise ScrapeInputError,"please enter a url that starts with http://"

        import urllib2
        #from BeautifulSoup import BeautifulSoup
        self._page = urllib2.urlopen(self._file) #fetching the page
        BeautifulSoup.__init__(self,self._page)
        #self._soup = BeautifulSoup(self._page) #calling the html parser

this way I can just initiate the class with

x = Scrape("http://someurl.com")

and be able to traverse the tree with x.elem or x.find

this works wonderfull with some beautifulsoup methods (see above) but fails with others - those using iterator like "for e in x:"

the error message:

 Traceback (most recent call last):
  File "<pyshell#86>", line 2, in <module>
    print e
  File "C:\Python27\lib\idlelib\rpc.py", line 595, in __call__
    value = self.sockio.remotecall(self.oid, self.name, args, kwargs)
  File "C:\Python27\lib\idlelib\rpc.py", line 210, in remotecall
    seq = self.asynccall(oid, methodname, args, kwargs)
  File "C:\Python27\lib\idlelib\rpc.py", line 225, in asynccall
    self.putmessage((seq, request))
  File "C:\Python27\lib\idlelib\rpc.py", line 324, in putmessage
    s = pickle.dumps(message)
  File "C:\Python27\lib\copy_reg.py", line 77, in _reduce_ex
    raise TypeError("a class that defines __slots__ without "
TypeError: a class that defines __slots__ without defining __getstate__ cannot be pickled

I researched the error message but couldn't find anything I could work with - becasue I don't want to play with the inner implantation of BeautifulSoup (and honestly I don't know or understand __slot__ or __getstate__..) I just want to use the functionality.

instead of subclassing I tried returning a beautifulsoup object from the __init__ of the class but __init__ method returns None

Be glad for any help here.

Sidenote: Don't use `re` to test if a string starts with a substring, that's overkill. Use `str.startswith()` instead. (`if not file.startswith("http://"):`). — Ferdinand Beyer, Oct 07 '11 at 08:42
Another sidenote: do you really want to disallow `https://`? (Or `ftp://`, or `file://`?) You might want to rely on `urlopen`'s own validation; it raises `urllib2.URLError` on invalid URLs. — Petr Viktorin, Oct 07 '11 at 10:52

score 1 · Accepted Answer · edited May 23 '17 at 12:27

1

The error is not happening in BeautifulSoup code. Rather, your IDLE is not able to retreive and print the object. Try print str(e) instead.

Anyway, subclassing BeautifulSoup in your situation may not be the best idea. Do you really want to inherit all of the parsing methods (like convert_charref, handle_pi or error)? Worse, if you override something that BeautifulSoup uses, it may break in a hard-to-find way.

I don't know your situation, but I suggest preferring composition over inheritance (i.e. having a BeautifulSoup object in an attribute). You can easily (if in a slightly hacky way) expose specific methods like this:

class Scrape(object):
    def __init__(self, ...):
        self.soup = ...
        ...
        self.find = self.soup.find

edited May 23 '17 at 12:27

Community

1
1

answered Oct 07 '11 at 09:23

Petr Viktorin

65,510
9
81
81

will this method also work for the __iter__ and __key__ methods? – alonisser Oct 07 '11 at 14:52
[No](http://docs.python.org/reference/datamodel.html#special-method-lookup-for-new-style-classes), but you still can do `def __iter__(self): return iter(self.soup)`. – Petr Viktorin Oct 07 '11 at 18:26

subclassing beautifulsoup html parser, getting type error

1 Answers1

Linked