0

I'm scraping some information off a site and one of the fields were stored in my list like this: [u'Dover Park', u'30 \u2013 38 Dover Rise']

The \2013 should be a –.

When trying to write to a .csv file, I get the following error:

UnicodeEncodeError: 'ascii' codec can't encode character u'\u2013' in position 3: ordinal not in range(128).

This is my code:

import re
import mechanize
from BeautifulSoup import BeautifulSoup

url = 'http://www.dummy.com'

br = mechanize.Browser()

page = br.open(url)

html = page.read()
html = html.decode('utf-8')
soup = BeautifulSoup(html)

table = soup.find('table', width='800')

property_list = []

for row in table.findAll('tr')[1:]:
    for field in row.findAll('td', width='255'):
        property_list.append(field.findAll(text=True))

for condo in property_list:
    for field in condo:
        if field == '  ':
            condo.remove(field)

for condo in property_list:
    if len(condo) < 2:
        condo.append(condo[0])
    if condo[1]:
        condo[1] = condo[1].replace(',','')

for condo in property_list:
    for field in condo:
        field = field.encode('utf-8')

import csv

myfile = open('condos.csv', 'wb')
try:
    wr = csv.writer(myfile)
    wr.writerow(('Name','Address'))
    for condo in property_list:
        print condo
        wr.writerow(condo)
finally:
    myfile.close()
super9
  • 29,181
  • 39
  • 119
  • 172
  • 2
    This is urlencoded data, you can use that lib to decode it. For your encoding problem, explicitly encode your strings to utf8. – Thomas Orozco Nov 16 '11 at 14:53
  • I've tried various combinations but I still can't seem to get it to work. Please see the updated question. – super9 Nov 16 '11 at 16:01

1 Answers1

0

Maybe you can try : 

#!/usr/bin/python
# --*-- coding:UTF-8 --*--

import codecs
streamWriter = codecs.lookup('utf-8')[-1]
sys.stdout = streamWriter(sys.stdout)

your_var.decode('utf-8')
Carto_
  • 577
  • 8
  • 28