12

I'm not entirely sure what I need to do about this error. I assumed that it had to do with needing to add .encode('utf-8'). But I'm not entirely sure if that's what I need to do, nor where I should apply this.

The error is:

line 40, in <module>
writer.writerows(list_of_rows)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2013' in position 1
7: ordinal not in range(128)

This is the base of my python script.

import csv
from BeautifulSoup import BeautifulSoup

url = \
'https://dummysite'

response = requests.get(url)

html = response.content

soup = BeautifulSoup(html)

table = soup.find('table', {'class': 'table'})

list_of_rows = []
for row in table.findAll('tr')[1:]:
list_of_cells = []
for cell in row.findAll('td'):
    text = cell.text.replace('[','').replace(']','')
    list_of_cells.append(text)
list_of_rows.append(list_of_cells)

outfile = open("./test.csv", "wb")
writer = csv.writer(outfile)
writer.writerow(["Name", "Location"])
writer.writerows(list_of_rows)
f00d
  • 581
  • 1
  • 6
  • 21

3 Answers3

25

Python 2.x CSV library is broken. You have three options. In order of complexity:

  1. Edit: See below Use the fixed library https://github.com/jdunck/python-unicodecsv (pip install unicodecsv). Use as a drop-in replacement - Example:

    with open("myfile.csv", 'rb') as my_file:    
    r = unicodecsv.DictReader(my_file, encoding='utf-8')

  1. Read the CSV manual regarding Unicode: https://docs.python.org/2/library/csv.html (See examples at the bottom)

  2. Manually encode each item as UTF-8:

    for cell in row.findAll('td'):
    text = cell.text.replace('[','').replace(']','')
    list_of_cells.append(text.encode("utf-8"))

Edit, I found python-unicodecsv is also broken when reading UTF-16. It complains about any 0x00 bytes.

Instead, use https://github.com/ryanhiebert/backports.csv, which more closely resembles Python 3 implementation and uses io module..

Install:

pip install backports.csv

Usage:

from backports import csv
import io

with io.open(filename, encoding='utf-8') as f:
    r = csv.reader(f):
Alastair McCormack
  • 26,573
  • 8
  • 77
  • 100
  • oh wow didn't know that the CSV library was broken in python. Thank you very much! That's a huge help – f00d Oct 05 '15 at 14:15
  • 2
    Why isn't this voted up enough ? I've seem people using all kinds of workarounds for solving this, and honestly, none works better than simply using the "unicodecsv" module. – Marcello Grechi Lins Jan 26 '16 at 13:58
1

The issue lies with the csv library in python 2. From the unicodecsv project page

Python 2’s csv module doesn’t easily deal with unicode strings, leading to the dreaded “‘ascii’ codec can’t encode characters in position …” exception.

If you can, just install unicodecsv

pip install unicodecsv

import unicodecsv

writer = unicodecsv.writer(csvfile)
writer.writerow(row)
anantdd
  • 125
  • 7
0

I found the easiest option, in addition to Alastair's excellent suggestions, to be using python3 instead of python 2. all it required in my script was to change wb in the open statement to simply w in accordance with Python3's syntax.

Community
  • 1
  • 1
jcomeau_ictx
  • 37,688
  • 6
  • 92
  • 107