Python 3.1.3 Win 7: csv writerow Error "must be bytes or buffer, not str"

Question

Got a simple script which worked perfectly under Python 2.7.1 at my Win xp machine. Now got a win 7 machine with python 3.1.3.

The code is:

owriter.writerow(dtime[1][1])

dtime[1][1]=['30-Aug-10 16:00:00', '2.5', '15']

Got this error message: TypeError: must be bytes or buffer, not str

What changes should I make?

thanks.

When you upgraded from Python 2 to Python 3, did you run [2to3](http://docs.python.org/library/2to3.html) on your script? — Greg Hewgill, Jan 19 '11 at 18:27
and what is `outputfile`? post your actual code. This is not even remotely comprehensible. — SilentGhost, Jan 19 '11 at 18:35

score 16 · Answer 1 · answered Jan 20 '11 at 04:04

In Python 2.X, it was required to open the csvfile with 'b' because the csv module does its own line termination handling.

In Python 3.X, the csv module still does its own line termination handling, but still needs to know an encoding for Unicode strings. The correct way to open a csv file for writing is:

outputfile=open("out.csv",'w',encoding='utf8',newline='')

encoding can be whatever you require, but newline='' suppresses text mode newline handling. On Windows, failing to do this will write \r\r\n file line endings instead of the correct \r\n. This is mentioned in the 3.X csv.reader documentation only, but csv.writer requires it as well.

+1 As it happens, I should be trying to answer the request for a doc-patch after I raised a bug report about the 3.X csv.writer docs -- http://bugs.python.org/issue7198 -- thanks for the reminder :-) — John Machin, Jan 20 '11 at 05:31

score 6 · Accepted Answer · answered Jan 19 '11 at 18:33

6

Probably you need to open the file in text mode. If not, include enough of your code so it's runnable and demonstrates the problem.

answered Jan 19 '11 at 18:33

Lennart Regebro

167,292
41
224
251

Very nice remark; Windows' text/binary file modes are common gotcha. – ulidtko Jan 19 '11 at 18:34
Thanks. the code is: outputfile=open("out.csv",'wb') owriter=csv.writer(outputfile) owriter.writerow(dtime[1][1]) outputfile.close() – steve Jan 19 '11 at 18:36
2

@steve So take away the `b` flag. CSV files are not binary, they should be opened in text mode. – ulidtko Jan 19 '11 at 18:38
@steve: and why are you opening file with `'wb'`? – SilentGhost Jan 19 '11 at 18:38
@steve: And indeed you do open it in binary mode, as suspected. Changing it to text mode solves the problem. – Lennart Regebro Jan 19 '11 at 18:39
Well, it *is* 100% correct. It's not just complete. :-) This is interesting as this is an example of a module in the stdlib that breaks the Python 3 rule of not changing it's API, while there is no fixer for it! – Lennart Regebro Jan 20 '11 at 07:06
@Lennart Regebro: The definition of text mode vs binary mode changed from 2.X to 3.X. In 2.x all it meant was: in text mode on Windows aka Visual CP/M, convert external CRLF <-> `\n` and stop reading on '\x1a`. In 3.x its meaning was changed: on any platform, convert external bytes <-> str. Newline handling was split out into the `newline` option. Consequently the csv API had to change. – John Machin Jan 20 '11 at 07:48
@Lennart Regebro: [continued] The best one could hope for from 2to3 would be warning messages "you may need to change the way that 'csvfile' is opened" on detecting csv.reader|writer. It would be nice if csv.writer checked immediately whether the file looked OK instead of getting the unhelpful message that the OP got -- WHAT "must be bytes or buffer"? – John Machin Jan 20 '11 at 07:50
@John Machin: Sure, bt if the API changes, there should be a fixer or a Python2.6 -3 warning. The problem is that the parameters are to the file opening, which means making a fixer is extremely difficult. That mmeans -3 should warn, which it doesn't, or opening in binary mode in Python 3 should warn or fail, which it doesn't. This is IMO a bug in Python. – Lennart Regebro Jan 20 '11 at 10:19
@Lennart Regebro: When you submit a bug report, please add `sjmachin` to the "nosy" list :-) – John Machin Jan 20 '11 at 11:44
@Lennart Regebro: [OT] There are other situations where the real problem is one step away and it's hard to make a fixer e.g. `ord(raw_bytes[i])` which needs to become merely `raw_bytes[i]`. I regard changing the result of `raw_bytes[i]` from 1-byte `str` to `int` a major change in API; what do you think? – John Machin Jan 20 '11 at 11:58
@John Machin: Done! http://bugs.python.org/issue10954 I dont' think the difference in api between Python 2 str and Python 3 bytes is an API change as such. :) Maybe it is, but it's also one of the fundamental changes, no point in warning for that. But libraries API shouldn't change without warning. – Lennart Regebro Jan 20 '11 at 12:08

ulidtko · Answer 3 · 2011-01-19T18:44:29.397

0

Change to str.encode("ascii").

The point is that Python 2.x had somewhat mixed usage of str type for storing byte buffers and for storing character strings. Now in Python 3.x we have proper Unicode support and byte buffers are now separate type. You can convert between them using str.encode() and bytes.decode() specifying each time a character encoding as parameter.

edited Jan 19 '11 at 18:44

answered Jan 19 '11 at 18:21

ulidtko

14,740
10
56
88

@steve it is a method of string objects. You must use them as `"foo bar".encode("utf-8")`. – ulidtko Jan 19 '11 at 18:41
@steve note that in your case you shouldn't open CSV files in binary mode and [Lennart's answer][http://stackoverflow.com/questions/4739066/python-3-1-3-win-7-csv-writerow-error-must-be-bytes-or-buffer-not-str/4739193#4739193 ] is the right solution. – ulidtko Jan 19 '11 at 18:43

Python 3.1.3 Win 7: csv writerow Error "must be bytes or buffer, not str"

3 Answers3

Linked