16

Using Python to append CSV file, I get data every other row. How do I fix?

import csv

LL = [(1,2),(3,4)]

Fn = ("C:\Test.csv")
w = csv.writer(open(Fn,'a'), dialect='excel')
w.writerows(LL)

C:\test.csv when opened looks like this:

1,2

3,4

1,2

3,4
martineau
  • 119,623
  • 25
  • 170
  • 301
Merlin
  • 24,552
  • 41
  • 131
  • 206

2 Answers2

45

Appending is irrelevant to the problem; notice that the first two rows (those from the original file) are also double-spaced.

The real problem is that you have opened your file in text mode.

CSV is a binary format, believe it or not. The csv module is writing the misleadingly-named "lineterminator (should be "rowseparator") as \r\n as expected but then the Windows C runtime kicks in and replaces the \n with \r\n so that you have \r\r\n between rows. When you "open" the csv file with Excel it becomes confused

Always open your CSV files in binary mode ('rb', 'wb', 'ab'), whether you are operating on Windows or not. That way, you will get the expected rowseparator (CR LF) even on *x boxes, your code will be portable, and any linefeeds embedded in your data won't be changed into something else (on writing) or cause dramas (on input, provided of course they're quoted properly).

Other problems:

(1) Don't put your data in your root directory (C:\). Windows inherited a hierarchical file system from MS-DOS in the 1980s. Use it.

(2) If you must embed hard-wired filenames in your code, use raw strings r"c:\test.csv" ... if you had "c:\test.csv" the '\t' would be interpreted as a TAB character; similar problems with \r and \n

(3) The examples in the Python manual are aligned more towards brevity than robust code.

Don't do this:

w = csv.writer(open('foo.csv', 'wb'))

Do this:

f = open('foo.csv', 'wb')
w = csv.writer(f)

Then when you are finished, you have f available so that you can do f.close() to ensure that your file contents are flushed to disk. Even better: read up on the new with statement.

John Machin
  • 81,303
  • 11
  • 141
  • 189
  • Darn. I just figured this out and was halfway through an answer saying the same. +1. – Steven Rumbalski Nov 22 '10 at 20:39
  • Interestingly, while most of the examples in the `csv` docs for Python 2 docs use binary mode, the [`DictReader`](https://docs.python.org/2/library/csv.html#csv.DictReader) examples don't, and nor do [any of the examples for Python 3](https://docs.python.org/3/library/csv.html). Is there a good reason for this? If not, perhaps you'd like to try getting the docs changed? – Mark Amery Jun 30 '15 at 10:32
  • 1
    In python3, opening in binary mode: `_csv.Error: iterator should return strings, not bytes (did you open the file in text mode?)` – mafrosis Jul 22 '16 at 06:25
  • As the previous comments have already pointed out: **With python3 it seems indeed to be preferred to open csv files in text mode** [see also this question](https://stackoverflow.com/questions/34283178/typeerror-a-bytes-like-object-is-required-not-str-in-python-and-csv) Maybe @John could add a corresponding notice to his answer? – nuala Oct 04 '22 at 20:17
5

I have encountered a similar problem with appending an already created csv file, while running on windows.

As in this case writing and appending in "binary" mode avoids adding extra line to each rows written or appended by using the python script. Therefore; 

w = csv.writer(open(Fn,'ab'),dialect='excel')
kenan bolat
  • 51
  • 1
  • 1