2

I have two csv files that give different results when I use wc -l (gives 65 lines for the first, 66 for the second) and when I use vim file.csv and then :$ to go to the bottom of the file (66 lines for both). I have tried viewing newline characters in vim using :set list and they look identical.

I have created the second (which shows one extra line with wc) was created from the first using pandas in Python and to_csv.

Is there anything within pandas that might generate new lines or other bash/vim tools I can use to verify the differences?

p-robot
  • 4,652
  • 2
  • 29
  • 38
  • Have you tried `diff file.csv file2.csv`. If there is an apparent difference, it should tell you where. – Javier Elices Nov 09 '17 at 20:58
  • Thanks. The rows have been replaced/reordered so there are many differences. However, when I do a `diff` It does say that there's `No newline at end of file` but I can still sit my cursor on the "line 66". – p-robot Nov 09 '17 at 21:02
  • That may be a "feature" of your editor. If the contents of your file are equivalent, that is all I would ask for. – Javier Elices Nov 10 '17 at 09:04

1 Answers1

4

If the last character of the file is not a newline, wc won't count the last line:

$ printf 'a\nb\nc' | wc -l
2

In fact, that's how wc -l is documented to work: from man wc

  -l, --lines
          print the newline counts
                    ^^^^^^^^^^^^^
glenn jackman
  • 238,783
  • 38
  • 220
  • 352
  • Many thanks for the response. I'm still stumped. The vim line counter clearly states the 66th line. – p-robot Nov 09 '17 at 20:55
  • It seems pandas is adding a new line at the end of the file. Related question [here](https://stackoverflow.com/questions/39237755/how-to-stop-writing-a-blank-line-at-the-end-of-csv-file-pandas) for those in the future interested in work-arounds. Thanks all. – p-robot Nov 09 '17 at 21:05