How to determine the line ending of a file

Question

I have a bunch (hundreds) of files that are supposed to have Unix line endings. I strongly suspect that some of them have Windows line endings, and I want to programmatically figure out which ones do.

I know I can just run

flip -u

or something similar in a script to convert everything, but I want to be able to identify those files that need changing first.

score 72 · Answer 1 · answered Sep 23 '08 at 14:40

72

You can use the file tool, which will tell you the type of line ending. Or, you could just use dos2unix -U which will convert everything to Unix line endings, regardless of what it started with.

answered Sep 23 '08 at 14:40

Adam Rosenfield

390,455
97
512
589

5

file doesn't show line ending. Ex. : "file .bashrc => .bashrc: ASCII English text" Need some extra keys ? – Fedir RYKHTIK Feb 09 '12 at 10:32
10

@Fedir: Yes, it does, it's just that if the file has regular LF line endings, then it won't print any output. But if the file has CRLF, bare CR, or mixed line endings, it will tell you that. – Adam Rosenfield Feb 09 '12 at 21:55
2

Didn't work for me on a CRLF-only Perl script on OS X. Might be a GNU extension? – Tim Yates Jun 11 '12 at 20:34
3

This works on some file types but not others. On Linux, it doesn't report the line endings for html files for example. – Edward Anderson Apr 09 '13 at 04:48
"file foo.txt" worked fine on OS X 10.9. It printed "foo.txt: ASCII text, with CRLF line terminators" – Bogdan Calmac Mar 14 '14 at 17:19
Almost 10 years later, it's certainly worth mentioning that `dos2unix` features the `-i` option to get the information on the file. I have found this to be the most reliable way to identify the culprits. This is also described in https://unix.stackexchange.com/a/231408/17800 – ikaerom Nov 04 '17 at 23:16

score 29 · Accepted Answer · edited Apr 09 '13 at 04:47

29

You could use grep

egrep -l $'\r'\$ *

edited Apr 09 '13 at 04:47

Edward Anderson

13,591
4
52
48

answered Sep 23 '08 at 14:42

stimms

42,945
30
96
149

3

for some reason, when I run this command in a MacOS X shell, I get a list of all files in the directory. Even one that I newly generate with "echo "test" >torderform6.cpp". Any idea what might be going wrong? – Adrian Grigore Feb 25 '09 at 18:33
7

It just lists all files in the folder for me on Ubuntu as well. – rjmunro May 09 '11 at 11:52
2

This command will still list files that have had dos2unix run on them. – Phyxx Jan 17 '12 at 02:16
[Here's an explanation of what's happening with $'](http://unix.stackexchange.com/questions/48106/what-does-it-mean-to-have-a-dollarsign-prefixed-string-in-a-script) – Bryce Guinta Jun 15 '16 at 21:16
use `$(echo -e '\r')` If you can't have bash do ASCII expansion for you (replace `$'\r'\$` with it) – Bryce Guinta Jun 15 '16 at 21:39
1

On mac I use: grep -E -rl '\r' . – LanDenLabs Aug 18 '16 at 13:52

score 14 · Answer 3 · edited Apr 01 '14 at 04:29

14

Something along the lines of:

perl -p -e 's[\r\n][WIN\n]; s[(?<!WIN)\n][UNIX\n]; s[\r][MAC\n];' FILENAME

though some of that regexp may need refining and tidying up.

That'll output your file with WIN, MAC, or UNIX at the end of each line. Good if your file is somehow a dreadful mess (or a diff) and has mixed endings.

edited Apr 01 '14 at 04:29

Cheran Shunmugavel

8,319
1
33
40

answered Jan 14 '10 at 15:24

joachim

28,554
13
41
44

Worked for me on Ubuntu, the accepted answer seems to just list all files – Noel Kennedy Jul 01 '11 at 12:14
Doesn't work for me, gives: `Unmatched ) in regex; marked by <-- HERE in m/(?<!WIN) <-- HERE \n/ at -e line 1.` – moshen May 13 '13 at 19:24
you need to replace the < with `<` – Joseph Jan 15 '14 at 11:14
The `<` symbol was messed up in a previous edit. I've fixed it now. – Cheran Shunmugavel Apr 01 '14 at 04:29

score 5 · Answer 4 · edited Oct 07 '16 at 07:48

5

Here's the most failsafe answer. Stimms answer doesn account for subdirectories and binary files

find . -type f -exec file {} \; | grep "CRLF" | awk -F ':' '{ print $1 }'

Use file to find file type. Those with CRLF have windows return characters. The output of file is delimited by a :, and the first field is the path of the file.

edited Oct 07 '16 at 07:48

Ulf Lindback

13,974
3
40
31

answered Jun 15 '16 at 21:41

Bryce Guinta

3,456
1
35
36

Indeed the most failsafe way. To convert only all found files just run `find . -type f -exec file {} \; | grep "CRLF" | awk -F ':' '{ print $1 }' | xargs flip -ub` afterwards. – pixelbrackets Jan 24 '17 at 09:36
3

Most failsafe it is not -- `file` does not always even tell "CRLF" in its output, that depends on what kind of file it is. I have discovered that for SVG files -- containing text much like plaintext files -- `file` does not mention the kind of line ending used. This script thus is not file type agnostic. Just saying. Otherwise looks like a sane one-liner, aforementioned limitation non-withstanding. – Armen Michaeli Apr 16 '17 at 12:51

score 2 · Answer 5 · edited Dec 08 '16 at 08:55

2

Unix uses one byte, 0x0A (LineFeed), while windows uses two bytes, 0x0D 0x0A (Carriage Return, Line feed).

If you never see a 0x0D, then it's very likely Unix. If you see 0x0D 0x0A pairs then it's very likely MSDOS.

edited Dec 08 '16 at 08:55

Hugo

27,885
8
82
98

answered Sep 23 '08 at 14:42

Adam Davis

91,931
60
264
330

score 0 · Answer 6 · answered Sep 23 '08 at 14:36

0

Windows use char 13 & 10 for line ending, unix only one of them ( i don't rememeber which one ). So you can replace char 13 & 10 for char 13 or 10 ( the one, which use unix ).

answered Sep 23 '08 at 14:36

TcKs

25,849
11
66
104

score 0 · Answer 7 · answered May 09 '15 at 09:00

When you know which files has Windows line endings (0x0D 0x0A or \r \n), what you will do with that files? I supose, you will convert them into Unix line ends (0x0A or \n). You can convert file with Windows line endings into Unix line endings with sed utility, just use command:

$> sed -i 's/\r//' my_file_with_win_line_endings.txt

You can put it into script like this:

#!/bin/bash

function travers()
{
    for file in $(ls); do
        if [ -f "${file}" ]; then
            sed -i 's/\r//' "${file}"
        elif [ -d "${file}" ]; then
            cd "${file}"
            travers
            cd ..
        fi
    done
}

travers

If you run it from your root dir with files, at end you will be sure all files are with Unix line endings.

How to determine the line ending of a file

7 Answers7

Linked