1

I am trying to remove blank lines from large text files. For some reason it seems that neither

sed "/^$/d" file.txt > trimmed.txt

nor

grep -v "^$" file.txt > trimmed.txt

nor 

awk /./ file.txt > trimmed.txt

do anything. Any thoughts?

UPDATE

Thanks to the great comments by @fedorqui & @Sebastian Stigler the problem was quickly identified as DOS/Windows carriage returns (^M$) at the end of each line.

While I appreciate Sebatian's suggestion to reformat the files using dos2unix I would rather have a solution using the tools generally available in most linux distributions.

The solution that worked for me was an answer given by @Jeremy Stein to this question [Can't remove empty lines with sed regex:

sed -n '/[!-~]/p' file.txt > trimmed.txt
Community
  • 1
  • 1
Tomm
  • 2,142
  • 1
  • 15
  • 19
  • All the commands are fine. Could you indicate what is the output? – fedorqui Jan 29 '15 at 09:59
  • Well, neither of the commands removes the blank lines. However, file size seems to have nothing to do with it as a shortened version of the file in question (7kb) also has the blanks not removed. – Tomm Jan 29 '15 at 10:03
  • 2
    Could it be that the files have some weird characters? Check it with `cat -vet file`. Could be that they come from Windows... – fedorqui Jan 29 '15 at 10:07
  • You're right, that's exactly the problem. A Carriage Return ^M$ has snuck in. – Tomm Jan 29 '15 at 10:11
  • 2
    @fedorqui Thanks for the `-vet`; I see that `-A` is a new option which does all the three: show non-printing chars, line endings and tabs. – legends2k Jan 29 '15 at 10:15
  • @legends2k that `-A` sounds good and easier to remember, thanks for reporting :) – fedorqui Jan 29 '15 at 10:45

1 Answers1

4

I just tried the commandos with a toy example and they work fine as long the file.txt was a file with unix newlines. If the file contains windows newlines then none of the commands were able to remove the blank lines.

You can use the dos2unix linux tool to convert the newlines in file.txt to unix newlines. If you need the output on a windows system then you can use unix2dos to convert trimmed.txt into a file with windows newlines.

Sebastian Stigler
  • 6,819
  • 2
  • 29
  • 31
  • I found another solution that actually suits me more (see Jeremy Stein's solution [here](http://stackoverflow.com/questions/10755692/cant-remove-empty-lines-with-sed-regex) – Tomm Jan 29 '15 at 10:15