0

I have a file with CRLF line endings, which I need to convert to LF. This sed command can do that: sed -E -i "s/\r\n/\n/" file.txt.

I have sed installed in my Windows 11 using msys2. So, if I execute the above sed command in Powershell, it executes successfully and converts CRLF to LF.

But instead, if I execute the above sed command inside bash in Powershell, it completes with no output to the console. But it fails to convert CRLF to LF.

Instead if I run a sed command to word character replacement, it executes correctly both in Powershell and Powershell bash. So I think this has something to do with how CRLF is handles by bash.

So now, how to convert CRLF to LF in Powershell bash using sed?

Sourav Kannantha B
  • 2,860
  • 1
  • 11
  • 35

1 Answers1

3

Use the following command in your WSL bash session in order to perform in-place conversion of a file from \r\n (CRLF, Windows-format) to \n (LF, Unix-format) newlines (line endings):[1]

sed -i 's/\r$//' file.txt

Alternatively, if you don't mind reading the file as a whole, using GNU sed's -z option and the g regex option for global matching (finding all matches):

sed -i -z 's/\r\n/\n/g' file.txt

Your problem was unrelated to PowerShell; instead, there was a problem with the logic of your sed command:

  • sed reads input line by line by default, and removes the trailing newline from each.

    • Therefore, no line that a sed script operates on contains \n by default, so a substitution (s command) trying to match \r\n is invariably a no-op.

  • Because sed considers only \n (LF) alone a newline:

    • Those lines originally terminated with \r\n will end in \r (CR) when a sed operation is performed on them.

  • s/\r$// therefore removes that trailing \r from each line, and - on writing the modified lines back to the file - terminates each line with just \n, so that in effect only \n (LF) newlines remain.

Note:

  • The only reason why sed -E -i "s/\r\n/\n/" file.txt worked for you is that the sed implementation that comes with MSYS has special behavior for Windows:

    • On reading files, it recognizes both \n (LF) and \r\n (CRLF) as newlines.

    • On writing files, it only uses \n (LF)

    • As such, it converts CRLF sequences to LF by default, and the following command would suffice, i.e. providing an empty script:

      • sed -i ' ' file.txt

        • Note: sed -i '' file.xt should work, i.e. passing an empty string as the script argument, but doesn't up to PowerShell 7.2.x, due to a long-standing bug that accidentally removes empty-string arguments when calling external programs - see GitHub issue #6280 for details.
          The underlying cause of this bug more prominently manifests in the inability to pass arguments with embedded " chars. to external programs - see this answer.

    • For the reasons explained above, s/\r\n/\n/ actually has no effect - in any sed implementation. In other words: it was simply the default behavior of the MSYS sed implementation that happened to perform the desired conversion for you.

[1] To perform the opposite conversion - from LF-only to CRLF newlines, use sed -i 's/$/\r/' file.txt, as jdweng notes.

mklement0
  • 382,024
  • 64
  • 607
  • 775