10

How do you create a unix file format in Powershell? I am using the following to create a file, but it always creates it in the windows format.

"hello world" | out-file -filepath test.txt -append

As I understand, the new line characters CRLF make it to be a Windows format file whereas the unix format needs only a LF at the end of the line. I tried replacing the CRLF with the following, but it didn't work

"hello world" | %{ $_.Replace("`r`n","`n") } | out-file -filepath test.txt -append
aldrin
  • 4,482
  • 1
  • 33
  • 50
  • 2
    Your test doesn't work because there is no CRLF in "hello world" – x0n Feb 24 '11 at 21:50
  • 1
    @x0n when you write it out to a file, then the cr/lf is appended to the line – aldrin Feb 25 '11 at 08:34
  • 1
    that's the point I was trying to make. clearly your replace operation is _before_ the write (out-file) so if out-file adds the CRLF, it happens after your attempt to replace it. – x0n Feb 25 '11 at 16:58
  • I know this is not answer to your question, but you can check if it is affordable to call dos2unix on the target. – Barny Mar 29 '21 at 14:58

5 Answers5

8

There is a Cmdlet in the PowerShell Community Extensions called ConvertTo-UnixLineEnding

Andy Schneider
  • 8,516
  • 6
  • 36
  • 52
6

One ugly-looking answer is (taking input from dos.txt outputting to unix.txt):

[string]::Join( "`n", (gc dos.txt)) | sc unix.txt

but I would really like to be able to make Set-Content do this by itself and this solution does not stream and therefore does not work well on large files...

And this solution will end the file with a DOS line ending as well... so it is not 100%

Anders Zommarin
  • 7,094
  • 2
  • 25
  • 24
  • 5
    Extending your solution, there is in fact a way to write the bytes and avoids the DOS line ending: sc unix.txt ([byte[]][char[]] "$contenttext") -Encoding Byte – aldrin Feb 24 '11 at 11:26
  • Good thinking, thanx for that! – Anders Zommarin Feb 24 '11 at 11:34
  • 1
    @x0n this also ends the file with a Windows-style line ending (\r\n) – Cocowalla May 31 '14 at 20:12
  • 1
    Indeed, this invariably creates a trailing CRLF on Windows. With the benefit of hindsight: in PowerShell v5+, you can use `Set-Content -NoNewLine` to avoid this problem. – mklement0 Feb 08 '23 at 02:40
4

I've found that solution:

sc unix.txt ([byte[]][char[]] "$contenttext") -Encoding Byte

posted above, fails on encoding convertions in some cases.

So, here is yet another solution (a bit more verbose, but it works directly with bytes):

function ConvertTo-LinuxLineEndings($path) {
    $oldBytes = [io.file]::ReadAllBytes($path)
    if (!$oldBytes.Length) {
        return;
    }
    [byte[]]$newBytes = @()
    [byte[]]::Resize([ref]$newBytes, $oldBytes.Length)
    $newLength = 0
    for ($i = 0; $i -lt $oldBytes.Length - 1; $i++) {
        if (($oldBytes[$i] -eq [byte][char]"`r") -and ($oldBytes[$i + 1] -eq [byte][char]"`n")) {
            continue;
        }
        $newBytes[$newLength++] = $oldBytes[$i]
    }
    $newBytes[$newLength++] = $oldBytes[$oldBytes.Length - 1]
    [byte[]]::Resize([ref]$newBytes, $newLength)
    [io.file]::WriteAllBytes($path, $newBytes)
}
evg656e
  • 644
  • 7
  • 10
2

Two more examples on how you can replace CRLF by LF:

  1. Example:

    (Get-Content -Raw test.txt) -replace "`r`n","`n" | Set-Content test.txt -NoNewline

  2. Example:

    [IO.File]::WriteAllText('C:\test.txt', ([IO.File]::ReadAllText('C:\test.txt') -replace "`r`n","`n"))

Be aware, this does really just replace CRLF by LF. You might need to add a trailing LF if your Windows file does not contain a trailing CRLF.

stackprotector
  • 10,498
  • 4
  • 35
  • 64
1

make your file in the Windows CRLF format. then convert all lines to Unix format in new file:

$streamWriter = New-Object System.IO.StreamWriter("\\wsl.localhost\Ubuntu\home\user1\.bashrc2")
$streamWriter.NewLine = "`n"
gc "\\wsl.localhost\Ubuntu\home\user1\.bashrc" | % {$streamWriter.WriteLine($_)}
$streamWriter.Flush()
$streamWriter.Close()

not a one-liner, but works for all lines, including EOF. new file now shows as Unix format in Notepad on Win11.

delete original file & rename new file to original, if you like:

ri "\\wsl.localhost\Ubuntu\home\user1\.bashrc" -Force
rni "\\wsl.localhost\Ubuntu\home\user1\.bashrc2" "\\wsl.localhost\Ubuntu\home\user1\.bashrc"
RoelDS
  • 997
  • 7
  • 10
  • 1
    Nice - good to know that the newline sequence can be controlled via the [`System.IO.StreamWriter`](https://learn.microsoft.com/en-US/dotnet/api/System.IO.StreamWriter) type's `.NewLine` property. As an inconsequential aside: you don't need to call `.Flush()` before `.Close()`. – mklement0 Feb 08 '23 at 02:45
  • Flush was need for me, in Windows PowerShell 5.1 on Win11 – RoelDS Feb 08 '23 at 03:17
  • 1
    As for flushing: at least the .NET (Core) [source code](https://github.com/dotnet/runtime/blob/8f3a34da468f970e268bd3b17247d77732991b00/src/libraries/System.Private.CoreLib/src/System/IO/StreamWriter.cs#L184-L209) shows that flushing is _automatically_ performed on `.Close()`. If that weren't the case, it would amount to a serious bug/design flaw. Even in W11 / WinPS v5.1 I don't see a problem with _not_ calling `.Flush()`, so if there is a problem, after all, it would be good to know what specific OS / PowerShell versions are affected. – mklement0 Feb 08 '23 at 04:06