1

I have a log file in a custom binary format sitting on a remote machine. I want to sync it with a file on my local machine, so when new bytes are appended to the remote binary file, the file on my local machine will be updated as well.

I learned about how to use tail over ssh here: https://serverfault.com/questions/608723/tail-a-file-from-ssh-and-mirror-to-a-local-file

Then I learned about how to use tail on binary files here: Binary "tail" a file

I tried combining them into

ssh -t remotebox "tail -c +1 -f /path/to/file.bin" > ./mirror.bin

But then I realized that mirror.bin is corrupted. Looking at the hex dump, I see that all 0d0a bytes are truncated into 0as (\r\n got replaced by just \n). However, if I run tail locally (tail -c +1 -f file1.bin > file2.bin), this truncation does not happen.

I also tried to use tee instead of redirection; the problem persists.

Are there shell tricks I can do with tail to prevent this from happening, or are there other programs that suites my needs?

Thanks.

P.S. Both remote and local machines are Linux, running bash.

nerrons
  • 130
  • 8
  • Are there two different systems with different line ending involved (e.g. Linux and Windows)? – Thomas Sablik Mar 02 '21 at 09:01
  • @ThomasSablik No, both are Linux. I've added that to the description. – nerrons Mar 02 '21 at 09:02
  • 1
    You should ask this question on Serverfault or Unix/Linux. It's unrelated to programming. – Thomas Sablik Mar 02 '21 at 09:03
  • I think the problem is that `tail` and `tee` are focused on text. Therefore the line endings are converted. You could check the documentation if there is a binary option. – Thomas Sablik Mar 02 '21 at 09:07
  • @ThomasSablik Thanks for the suggestion, I've posted the question on Unix/Linux. The thing is tail on the local machine doesn't have this problem, so I suspect something in SSH is to blame. – nerrons Mar 02 '21 at 09:10
  • Please post `tail --version` on both machines... but I strongly do not believe it's `tail` fault. This is very strange. Would you be able to setup a [MCVE]? Create a test files with `\r\n` ndings, `scp` to remote machine, `ssh+tail` it back to local and `diff`? I just did `echo -e '123\r\n234\r\n' > /tmp/1; scp /tmp/1 server:/tmp/1 ; timeout 1 ssh server tail -c +1 -f /tmp/1 > /tmp/2 ; diff /tmp/1 /tmp/2` and got back the same file. Och `ssh -t` is at fault here! – KamilCuk Mar 02 '21 at 09:11
  • Seems like when I turn off `-t` everything is fine... I will play with it a bit more to verify – nerrons Mar 02 '21 at 09:17

1 Answers1

1

The following reproduces the problem:

$ echo -e '123\r\n234\r\n' > /tmp/1
$ scp /tmp/1 server:/tmp/1
$ ssh -t server cat /tmp/1 > /tmp/2
Shared connection to closed.
$ diff /tmp/1 /tmp/2
1,3c1,3
< 123
< 234
< 
---
> 123
> 234
>

It's possible that ssh -t changes \r\n to \n because it might do stty +igncr on the line. Mine ssh -t does the opposite (I think stty +inlcr) and I have the character doubled:

$ ssh -t server cat /tmp/1 | hexdump -C
Shared connection to  closed.
00000000  31 32 33 0d 0d 0a 32 33  34 0d 0d 0a 0d 0a        |123...234.....|
0000000e
$ ssh server cat /tmp/1 | hexdump -C
00000000  31 32 33 0d 0a 32 33 34  0d 0a 0a                 |123..234...|
0000000b

Anyway. remove the -t option if you do not have the intention to use terminal features.

And your command line suggest that you are copying a file. I have no idea why do you use tail for that - tail it's for outputting last part of file, not for printing the whole file... To print the whole file use cat. To copy a file, use scp.

KamilCuk
  • 120,984
  • 8
  • 59
  • 111
  • 1
    The reason I'm using tail is that the remote file is constantly being appended and I want to get the new content as soon as possible on my local machine. Since it's a log file, tail seems to perform a bit better than rerunning rsync. – nerrons Mar 02 '21 at 09:24