1

In my Python script I needed a fast/efficient way to set a max filesize on a file I'm constantly writing to. Rather than bring the whole thing into py's RAM, I ran this shell command:

sed -i '1d' file.csv

I monitor the filesize periodically and run the command as needed. Problem is that now if I tail -f file.csv, tail stops tailing the file as soon as sed removes a line from it. Any solution?

xendi
  • 414
  • 5
  • 10
  • 22

2 Answers2

1

The basic problem is sed creates a new file. This is shown by the extract below:

$ strace -fe trace=file,read,write,close sed -i '1d'  /tmp/x.csv 
....
openat(AT_FDCWD, "/tmp/x.csv", O_RDONLY) = 3
...
openat(AT_FDCWD, "/tmp/sed1nFxY1", O_RDWR|O_CREAT|O_EXCL, 0600) = 4
write(5, NULL, 0)                       = 0
read(3, "1,1,1,1\n2,2,2,2\n3,3,3,3\n", 4096) = 24
write(4, "2,2,2,2\n", 8)                = 8
write(4, "3,3,3,3\n", 8)                = 8
read(3, "", 4096)                       = 0
close(3)                                = 0
close(4)                                = 0
rename("/tmp/sed1nFxY1", "/tmp/x.csv")  = 0
close(1)                                = 0
close(2)                                = 0

There isn't, as far as I know, an easy way to truncate the start of a file.

I'd suggest getting your python application to listen on a unix/tcp socket, and once accepted put that stream into the logging framework and then use nc ... | tail to watch output.

danblack
  • 1,249
  • 13
  • 15
1

Michael Hampton suggests the obvious tail -F, but I personally mostly use less nowadays:

less --follow-name +F file.csv

There being no shorthand alternative to --follow-name, I have to use an alias:

alias less='less --follow-name -XFR'
simlev
  • 1,105
  • 3
  • 14
  • 22