0

I am looking for the best optimized way we can use to transfer the large log files from local path to NFS path.

Here the log files will keep on changing dynamically with time.

What i am currently using is a java utility which will read the file from local path and will transfer it to NFS path. But this seems to be consuming high time.

We cant use copy commands, as the log file are getting appended with more new logs. So this will not work.

What i am looking for is .. Is there any way other than using a java utility which will transfer the details of log file from local path to NFS path.

Thanks in Advance !!

vijay joshi
  • 63
  • 7
  • Unclear what you're asking. If you're limited by network speed, you can try to use some fast compression/decompression. If not, what's problem with ordinary `cp`? – keltar Jun 04 '15 at 12:47
  • @keltar plz find the updated Query .. let me know if its still not clear. – vijay joshi Jun 04 '15 at 13:06

1 Answers1

0

If your network speed is higher than log growing speed, you can just cp src dst.

If log grows too fast and you can't push that much data, but you only want to take current snapshot, I see three options:

  1. Like you do now, read whole file into memory, as you do now, and then copy it to destination. With large log files it may result in very large memory footprint. Requires special utility or tmpfs.
  2. Make a local copy of file, then move this copy to destination. Quite obvious. Requires you to have enough free space and increases storage device pressure. If temporary file is in tmpfs, this is exactly the same as first method, but doesn't requires special tools (still needs memory and large enough tmpfs).
  3. Take current file size and copy only that amount of data, ignoring anything that will be appended during copying.

E.g.:

dd if=src.file of=/remote/dst.file bs=1 count=`stat -c '%s' src.file`

stat takes current file size, and then this dd is instructed to copy only that amount of bytes.

Due to low bs, for better performance you may want to combine it with another dd:

dd status=none bs=1 count=`stat -c '%s' src.file` | dd bs=1M of=/remote/dst.file
keltar
  • 17,711
  • 2
  • 37
  • 42