1

I need to load big text files from another machine through a 10Gb link, files been creating by external/closed-source software that i don't have access to change (for example to make it to compress output files)

Currently network and disk IO usage is 100% on these machines (source and destination machines) so bottleneck of the system.

If the source machine could have compress text files in the first place there would be no problem at all, but i don't have access to source machine (files being rsync-ed to destination machine which i have access)

Is there anything at the destination machine that i can do to at least reduce the pressure on destination machine's disk.

I came up with a kind of ridiculous idea which is: to have a temp memory mapped drive with a few GBs, mount it as input directory of the rsync from source machine, then write a program to compress couple of text files each time, write the compressed output file (10X smaller size in compare to original files) to HDD and delete original files from memory mapped drive.

Is there a tool already doing this?! Any other recommendations?

I'm using ubuntu 18.04.

Best Regards

f.ald
  • 111
  • 1
  • Product recommendations are off topic, but [fusecompress looks promising](https://www.systutorials.com/docs/linux/man/1-fusecompress/). – Gerald Schneider Jan 18 '22 at 13:22
  • fusecompress looke like a abandoned project (not updated for a long time) and i couldn't find a way to install it on linux – f.ald Jan 18 '22 at 14:45

1 Answers1

0

Is there anything at the destination machine that i can do to at least reduce the pressure on destination machine's disk.

When disk IO is the bottleneck the obvious solution is of course "get faster disks".

Alternatively when you have some CPU capacity to spare: several Linux filesystems support "transparent" compression where the file system driver performs on-the-fly compression while writing the data to the underlying block storage and decompression when the data is read again. Then you don't have to change anything at the application level.

Bob
  • 5,805
  • 7
  • 25
  • If we mount a ZFS drive over NFS and access files remotely, compressed/small files would be passed through network and decompressed on remote machine or vice versa? – f.ald Jan 18 '22 at 14:47
  • you csn use tar and xz over nc or ssh – djdomi Jan 18 '22 at 16:53