1

We have several files larger than 2GB tared using tar command UNIX version (SunOS 5.5 with last change: 1995!). There is "E" flag to tar files > 2GB. This is how it was been done: 

rsh sun_server tar cvEf - abc/file1 abc/file2 | rsh archive_server archive_cmd

Notice that archive_server is also a UNIX server and uses the same version of tar command.

In retrieving, we do so:

cd /destination_path/
rsh archive_server retrieve_cmd | tar xvf -

What happened is the files are extracted to /abc/ (in parent directory); not in /destination_path/abc/

The strange thing is we changed the retrieving method to gtar:

cd /destination_path/
rsh archive_server retrieve_cmd | gtar xvf -

For files > 2GB, it gives many errors of "bad header" and "memory exhausted" error at the end. However, for files that are less than 2GB, they are extracted normally to /destination_path/abc/ !

We need to extract the >2GB files to /destination_path/abc/ because there are many files under /abc/ have the same names and we afraid that the extracted files would overwrite them..

I also tried

rsh archive_server retrieve_cmd | tar xvf - /destination_path/

and 

cd /destination_path/
rsh archive_server retrieve_cmd | tar xvf - .

But the result is: no result! It shows that the extraction is completed successfully but no files extracted neither in /abc/ nor in /destination_path/ !

Is there a way to extract these files to the desired destination path..?

4 Answers4

2

rsh is faster than ssh, but I prefer ssh. this should work substituting ssh with rsh.

gtar -cf - files | ssh user@host "(cd /destPath ; gtar -xf -)"

The key here is enclosing in parenthesis.

  • 1
    Just to expand: The enclosing parenthesis make the 2 commands run in a sub-shell, so you change dir and extract, regardless of your ssh CWD – katriel Aug 02 '09 at 19:55
  • Didn't work either... again the extracted files are not in the destination directory.. –  Aug 03 '09 at 07:15
0

A thought, you could use the command split to break up the tar archive into several pieces....For example, consider something like: 

tar cf - <dirspec> | split --bytes=1GB  <destdir>/archive.

The destination could be an NFS mounted filesystem. Or you could then send them from the local source with many remote copy commands running concurrently if you wish afterwards.

Then you could untar the thing at any time on the destination location using:

cd <destdir>
cat archive.* | tar xf -

On a network, I'd have a little concern with tranmission of a continuous stream of bytes this long depending on the distance. Smaller segment transfers is probably preferable anyway to more easily recover from network glitches.

To make things smaller you could add gzip/gunzip into the mix depending on how much cpu time you can afford.

mdpc
  • 11,856
  • 28
  • 53
  • 67
0

You need to use GNU's tar (gtar) on both sides. Solaris' tar will silently screw up files >2GB. Additionally, it will silently screw up files with long file names.

brianegge
  • 1,064
  • 2
  • 14
  • 23
0

several of the above answers would work, however, check out pax. it was supposed to be a replacement for tar and cpio and may handle the job more elegantly. most people have not adopted the tool because of the wide spread popularity of the tool, but its actually pretty nice.

cptfrl