1

We use curl on our openvms system to successfully download many files - no problems. We have a particularly big ZIP file that I wanted to try and multi-part download in parallel using the --range flag of curl to get different parts of the file that we can then append to one large ZIP.

So as a test I tried it out on a smaller file of approx 50 Mbtyes. Using this sequence of commands it worked perfectly: Note that the curl commands will normally be run in parallel, not just one after the other as shown

$ curl --range 0-5000000 bigfile.zip -o part1.zip 
$ curl --range 5000001-50000000 bigfile.zip -o part2.zip

When the above two commands complete I do

$ copy part1.zip,part2.zip final.zip

and the following unzip works as expected

$ unzip -ao final.zip

Ok, so I thought I would try and split it 3 ways now e.g

$ curl --range 0-5000000 bigfile.zip -o part1.zip  
$ curl --range 5000001-30000000 bigfile.zip -o part2.zip 
$ curl --range 30000001-50000000 bigfile.zip -o part3.zip

Three zips are produced as expected , but this time after 

$ copy part1.zip,part2.zip,part3.zip final.zip

on the unzip I get ....

$ unzip -ao final.zip
Archive: final.zip;1
**warning final.zip;1:  1 extra byte at beginning or within zipfile**
  (attempting to process anyway)
file #1:  bad zipfile offset (local header sig):  1  
(attempting to re-compensate)
  inflating: CompanyRel.txt          [text]
  error:  invalid compressed data to inflate
[ WriteRecord: sys$put failed ]
[ %RMS-F-RSZ, invalid record size ]
[ %NONAME-W-NOMSG, Message number 00000000 ]

Any suggestions as to how to fix would be welcome

user2699504
  • 195
  • 1
  • 4
  • 18
  • (welcome to Stackoverflow). In the future, use the `{}` tool at the top left of the edit tool on highlighted text to format correctly as code/data/error msgs/etc. Good luck. – shellter Jan 14 '16 at 16:12
  • So, this is on openvms using DCL. Copy is concatenating the zips into one final zip ( like the Unix CAT command) – user2699504 Jan 14 '16 at 16:15
  • Ah, DCL, haven't seen that in a looongg time ;-) . I just wanted to confirm `,` was legitimate part of cmd. I don't know what utils are available in openvms that could help you debug this problem. Noting that there are only 57 followers for tag openvms, maybe you should look for other user support forums on the net? Good luck! – shellter Jan 14 '16 at 16:17

1 Answers1

1

Your part*.zip files are very likely of record format Stream_LF, which you can check with a DIR/FULL command. For whatever reason, the VMS copy command appends a line feed (the above LF) to the contents of each file after the first comma (or plus) in the source file list. You can check that for your successfully concatenated zip file out of two parts. The new file is one byte longer and that is the LF at the end.

That additional LF at the end doesn't disturb unzip. But with three files in the list there are two LFs, one after the second part and one at the end. And the one "in the middle" is the one unzip complains about: file offsets are wrong, etc.

After copying the files with curl, try to set a different record format: udf. Something like 

$ set file/attribute=(rfm=udf) part%.zip

then do the concatenation with copy. This should prevent the copy command from appending the LF and should make unzip happy.

user2116290
  • 1,062
  • 5
  • 6
  • For a belt and braces approach I recommend using the -b option of UNZIP also – user2699504 Jan 15 '16 at 09:22
  • The -b is an output option for unzip, which on VMS overwrites the default record format Stream_LF (or the record format which was saved with -V) with the fixed length format. For the original problem, you would need such a -b for curl. But as far as I can see, using copy to concatenate fixed length files (where the last record is only partially filled) seems to have problems as well. -- Regarding unzip, if the original zip file was created on VMS, the best approach is to create it with -V. Otherwise the default output format Stream_LF is usually OK, even for binary files. – user2116290 Jan 15 '16 at 10:12
  • user2116290 I have a follow-up question, hope you can help me out with it too. I will title it Another curl usage on openvms question – user2699504 Jan 15 '16 at 14:25