0

This is a follow-up to my previous question - Curl Usage on OpenVMS. So I got everything working OK on my smallish 39Mbyte file. I then changed my command file to try and deal with the proper target file which is just over 8Gig. I have 9 curl commands of the type

$ pipe curl -range 0-100000000 -o part1.zip etc ... &
$ pipe curl -range 1000000001-2000000000 -o part2.zip ... &
$ pipe curl -range 2000000001-3000000000 -o part3.zip... &
$ pipe curl -range 3000000001-4000000000 -o part4.zip... &
$ pipe curl -range 4000000001-5000000000 -o part5.zip... &
$ pipe curl -range 5000000001-6000000000 -o part6.zip... &
$ pipe curl -range 6000000001-7000000000 -o part7.zip... &
$ pipe curl -range 7000000001-8000000000 -o part8.zip... &
$ pipe curl -range 8000000001- ... -o part9.zip &

Now I get the following error message on parts 4, 5 and 9 - always the same parts

> > PWD < 257 "/" is current directory.
> * Entry path is '/'
> > CWD Products < 250 Requested file action okay, completed.
> > CWD OwnershipDetailV2 < 250 Requested file action okay, completed.
> > PASV
> * Connect data stream passively < 227 Entering Passive Mode (204,8,135,60,43,24)
> *   Trying 204.8.135.60... connected
> * Connecting to 204.8.135.60 (204.8.135.60) port 11032
> > TYPE I < 200 Type set to I.
> > SIZE OwnershipDetailV2Full20160110.zip < 451 Requested action aborted: session in inconsistent state.
> * ftp server doesn't support SIZE
> * Instructs server to resume from offset -1294967295
> > REST -1294967295 < 554 Invalid REST argument.
> * Couldn't use REST
> ** Resuming transfer from byte position -1294967295   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
>                                  Dload  Upload   Total   Spent    Left  Speed   0     0    0     0    0     0      0      0 --:--:--  0:00:01
> --:--:--     0* Closing connection #0
> 
> %CURL-E-FTP_COULDNT_USE, FTP REST command failed

At first I was thinking it was some kind of 2Gig limit on the range parameters but parts 6 and 7 work OK. Any thoughts, ideas or work-arounds would be welcome.

Incidentally parts 1,2 3, 6 and 7 download just fine

user2699504
  • 195
  • 1
  • 4
  • 18

2 Answers2

2

It looks like a 32/64 bit signed/unsigned integer problem. 3000000001 is 0xb2d05e01 and as an unsigned value fits into 32 bits, interpreted as a signed 32 bit value it is -1294967295. The offsets you use don't fit into 32 bit integers, neither signed nor unsigned. From the output it is not obvious to me whether the error is in the client, or the server.

edit: I may have missed the curl version, I just noticed that for VMS curl version 7.46 was released a couple of days ago. However I have no info on a changelog. But it may be worth to try a new client.

user2116290
  • 1,062
  • 5
  • 6
  • @user2699504 To add to that: The value `4000000001` (part5.zip start) also overflows to a negative value. `5000000001` (part6.zip start) overflows to a positive value, as does `6000000001` (part7.zip start). `7000000001` (part8.zip start) and `8000000001` (part9.zip start) are negative. I'll go out on a limb and guess that values are being converted to 32-bit signed values without overflow checking and as long as the starting offset appears to be positive it does _something_. – HABO Jan 15 '16 at 22:11
  • We are currntly using version ... $ curl --version curl 7.19.5 (IA64-HP-VMS) libcurl/7.19.5 OpenSSL/0.9.7d Protocols: tftp ftp telnet dict http file https ftps Features: NTLM SSL I am in the process of updating to a newer version and will let you know if that solves my problem – user2699504 Jan 18 '16 at 09:28
0

Ok, for the benefit of anyone else who comes across this, the answer was to upgrade our curl to the latest version - which in our case this was ...

$ curl2 --version
curl 7.46.0 (IA64-HP-VMS) libcurl/7.46.0 OpenSSL/0.9.8z zlib/1.2.8
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smtp smtps telnet tftp

Features: IPv6 Largefile GSS-API Kerberos SPNEGO NTLM SSL libz

I was then able to set off 9 concurrent curls extracting about 1 G of data each, recombine them into one large ZIP and get the required text file out of the archive via UNZIP.

This took the extract down from about 3 hours to about 25 minutes

Many thanks to all who offered advice and help on this issue.

user2699504
  • 195
  • 1
  • 4
  • 18