1

I am trying to upload file to a file hosting service with cURL command line utility. At first I uploaded a file with the Chromium browser, with an opened web developer console: at the network tab I looked for the appropriate line, and I clicked on "copy all as curl". I imitated the same request - after proper login and attaching the saved cookie file - with cURL, but when I download the uploaded file, the file content unfortunately always start with a few lines of the HTTP headers, which I send explicitely (set with -H, like Content-Type), or added implicitly by cURL (like the boundary).

An example of the beginning of the file contents:

--------------------------1dbea6717e57a1ab
Content-Disposition: attachment; name="files[]"; filename="data.bin"
Content-Type: application/octet-stream

<...binary file data then...>

Where line endings are CR/LF (0D0A in the hexadecimal viewer), and double CR/LF 0D0A0D0A ath the very and of this unneeded header.

What can cause this strange behaviour? It seems that the server side program can't separate the binary file data from the header lines. Maybe should I manually set the boundary or using only LF (line-feed, 0A) as the line endings of the header? I don't find such a cURL option which can set that character which separates the header lines (CR/LF <-> LF).

The cURL command I used for uploading:

curl 'https://example.com/upload' -H 'Origin: https://example.com' -H 'Accept-Encoding: gzip, deflate, br' -H 'Accept-Language: en-US,en;q=0.9,hu;q=0.8' -H 'User-Agent: Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36' -H 'Content-Type: application/octet-stream' -H 'Accept: */*' -H 'Referer: https://example.com/' -b cookie.txt -H 'Connection: keep-alive' -F "files[]=@data.bin" --compressed -L

If I use the web uploader of the service - some JS library - files are OK of course.

If I use this command, then it basically works, the problem is, that the filesize will be zero on the server. However when I download the file, it is intact, md5sum passes. This also doesn't happen when I use the web uploader, in that case filesize is OK on the server, again.

curl 'https://example.com/upload' -H 'Origin: https://example.com' -H 'Accept-Encoding: gzip, deflate, br' -H 'Accept-Language: en-US,en;q=0.9,hu;q=0.8' -H 'User-Agent: Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36' -H 'Accept: */*' -H 'Referer: https://example.com/'  -H 'Connection: keep-alive' -b cookie.txt -F "files[]=@data.bin;type=application/octet-stream" --compressed -L
Konstantin
  • 2,983
  • 3
  • 33
  • 55
  • 1
    When you ask such questions, please include the command you used. – root Sep 02 '19 at 06:34
  • 1
    my best guess: your server does not support the `Multipart/form-data`-format, and doesn't parse the file upload correctly. does the documentation for `https://example.com/upload` state that you are supposed to upload files in the `multipart/form-data` format? – hanshenrik Sep 02 '19 at 21:30
  • 1
    don't set the `Accept-Encoding` header yourself. if your curl installation was not compiled with `br` support, and the server actually decides to use `br` for the transfer, you will just get unreadable binary data back. just use the `--compressed` argument instead, and curl will craft the Accept-Header header automatically, with all encodings that your curl installation was built to support. – hanshenrik Sep 02 '19 at 21:38

1 Answers1

2

It seems that the server side program can't separate the binary file data from the header lines

yeah, that's most likely it, the server side program does not understand the multipart/form-data-format, and thinks the multipart headers are part of the actual file. that suggests the server expect you to upload the file raw in the request body. to get curl to upload the file raw in the request body, use --data-binary @filename instead.

the documentation for https://example.com/upload should state how the file is supposed to be uploaded tho.

hanshenrik
  • 19,904
  • 4
  • 43
  • 89
  • There is no documentation, this a file sharing service. They have a web uploader, and it works flawlessly, I try to mimic that. – Konstantin Sep 02 '19 at 22:01