1

We are using scalatra with Jetty 9. We are have an endpoint which serves static files. But I've noticed that anything that is greater than 2GB seems to get truncated. I can see that the file gets sent chunked. Curl and wget can properly start the download and they finish without an error. I've looked for issues with scalatra and Jetty and haven't seen anything like the issue I'm seeing. But it looks like someone used an int when they needed a long

jamesatha
  • 7,280
  • 14
  • 37
  • 54
  • Some filesystems can't handle 2GB+ files. Is this the case for you too? Describe the client in detail that is having issues. Are you running Java 64-bit on the client? – Joakim Erdfelt Jun 09 '16 at 17:38
  • We are using ext4, so I'm confident we can handle 2GB+ files. And we are running java 64-bit for the server – jamesatha Jun 09 '16 at 17:45
  • The server isn't the issue, as you said curl and wget handled it fine. Focus on the client. – Joakim Erdfelt Jun 09 '16 at 17:48
  • wget and curl complete with no errors but the file is still truncated which makes me thing that something is wrong with the server – jamesatha Jun 09 '16 at 18:51
  • Are you using the jetty built-in static file serving? or something custom? (if custom, can you post the technique you are using?) – Joakim Erdfelt Jun 09 '16 at 19:23
  • You should file a bug about this at https://github.com/scalatra/scalatra/issues – Joakim Erdfelt Jun 10 '16 at 14:09

2 Answers2

2

For those of you that have stumbled across this question via a search result ...

This was handled at the Jetty issue tracker at https://github.com/eclipse/jetty.project/issues/630

The answer is that this is a scalatra bug, which is incorrectly making the assumption that .transferTo() has a guarantee to send all of the bytes handed to it.

The scalatra implementation needs to put its use of .transferTo() in a loop that verifies that the entire size has been sent or not.

To answer the reason why 2GB?, that's because the transferTo() use java's ByteBuffer, which itself has a 2GB limit.

Joakim Erdfelt
  • 46,896
  • 7
  • 86
  • 136
  • Yes, this ended up being an issue with scalatra. We have a fix for it that is out for review. You can track it's progress on the bug: https://github.com/scalatra/scalatra/issues/575 – jamesatha Jun 10 '16 at 18:48
1

This is fixed in scalatra 2.5.0+ http://scalatra.org/2016/11/21/2016-11-21-scalatra-2-5-released.html

jamesatha
  • 7,280
  • 14
  • 37
  • 54