3

I want to compress response body in javax.servlet.Filter. Here is my code

byte[] bytes =  // compressing response body
response.addHeader("Content-Encoding", "gzip");
response.addHeader("Content-Length", String.valueOf(bytes.length));
response.setContentLength(bytes.length);
response.setBufferSize(bytes.length * 2);
ServletOutputStream output = response.getOutputStream();
output.write(bytes);
output.flush();
output.close();

But actual response I see in Chrome Dev Tool is 

Accept-Ranges: bytes
Cache-Control: max-age=2592000
Content-Type: application/javascript;charset=UTF-8
Date: Fri, 14 Dec 2018 15:34:25 GMT
Last-Modified: Tue, 09 Oct 2018 13:42:54 GMT
Server: Apache-Coyote/1.1
Transfer-Encoding: chunked

I do not expect Transfer-Encoding: chunked, because I declare "Content-Length". I wrote a simple test on java

URLConnection connection = new URL("http://127.0.0.1:8081/js/ads.js").openConnection();
connection.addRequestProperty("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8");
connection.addRequestProperty("Accept-Encoding", "gzip, deflate");
connection.addRequestProperty("Accept-Language", "ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7");
connection.addRequestProperty("Cache-Control", "no-cache");
connection.addRequestProperty("Connection", "keep-alive");
connection.addRequestProperty("Host", "127.0.0.1:8081");
connection.addRequestProperty("Pragma", "no-cache");
connection.addRequestProperty("Upgrade-Insecure-Requests", "1");
connection.addRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36"); 
connection.connect();
connection.getHeaderFields().forEach((s, strings) ->
        System.out.println(s + ":" + String.join(",", strings)));

and here is what I found:

  • if I comment setting "User-Agent" header or change "User-Agent" to any other value then I get response with "Content-Length"
  • if "User-Agent" points on Chrome then I get Transfer-Encoding: chunked.

I debugged up to sun.nio.ch.SocketChannel#write method and it gets correct ByteBuffers with Content-Length header values.

I cant undestand where is this magic transormation to chunked happening?

Update

The strange thing is that I write gziped bytes into Socket (I'm sure as I debugged up to call of native method write in SocketChannel implementation). But URLConnection returns my unzipped byte array with Chrome's User-Agent and correct gziped byte array if I dont specify User-Agent header or put some random string. SO seems like magic happening somewhere in Windows socket implementation.

user207421
  • 305,947
  • 44
  • 307
  • 483
Арсен Мирзаян
  • 167
  • 10
  • Why are you setting the buffer size to be double the size of your file? Also, try removing the explicit call to `flush` on the output stream. – Christopher Schultz Dec 17 '18 at 15:40
  • I did not know if buffer used only for response body or it contains headers as well. So I set buffer size that is enough for both. – Арсен Мирзаян Dec 17 '18 at 22:12
  • removing flush does not helps. My debug shows that call of write method invokes writing to SocketChannel. – Арсен Мирзаян Dec 17 '18 at 22:14
  • Hmm. Usually chunked-encoding only happens when you try to do "extra" stuff like calling `flush()` or `close()` on the `OutputStream`. If you set the `Content-Length` and then dump the file yourself to the output stream, Tomcat really should respect your code. Tomcat version? Any reverse-proxy between your client and Tomcat? – Christopher Schultz Dec 17 '18 at 22:16
  • The buffer size is for the response in general. There is usually no reason to set the response buffer size unless you are having some odd performance problem. Changing the response buffer size ends up permanently changing the response buffer for that (re-used) response object, so if you have a few large files being returned, you might end up wasting large amounts of heap with buffers that *stay large*. – Christopher Schultz Dec 17 '18 at 22:19
  • I'm debbuging this code at the moment and made some updates to ticket. Looks like magic happening somewhere in Windows socket implementation. – Арсен Мирзаян Dec 17 '18 at 22:22
  • I'm testing on Tomcat 7 and Jetty 9 with Intellig Idea integration. And send requests direct to Tomcat/Jetty without any intermediate Apache or Nginx. – Арсен Мирзаян Dec 17 '18 at 22:24
  • Tomcat has no agent-specific behavior AFAIK. Something else is going on. – Christopher Schultz Dec 17 '18 at 22:54

0 Answers0