Questions tagged [chunked-encoding]

Chunked encoding involves transferring the body of a message as a series of chunks, each with its own chunk size header.

This defined process means that an application-generated entity body, or a large entity body, can be sent in convenient segments. The client or server knows the chunked message is complete when the zero length chunk is received.

The body of a chunked message can be followed by an optional trailer that contains supplementary HTTP headers, known as trailing headers. Clients and servers are not required to accept trailers, so the supplementary HTTP headers should only provide non-essential information, unless a server knows that a client accepts trailers.

To use chunked transfer-coding, both the client and server must be using HTTP/1.1. A chunked message cannot be sent to an HTTP/1.0 client. The requirements that apply to chunked transfer-coding and the use of trailing headers are defined in the HTTP/1.1 specification

Source

421 questions
3
votes
1 answer

Improve performance of running large files

I know there are a few questions on this topic but I can't seem to get it going efficiently. I have large input datasets (2-3 GB) running on my machine, which contains 8GB of memory. I'm using a version of spyder with pandas 0.24.0 installed. The…
jonboy
  • 415
  • 4
  • 14
  • 45
3
votes
2 answers

Unexpected content at the end of chunk - what builds the corrupt stream?

I am hitting an endpoint that returns roughly 900k of XML. Every now and again (less than 1 in 5000 on today's tests) I get a MalformedChunkCodingException This is happening from a fairly old webapp (~10 years), built on Spring 3. I switched to…
Dave Thorn
  • 61
  • 1
  • 6
3
votes
1 answer

Error code redirect when returning a response with chunked encoding?

My web application uses chunked encoding. I'd like to have the behavior where if any chunk generates an error, I can properly set the error code and redirect to an error page using that. Right now it seems like that can only happen if the error…
Heinrich Schmetterling
  • 6,614
  • 11
  • 40
  • 56
3
votes
3 answers

Download chunked resource with wget?

There's a sample ASP.NET project with this controller: using System; using System.Collections.Generic; using System.Web; using System.Web.UI; using System.Web.UI.WebControls; public partial class _Default : System.Web.UI.Page { protected void…
Guard
  • 6,816
  • 4
  • 38
  • 58
3
votes
3 answers

Handling chunked JSON on progress event?

I have a server request which may return a huge json list (~100K records, ~50 Mb) of points I have to draw on a canvas using D3js. I'd like to draw them as they arrive in order to favor interactivity and spare memory, So : I enabled Chunked transfer…
Dr. Goulu
  • 580
  • 7
  • 21
3
votes
1 answer

How to read an HTTP chunked response and send chunked response to client in Elixir?

I'm using Elixir/Phoenix and I have an endpoint that returns a chunked response like say a never ending stream of log lines. The log lines, however, come from another service A which also returns a chunked response. I want my endpoint to read the…
Jesse Shieh
  • 4,660
  • 5
  • 34
  • 49
3
votes
1 answer

cpprestsdk: handle chunked response

How should I handle chunked response using cpprestsdk? How to request the next chunk? Is there required functionality at all there? Here is how we are performing http requests: web::http::http_request…
Dmitry Katkevich
  • 883
  • 7
  • 26
3
votes
2 answers

How do I do a chunked transfer-encoding upload with WWW:Mechanize?

I'm attempting to use a particular web service, and I can successfully perform the upload with the following command: curl -X POST --header "Transfer-Encoding: chunked" -d @Downloads/file.pdf https://some.webservice/upload I get back a json…
John O
  • 4,863
  • 8
  • 45
  • 78
3
votes
3 answers

Handling 'chunked' encoded response with HTTPBuilder/Groovy

I'm new to using groovy and have started to use it to test some REST services. I'm having an issue parsing my XML response from our service due to 'Content not allowed in prolog.' After awhile searching I came across a post saying there might be a…
Caleb S
  • 163
  • 11
3
votes
0 answers

Chunked encoding, streams and content-length

I need to upload a gzipped file. For performance in case my string gets too big I decided to use streams but ran into an issue with the server requiring a content-length header which cannot be calculated as the gzipping is inline. I then decided to…
cyberwombat
  • 38,105
  • 35
  • 175
  • 251
3
votes
1 answer

What are valid chunks extensions?

RFC 7230 defines chunked encoding and states : The chunked encoding allows each chunk to include zero or more chunk extensions, immediately following the chunk-size, for the sake of supplying per-chunk metadata (such as a signature or…
Jeff Bencteux
  • 1,406
  • 16
  • 27
3
votes
1 answer

ColdFusion - HTTP chunk missing first character

This question is a continuation to my previous question regarding HTTP chunk transfer in ColdFusion. Now, I have used java.net.URL to read the chunks and this is what I have tried:
Abhishekh Gupta
  • 6,206
  • 4
  • 18
  • 46
3
votes
2 answers

CFHTTP - Read data from URL transferring in chunks

I am trying to get a file from a URL using cfhttp but it seems that the provider is sending the data in chunks. The response header is having Transfer-Encoding as…
Abhishekh Gupta
  • 6,206
  • 4
  • 18
  • 46
3
votes
1 answer

Disable chunked transfer encoding in go without using content-length

Is there a way to disable chunked transfer encoding in go without using a content-length? It's for Server-Sent Events which can't have either.
Timmmm
  • 88,195
  • 71
  • 364
  • 509
3
votes
1 answer

downloading a large file in chunks with gzip encoding (Python 3.4)

If I make a request for a file and specify encoding of gzip, how do I handle that? Normally when I have a large file I do the following: while True: chunk = resp.read(CHUNK) if not chunk: break writer.write(chunk) writer.flush() where…
code base 5000
  • 3,812
  • 13
  • 44
  • 73