how to handle a chunked response from a server?

Question

I'm on the last major hurdle of writing a basic proxy server in C.

I've got my server successfully receiving requests from the browser, then successfully sending them to the host, wherever they are. And I'm successfully receiving responses from the host! Except, here's what I get when I try to connect to Google through the server:

Rcvd message from server: 

----

HTTP/1.1 200 OK
Date: Thu, 15 Mar 2012 20:35:11 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=UTF-8
Set-Cookie: PREF=ID=83a7c2e6675a9a9f:FF=0:TM=1331843711:LM=1331843711:S=7I7RIVV1B-HxhWJR; expires=Sat, 15-Mar-2014 20:35:11 GMT; path=/; domain=.google.com
Set-Cookie: NID=57=KvqnXtYNkJZBryXL5zzhG5eH8Or2_PDWDqT_kU35PvOro_mAFiLiTSjPHOnWWxxm3R0vKYnzEeVkAPFKK366lZiNZGpjhO2-II5OeZQnWe09H-jZdePsrN-SnBdQ2ENT; expires=Fri, 14-Sep-2012 20:35:11 GMT; path=/; domain=.google.com; HttpOnly
P3P: CP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more info."
Server: gws
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Transfer-Encoding: chunked

1000
<!doctype html><html itemscope itemtype="http://schema.org/WebPage"><head><meta http-equiv="content-type" content="text/html; charset=UTF-8"><meta name="description" content="Search the world&#39;s information, including webpages, images, videos and more. Goo

you see how it cuts off? Because of "Transfer-Encoding: chunked". Now how do I get the server to continue sending me the rest of the chunks? Because after that first one, it stops. Do I need to put read() inside a while loop, and say to continue reading and forwarding to the client as long as there's something to read?

EDIT:

Okay, here is my present code. This makes one initial read of the server's response (on socket named "sock"), writes it to the client ("newsock"), then enters a while loop to continue reading more server responses and sending them to the client. i have not tested this form of the code just yet. do you see any blatant problems besides some error-checking gaps?

/*WRITING SERVER RESPONSE TO CLIENT*/

  char buffer2[1024];

  n = read(sock, buffer2, 1024 );

  if ( n < 1 )
  {
    perror( "read() failed" );
    return EXIT_FAILURE;
  }

  else
  {
    buffer2[n] = '\0';
    printf( "Rcvd message from server: \n\n----\n\n%s\n\n----\n\n", buffer2 );
  }

  n = write( newsock, buffer2, strlen( buffer2 ) );

  while((n = read(sock, buffer2, 1024 )) >= 1)
  {
      buffer2[n] = '\0';
      printf( "Rcvd message from server: \n\n----\n\n%s\n\n----\n\n", buffer2 );
      n = write( newsock, buffer2, strlen( buffer2 ) );
  }

Can you elaborate on what do you do (place your relevant C code), why do you say it cut? — roni bar yanai, Mar 15 '12 at 21:00
i couuullld attach some code, except that I've been editing it this whole time trying to solve it on my own, and I haven't tested it yet, so at this moment I don't know what it will do yet. need some time. and i think you'll see the cutoff if you scroll to the right-- obviously there's more to Google.com's code than that. it just stops in the middle of the meta tags. there's much more to come. — temporary_user_name, Mar 15 '12 at 21:01
@Aerovistae, how are you using the content length in response? — perreal, Mar 15 '12 at 21:07
I'm sorry, could you elaborate on that question? I get where you're going but I'm not sure exactly how to use content-length. Is it really as simple as only reading as much as the content length header says to read, then stopping? — temporary_user_name, Mar 15 '12 at 21:12
Regarding images (and binary data in general), don't NUL-terminate `buffer2` and use `strlen` - that only works for text data. In all cases, just use `n` for the length of data to send. (NUL-terminating `buffer2` when `n` is 1024 is also stepping beyond the end of the array.) — Alex Measday, Mar 15 '12 at 22:06
@AlexMeasday But the chunks are a mix of text and image, are they not? Should I just never null-terminate, or will that then cause errors with the text portions? — temporary_user_name, Mar 15 '12 at 22:15
@Aerovistae will you please post your final solution that worked for you? Did you end up getting rid of that null termination and strlen()? — CFL_Jeff, Aug 24 '12 at 09:19

score 1 · Accepted Answer · answered Mar 16 '12 at 02:47

1

You can't use strlen to get the size of your buffer, strlen is only for getting the size of a string, you're probably reading some binary data, because there is not only text going through your proxy (images...). Try using instead the value returned by read, which is the number of bytes actually read.

And ending your string by '\n' becomes useless.

answered Mar 16 '12 at 02:47

Mualki

186
6

But the chunks are a mix of text and image, are they not? Should I just never null-terminate, or will that then cause errors with the text portions? – temporary_user_name Mar 16 '12 at 02:51
1

You have no reason to null-terminate the problem is that you're reasoning as if it was c sting which it's not it is raw data. So in the middle of your data you might get a byte equals to 0. Which is the reason why your strlen is not working. Don't consider this as text and images but as data. So no reason to null-terminate – Mualki Mar 16 '12 at 03:03
Is it okay to hold raw data in a char array? I don't know what other sort of variable to put it in. – temporary_user_name Mar 16 '12 at 03:04
1

yes, it's perfectly fine. Don't think of char as a character but as a container of 1byte for your data. If you haven't done it already, try opening a binary file in a text editor, you'll see that even tho it's completely unreadable for a human, it's working fine. – Mualki Mar 16 '12 at 03:17

how to handle a chunked response from a server?

1 Answers1