1

I am using libcurl for performing HTTP request and response to my webpage. But when getting the response, I am not getting the complete server response. the response lines get truncated at one point. e.g 

10-25 15:53:22.264: XXX(14847): Security.8021x:true
10-25 15:53:22.264: XXX(14847): Event.AlarmInput:true
10-25 15:53:22.264: XXX(14847): Event.AlarmInput.Notification.HTTP.CGI:true
10-25 15:53:22.264: XXX(14847): Event.AlarmInput.Notification.HTTP.CGI.SingleSession:true
10-25 15:53:22.264: XXX(14847): Event.AlarmInpu   .............

Note: this is a custom android log. please don't confuse with the output lines

There are around 10 more lines in the response, but my current log shows that after "Event.AlarmInpu" there is no output.

When I give the curl command line option 

curl http://www.google.com > output.txt

I can find the complete output lines inside the file.

When I use the curl command line tool and I moved the response to an output file, I could see the entire response data. So I can agree that curl command is retrieving the entire response properly. I get the whole response, but the response is longer than my buffer length and hence it is getting truncated.

Do let me know how I can increase my buffer length to get the complete response line.
The code snippet is shown below

typedef struct pageInfo_t {
    char *data;
    int  len;
} pageInfo_t;

static size_t HTTPData(void *buffer, size_t size, size_t nmemb, void *userData)
{
    int len = size * nmemb;
    pageInfo_t *page = (pageInfo_t *)userData;
    page->data = realloc(page->data,page->len + len +1);
    memcpy(&page->data[page->len], buffer, len);
    page->len += len;
    page->data[page->len] = 0;
    return len;
}

//Inteface funciton that will recieve web page fom Java
jstring Java_com_samsung_jnitest_MainActivity_JNIGetWebpage( JNIEnv* env,jobject entryObject,jstring webpageJStr)
{
    pageInfo_t page;
    CURL *curl;
    CURLcode res;
    char *buffer;

    int memorysize = 19189;
    page.data = (char *)malloc(16 * memorysize);
    page.len = 0;

    if (page.data)
        memset(page.data, 32, 16 * memorysize);

    buffer = (char *)malloc(memorysize);

    curl = curl_easy_init();
    if(curl)
    {
        curl_easy_setopt(curl, CURLOPT_URL, webpage);
        curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, HTTPData);
        curl_easy_setopt(curl, CURLOPT_WRITEDATA, &page);
        res = curl_easy_perform(curl);
        curl_easy_cleanup(curl);
        (*env)->ReleaseStringUTFChars(env, webpageJStr, webpage);
        if(response_code == 200) 
        {
            if (buffer) 
            {
                page.data[page.len] = '\0';
                sprintf(buffer, "%ld:%s \n", response_code, page.data);
                return (*env)->NewStringUTF(env, buffer);
            }
        }
    }
}

Please let me know how to increase the buffer size to hold the complete response lines.
Let me know if I am over complicating the code

Suraj Rao
  • 29,388
  • 11
  • 94
  • 103
Sunil
  • 521
  • 1
  • 7
  • 24

2 Answers2

0

You allocate buffer to have a size of memorysize (19189 bytes).

However, the buffer pointed to by page.data grows according to the the amount of data returned by the HTTP request. But then you copy the null terminated string pointed to by page.data into buffer regardless of whether or not buffer is large enough.

Perhaps you should get rid of all the code related to buffer and exit your function with something like:

jstring retval = (*env)->NewStringUTF(env, page.data);
free(page.data);
return retval;

Don't forget to clean up properly in the case where response_code != 200.

A few other comments:

  • I don't see how webpage is set up or declared - I assume there's an elided call to GetStringUTFChars() in your real code?
  • I'm not familiar with JNI, so I might not be properly handling the jstring result in my suggestion above.
  • I'm curious - when you use the curl command line tool to place the HTTP response in a file, how big does the file end up being?
Michael Burr
  • 333,147
  • 50
  • 533
  • 760
  • i tried just returning "page.data" but still it just contains the same output lines as previous, when i was using "buffer". – Sunil Nov 12 '13 at 09:04
  • When I try a simple program (based on libcurls' simple.c example) that's basically the same as yours with the JNI stuff stripped out I get the expected data in `page.data`. Maybe you can try it yourself on your problem URL: https://gist.github.com/anonymous/7441125 – Michael Burr Nov 13 '13 at 00:10
  • i followed your example..but i couldnt get the complete response in page.data I am using a sample.pl file installed inside apache server for testing. i try to send request to this script and fetch the response using curl . i will add my perl script code. can you try to install apache server and add the lines into a sample.pl file . then use browser to check if it prints the output when you access it. if all done..then can you try the curl program to get and print the output . If succesful please let me know. – Sunil Nov 13 '13 at 06:07
  • #!"D:\xampp\perl\bin\perl.exe" ## ## printenv -- demo CGI program which just prints its environment ## print "Content-type: text/plain; charset=utf-8\n\n"; foreach $var (sort(keys(%ENV))) { $val = $ENV{$var}; $val =~ s|\n|\\n|g; $val =~ s|"|\\"|g; print "${var}=\"${val}\"\n"; } this is the program. just copy this into a notepad and save as .pl file. the please add the foreach loop as many times to make the response output bigger in size. then test with the current libcurls sample.c example – Sunil Nov 13 '13 at 06:10
  • do i need to give some delay because i tried simple.c code into my application. but still i get only first section of my response. is there any delay in getting the complete response.. I tried to add printf statement inside the size_t method and found that len variable is being print 3 times ..i understand that the method is called many times until the complete chunk of data is written – Sunil Nov 13 '13 at 07:48
0

I had the same issue and it got fixed for me by extending CURLOPT_TIMEOUT_MS.

Vladimir Pavluk
  • 143
  • 4