API Data Returning Unicode Characters in Console

Question

I am facing a rather confusing problem since the last two days. I am working on a document management system, that uses an API that pulls in data from SOLR. The data is in tune of around ~15Mbs, and pulls records of more than 4000+ documents. The API has response in this format -

{
    "documents": [
        {
            id: 123,
            some_field: "abcd",
            some_other_field: "abcdef"
        },
        {
            id: 124,
            some_field: "abcd1",
            some_other_field: "abcdef1"
        }
    ]
}

Everything works fine in browser. If I hit the endpoint in Chrome or Firefox browser, it gives me the correct output and I am able to see the JSON output.

However, if I try hitting the same API endpoint with a Java or JS code - the response code is 200, but the output in console (Terminal or Eclipse) shows unicode characters like \u0089 \u0078 U+0080 - all the output comes in this way, and since there are around 4000+ records being fetched by the API, the console kinda fills with all of these unicode characters.

The only difference that I see between the requests made from browser and the code is that in browser I can see Content-Encoding : gzip, while I cannot find this header from the code that I written . For eg - in JS code, through Chakram framework, I can check

expect(response).to.be.encoded.with.gzip

mentioned here. However, this returns a failure stating expected undefined to match gzip

What am I missing here? Is this something related to encoding/decoding or something entirely different?

Edit 1 : The Response Headers as seen in Network tab of Chrome :

cache-control: max-age=0, private, must-revalidate, max-age=315360000
content-encoding: gzip
content-type: application/json; charset=utf-8
date: Tue, 22 May 2018 06:07:26 GMT
etag: "a07eb7c1eef4ab97699afc8d61fb9c5d"
expires: Fri, 19 May 2028 06:07:26 GMT
p3p: CP="NON CUR OTPi OUR NOR UNI"
server: Apache
Set-Cookie : some_cookie
status: 200 OK
strict-transport-security: 
transfer-encoding: chunked
vary: Accept-Encoding
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-request-id: abceefr4-1234-acds-100b-d2bef2413r47
x-runtime: 3.213943
x-ua-compatible: chrome=1
x-xss-protection: 1; mode=block

The Request Headers as seen in Network tab of Chrome

Accept: application/json, text/plain, */*
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9
Connection: keep-alive
Cookie: some_cookie
Host: abcd.bcd.com
IV_USER: demouser123
IV_USER_L: demouser123
MAIL: demouser@f.com
PERSON_ID: 123
Referer: http://abcd.bcd.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36
X-CSRF-TOKEN: some_csrf_token

Edit 2 : The tests that I am using

describe('Hits required API',()=>{

    before(()=>{
        return chakram.wait(api_response = chakram.get(url,options));
    });

    it('displayes response',()=>{
        return api_response.then((t_resp)=>{
            console.log(JSON.stringify(t_resp));
            expect(t_resp).to.have.header('Content-Encoding','gzip');
        });
    });

Can you show us all the headers for the request – both the request and response? (Either copy from the browser or use cURL) — Zoe Edwards, May 21 '18 at 16:41
See edit 2 @ThomasEdwards. This request is made using `Chakram` — demouser123, May 22 '18 at 08:56
What does the console log print out for `t_resp`? Does it contain any headers? — Zoe Edwards, May 22 '18 at 09:23
What do you mean by the hexadecimal characters? If you `console.log(api_response)` before resolving the promise, what does it log? — Zoe Edwards, May 22 '18 at 09:29
It logs `{ state: 'pending' }` if I do a console without resolving the promise. If I resolve the promise , then console is filled with something like this `u0014^�&��g�h�d8��ݐ\'\n\u00190 ��+� �\u001d�O�\u001cP��\u000e�E�`G�\u0013��1��{n�H�"\u0017\u001eqh]�\\` — demouser123, May 22 '18 at 10:01
Do you have anyway to control the server – could you turn of gzip to see if that’s the issue? — Zoe Edwards, May 22 '18 at 15:56
I think you’ve done everything right here, it’s very possibly it’s a bug with chakram not understanding gzip. I’d [open an issue](https://github.com/dareid/chakram/issues) and walk them through the debugging you’ve done here. — Zoe Edwards, May 23 '18 at 08:25
Looking at their code it seems they do have gzip decoding support. So there may be some edge case around here or something weird happening. It could be like getting gzip response even when `chakram` didn't ask for it — Tarun Lalwani, May 30 '18 at 10:38

karthik · Answer 1 · 2018-06-05T13:04:27.147

This has nothing to do with encoding. The web server in general compresses to gzip to save the bandwidth since its redundant to transfer the whole 15MB file as is refer this article for more about gZip and the its working ( https://betterexplained.com/articles/how-to-optimize-your-site-with-gzip-compression/ ). So where does it went wrong and how it worked in chrome is pretty simple chrome has an inbuilt unicode parser(even an HTML parser) in its devTools which can show you the parsed content rather showing you the wiered text (same can be seen in response tab next to preview tab). why you see wierd text is that you are stingfying the response which will escape special character if any console.log(JSON.stringify(t_resp));. You cannot use something like console.log("response", t_resp); without stringifying in terminal since the terminal doesn't have a JSON or an unicode parser it just prints in text. try removing that console since stringifying a 15mb file is a costly process.

Edit 1:- if you still want to output in the console here whats to be done. Since NODE cannot decode gzip by default directly (not with chakram, its just a APItesting platform) you can use zlib to do this. Please find the example snippet

    const zlib = require('zlib');

describe('Hits required API',()=>{

    before(()=>{
        return chakram.wait(api_response = chakram.get(url,options));
    });

    it('displayes response',()=>{
        return api_response.then((t_resp)=>{
            zlib.gunzip(t_resp, function(err, dezipped) {
                console.log(dezipped);
            });
        });
    });

hoping the example code works. Haven't tested though. you can use fiddler to check the responces right insted logging the etire responce and filling the console. @demouser123 ? — karthik, Jun 05 '18 at 09:00
errors on the pipe function - `t_resp.pipe(gunzip);` - pipe is not a function. — demouser123, Jun 05 '18 at 11:08
can you check now ... if possible can you provide a sandbox example — karthik, Jun 05 '18 at 13:05
This returns `undefined`. Sorry I can't provide a sandboxed example :( — demouser123, Jun 05 '18 at 13:32
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/172556/discussion-between-demouser123-and-karthik). — demouser123, Jun 06 '18 at 05:34

Michael · Answer 2 · 2018-06-05T08:22:50.617

-1

Try with console.dir to display your values

describe('Hits required API',()=>{

    before(()=>{
        return chakram.wait(api_response = chakram.get(url,options));
    });

    it('displayes response',()=>{
        return api_response.then((t_resp)=>{

            console.dir(t_resp, { depth: null });

        });
    });

Console.dir

edited Jun 05 '18 at 08:22

answered Jun 04 '18 at 19:29

Michael

556
2
8

isn't `console.log(JSON.parse(t_resp)); ` on a terminal show something like [Object] ?(if the output sucessfully parses to object) – karthik Jun 05 '18 at 07:18
@karthik you are true. Didn't recognize the treminal in text. Console.dir can then help. I update it now – Michael Jun 05 '18 at 07:27
isn't console.dir and console.log returns same in terminal ? – karthik Jun 05 '18 at 07:45
@karthik did you give it a try? In node environment it gives a different output. – Michael Jun 05 '18 at 08:18
This gives me the same response with the response coming in gibberish – demouser123 Jun 05 '18 at 11:16
@demouser123 if you provide a smaller pice of your json code, I can check it later on? What environment do you use? – Michael Jun 05 '18 at 12:35

API Data Returning Unicode Characters in Console

2 Answers2