why curl cannot get the same response as in ie browser?

Question

I have an api which run like this:

so I want to get the result in my ubuntu machine with curl, then I try this:

curl -v \
--header 'Accept: text/html, application/xhtml+xml, */*' \
--header 'Accept-Language: zh-CN' \
-A 'Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko' \
--header 'Accept-Encoding: gzip, deflate' \
--header 'Host: 10.202.15.197:20176' \
--header 'DNT: 1' \
--header 'Connection: Keep-Alive' \
http://10.202.15.197:20176?user_id=1&query_type=GEOSPLIT&address=广东省深圳市宝安&ret_splitinfo=1

then strange thing happened: as you can see, I get total different result from the ie browswer, so I thought it must be encoding problem, then I try this:

curl -v \
--header 'Accept: text/html, application/xhtml+xml, */*' \
--header 'Accept-Language: zh-CN' \
-A 'Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko' \
--header 'Accept-Encoding: gzip, deflate' \
--header 'Host: 10.202.15.197:20176' \
--header 'DNT: 1' \
--header 'Connection: Keep-Alive' \
http://10.202.15.197:20176 --data-urlencode 'user_id=1&query_type=GEOSPLIT&address=广东省深圳市宝安&ret_splitinfo=1'

but no, it returns the same result, and I catch my request by fiddler in my windows ie browser, I get the request data:

GET http://10.202.15.197:20176/?user_id=1&query_type=GEOSPLIT&address=广东省深圳市宝安&ret_splitinfo=1 HTTP/1.1
Accept: text/html, application/xhtml+xml, */*
Accept-Language: zh-CN
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko
Accept-Encoding: gzip, deflate
Host: 10.202.15.197:20176
DNT: 1
Connection: Keep-Alive
Pragma: no-cache


HTTP/1.0 200 OK
Content-Type: application/octet-stream
Connection: close
Content-Length: 222

<?xml version='1.0' encoding='GBK'?>
<addrSplitInfo>
<status>0</status><as_info prop="1" level="1">广东省</as_info>
<as_info prop="1" level="2">深圳市</as_info>
<as_info prop="3" level="18">宝安</as_info>
</addrSplitInfo>

Answer 1

I do much further research in this phenomenon(see here). The reason of this is that IE browser will use GBK as default encoding meanwhile Chrome Firefox cURL python-requests just use UTF-8 encoding.

Here is the solution to use this API:

with cURL:

echo "http://10.202.15.197:20176\?user_id\=1\&query_type\=GEOSPLIT\&address\=广东省深圳市宝安\&ret_splitinfo\=1" | iconv -f utf-8 -t gbk | xargs curl

with python-requests

payload = {"user_id": 1, "query_type": "GEOSPLIT", "address": u"广东省深圳市宝安".encode('gbk'), "ret_splitinfo": 1}
r = requests.get("http://10.202.15.197:20176", payload)
print r.text