cURL return "There was an error processing the request"

Question

everyone!

I wanted to scrape this site using SocialSite platform by cURLing on the XHR link in it but it returns:

{"Message":"There was an error processing the request.","StackTrace":"","ExceptionType":""}

This is the cURL code:

curl "http://www.dbmanetwork.com/WebServices/PlannerFace.asmx/GetAppointments" \
  -H "Cookie: ASP.NET_SessionId=vdazkdnenkpzqjgdnjl24pz0; perfectmindmobilefeature=0" \
  -H "Origin: http://www.dbmanetwork.com" \
  -H "Accept-Encoding: gzip, deflate" \
  -H "Accept-Language: en-US,en;q=0.8,id;q=0.6,ms;q=0.4,jv;q=0.2" \
  -H "User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36" \
  -H "Content-Type: application/json; charset=UTF-8" \
  -H "Accept: application/json, text/javascript, */*; q=0.01" \
  -H "Referer: http://www.dbmanetwork.com/8661/Office/Planner/ScheduleView?layoutType=wide&objectId=63be552d-cb71-40e7-aa6f-5289d7e766e9&viewId=2cb73398-53ff-4a9e-adda-3444fb771702&text=Book&skinId=4494e024-c590-4daa-9bfd-e14a7709d23b" \
  -H "X-Requested-With: XMLHttpRequest" \
  -H "Connection: keep-alive" \
  --data-binary "{""schedulerInfo"":{""ViewStart"":""\\/Date(1430092800000)\\/"",""ViewEnd"":""\\/Date(1430697600000)\\/"",""LocationId"":""All Locations"",""IsReadOnly"":true,""UseAppointmentLocationTimezone"":true,""ApplicationType"":2,""OrgId"":""29f8130e-ea28-4cd7-8bb0-298f753d9d17"",""ObjectViewIdCombos"":""63be552d-cb71-40e7-aa6f-5289d7e766e9.2cb73398-53ff-4a9e-adda-3444fb771702""}}" \
  --compressed -k

The advice from here and here don't work since the cURL already fulfilled the required condition (json_encode and Content-Type).

Thx!

UPDATE!

Using the --verbose flag, I got this result:

* Adding handle: conn: 0x214d7e0
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x214d7e0) send_pipe: 1, recv_pipe: 0
* About to connect() to www.dbmanetwork.com port 80 (#0)
*   Trying 50.112.169.144...
* Connected to www.dbmanetwork.com (50.112.169.144) port 80 (#0)
> POST /WebServices/PlannerFace.asmx/GetAppointments HTTP/1.1
> Host: www.dbmanetwork.com
> Cookie: ASP.NET_SessionId=vdazkdnenkpzqjgdnjl24pz0; perfectmindmobilefeature=0
> Origin: http://www.dbmanetwork.com
> Accept-Encoding: gzip, deflate
> Accept-Language: en-US,en;q=0.8,id;q=0.6,ms;q=0.4,jv;q=0.2
> User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.
36
> Content-Type: application/json; charset=UTF-8
> Accept: application/json, text/javascript, */*; q=0.01
> Referer: http://www.dbmanetwork.com/8661/Office/Planner/ScheduleView?layoutType=wide&objectId=63be552d-cb71-40e7-aa6f-
5289d7e766e9&viewId=2cb73398-53ff-4a9e-adda-3444fb771702&text=Book&skinId=4494e024-c590-4daa-9bfd-e14a7709d23b
> X-Requested-With: XMLHttpRequest
> Connection: keep-alive
> Content-Length: 346
>
* upload completely sent off: 346 out of 346 bytes
< HTTP/1.1 500 Internal Server Error
< Cache-Control: private
< Content-Type: application/json; charset=utf-8
* Server Microsoft-IIS/8.5 is not blacklisted
< Server: Microsoft-IIS/8.5
< jsonerror: true
< X-AspNet-Version: 4.0.30319
< X-Powered-By: ASP.NET
< P3P: CP="CAO PSA OUR"
< Date: Tue, 28 Apr 2015 04:01:25 GMT
< Content-Length: 91
< Connection: Keep-Alive
<
{"Message":"There was an error processing the request.","StackTrace":"","ExceptionType":""}* Connection #0 to host www.d
bmanetwork.com left intact

Can you add the `--verbose` option and post the output? – Jordan Running Apr 28 '15 at 03:39 — Jordan Running, Apr 28 '15 at 03:39
Sure @Jordan, I've added it to the bottom of the post! – Aminah Nuraini Apr 28 '15 at 04:06 — Aminah Nuraini, Apr 28 '15 at 04:06

score 0 · Answer 1 · answered Apr 28 '15 at 04:09

0

seem that the target server http://www.dbmanetwork.com. 3594 IN A 50.112.169.144 is down.

salam dari jakarta :)

answered Apr 28 '15 at 04:09

rezashamdani

577
6
11

Btw, @rezashamdani, the XHR link works on the website itself, at this link http://www.dbmanetwork.com/SocialSite/schedule I think the website developer is genius or something, don't want us to access their data. The verbose result shows that I succeeded to connect to dbmanetwork.com anyway – Aminah Nuraini Apr 28 '15 at 04:15
sorry :) apparently they block the port used by curl :) Starting Nmap 6.47 ( http://nmap.org ) at 2015-04-28 11:15 WIB Nmap scan report for www.dbmanetwork.com (50.112.169.144) Host is up (0.21s latency). rDNS record for 50.112.169.144: ec2-50-112-169-144.us-west-2.compute.amazonaws.com Not shown: 992 filtered ports PORT STATE SERVICE 80/tcp open http 443/tcp open https 808/tcp closed ccproxy-http 8080/tcp closed http-proxy 8081/tcp closed blackice-icecap 8082/tcp closed blackice-alerts 8086/tcp closed d-s-n 9090/tcp closed zeus-admin – rezashamdani Apr 28 '15 at 04:16
Hmmm, the first time I know cURL can be blocked. Do you ever try to force scrape to this kind of URL? How to make them don't know we are pulling data from cURL? And how did you check that? I wanted to be able to do it myself. Thx! :D – Aminah Nuraini Apr 28 '15 at 04:20
one moment, i'll dig into this issue – rezashamdani Apr 28 '15 at 04:23
Yup, that's the result I got when curling using PHP. What are you trying to say? – Aminah Nuraini Apr 28 '15 at 04:25
just doing a quick dig. the curl doesnt run javascript function within the target, while the data list was loaded by http://www.dbmanetwork.com/asset.axd?id=WgEAAB-LCAAAAAAABABtkNEKgjAUht_lXEtqLoPdBl0VFEY34cWYqybLydysEN-9k0Eqered__v-A6eBDOilgRIoJNzI0lZ-fnTCvDfaCPDg2sUFxv14L4tFXmGogQZt2npz_kG5myxmGhDTSp30sCX8dvTITnOmftyQWk53OStVNVqSMSu22jyYHarRRB1JCb-LzClxluL5dwg6qQfcImDFy_o5q1nV2QjUOA1IGAeErPHL8Y5RvMIHUGucaD9OmlWlWgEAAA%3d%3d , so curl is not possible in this issue CMIIW, try something that simulate javascript actions. – rezashamdani Apr 28 '15 at 04:42

score 0 · Accepted Answer · answered Apr 28 '15 at 09:29

First, I want to thank @rezashamdani His answer wasn't the exact answer, but it gave me a strong hint until I could finally found it.

He said that the problem was with the "\\/Date(1430697600000)\\/" in the json string I sent. I might need to simulate the javascript date function. I found out in its javascript file where the XHR executed that the date string was reformatted using local function toAspDateFormat which changes "/Date(1430697600000)/" to "\\/Date(1430697600000)\\/". It shows that the escape character is important. I change the --data-binary from my cURL to this: "{""schedulerInfo"":{""ViewStart"":""\/Date(1430092800000)\/"",""ViewEnd"":""\/Date(1430697600000)\/"",""LocationId"":""All Locations"",""IsReadOnly"":true,""UseAppointmentLocationTimezone"":true,""ApplicationType"":2,""OrgId"":""29f8130e-ea28-4cd7-8bb0-298f753d9d17"",""ObjectViewIdCombos"":""63be552d-cb71-40e7-aa6f-5289d7e766e9.2cb73398-53ff-4a9e-adda-3444fb771702""}}"

It works perfectly now.

score 0 · Answer 3 · answered Mar 30 '23 at 13:07

I was facing a similar issue where I was getting the following response from the source for the curl request that I was using:

{"Message":"There was an error processing the request.","StackTrace":"","ExceptionType":""}

when I checked the verbose info adding the flag: --verbose

I got to know that I too was sending an incorrect json data payload with my curl request which was working in the terminal when I tried it, but was failing when I was making the curl request through the following python library:

from curllibrary.curl import curl

sending the curl request as:

# using the same curl request given in the question to demonstrate an example

curl_request = """curl "http://www.dbmanetwork.com/WebServices/PlannerFace.asmx/GetAppointments" \
  -H "Cookie: ASP.NET_SessionId=vdazkdnenkpzqjgdnjl24pz0; perfectmindmobilefeature=0" \
  -H "Origin: http://www.dbmanetwork.com" \
  -H "Accept-Encoding: gzip, deflate" \
  -H "Accept-Language: en-US,en;q=0.8,id;q=0.6,ms;q=0.4,jv;q=0.2" \
  -H "User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36" \
  -H "Content-Type: application/json; charset=UTF-8" \
  -H "Accept: application/json, text/javascript, */*; q=0.01" \
  -H "Referer: http://www.dbmanetwork.com/8661/Office/Planner/ScheduleView?layoutType=wide&objectId=63be552d-cb71-40e7-aa6f-5289d7e766e9&viewId=2cb73398-53ff-4a9e-adda-3444fb771702&text=Book&skinId=4494e024-c590-4daa-9bfd-e14a7709d23b" \
  -H "X-Requested-With: XMLHttpRequest" \
  -H "Connection: keep-alive" \
  --data-binary $"{""schedulerInfo"":{""ViewStart"":""\\/Date(1430092800000)\\/"",""ViewEnd"":""\\/Date(1430697600000)\\/"",""LocationId"":""All Locations"",""IsReadOnly"":true,""UseAppointmentLocationTimezone"":true,""ApplicationType"":2,""OrgId"":""29f8130e-ea28-4cd7-8bb0-298f753d9d17"",""ObjectViewIdCombos"":""63be552d-cb71-40e7-aa6f-5289d7e766e9.2cb73398-53ff-4a9e-adda-3444fb771702""}}" \
  --compressed -k"""

curl_response = curl(curl_request)

since I was using the '$' in front of the raw data that I was sending which was causing the request to fail, so as soon as I removed the dollar sign, it worked and I got the expected response that I was getting in the terminal.

Hope this ends up resolving an issue for someone who is facing this type of problem.

cURL return "There was an error processing the request"

UPDATE!

3 Answers3