2

I've tried to figure this out myself, but I can't get any useful solution. I want to send a request to a website and get the processed results. I've had a problem with this already (see How do I fill in a website form and retrieve the result in C#?) but I was able to solve it with the website stated there. Now I'm trying to access yet another website (http://motif-x.med.harvard.edu/motif-x.html) with the following code:

ServicePointManager.Expect100Continue = false;

HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://motif-x.med.harvard.edu/cgi-bin/multimotif-x.pl");
request.Credentials = CredentialCache.DefaultCredentials;
request.ProtocolVersion = HttpVersion.Version10; // Motif-X uses HTTP 1.0
request.KeepAlive = false;
request.Method = "POST";
string motives = "SGSLDSELSVSPKRNSISRTH";
string postData = "fgdata=" + motives + "&fgcentralres=S&width=21";
byte[] byteArray = Encoding.UTF8.GetBytes(postData);
request.ContentType = "multipart/form-data";
request.ContentLength = byteArray.Length;
Stream dataStream = request.GetRequestStream();
dataStream.Write(byteArray, 0, byteArray.Length);
dataStream.Close();

WebResponse response = request.GetResponse();

This gives me the following exception:

The remote server returned an error: (500) Internal Server Error.

Is there anything you can do to prevent this?

In case you want to manually make an input to the site:

Text area for the data set: SGSLDSELSVSPKRNSISRTH

Central character (in basic options): S

You'll get redirected to the result's site - and it may take a while to process. Could that be actually the reason for the exception?

Community
  • 1
  • 1
Anna
  • 89
  • 1
  • 1
  • 11
  • Setting `System.Net.ServicePointManager.Expect100Continue` [doesn't work for HTTP 1.0 requests](http://msdn.microsoft.com/en-us/library/system.net.servicepointmanager.expect100continue.aspx): "The 100-Continue behavior is not used for HTTP 1.0 requests even if this property is set to true.". – Filburt Sep 01 '11 at 13:40
  • Ok sorry, this line is from before I found out about the HTTP 1.0 property. Please ignore it. – Anna Sep 01 '11 at 13:44
  • I tried the web form and the result isn't just a redirect but a popup/new window. I'm not sure if HttpWebRequest can handle this or if this can result in an error on the server side. – Filburt Sep 01 '11 at 14:03

4 Answers4

5

If you look at the documentation you can see that for a "multipart/form-data" sending data "POST" is very different than "application/x-www-form-urlencoded". For your case, you would have to send something like this:

Content-Type:

Content-Type: multipart/form-data; boundary=---------------------------7db1af18b064a

POST:

-----------------------------7db1af18b064a
Content-Disposition: form-data; name="fgdata"

SGSLDSELSVSPKRNSISRTH
-----------------------------7db1af18b064a
Content-Disposition: form-data; name="fgcentralres"

S
-----------------------------7db1af18b064a
Content-Disposition: form-data; name="width"

21
-----------------------------7db1af18b064a--

these links can help you for sending this data format, but in your case you should avoid sending the file:

POSTING MULTIPART/FORM-DATA USING .NET WEBREQUEST

http://www.groupsrv.com/dotnet/about113297.html

With some HTTP analyzer, you can check the sending data this simple HTML code

<html>
<body>
<form action="http://motif-x.med.harvard.edu/cgi-bin/multimotif-x.pl" enctype="multipart/form-data" method="post">
   <input type="text" name="fgdata" value="SGSLDSELSVSPKRNSISRTH" /><br />
   <input type="text" name="fgcentralres" value="S" /><br />
   <input type="text" name="width" value="21" /><br />

   <input type="submit" value="Send" />
 </form>
</body>
</html>
andres descalzo
  • 14,887
  • 13
  • 64
  • 115
  • Thank you for your reply. The request is working, but now I have the problem that I need to get the results from the popup window that opens after submitting. Is there any way of doing that? The thing is that the popup window has an URL that consists of the date and some sort of hashcode. Can you somehow get that address? – Anna Sep 02 '11 at 10:20
  • 1 - if the answer solved the problem you should accept. 2 - for your new question I would need more information. you can use the tool Fiddler2 to see the traffic and see how it calls another url or parse the response and see if the data are to call the other window. – andres descalzo Sep 02 '11 at 10:30
  • 1 - done :) 2 - I followed your suggestion of using Fiddler and I found that there's a href that leads to the result page. It's working now. Thank you so much for your help!! – Anna Sep 02 '11 at 11:36
  • Thank you very much andres descalzo. This helped me – Sudha Apr 04 '13 at 09:55
1

I set the UserAgent and my problem solved.

 request.UserAgent = "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)";
Mohammad
  • 97
  • 1
  • 9
0

I'm assuming you're trying to access motif-x programatically. There is an implemented version in R you can use:

https://github.com/omarwagih/motifx

Good luck.

Omar Wagih
  • 8,504
  • 7
  • 59
  • 75
0

Thanks a lot for your help! I could make it work with the following code, in case anyone else encounters this problem:

HttpWebRequest request = (HttpWebRequest)WebRequest.Create(
                        "http://motif-x.med.harvard.edu/cgi-bin/multimotif-x.pl");
request.ProtocolVersion = HttpVersion.Version10;
request.Method = "POST";
string boundary = "---------------------------7db1af18b064a";
string newLine = "\r\n";
string postData = "--" + boundary + newLine +
                  "Content-Disposition: form-data; name=\"fgdata\"" + newLine + newLine + "SGSLDSELSVSPKRNSISRTH" + newLine +
                  "--" + boundary + newLine +
                  "Content-Disposition: form-data; name=\"fgcentralres\"" + newLine + newLine + "S" + newLine +
                  "--" + boundary + newLine +
                  "Content-Disposition: form-data; name=\"width\"" + newLine + newLine + "21" + newLine +
                  "--" + boundary + "--";

byte[] byteArray = Encoding.UTF8.GetBytes(postData);
request.ContentType = "multipart/form-data; boundary=" + boundary;
request.ContentLength = byteArray.Length;
Stream dataStream = request.GetRequestStream();
dataStream.Write(byteArray, 0, byteArray.Length);
dataStream.Close();

using (WebResponse response = request.GetResponse())
using (Stream resSteam = response.GetResponseStream())
using (StreamReader sr = new StreamReader(resSteam))
    File.WriteAllText("SearchResults.html", sr.ReadToEnd());
System.Diagnostics.Process.Start("SearchResults.html");
Anna
  • 89
  • 1
  • 1
  • 11