I have scrapyd and spider installed on a Unix machine, and everything works fine when I run
curl http://localhost:6800/schedule.json -d project=myproject -d spider=somespider
I can see the job status,logs and items on web interface of scrapyd API. in short everything is working as expected.
Now I want to start a spider programmatically by making a http post to the API in ASP.Net using C# as scrapyd will be part of my .NET project but I get
{"status": "error", "message": "'project'"}
I have found an example http://mahmoud.abdel-fattah.net/2012/07/04/super-simple-and-basic-scrapyd-web-interface/comment-page-1/ it makes a Jquery post and this example works for me but below one is not working for me
public void StartCrawler()
{
var httpWebRequest = (HttpWebRequest)WebRequest.Create("http://mydomain.com:6800/schedule.json");
httpWebRequest.ContentType = "application/json; charset=utf-8";
//httpWebRequest.ContentType = "text/json;; charset=utf-8";
httpWebRequest.Method = "POST";
using (var streamWriter = new StreamWriter(httpWebRequest.GetRequestStream()))
{
string json = "{\"project\":\"projectname\",\"spider\":\"spidername\"}";
streamWriter.Write(json);
}
var httpResponse = (HttpWebResponse)httpWebRequest.GetResponse();
using (var streamReader = new StreamReader(httpResponse.GetResponseStream()))
{
var responseText = streamReader.ReadToEnd();
}
}
please tell me what I am doing wrong