I've been working on my project from the last two months, everything works fine but I have ended up with a problem here. I have been running my script in c#:
using OpenQA.Selenium;
using OpenQA.Selenium.PhantomJS;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
namespace console
{
class Program
{
static void Main(string[] args)
{
IWebDriver driver = new PhantomJSDriver();
var url = "http://example1.com";
driver.Navigate().GoToUrl(url);
var r = driver.FindElements(By.CssSelector("css selector of example1.com"));
var d = r.First().Text;
Console.WriteLine(d);
var phantomJSDriverService = PhantomJSDriverService.CreateDefaultService();
IWebDriver driver_p = new PhantomJSDriver(phantomJSDriverService);
driver_p.Manage().Timeouts().ImplicitlyWait(new TimeSpan(0, 0, 30));
var url_p = "http://example2.com";
driver.Navigate().GoToUrl(url_p);
var title = driver.FindElements(By.CssSelector("css selector of example2.com"));
var myFirstScrape = title[0].Text;
Console.WriteLine(myFirstScrape);
Console.ReadLine();
driver_p.Quit();
}
}
}
on Visual studio express for desktop in console application . I get wonderful result in 40 sec. Both gives 40 sec while the first one gives result in lessthan 12 seconds.
The problem is here, While running the script based on the above in asp.net project of visual studio for web,
The script in asp.net project of visual studio is:
using OpenQA.Selenium;
using OpenQA.Selenium.PhantomJS;
using System;
using System.Web;
using System.Web.UI;
using System.Web.UI.WebControls;
namespace WebScraperapp
{
public partial class _Default : Page
{
protected void Page_Load(object sender, EventArgs e)
{
var phantomJSDriverService = PhantomJSDriverService.CreateDefaultService();
IWebDriver driver = new PhantomJSDriver(phantomJSDriverService);
driver.Manage().Timeouts().ImplicitlyWait(new TimeSpan(0, 0, 30));
var url = "http://www.example1.com";
driver.Navigate().GoToUrl(url);
var title = driver.FindElements(By.CssSelector("css selector of example 1"));
var myFirstScrape = title[0].Text;
Label2.Text = myFirstScrape;
driver.Quit();
}
}
}
The result I got from the above asp.net project code is over 150s which is not fare for my business. I have been struggling in breaking this down to minimum of < 50 sec.
Note: I am very happy if I will get the final output in php script, I am not aware of phantomjs on web hosting server, hence preferred asp.net for this project.
After removing the implicitlywait
function, I'm getting the same results!
Thanks in advance!