0

I've been working on my project from the last two months, everything works fine but I have ended up with a problem here. I have been running my script in c#:

using OpenQA.Selenium;
using OpenQA.Selenium.PhantomJS;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace console
{
    class Program
    {
        static void Main(string[] args)
        {

        IWebDriver driver = new PhantomJSDriver();
        var url = "http://example1.com";
        driver.Navigate().GoToUrl(url);
        var r = driver.FindElements(By.CssSelector("css selector of example1.com"));
        var d = r.First().Text;
        Console.WriteLine(d);

        var phantomJSDriverService = PhantomJSDriverService.CreateDefaultService();

        IWebDriver driver_p = new PhantomJSDriver(phantomJSDriverService);
        driver_p.Manage().Timeouts().ImplicitlyWait(new TimeSpan(0, 0, 30));
        var url_p = "http://example2.com";
        driver.Navigate().GoToUrl(url_p);
        var title = driver.FindElements(By.CssSelector("css selector of example2.com"));
        var myFirstScrape = title[0].Text;

        Console.WriteLine(myFirstScrape);
        Console.ReadLine();

        driver_p.Quit();


    }
  }
}

on Visual studio express for desktop in console application . I get wonderful result in 40 sec. Both gives 40 sec while the first one gives result in lessthan 12 seconds.

The problem is here, While running the script based on the above in asp.net project of visual studio for web,

The script in asp.net project of visual studio is:

using OpenQA.Selenium;
using OpenQA.Selenium.PhantomJS;
using System;
using System.Web;
using System.Web.UI;
using System.Web.UI.WebControls;

namespace WebScraperapp
    {
        public partial class _Default : Page
        {
            protected void Page_Load(object sender, EventArgs e)
            {


           var phantomJSDriverService = PhantomJSDriverService.CreateDefaultService();

        IWebDriver driver = new PhantomJSDriver(phantomJSDriverService);
        driver.Manage().Timeouts().ImplicitlyWait(new TimeSpan(0, 0, 30));
        var url = "http://www.example1.com";
        driver.Navigate().GoToUrl(url);
        var title = driver.FindElements(By.CssSelector("css selector of example 1"));
        var myFirstScrape = title[0].Text;
       Label2.Text = myFirstScrape;
        driver.Quit();


        }

    }
}

The result I got from the above asp.net project code is over 150s which is not fare for my business. I have been struggling in breaking this down to minimum of < 50 sec.

Note: I am very happy if I will get the final output in php script, I am not aware of phantomjs on web hosting server, hence preferred asp.net for this project.

After removing the implicitlywait function, I'm getting the same results!

Thanks in advance!

marc_s
  • 732,580
  • 175
  • 1,330
  • 1,459
abdul riyaz
  • 80
  • 1
  • 1
  • 10
  • Did you tried it in a real IIS server? Remember you're using IIS Express in debug mode which will be a lot slower than in a real production environment – Gusman Jan 08 '16 at 18:30
  • Ya IIS is the server.Does it really? Everything will be fine after deploying it to production? – abdul riyaz Jan 08 '16 at 18:32
  • No, I cannot guarantee it, but for sure it will be a lot faster, if you have Windows 7/8/10 professional just install IIS and test it :) – Gusman Jan 08 '16 at 18:33
  • Currently i am running it on windows 10, but console application works really faster and amazing results. any ideas of running invisible console application for asp.net? – abdul riyaz Jan 08 '16 at 18:36
  • wow, don't do it. Really a bad idea, you don't know permissions and security settings in your production environment, so it's a really bad idea trying to spawn a new process – Gusman Jan 08 '16 at 18:38
  • OK.I will Never do that. Thanks for information. – abdul riyaz Jan 08 '16 at 19:34

0 Answers0