multiple headless firefox instances using watir-webdriver within rails application

Question

I have a simple rails application written for scraping a web page. The controller calls the scraper utility in which I am using firefox in headless mode using watir-webdriver. The application works and returns the results properly. The way I would call the application is something like this:

http://somedomain.com:3000/scrapers.json?session=1349426645_562&l=test

and it returns a json string.

It takes about 15 seconds for the scraper to complete. While one request is in progress, when I try to launch an other request, the request is queued until the previous one completes. I am not sure if its a limitation on the rails application side or on watir-webdriver or headless gem using Xvfb.

Any pointers would help.

Thanks, Sridhar

Answer 1

There are better libraries for screen scraping such as mechanize. In fact, there are some applications made just for scraping.

Answer 2

I found out that I can use the CGI module to do my work. Since I didnt need a rails application, I used ruby CGI to call the ruby script and pass parameters through URL. I can also launch multiple instances in a non-blocking fashion using this approach. I had to re-write my controller as a stand alone ruby program in order to do this.