How to search YouTube with HtmlUnit

Question

I wonder if YouTube could be searched with HtmlUnit. I started to write code, here it is:

import java.io.IOException;
import java.net.MalformedURLException;
import com.gargoylesoftware.htmlunit.FailingHttpStatusCodeException;
import com.gargoylesoftware.htmlunit.WebClient;
import com.gargoylesoftware.htmlunit.html.HtmlForm;
import com.gargoylesoftware.htmlunit.html.HtmlPage;
import com.gargoylesoftware.htmlunit.html.HtmlSubmitInput;

public class HtmlUnitExampleTestBase {
    private static final String YOUTUBE = "http://www.youtube.com";
    public static void main(String[] args) throws FailingHttpStatusCodeException, MalformedURLException, IOException {
        WebClient webClient = new WebClient();
        webClient.setThrowExceptionOnScriptError(false);

        //This is equivalent to typing youtube.com to the adress bar of browser
        HtmlPage currentPage = webClient.getPage("http://www.youtube.com");

        //Get form where submit button is located
        HtmlForm searchForm = (HtmlForm) currentPage.getElementById("masthead-search");
        //Printing result form
        System.out.println(searchForm.asText());
        final List<HtmlAnchor> listLinks = (List<HtmlAnchor>) newPage.getByXPath("//a[@class='ux-thumb-wrap result-item-thumb']");
        for (int i=0; i<listLinks.size(); i++){
            System.out.println(YOUTUBE + listLinks.get(i).getAttribute("href"));
        }
    }   
}

Now I don't know how to type some text into a search field and press Search button.

I saw tutorials about HtmlUnit but I'm having a problem because they use a method named: getElementByName but the search button on YouTube doesn't have a name, just an id. Could someone help me?

EDIT: I edited code above code and now I am getting youtube links from first page. But before that I need to sort by upload date and then to grab links. Can someone help me to do sorting?

score 3 · Accepted Answer · answered Jul 21 '11 at 10:05

I'm no HtmlUnit expert, but there is a workaround. You can add your own button to the form and use it to submit the form.

Here's a code sample with comments:

import java.io.IOException;
import java.net.MalformedURLException;

import com.gargoylesoftware.htmlunit.FailingHttpStatusCodeException;
import com.gargoylesoftware.htmlunit.WebClient;
import com.gargoylesoftware.htmlunit.html.HtmlButton;
import com.gargoylesoftware.htmlunit.html.HtmlForm;
import com.gargoylesoftware.htmlunit.html.HtmlPage;
import com.gargoylesoftware.htmlunit.html.HtmlTextInput;

public class HtmlUnitExampleTestBase {
   public static void main(String[] args) throws FailingHttpStatusCodeException, MalformedURLException, IOException {
      WebClient webClient = new WebClient();
      webClient.setThrowExceptionOnScriptError(false);

      // This is equivalent to typing youtube.com to the adress bar of browser
      HtmlPage currentPage = webClient.getPage("http://www.youtube.com");

      // Get form where submit button is located
      HtmlForm searchForm = (HtmlForm) currentPage.getElementById("masthead-search");

      // Get the input field.
      HtmlTextInput searchInput = (HtmlTextInput) currentPage.getElementById("masthead-search-term");
      // Insert the search term.
      searchInput.setText("Nyan Cat");

      // Workaround: create a 'fake' button and add it to the form.
      HtmlButton submitButton = (HtmlButton) currentPage.createElement("button");
      submitButton.setAttribute("type", "submit");
      searchForm.appendChild(submitButton);

      // Workaround: use the reference to the button to submit the form. 
      HtmlPage newPage = submitButton.click();

      System.out.println(newPage.asText());
   }
}

It works. Now I just want to sort results with some criteria other than default. Default is sort by relevance, how to change to sort by upload date? — Иван Бишевац, Jul 21 '11 at 21:20
@Иван Бишевац: Perhaps it would be better to open a new question? — Jasper, Jul 22 '11 at 07:45

score 1 · Answer 2 · edited Jul 21 '11 at 10:04

1

HtmlUnit is OK, but I vastly prefer Watir or Selenium for web automation.

One of HtmlUnit's weaknesses is its lack of selector methods for getting at DOM elements in a jQuery-like way. Check out the css-selector project, which will add on to HtmlUnit to help you do what you need very easily. There's an intro at Gooder Code.

Once you get that working, the selector for the YouTube search form would be ".search-term" and the submit button's selector would be ".search-button"

edited Jul 21 '11 at 10:04

Jasper

2,166
4
30
50

answered Jul 21 '11 at 01:15

jkraybill

3,339
27
32

Do you recommend me to take Watir or Selemium? I heard here on forum that many people recommends Selenuim. – Иван Бишевац Jul 21 '11 at 11:08
@Иван Бишевац: Here's a [comparison of Watir vs Selenium vs Sahi](http://stackoverflow.com/questions/606550/watir-vs-selenium-vs-sahi/643124#643124). – Jasper Jul 21 '11 at 12:18
@Иван Бишевац: I like them both. If I want really quick-and-easy record/playback and simple functional tests I want to watch, I use Selenium; if I want more powerful control structures, logic, and/or automation, I use Watir. – jkraybill Jul 21 '11 at 23:28

How to search YouTube with HtmlUnit

2 Answers2

Linked