Way to get more than 999 results from a Youtube video feed?

Question

I'm writing right now a bachelorthesis about the Long Tail and want to get data to research it's behaviour. That's why I wanted to retrieve information on the viewcounts of youtube videos. The only problem is that one video feed to a specific topic like "most_popular" has only 999 entries. Is there a way to retrieve more data to a specific category or in general? I'll post my current code (it's a try to retrieve data for the category "sports") here:

public static void printVideoEntry(VideoEntry videoEntry, boolean detailed) {
      System.out.println("Title: " + videoEntry.getTitle().getPlainText());

      if(videoEntry.isDraft()) {
        System.out.println("Video is not live");
        YtPublicationState pubState = videoEntry.getPublicationState();
        if(pubState.getState() == YtPublicationState.State.PROCESSING) {
          System.out.println("Video is still being processed.");
        }
        else if(pubState.getState() == YtPublicationState.State.REJECTED) {
          System.out.print("Video has been rejected because: ");
          System.out.println(pubState.getDescription());
          System.out.print("For help visit: ");
          System.out.println(pubState.getHelpUrl());
        }
        else if(pubState.getState() == YtPublicationState.State.FAILED) {
          System.out.print("Video failed uploading because: ");
          System.out.println(pubState.getDescription());
          System.out.print("For help visit: ");
          System.out.println(pubState.getHelpUrl());
        }
      }

      if(videoEntry.getEditLink() != null) {
        System.out.println("Video is editable by current user.");
      }

      if(detailed) {



        YtStatistics stats = videoEntry.getStatistics();
        if(stats != null ) {
          System.out.println("View count: " + stats.getViewCount());
        }
        System.out.println();


      }
    }

  public static void printVideoFeed(VideoFeed videoFeed, boolean detailed) {
      for(VideoEntry videoEntry : videoFeed.getEntries() ) {
        printVideoEntry(videoEntry, detailed);
      }
    } 

  public static void printEntireVideoFeed(YouTubeService service, 
          VideoFeed videoFeed, boolean detailed) throws MalformedURLException, 
          IOException, ServiceException {
         do {
           printVideoFeed(videoFeed, detailed);
           if(videoFeed.getNextLink() != null) {
             videoFeed = service.getFeed(new URL(videoFeed.getNextLink().getHref()), 
               VideoFeed.class);
           }
           else {
             videoFeed = null;
           }
         }
         while(videoFeed != null);
        }


  public static void main(String[] args) {

  try {

      YouTubeService service = new YouTubeService("test");

      YouTubeQuery query = 
              new YouTubeQuery(new URL("http://gdata.youtube.com/feeds/api/videos"));
            query.setFullTextQuery("Sports");
            VideoFeed videoFeed = service.query(query, VideoFeed.class);
            printEntireVideoFeed(service, videoFeed, false);
  }
  catch(AuthenticationException e) {
    e.printStackTrace();
  }
  catch(MalformedURLException e) {
    e.printStackTrace();
  }
  catch(ServiceException e) {
    e.printStackTrace();
  }
  catch(IOException e) {
    e.printStackTrace();
  }
}

Answer 1

The 999 entries appears to be a limit by design. See the YouTube API v2.0 – Video Feed Types:

"The API returns a videos feed in response to a request to search for videos. A videos feed contains a maximum of 999 entries."

It looks like the same limit apply to the web site as well.

You could try to add custom query parameters for the YouTube Data API to limit the amount of results you get, like for example:

• caption=true|false
• duration=short|medium|long
• format=1|5|6
• time=today|this_week|this_month (instead of the default: all_time)

This way you can use a combination of queries to get more of the results that you're interested in.

Edit: While searching how to use the duration, I came across YouTube GData API - Query for videos with a specific duration on Stack Overflow, which refers to YouTube API v2.0 – Retrieving a Partial Response. You can for example specify a maximum view count (which might be useful to get the Long Tail) using this experimental "Retrieving a Partial Response API" and that's only one of the many things you can do with it.