Fetch list of 50,000 most subscribed channels

Question

I'm trying to figure out a way to grab the top 50,000 most subscribed youtube channels using javascript. These only need to be grabbed once and will be stored in a file to be used for an autocomplete input in a webpage.

I've gotten pretty close to getting the first top 50 by using search:list (/youtube/v3/search) by searching with parameters maxResults=50, order=viewCount, part=snippet, type=channel, fields=nextPageToken,items(snippet(channelId,title))

Returning:

{
 "nextPageToken": "CDIQAA",
 "items": [{
   "snippet": {
    "channelId": "UC-9-kyTW8ZkZNDHQJ6FgpwQ",
    "title": "Music"
   }
  },{
   "snippet": {
    "channelId": "UC-lHJZR3Gqxm24_Vd_AJ5Yw",
    "title": "PewDiePie"
   }
  },{
   "snippet": {
    "channelId": "UCVPYbobPRzz0SjinWekjUBw",
    "title": "Анатолий Шарий"
   }
  },{
   "snippet": {
    "channelId": "UCam8T03EOFBsNdR0thrFHdQ",
    "title": "VEGETTA777"
   }
  },...

Then all I'd have to do is fetch that 1000 more times using the nextPageToken to get a list of the top 50,000.

Unfortunately, sorting by relevance, rating, viewCount, or nothing is not yielding the 50 most subscribed channels, and there doesn't seem to be any sort of way to order them by subscriber count according to the documentation; so it seems like i am stuck.

Answer 1

Just before you writing your 50 results in file (or database), you can make one more API call, using channelId field from your result, and merge all of them with comma delimited and make another API call Channels: list.
On that page for example you can use following parameters:
(these are IDs from your example above)

part=statistics
id=UC-9-kyTW8ZkZNDHQJ6FgpwQ,UC-lHJZR3Gqxm24_Vd_AJ5Yw,UCVPYbobPRzz0SjinWekjUBw,UCam8T03EOFBsNdR0thrFHdQ`

And result will look something like this:

{
"kind": "youtube#channel",
"etag": "\"m2yskBQFythfE4irbTIeOgYYfBU/MG6zgnd09mqb3nAdyRnPDgFwfkE\"",
"id": "UC-lHJZR3Gqxm24_Vd_AJ5Yw",
"statistics": {
     "viewCount": "15194203723",
     "commentCount": "289181",
     "subscriberCount": "54913094",
     "hiddenSubscriberCount": false,
     "videoCount": "3175"
    }
}

And you can take subscriberCount from result for each channel.

I know, this is not the way to sort your 50 results while writing into the file, but with this you can sort later your results by "subscriber count" while fetching from file for your autocomplete input.

I didn't find any other way to sort results by subscriber count, so maybe this can be helpful.

Answer 2

The idea to do is to run a server side script, that makes RESTful api calls in a loop, and writes the results to .JSON file, to save results. For that you can create PHP script, that makes REST API call to google, and fetch first 50 results, and then use file write operations to write your results. Run that PHP script as corn job to update results at regular intervals. Executing corn job at every specific time interval you set keeps results fresh.

Hit CURL command with loop for next, to fetches 50 results every time and create temp file with all the results saved in .JSON file. Once your results are fetched, replace your old JSON file with newly created temporary file. This will generate fresh JSON file are regular, with new results if any changes are made to data.

However, the idea to use temporary file is to avoid script avoid wait/slow of AJAX down due to consistent read and write operations on same file. Once temporary file is written, simply use move command to replace the actual file.

Make sure, you use cache control headers in AJAX results to keep its freshness of data.