14

Not having any experience with couch and redis, this is becoming more than I can handle at this point.

The npm website allows you to search for packages by a keyword - https://npmjs.org/browse/keyword/awesome

However, it doen't provide any way of obtaining this information in json format - ideally, we could just do https://npmjs.org/browse/keyword/awesome.json but that is not the case :(

I know that the npm website is powered by couchdb and a local redis instance. The remote couchdb installation is http://registry.npmjs.org/ and powered by https://github.com/isaacs/npmjs.org

However, spending the day looking into this, I just cannot figure out how to get all packages of a particular keyword. Does anyone know how? Bonus points if you also explain the process that you went about finding out how to do it so I know for next time :)

balupton
  • 47,113
  • 32
  • 131
  • 182

3 Answers3

22

Great question!

This will give you what you're looking for about a specific module:

npm view request

To get what you want for all modules you can hit the URL: https://registry.npmjs.org/-/all/

After pouring through these two files:

  1. https://github.com/isaacs/npm/blob/master/lib/search.js
  2. https://github.com/isaacs/npm-www/blob/master/models/browse.js

I came to the following conclusions:

  1. I'm super surprised there's not a better way to do search without hitting couchdb directly.
  2. The command-line NPM client does search inside of node.js by sorting and filtering through the full results of that /all/ search listed above.
  3. The website doesn't even bother with real search as it pawns it to a search engine
  4. The search by keyword thing you want won't get the same results as command-line NPM. It's really limited in scope to the keyword attribute, other search options may be available through (see search.js above)
  5. The query is going to look really weird.

Try this: https://registry.npmjs.org/-/_view/byKeyword?startkey=["keyword"]&endkey=["keyword",{}]&group_level=3

Also, one quick note, this is the kind of question that would probably get answered in the node.js chat room or mailing list in about 4 seconds :)

Hope that helps.

Jamund Ferguson
  • 16,721
  • 3
  • 42
  • 50
  • YATAAAA! Thanks so much! Man, you're my hero right now. Btw, tried the IRC channel but no one said anything, didn't realise people used the mailing list - what's the benefit of that over stackoverflow? Again, thanks so much! – balupton Dec 01 '12 at 08:32
  • Is there a way to get the results to include the entire package.json? Happy to do that as a different post, guessing I have to query each one individual, not that much of a biggie. – balupton Dec 01 '12 at 08:38
  • Definitely try the node mailing list. The people that respond in that list are the main node programmers and I doubt many if them come on stackoverflow – Jamund Ferguson Dec 01 '12 at 16:10
  • 2
    @balupton once you know the name of the module you can hit this URL to get the details for it http://registry.npmjs.org/wordfreq. Don't ask me how it works :) I know you're looking for a single query though....I'll keep looking around. – Jamund Ferguson Dec 01 '12 at 22:38
  • 1
    Is there a way to get a list of all packages for a given user? Basically, the registry equivalent of this page: https://www.npmjs.com/~substack – James Messinger Jul 14 '15 at 17:50
  • @JamundFerguson Man... This is treasure. You saved me a tons of time, and a lots of pulled out hair. Thanks! – Balázs Édes Dec 23 '15 at 21:32
  • @JamundFerguson I've tried modifying this to just search based on single key strings but I keep getting empty results `https://registry.npmjs.org/-/_view/byKeyword?key=["keyword"]&limit=6&group_level=3` or `https://registry.npmjs.org/-/_view/byKeyword?key="keyword"&limit=6&group_level=3`. What am I missing? – Pirijan Jan 26 '16 at 05:28
  • also, if I try and replace the `keyword` strings in your answer with anything else I know returns results on npm, I also get back an empty object `Try this: https://registry.npmjs.org/-/_view/byKeyword?startkey=["gulp"]&endkey=["gulp",{}]&group_level=3 ` – Pirijan Jan 26 '16 at 05:33
  • I have the same question as @JamesMessinger . Does someone know what the API call that will fetch me something like npmjs.com/~author – balajikris Oct 27 '16 at 18:56
  • Seems npm have nuked this technique. Anyone have a workaround? – balupton Jul 11 '18 at 10:40
  • Asked on the npm forum: https://npm.community/t/how-do-i-fetch-all-npm-packages-that-contain-a-given-keyword/400?u=balupton – balupton Jul 11 '18 at 10:46
2

Based on this answer I wrote a small lib for node, https://github.com/wires/npm-keywordsearch.

npm install npm-keywordsearch

then

var search = require('npm-keywordsearch')

search('my-plugin', function (error, packages) {
  packages.forEach(function (pkg) {
    console.log(pkg.name + ': ' + pkg.description)
  })
})

Maybe useful for you

wires
  • 4,718
  • 2
  • 35
  • 31
  • Love the module.. wish it checked my `npm config` for things like `registry`, though... Will create an issue... – Alex Gray May 16 '16 at 09:37
  • great! if you can describe what it should do and maybe refer to some docs I will see what I can do – wires May 16 '16 at 23:18
1

Sometime in 2018, npm retired the /-/_view/byKeyword API:

https://registry.npmjs.org/-/_view/byKeyword?startkey=%5B%22docpad-plugin%22%5D&endkey=%5B%22docpad-plugin%22,%7B%7D%5D&group_level=2

The new API is now: /-/v1/search:

https://registry.npmjs.org/-/v1/search?text=keywords:docpad-plugin&size=250

Documentation for it is here:

https://github.com/npm/registry/blob/master/docs/REGISTRY-API.md#get-v1search

I've updated the pluginclerk package accordingly. It is a node package that provides an API to hit that call, while providing caching, as well as dependency compatibility resolution.

balupton
  • 47,113
  • 32
  • 131
  • 182