1

Is there any easier way to filter data in Wikidata and download a portion of claims?

For e.g., let us say that I want a list of all humans that are alive currently and have an active Twitter profile.

I would like to download a file containing their Q-ids, names and Twitter usernames (https://www.wikidata.org/wiki/Property:P2002).

I expect there to be hundreds of thousands of results, if not millions.

What is the best way to obtain this information?

I am not sure if by submitting a SPARQL query, one can collect results in a file.

I looked at MediaWiki API, but not sure if it allows accessing multiple entities in one go.

Thanks!

abhinavkulkarni
  • 2,284
  • 4
  • 36
  • 54
  • of course you can get the result of a SPARQL SELECT query as CSV for example. Just do it from commandline via curl or use a commandline API - indeed, the public service might have some limitations regarding resultset size per query and query execution time – UninformedUser Apr 14 '22 at 10:15

1 Answers1

1

Wikidata currently has around 190,000 Twitter IDs linked to people. You can easily get them all using the SPARQL Query Interface: Web Interface (with a LIMIT you can remove or increase). In the dropdown on the right, choose SPARQL Endpoint for the Direct Link (no limit, 35MB .csv).

But, in case you run into timeouts with more complicated queries, you can first try LIMIT and OFFSET, or one of:

Wikibase Dump Filter is a CLI tool that downloads the full wikidata dump but filters the stream as it comes in according to your needs. You can put very much the same thing together with some creative pipe|ing and it tends to work better than one would expect.

https://wdumps.toolforge.org wdumps.toolforge.org does more or less the same thing but on-premise, then allows you to download the filtered data.

The linked data interface also works rather well for "simple query, high volume" access needs. Example here gives all Twitter IDs (326,000+) and you can read it in pages as fast as you can generate get requests (set an appropriate Accept header to get json)

Matthias Winkelmann
  • 15,870
  • 7
  • 64
  • 76
  • Your queries are for all humans, not all alive. Here is a query that removes about 2000 results, through some common signs that they are not alive: https://w.wiki/54mC – Ainali Apr 18 '22 at 08:32