Socrata SODA and Python

Question

Socrata confirms there is no official Python API. I would like to know if there is some reason for this that would make it worthwhile to learn one of the languages which does have official API support, ie is Python (and associated packages like Pandas) not the best choice for data wrangling.

Specific things I'd like to do include looking at sites like data.lacity.gov Whatever databases I look at on the browser dont let me set dates or other parameters. I get a few months of results where I'd expect years or decades. That led to look for the API and to the lack of Python support apart from an emerging fork of a deprecated project. I'm assuming that the APIs allow this higher visibility, of course. ..

The Old Guy in the Club

score 6 · Accepted Answer · answered Nov 04 '14 at 22:05

For a little bit more background about why we don't have a Python library yet, you might want to check out this thread on Github.

The short version is - it's not because there is anything in particular wrong with Python or because we have any dislike of it (I think it's a great language), it's more that we just don't have the in-house expertise to build one right now. I'd love to write one but frankly I'm more of a Rubyist than a Pythonista so I'd probably mess it up in some way that would make everybody unhappy. :)

That said, it is pretty easy to use the Socrata API directly using something like Requests or urllib2. For example, here's how to make a simple call using the Requests library against the [data.lacity.org Building Inspections dataset]:

import requests
r = requests.get(
    "https://data.lacity.org/resource/9w5z-rg2h.json?$where=within_circle(lat_lon, 34.053714, -118.242653, 1000)", 
    headers={"X-App-Token":"[YOUR APP TOKEN]"}
)
r.json()[0] # {u'permit_status': u'Permit Finaled', u'inspection_result': u'Partial Inspection', u'lat_lon': {u'latitude': u'34.04866', u'needs_recoding': False, u'longitude': u'-118.23787'}, u'address': u'100 S ALAMEDA ST', u'inspection': u'Excavation/Setback/Form/Re-Bar', u'inspection_date': u'2013-08-27T00:00:00', u'permit': u'13016-30000-09747'}

You'll need to structure your own SoQL query to use with Requests, but there are lots of resources on how to do so on dev.socrata.com.

Because I was curious I also wrote up a quick example of how to use the SODA API with Python, Pandas, and Bokeh to visualize some data. It might be up your alley: http://dev.socrata.com/consumers/examples/data-visualization-with-python.html — chrismetcalf, Nov 05 '14 at 03:16
Very good pointers and I appreciate the clarification. I'll look into the Python options. — , Nov 05 '14 at 16:24

score 1 · Answer 2 · answered Aug 29 '18 at 22:00

For anyone stumbling across this question in the current era, sodapy is referenced in some of the Socrata documentation as the 'unoffical Python API'. There is a bit lacking on the publishing side, and the Discovery API is unsupported, but gathering known datasets is quick and smooth.

score 0 · Answer 3 · answered Feb 05 '15 at 07:24

I don't know if this qualifies as an answer, but I recently wrote some Python to interact with SODA and (after going through the same initial confusion) found that you really don't need a "Python API". Here's my code: https://github.com/stevage/meshlium-soda

Samples:

headers = {
  'X-App-Token': config.app_token,
  'Content-Type': 'application/json'
  }

...

params = {#'$select': 'max(timestamp)',
            '$limit': 1, 
            '$order': 'timestamp DESC',
            '$where': 'timestamp IS NOT NULL'}
  r = requests.get(config.dataset + '.json', params=params, headers=headers, auth=config.auth)
  if r.status_code != 200:
    raise RuntimeError('Can''t retrieve latest timestamp.' + r.text)

...

r = requests.post(config.dataset, data=simplejson.dumps(rows), headers = headers, auth=config.auth)
  j = r.json()
  print
  if r.status_code != 200:
    raise RuntimeError ( "%d Socrata error: %s" % (r.status_code, j['message']))
  return j

I definitely recommend the Requests library.

Socrata SODA and Python

3 Answers3