0

Im trying to pull financial data from the polygon.io API to organize it and later inject it into an Azure database. In order to accomplish this I have been using the "petl" python package but I am having issues starting with an example table because of an invalid source argument that I believe to be caused by how I am retrieving and writing the JSON result.

import petl as etl
import websocket
import json
import requests

r =requests.get("https://api.polygon.io/v1/historic/forex/AUD/USD/2018-2-2?limit=1&apiKey=*********") 
#gets data includes plenty of responses that arnt text
data = json.loads(r.text)

table1 = etl.fromjson(data, header=['day','map','msLatency','pair','status','ticks','type'])
print(table1)

I expected a table with the column headers being the ones listed with their values underneath but instead received this error message: "AssertionError: invalid source argument, expected None or a string or an object implementing open()"

When printing the data from the API it looks like this: 

{'day': '2018-2-2', 'map': {'a': 'ask', 'b': 'bid', 't': 'timestamp'}, 'msLatency': 1, 'pair': 'AUD/USD', 'status': 'success', 'ticks': [{'b': 0.80392, 'a': 0.80392, 'x': 0, 't': 1517529600225}], 'type': 'forex'}

I have tried converting the data or using the api call line as the data argument but have had little success

jknotek
  • 1,778
  • 2
  • 15
  • 23
ADHDTurtle
  • 3
  • 2
  • It's telling you that is expects `None` or `a string` or `an object ...`, so what you get from `r.text` (better use `r.content.decode()`) is a string, so no need to use `json.loads`, try calling: `table1 = etl.fromjson(r.content.decode(), ...)` – c0x6a Jun 11 '19 at 20:46

1 Answers1

0

The petl.fromjson call expects a string, (the path to a local file), or None, (stdin). You have pulled the JSON (text) from the URL and converted it to a Python dictionary. So you should be thinking in terms of dictionaries, not the JSON format of the source. I can see several possible solutions:

  • Don't run json.load, just write the response to a temporary file, then call petl.fromjson on the file.
  • Don't use petl.fromjson, since your data is no longer JSON, use petl.fromdicts, which can load from dictionaries. (In fact, inside the code, petl.fromjson essentially calls json.load, then calls petl.fromdicts.
  • If you work with a lot of data, then you might want to wrap the request in an io Helper so it pulls from the URL while reading and passing to further PETL functions.
swstephe
  • 1,840
  • 11
  • 17