How do you parse json data that is under multiple layers?

Question

The goal of the code is take json information I am getting and parsing it into its raw location data( address,postal,etc). I am pretty new to coding and this is a one off task I got stuck with for a school project as I studying Geography and need the locations of all McDonald in Canada.So any other learning tools are welcomed. However the main issue I am having is that I want to write

for blank in blanks['']:

so I can grab the data for a csv output. However I have noticed my data is under multiply layers. For example:

{
    "features": [
        {
            "geometry": {
                "coordinates": [
                    -79.28662,
                    43.68758
                ]
            },
            "properties": {
                "name": "Vic Park/Gerrard",
                "shortDescription": "VIC PARK/G",
                "longDescription": "VIC PARK/GERRARD",
                "todayHours": "06:00 - 22:00",
                "driveTodayHours": "00:00 - 00:00",
                "id": "195500517230-en-ca",
                "filterType": [
                    "ALL_DAY_BREAKFAST",
                    "BAKERY",
                    "BREAKFAST",
                    "CYT",
                    "DRIVETHRU",
                    "INDOORDINING",
                    "MCCAFE",
                    "MOBILEOFFERS",
                    "MOBILEORDERS",
                    "PARKINGAREA",
                    "TWENTYFOURHOURS",
                    "WIFI"
                ],
                "addressLine1": "2480 GERRARD STREET EAST",
                "addressLine2": "",
                "addressLine3": "SCARBOROUGH",
                "addressLine4": "Canada",
                "subDivision": "",
                "postcode": "M1N 4C3",
                "customAddress": "SCARBOROUGH, M1N 4C3",
                "telephone": "4166903659",

The information I want is under properties to what it looks to me (not sure) but my

for store in stores['features']:

statement. Does Not allow me to grab the 'addressLine1' information or the other information individually for the csv. I am wondering if anyone has a solution to parsing through data such as this.

P.S I included my whole code just in case there is a deeper problem.

import requests
import csv
import json

url = "https://www.mcdonalds.com/googleapps/GoogleRestaurantLocAction.do?method=searchLocation&latitude=43.6936965&longitude=-79.2969938&radius=1000000&maxResults=1700&country=ca&language=en-ca&showClosed=&hours24Text=Open%2024%20hr"

payload={}
files={}
headers = {
  'authority': 'www.mcdonalds.com',
  'sec-ch-ua': '" Not;A Brand";v="99", "Google Chrome";v="91", "Chromium";v="91"',
  'accept': '*/*',
  'x-requested-with': 'XMLHttpRequest',
  'sec-ch-ua-mobile': '?0',
  'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.106 Safari/537.36',
  'sec-fetch-site': 'same-origin',
  'sec-fetch-mode': 'cors',
  'sec-fetch-dest': 'empty',
  'referer': 'https://www.mcdonalds.com/ca/en-ca/restaurant-locator.html',
  'accept-language': 'en-GB,en-US;q=0.9,en;q=0.8',
  'cookie': 'bm_sz=C04645E7F7A956C5F9D9C5A20DEAEC97~YAAQ1Cv2SEtfMBN6AQAAItxfEwwTVV2V2Tr7UWpPt1Ps7gl84FzQlmbWIm4kBBh5dxlK3w8RenwiEiKtvERE6dLmrwPwJUuy+14gU/LeEZvP+uxzyBr04oQXdcSEQuiOgdkAGasqnBrTw1mp5E5iehnRpvHBDdSqh8wRSgJV0eG4f8YwSz66BfntCBALtQNCAFK2; _abck=F05779F2345218EA4989FF467D897C5A~0~YAAQ1Cv2SExfMBN6AQAAItxfEwaIwCrBeP25JBhBb7TX+HmnLQgrj1TkosrB+oHSv9ctrxRukqEDUaHPL1KkjpqjY1XY1yyulQ0ZRhsEfhY968YVsTOqfiosAu3kykd3pJG/bQ37XHwWs5qXpIdhMXRwJwXmkYtl3ETG8kXK2iZ22Q31COaSjNVACLaa7s9tCk9ItgLvUj5x9Nldjnd8AdXR0pXicrQY1IaruJyNqwMcJv42AUHW7iH4Ex9ZOSYsgEjLMNd44mS525X/gSNUTSOzoqoWsnH4MU59vfgLTwc2hVncAv67LBViTLxbWw4eVAvz7Z5phQfCmvoIy0PD8gy5iwPDMaD3GASrK9xScDPAPUI2wquxmSJ+f2cQaxZQKhvJCeH9cz14OZfx8ksA2ss53E0l0kDvgmnw~-1~-1~-1; ak_bmsc=BA4817D8DEE20E92C1E6251C54FC124348F62BD48F5F00005F91C9608B679D5F~plUkbYfsvYr5dCayJ9dMGEJ3QDgkmkv2mLpE7pCY9vW0xrdawvmyxfSnupw/4F7C48Akdn8PKsBniqz+7F+RZb8v4AkvH3c0RuvnynqJoni+kJcDYtPOxdMvdtGdTlZGIkSQNfpcxHNQDVlzojdSBX0vyBh/8seKQv10U67M7m787olYzg9jnsUwk3/VHBrnMDogiWJT8rNV7saSXunN0pAgucZWo/XhCpTJL+tI9urt0=; MCDCountry_code=US; bm_mi=BEE06312635FD442995BC0237BAFDA7C~f/RxgMW/JJSUc/wB9ZRg9fPD/76+wq/TaoWEZR1/ttrAiVTO256xhDTsVYc/kdHIjWkxvfO4XDcBjqe4hQ4qXt8Anpfi09vna/zcC7l6OVWpWeRSoZNztl7h5VF407L3XG+9CpzjSHNcaqAPRk5d0J5gLMtL/KmR8XBkAC0Syim7ST97nxNrPfLdlkSPMGm4Oy86xvY5PH5Nu47zS/gwhanBFg69tAdrQdaZewE2eGuzoJPsZit3UsihTzhXc4LY92hfSdh3/kZRId+NE8Jp0w==; bm_sv=7CACE3495320A7C0A6CF8F41DFE0EB36~F9KzvznVNk/fE4+ijLD5H/szY7O161rWlemmShElumIW7HN49Gq2d9Sd2tqBjCa9sJOX4zoehAkc8WvsID5Idon/hDlDeLJZuqnEmff4PN4a9yst3R170rBCm1egzGvCBmB1jq9aCwQm5VgIJgloPOdpiIPfD3kDxFbKhqMuS5U=; JSESSIONID=64PZkBXhhpvNjM4NganzSZ0r1npIIaM7Fo84EsxN.eap7node7; _abck=F05779F2345218EA4989FF467D897C5A~-1~YAAQ1Cv2SExyMBN6AQAA5Et0EwZueCejZbKz1VDGCq2sB43Yx4dq0SiiGeUS6gVpXRIdw3rA3OdpNGHq7tVzQ+IvPpEKwLML9736x1qB5SQxV3jai89y2B2QF6K8nKtyrDAes0qbeTyIrHu0Rh1HLs7CjNxiLi0wswbCZfSsPI6fJZiEt+Itre3lfmua/HkhIRwpVTKqlVN5eQ8XIX+s1jJbINx/jUmMTW+jB5k4A5NARGChYH7rJQGYIT/oyZYpSbS3Yweqa4FRgGMW4gYZBN39+t2xSfewADLdpihfOnoZtakw9VhcvAKaf4mEzjB7WEfNJIZSjSE8DzvbJNIF41MGuAhhrnEBwBE8uVCZsA+2qjVPSADVp2Nn8JanJXCbucnLFOLsmPz3oVtGzentht1cHog4+eYOUlmw~0~-1~-1; bm_sv=7CACE3495320A7C0A6CF8F41DFE0EB36~F9KzvznVNk/fE4+ijLD5H/szY7O161rWlemmShElumIW7HN49Gq2d9Sd2tqBjCa9sJOX4zoehAkc8WvsID5Idon/hDlDeLJZuqnEmff4PN5ZCTzA250oKEeVeXaa6j4gEGJ9RRtrTXQdYXzzSx6fM9aLwif+We2vtIc1yLQgTt4=',
  'dnt': '1'
} 

response = requests.request("GET", url, headers = headers, data = payload, files = files)


stores = json.loads(response.text)

with open('Mcdonlocation.csv', mode='w') as CSVFile:
    writer = csv.writer(CSVFile, delimiter=",", quotechar='"', quoting=csv.QUOTE_MINIMAL)

    writer.writerow([
        "addressLine1",
        "addressLine2",
        "addressLine3",
        "subDivision",
        "postcode",
        "telephone"
        ])

    for store in stores['features']:
        row = []
        Match_Address1= store['properties']["addressLine1"]
        Match_Address2= store['properties']["addressLine2"]
        Match_Address3= store['properties']["addressLine3"]
        subDivision= store['properties']["subDivision"]
        Postalcode= store['properties']["postcode"]
        telephone= store['properties']["telephone"]

        row.append(Match_Address1)
        row.append(Match_Address2)
        row.append(Match_Address3)
        row.append(subDivision)
        row.append(Postalcode)
        row.append(telephone)
        writer.writerow(row)

store["addressLine1"] -> store['properties']["addressLine1"] and so on — Yuri, Jun 17 '21 at 09:39
@yuri thanks that worked!, However It isnt letting me get the phone number. Any idea on why it is like that ? — John_Muir, Jun 17 '21 at 16:32
The error I am receiving now is::: telephone = store['properties']["telephone"] KeyError: 'telephone' — John_Muir, Jun 17 '21 at 17:23
I don't know, John. `stores['features']` is list of dictionaries. The first dictionary, the one you show, definitely has 'telephone'. Maybe the 2nd or later one doesn't? — James K. Lowden, Jun 17 '21 at 21:27
@John_Muir maybe it is missing the telephone entry. Check if it exists with "if 'telephone' in store['properties']:" — Yuri, Jun 21 '21 at 08:47

score 1 · Accepted Answer · answered Jun 17 '21 at 17:20

I think the fundamental answer to your quesions is, "look at the type". The Python json conversion table tells you what to expect for each type. Let's load your file and see what we have, according to the Python interpreter:

>>> input = dat.read()
>>> stores = json.loads(input)
>>> type(stores)
<class 'dict'>
>>> type(stores['features'])
<class 'list'>
>>> type( stores['features'][0] )
<class 'dict'>
>>> type( stores['features'][0]['properties'] )
<class 'dict'>
>>> type( stores['features'][0]['properties']['telephone'] )
<class 'str'>
>>> stores['features'][0]['properties']['telephone']
'4166903659'

Every object has a type; every type has methods. Just work your way down the list.

So I updated the statement and everything is being moved over to excel except telephones. I get error telephone = store['properties']['telephone'] KeyError: 'telephone'. Am I getting this error because I just did use the right method for type? Also, You view the change in original post now if it helps answer my question. — John_Muir, Jun 17 '21 at 21:18

score 0 · Answer 2 · answered Jun 17 '21 at 17:34

It looks like your data is structured like this:

features:
- geometry:
    coordinates:
    - float
    - float
  properties:
    addressLine1: str
    addressLine2: str
    addressLine3: str
    addressLine4: str
    customAddress: str
    driveTodayHours: str
    filterType:
    - str
    - ...
    id: str
    longDescription: str
    name: str
    postcode: str
    shortDescription: str
    subDivision: str
    telephone: str
    todayHours: str

You want to pull out information from the individual "feature" elements, which appear to be stores, so you're off to a good start with code like your existing logic:

for store in data['features']:
   csv_row = process_store(store)

From there you just need to decide what you want to pull out of the info, e.g.:

coord_1 = store['coordinates'][0]
custom_address = store['properties']['customAddress']
...

. Compared to your existing code I think you just didn't notice there is a properties attribute to access beyond the initial features level.

So now I am getting everything except the phone number, I get the error code telephone = store['properties']['telephone'] KeyError: 'telephone. I updated my original post to show the change I made. Is the error I am getting because I just didn't find the right path to it or some other issue? — John_Muir, Jun 17 '21 at 21:23
At least in the data you provided, telephone is *sometimes* there, but perhaps it is not *always* there. Try `.get('telephone')` instead of `['telephone']` — Cireo, Jun 17 '21 at 21:40

How do you parse json data that is under multiple layers?

2 Answers2