KeyError(Key) when using append with defaultdict

Question

I am getting the following error when I am trying to append to a dictionary using defaultdict(list). From my understanding, defaultdict is suppose to prevent a keyerror.

    raise KeyError(key) from err

KeyError: 'id'

The following is my code:

weather_data = defaultdict(list)
m = len(_ids)
date = str(date.today())
i = 0

while i < m:

url = ("https://api.openweathermap.org/data/2.5/weather?id=%s&units=%s&appid=%s") % 
(_ids.loc[i], 'imperial', weather_key)
payload = r.get(url).json()
payload_from_json = pd.json_normalize(payload)

weather_data[date].append(date)
weather_data['id'].append(payload_from_json['id'])
weather_data['weather'].append(payload_from_json['weather'])
weather_data['base'].append(payload_from_json['base'])
weather_data['visibility'].append(payload_from_json['visibility'])
weather_data['dt'].append(payload_from_json['dt'])
weather_data['name'].append(payload_from_json['name'])
weather_data['cod'].append(payload_from_json['cod'])
weather_data['coord.lon'].append(payload_from_json['coord.lon'])
weather_data['coord.lat'].append(payload_from_json['coord.lat'])
weather_data['main.temp'].append(payload_from_json['main.temp'])
weather_data['main.feels_like'].append(payload_from_json['main.feels_like'])
weather_data['main.temp_min'].append(payload_from_json['main.temp_min'])
weather_data['main.temp_max'].append(payload_from_json['main.temp_max'])
weather_data['main.pressure'].append(payload_from_json['main.pressure'])
weather_data['main.humidity'].append(payload_from_json['main.humidity'])
weather_data['wind.speed'].append(payload_from_json['wind.speed'])
weather_data['wind.deg'].append(payload_from_json['wind.deg'])
weather_data['clouds.all'].append(payload_from_json['clouds.all'])
weather_data['sys.type'].append(payload_from_json['sys.type'])
weather_data['sys.id'].append(payload_from_json['sys.id'])
weather_data['sys.country'].append(payload_from_json['sys.country'])
weather_data['sys.sunrise'].append(payload_from_json['sys.sunrise'])
weather_data['sys.sunset'].append(payload_from_json['sys.sunset'])

i = i + 1

print(weather_data)

Here is the traceback error - can someone tell me how to interpret this:

   Traceback (most recent call last):
  File "/opt/anaconda3/lib/python3.8/site-packages/pandas/core/indexes/base.py", line 2895, in get_loc
    return self._engine.get_loc(casted_key)
  File "pandas/_libs/index.pyx", line 70, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 101, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 1675, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 1683, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'coord.lon'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "main.py", line 40, in <module>
    weather_data['coord.lon'].append(payload_from_json['coord.lon'])
  File "/opt/anaconda3/lib/python3.8/site-packages/pandas/core/frame.py", line 2902, in __getitem__
    indexer = self.columns.get_loc(key)
  File "/opt/anaconda3/lib/python3.8/site-packages/pandas/core/indexes/base.py", line 2897, in get_loc
    raise KeyError(key) from err

I'm going to guess that the error is actually being caused by `payload_from_json['id']`, not `weather_data['id'].`. Double check your data. — Carcigenicate, Dec 28 '20 at 16:27
The error should come from `payload_from_json['id']`, print your data, and see what are the keys — azro, Dec 28 '20 at 16:28
@Carcigenicate is probably right. Check to see if your payload has an "id" key. — Jack Moody, Dec 28 '20 at 16:33
And this, folks, is why you need to post the full traceback when asking about errors... — Tomerikoo, Dec 28 '20 at 16:34
I definitely think you should use a for loop for the whole last part of the code. — Jacob Lee, Dec 28 '20 at 16:36

ljuk · Answer 1 · 2020-12-28T20:09:37.773

[EDIT] your weather_data is your default dict, but payload_from_json not. So your error was raised by payload_from_json.

You can fix this by using get to access the key:

weather_data['id'].append(payload_from_json.get('id'))

if you dont want to include junk data, you can add some verifications before append:

if payload_from_json.get('id') is not None:
         weather_data['id'].append(payload_from_json.get('id'))

Also, you can add some default value like this:

weather_data['id'].append(payload_from_json.get('id', 'missing'))

or

weather_data['id'].append(payload_from_json.get('id', ''))

or by default:

weather_data['id'].append(payload_from_json.get('id', None))

In your specific problem, this should work:

weather_data = defaultdict(list)
m = len(_ids)
date = str(date.today())
i = 0

while i < m:

url = ("https://api.openweathermap.org/data/2.5/weather?id=%s&units=%s&appid=%s") % 
(_ids.loc[i], 'imperial', weather_key)
payload = r.get(url).json()
payload_from_json = pd.json_normalize(payload)

weather_data[date].append(date)
weather_data['id'].append(payload_from_json.get('id'))
weather_data['weather'].append(payload_from_json.get('weather'))
weather_data['base'].append(payload_from_json.get('base'))
weather_data['visibility'].append(payload_from_json.get('visibility'))
weather_data['dt'].append(payload_from_json.get('dt'))
weather_data['name'].append(payload_from_json.get('name'))
weather_data['cod'].append(payload_from_json.get('cod'))
weather_data['coord.lon'].append(payload_from_json.get('coord.lon'))
weather_data['coord.lat'].append(payload_from_json.get('coord.lat'))
weather_data['main.temp'].append(payload_from_json.get('main.temp'))
weather_data['main.feels_like'].append(payload_from_json.get('main.feels_like'))
weather_data['main.temp_min'].append(payload_from_json.get('main.temp_min'))
weather_data['main.temp_max'].append(payload_from_json.get('main.temp_max'))
weather_data['main.pressure'].append(payload_from_json.get('main.pressure'))
weather_data['main.humidity'].append(payload_from_json.get('main.humidity'))
weather_data['wind.speed'].append(payload_from_json.get('wind.speed'))
weather_data['wind.deg'].append(payload_from_json.get('wind.deg'))
weather_data['clouds.all'].append(payload_from_json.get('clouds.all'))
weather_data['sys.type'].append(payload_from_json.get('sys.type'))
weather_data['sys.id'].append(payload_from_json.get('sys.id'))
weather_data['sys.country'].append(payload_from_json.get('sys.country'))
weather_data['sys.sunrise'].append(payload_from_json.get('sys.sunrise'))
weather_data['sys.sunset'].append(payload_from_json.get('sys.sunset'))

i += 1

Here is an example of the traceback error: Since I am new to programming, can someone tell me how to read this - I guess I was reading it incorrectly: — pandoo, Dec 28 '20 at 18:06

KeyError(Key) when using append with defaultdict

1 Answers1