Proper way to iterate Python dictionary and build new one

Question

I'm trying to find a cogent way to check to see whether certain keys exist in a dictionary and use those to build a new one.

Here is my example json:

 "dmarc": {
    "record": "v=DMARC1; p=none; rua=mailto:dmarc.spc@test.domain; adkim=s; aspf=s",
    "valid": true,
    "location": "test.domain",
    "warnings": [
                "DMARC record at root of test.domain has no effect"
                ],
     "tags": {
            "v": {
                   "value": "DMARC1",
                   "explicit": true
                 },
            "p": {
                   "value": "none",
                   "explicit": true
                 },
            "rua": {
                   "value": [
                          {
                           "scheme": "mailto",
                           "address": "ssc.dmarc.spc@canada.ca",
                           "size_limit": null
                          }
                         ],
                   "explicit": true
                   },
            "adkim": {
                   "value": "s",
                   "explicit": true
                     },
            "aspf": {
                   "value": "s",
                   "explicit": true
                   },
            "fo": {
                   "value": [
                             "0"
                            ],
                   "explicit": false
                  },
           "pct": {
                   "value": 100,
                   "explicit": false
                  },
           "rf": {
                  "value": [
                            "afrf"
                           ],
                  "explicit": false
                 },
           "ri": {
                  "value": 86400,
                  "explicit": false
                 },
           "sp": {
                  "value": "none",
                  "explicit": false
                 }
             }
       }
 }

What I'm specifically looking to do, is pull record, valid, location, tags-p, tags-sp, and tags-pct in a programmatic way, instead of doing a bunch of try/excepts. For example, to get valid, I do:

try:
    res_dict['valid'] = jsonData['valid']
except KeyError:
    res_dict['valid'] = None

Now, this is easy enough to loop/repeat for top level key/values, but how would I accomplish this for the nested key/values?

score 2 · Answer 1 · answered Nov 17 '19 at 21:15

No, you don't need a try-except block for the same. You can check if the key exists using:

if jsonData.get("valid"):
    res_dict["valid"] = jsonData.get("valid")

The .get("key") method returns the value for the given key, if present in the dictionary. If not, then it will return None (if get() is used with only one argument).

If you want it to return something else if it doesn't find the key then suppose:

jsonData.get("valid", "invalid_something_else")

Walter Tross · Answer 2 · 2019-11-18T07:51:06.980

Simple: instead of dict['key'] use

dict.get('key', {}) for all nodes that are not leaves, and
dict.get('key', DEFAULT) for leaves, where DEFAULT is whatever you need.

If you omit DEFAULT and 'key' is absent, you get None. See the docs.

E.g.:

jsonData.get('record', "")    # empty string if no 'record' key
jsonData.get('valid', False)  # False if no 'valid' key
jsonData.get('location')      # None if no 'location'
jsonData.get('tags', {}).get('p')  # None if no 'tags' and/or no 'p'
jsonData.get('tags', {}).get('p', {})  # {} if no 'tags' and/or no 'p'
jsonData.get('tags', {}).get('p', {}).get('explicit', False)  # and so on

The above presumes that you don't traverse lists (JSON arrays). If you do, you can still use

dict.get('key', [])

but if you have to dive deeper from there, you will probably have to loop over list items.

Thanks for your input on this. – tparrott Nov 17 '19 at 21:50 — tparrott, Nov 17 '19 at 21:50

James Mchugh · Accepted Answer · 2019-11-17T21:29:21.850

One way of handling this is by taking advantage of the fact that the result of dict.keys can be treated as a set. See the following code.

my_keys = {'record', 'valid', 'location'} # you can add more here
new_dict = {}
available_keys = my_keys & jsonData.keys()
for key in available_keys:
    new_dict[key] = jsonData[key]

Above, we define the keys we are interested in within the my_keys set. We then get the available keys by taking the intersection of the keys in the dictionary and the keys we are interested in. This, in effect, only gets the keys that we are interested in that are also defined in the dictionary. Finally, we just iterate through the available_keys and build the new dictionary.

However, this does not set keys to None if they do not exist in the input dictionary. For that, it may be best to use the get method as mentioned in other answers, like so:

my_keys = ['record', 'valid', 'location'] # you can add more here
new_dict = {}
for key in my_keys:
    new_dict[key] = jsonData.get(key)

The get method allows us to attempt to get the value for a key in the dictionary. If that key is not defined, it returns None. You can also change the returned default by adding an extra argument to the get method like so new_dict[key] = jsonData.get(key, "some other default value")

Proper way to iterate Python dictionary and build new one

3 Answers3