convert pandas dataframe of multiple columns with NaN to a nested dictionary

Question

                         object_id                  time_id                      class         x       y
0  3db53411-c23b-49ec-8635-adc4e3ee2895  5G21A6P01L4100029:1570754223950071         NaN       NaN      NaN
1  3cea3cdc-883e-48d7-83de-e485da2e085a  5G21A6P01L4100029:1570754223950071        PERSON   528.868  2191.747
2  fc87a12f-a76a-4273-a712-6f56afc042c6  5G21A6P01L4100029:1570754223950071          CAR   512.238  2192.744
3  4edb4e32-0345-4f85-a4b1-e60903368fed  5G21A6S09K40039EX:1565470602550590          NaN      NaN       NaN
4  cd68a1d0-2470-4096-adb1-201017aadc9e  5G21A6S09K40039EX:1565470602550590         PERSON -1305.968 -2423.231

I have a nested dictionary detections with the following schema

detections = defaultdict(dict)
detections[key:time_id][key:object_id] = {'class_text':... , 'x': ..., 'y': ...}

For the above dataframe detections would be:

detections[5G21A6P01L4100029:1570754223950071] = 
{
`3db53411-c23b-49ec-8635-adc4e3ee2895`: {},
'3cea3cdc-883e-48d7-83de-e485da2e085a': {'class_text': 'PERSON', 'x': 528.8, 'y': 2191.7}, 
'fc87a12f-a76a-4273-a712-6f56afc042c6': {'class_text': 'CAR', 'x': 512.2, 'y': 2192.7}}
}

detections["5G21A6S09K40039EX:1565470602550590"] = 
{
`4edb4e32-0345-4f85-a4b1-e60903368fed`: {},
'cd68a1d0-2470-4096-adb1-201017aadc9e': {'class_text': 'PERSON', 'x': -1305.968, 'y': -2423.23}
}

detections has an empty value when values of (class, x and y) are NaN, otherwise it has the corresponding values.

I appreciate any comment on how can I make detections without for loop over each row?

score 2 · Accepted Answer · answered Jul 30 '20 at 08:14

Use groupby on time_id and apply a custom merge function merge_dicts to merge the grouped dataframe into dictionaries according to predefined requirements:

def merge_dicts(s):
    s = s.set_index('object_id')[['class', 'x', 'y']]
    return s.agg(lambda x: {} if x.isna().all() else dict(**x), axis=1).to_dict()

detections = df.groupby('time_id').apply(merge_dicts).to_dict()

Result:

print(detections)

{
    '5G21A6P01L4100029: 1570754223950071': 
    { 
        '3db53411-c23b-49ec-8635-adc4e3ee2895': {},
        '3cea3cdc-883e-48d7-83de-e485da2e085a': {'class': 'PERSON', 'x': 528.868, 'y': 2191.7470000000003},
        'fc87a12f-a76a-4273-a712-6f56afc042c6': {'class': 'CAR', 'x': 512.238, 'y': 2192.744}
    },
    '5G21A6S09K40039EX: 1565470602550590': 
    {
        '4edb4e32-0345-4f85-a4b1-e60903368fed': {},
        'cd68a1d0-2470-4096-adb1-201017aadc9e': {'class': 'PERSON', 'x': -1305.968, 'y': -2423.231}
    }
}

convert pandas dataframe of multiple columns with NaN to a nested dictionary

1 Answers1