networkX distance matrix generator to pandas DataFrame

Question

I downloaded graph via osmnx and I want to get distance between all the nodes. I call nx.all_pairs_dijkstra_path_length(G, weight='length') which returns generator of dicts.I then store it into DataFrame and then to .csv

How to cope with huge graphs (Stockholm, DC, Paris, etc.)?

I did this, which worked up to size of Amsterdam:

G = ox.graph_from_place('Amsterdam, Netherlands', network_type='drive')
skim_generator = nx.all_pairs_dijkstra_path_length(inData.G,weight='length') # this is generator of dicts
skim_dict = dict(inData.skim_generator) 
skim = pd.DataFrame(inData.skim_dict).fillna(_params.dist_threshold).T.astype(int) 
skim.to_csv(_params.paths.skim, chunksize=20000)

Yet now it kind of destroyed my RAM with bigger networks. So I try to tweak it and fit into memory, yet it is painstackingly slow, how to improve this?

    ret = dict()
    first = True
    j=0
    for i in nx.all_pairs_dijkstra_path_length(_inData.G, weight='length'):
        ret[i[0]]=dict(i[1])
        j+=1
        if divmod(j,chunk)[1]==0:  
            print(j,_inData.nodes.shape[0])
            df = pd.DataFrame(ret).reindex(_inData.nodes.index).fillna(999999).astype(int)
            df.T.to_csv(path, mode = 'w' if first else 'a', header = first)
            first = False
            ret = dict()

You could directly iterate over the generator of `all_pairs_dijkstra_path_length` and write it without `pandas` to the file. However, have you tried to analyse, which is the slow part of your code? The writing with `pandas` or retrieving the shortest path length? — Sparky05, Apr 07 '20 at 11:15
I think both are slow for huge networks @Sparky05, but I can speed up only the pandas part. How can I iterate over the generator to file? — Intelligent-Infrastructure, Apr 07 '20 at 13:37
@Intelligent-Infrastructure in general for this kind of intensive task, I recommend creating your OSMnx graph then dumping it to cugraph for faster GPU processing. — gboeing, Mar 29 '21 at 23:11
@gboeing - thanks for digging this old question. What do you mean `curgraph` is it some new feauture in OSMNX or external lib? — Intelligent-Infrastructure, Apr 01 '21 at 11:17

networkX distance matrix generator to pandas DataFrame

0 Answers0