-1

I am trying to parallelize my code.

What I want it to do is calculate for every target all the shortest paths between the target and the rest of the networks nodes.

For instance, if I have a network of 10000 nodes and I have 150 targets, then for every target I want to calculate the 10000 shortest paths of the network.

I don't know if I can do a for loop like I am doing.

import osmnx as ox
import igraph as ig
import matplotlib.pyplot as plt
import pandas as pd
import networkx as nx
import numpy as np
import matplotlib as mpl
import random as rd
import math
import ast
from IPython.display import clear_output
from copy import copy
import matplotlib.cm as cm
import matplotlib.ticker as mtick
import multiprocessing as mp
ox.config(log_console=True, use_cache=True)

cities = ox.geocode_to_gdf(['Município de Lisboa', 'Município de Oeiras', 'Município da Amadora', 'Município de Loures', 'Município de Odivelas'])

whole_polygon = cities.unary_union #unary union of both geometries

G = ox.graph_from_polygon(whole_polygon, network_type='drive', simplify=True)
G_nx = nx.relabel.convert_node_labels_to_integers(G)

def shortest_path(G, origin, target):
    
    try:
        return ox.shortest_path(G_nx, origin, target, weight = 'length')
    except:
        return None

cpus = mp.cpu_count() -1

nodes = np.array(G.nodes())
n=150
dests = np.random.choice(nodes, size=n, replace=True) 


list_routes=[]
for value in dests:
    params = ((G, node, value) for node in nodes) 
    pool = mp.Pool(cpus)
    sma = pool.starmap_async(shortest_path, params)
    routes = sma.get()
    list_routes.append(routes)
    pool.close()
    pool.join()

What I should get is the list_routes which will have inside 150 lists.

I question the code I wrote because, without the cycle and for only one target the code takes about 14s to run. But when I put everything inside the cycle it takes a long time and I don't know if it is correct.

martineau
  • 119,623
  • 25
  • 170
  • 301
DPM
  • 845
  • 7
  • 33

1 Answers1

1

Don't use a loop. There's an example of using multiprocessing in route solving in this OSMnx usage example notebook https://github.com/gboeing/osmnx-examples/blob/v0.16.0/notebooks/02-routing-speed-time.ipynb

gboeing
  • 5,691
  • 2
  • 15
  • 41
  • Hello! I saw that example and tried to adapt it to my situation here. In this situation since I want 150 lists I thought I had to loop through all the values I want. I think it all comes down to the variable params. Basically for the length of the nodes I want to keep my target the same. After that change the target. That is why I did the cycle. Do you reckon there is a better way? – DPM Oct 26 '20 at 00:05