7

I have an implementation of Dijkstra's Algorithm, based on the code on this website. Basically, I have a number of nodes (say 10000), and each node can have 1 to 3 connections to other nodes.

The nodes are generated randomly within a 3d space. The connections are also randomly generated, however it always tries to find connections with it's closest neighbors first and slowly increases the search radius. Each connection is given a distance of one. (I doubt any of this matters but it's just background).

In this case then, the algorithm is just being used to find the shortest number of hops from the starting point to all the other nodes. And it works well for 10,000 nodes. The problem I have is that, as the number of nodes increases, say towards 2 million, I use up all of my computers memory when trying to build the graph.

Does anyone know of an alternative way of implementing the algorithm to reduce the memory footprint, or is there another algorithm out there that uses less memory?

Rubens
  • 14,478
  • 11
  • 63
  • 92
John
  • 536
  • 5
  • 16
  • 2
    Since Dijkstra itself scales linearly with the number of nodes I suppose it is the graph representation itself which costs so much memory. Which data structure do you use to represent the graph? – Howard Dec 26 '12 at 11:45
  • More importantly, what sort of architecture are you running this on? A PC with a couple of GB of ram should be able to store A LOT more than 2M nodes, unless each node is taking up several hundred bytes each - at which point you probably want to reconsider your node information. – Mats Petersson Dec 26 '12 at 11:54
  • you may have memory leaks in building the graph phase. can you post your code? – emreakyilmaz Dec 26 '12 at 11:57
  • @emreakyilmaz , the code is pretty much identical to the code in the website in my first post. – John Dec 26 '12 at 13:14
  • @ Mats Petersson : My PC has 4 GB of memory!. As emreaky says, perhaps there is a memory leak. Looking at the code in the link I posted in the question, would it be capable of working with a graph of 2Million nodes? This for instance, long dist[GRAPHSIZE][GRAPHSIZE], wouldn't this would take a huge amount of memory with 2 million nodes? – John Dec 26 '12 at 13:17
  • @John, it looks like you figured it out yourself - look at interjay's answer for details on how to get rid of this array. – Ivan Vergiliev Dec 26 '12 at 13:43

3 Answers3

8

According to your comment above, you are representing the edges of the graph with a distance matrix long dist[GRAPHSIZE][GRAPHSIZE]. This will take O(n^2) memory, which is too much for large values of n. It is also not a good representation in terms of execution time when you only have a small number of edges: it will cause Dijkstra's algorithm to take O(n^2) time (where n is the number of nodes) when it could potentially be faster, depending on the data structures used.

Since in your case you said each node is only connected to up to 3 other nodes, you shouldn't use this matrix: Instead, for each node you should store a list of the nodes it is connected to. Then when you want to go over the neighbors of a node, you just need to iterate over this list.

In some specific cases you don't even need to store this list because it can be calculated for each node when needed. For example, when the graph is a grid and each node is connected to the adjacent grid nodes, it's easy to find a node's neighbors on the fly.

interjay
  • 107,303
  • 21
  • 270
  • 254
  • Interjay thank you very much for your answer! Thanks to everyone else for their contributions. – John Dec 26 '12 at 15:19
4

If you really cannot afford memory, even with minimizations on your graph representation, you may develop a variation of the Dijkstra's algorithm, considering a divide and conquer method.

The idea is to split data into minor chunks, so you'll be able to perform Dijkstra's algorithm in each chunk, for each of the points within it.

For each solution generated in these minor chunks, consider the it as an unique node to another data chunk, from which you'll start another execution of Dijkstra.

For example, consider the points below:

.B        .C
                  .E
 .A           .D
       .F                   .G

You can select the closest points to a given node, say, within two hops, and then use the solution as part of the graph extended, considering the former points as only one set of points, with a distance equal to the resulting distance of the Dijkstra solution.

Say you start from D:

  • select the closest points to D within a given number of hops;
  • use Dijkstra's algorithm upon the selected entries, commencing from D;
  • use the solution as a graph with the central node D and the last nodes in the shortest paths as nodes directly linked to D;
  • extend the graph, repeating the algorithm until all the nodes have been considered.

Although there's a costly extra processing here, you'd be able to surpass memory limitation, and, if you have some other machines, you can even distribute the processes.

Please, note this is just the idea of the process, the process I've described is not necessarily the best way to do it. You may find something interesting looking for distributed Dijkstra's algorithm.

Rubens
  • 14,478
  • 11
  • 63
  • 92
  • Hi Rubens, thanks for your answer. Divide and conquer sounds like a good way to go eventually. I'll try interjay's suggestion first since it currently fits how my nodes are generated and the connections stored (array of nodes, each node containing a list of connected nodes). Perhaps if a limitation is reached there, I can move onto your approach. Thanks! – John Dec 26 '12 at 15:24
  • @John That's very nice you found a suggestion that will not require too much changes on what you already have; if it happens the limitation is reached, I'll be glad to help! Good luck! – Rubens Dec 26 '12 at 15:44
1

I like boost::graph a lot. It's memory consumption is very decent (I've used it on road networks with 10 million nodes and 2Gb ram).

It has a Dijkstra implementation, but if the goal is to implement and understand it by yourself, you can still use their graph representation (I suggest adjacency list) and compare your result with theirs to be sure your result is correct.

Some people mentioned other algorithms. I don't think this will play a big role on the memory usage, but more likely in the speed. 2M nodes, if the topology is close to a street-network, the running time will be less than a second from one node to all others.

http://www.boost.org/doc/libs/1_52_0/libs/graph/doc/index.html

Tristram Gräbener
  • 9,601
  • 3
  • 34
  • 50