0

I have 10 grids (currently stored as ascii grids from a GIS), each of them with about 4.5GB uncompressed. In addition I have about 100,000 location with an x and y coordinate. I need to extract the grid value at each of this location. I am currently doing it with GRASS GIS which works, but is very slow. Can anyone recommend me a library or a programming language most suitable for such a task?

Thanks in advance!

johannes
  • 14,043
  • 5
  • 40
  • 51
  • What is the application? Whatever you are doing, there has to be a better way! – Benjamin Feb 04 '11 at 18:49
  • Are you a programmer or do you need something that is already written? Your data could be made a lot smaller and will most likely fit into memory. For example a 16 byte x,y,id structure can hold 67,108,864 points in 1 GB of ram. – madmik3 Feb 04 '11 at 18:55
  • @ Benjamin, I am doing some modelling with R and preparing data – johannes Feb 04 '11 at 20:49

1 Answers1

1

Sounds like the classic use-case for Hadoop MapReduce.

Hadoop MapReduce is a programming model and software framework for writing applications that rapidly process vast amounts of data in parallel on large clusters of compute nodes.

skaffman
  • 398,947
  • 96
  • 818
  • 769