2

Hi and thanks for all the good work on OpenMapTiles.

I'm trying to build tiles for Europe, North-America, maybe world.

I'm using the ./quickstart script and it's said to be taking 30 days to build the tiles for America and 192 days for Europe. This is running on a c5d.18xlarge EC2 instance (70 CPU, 180G RAM, SSD disks).

Am I missing something ?

I'm currently trying to use a database outside of Docker (on localhost) to see if I can speed things... but how are you guys doing ?

Prune
  • 345
  • 4
  • 11

2 Answers2

1

I'm using this
https://github.com/mapbox/mbutil/blob/5e1ac74fdf7b0f85cfbbc245481e1d6b4d0f440d/patch

This is one of my scripts, I'm merging it all to tmp and checking if there is still a tilelive-copy in progress on that file 

for i in *.mbtiles; do
    [ -f "$i" ] || break
    if [[ $i != *"final.mbtiles"* ]]; then
      if ! [[ `lsof -c /tilelive-copy/ $i` ]]; then
  exit=$(/usr/local/bin/merge_mbtiles.sh $i /tmp/final.mbtiles)
        echo $exit
        (( $rc )) && echo "merge failed $i" && exit 1
        echo "merge sucessfull"
      fi
    fi
done
loosi
  • 11
  • 1
0

I'm using openmaptiles too and speed extremly slowed down after the last updates (still have to figure out the changes that caused that). quickstart script is nice for trying and figuring out, in the end I started writing scripts to split and parallise the work. Right now we are processing the whole world with zoom 0-14(fast) and most of europe with 14-18 (that takes weeks)

Try the following:
* tune postgres (defaults are bad for large databases)
* try to split the areas and parralise the work.

You can see that the rendering process with tilelive-copy is not really using all cores. The whole process isn't that effective in using resources. After a couple of tries I figured out that starting multiple workers in parallel is faster (in the end) than supersizing your server with more CPU core speed.

see also:
https://github.com/openmaptiles/openmaptiles/issues/462
https://github.com/mapbox/tilelive/issues/181

loosi
  • 11
  • 1
  • I did tuned the Postgres, which now use far more RAM, but some queries are still single-threaded / not parallelized. – Prune Sep 25 '18 at 12:10
  • Would you mind to share some details on what you tuned? – Robin Sep 29 '18 at 07:07
  • 2
    just the general optimization stuff for osm postgres : shared_buffers, huge_pages, work_mem/maintenance_work_mem autovacuum settings for your workload io_concurrency depending on your disks max_worker_processes min/max wal_size depending on your server and import behaviour page_costs depending on your server for ssd/hdd it really depends on your server for ssd, memory. – loosi Oct 01 '18 at 13:22