2

Background

I am trying to make a reduced size vector mbtiles file for the world for a compact offline maps solution I am working on for Windows desktop. To do so I have found OsmAnd's reduced size World_basemap_2.obf to be exactly what I need, but in the mbtiles OSM format so I can generate an mbtiles. The obf was created using the following OSM files generated from shape files. They are available at http://builder.osmand.net/basemap/

The problem I have is that in order to make an mbtiles they need to be combined into one OSM file. osm2vectortiles' guide to generate an mbtiles found here http://osm2vectortiles.org/docs/own-vector-tiles/.

First I extract them to deal with the osm files.

Problem

When I use osmconvert to combine the files using the --complete-ways and --complex-ways flags, a lot of relations and ways are removed and the filesize is extremely different.

For ex:

osmconvert proc_line_tertiary_out.osm --complete-ways --complex-ways -o=tertiary.o5m

results in a file 133,691 kb in size

osmconvert tertiary.o5m --complete-ways --complex-ways -o=tertiary.csv

results in a CSV with a total of 13131405 nodes, ways, and relations

osmconvert tertiary.o5m proc_line_ferry_out.osm.o5m --complete-ways --complex-nodes -o=combined.o5m

merging tertiary.o5m with proc_line_ferr_out.osm.o5m (generated the same as shown above) and converting to CSV

results in a CSV with 1195239 total nodes, ways, and relations... which is significantly less and is very concerning. Combining two files should not make fewer nodes, ways, and relations than the original files.

Question

What am I doing wrong?

  • Is there an alternative way to combine osm files? (I've read that osmosis can't deal with negative nodes.)

  • Is there an alternative way to generate an mbtiles, possibly from multiple osm files?

Files trying to combine

  • points.osm.bz2
  • polygon_aeroway_military_tourism.osm.bz2
  • polygon_lake_water.osm.bz2
  • polygon_natural_landuse.osm.bz2
  • proc_line_admin_level_out.osm.bz2
  • proc_line_ferry_out.osm.bz2
  • proc_line_motorway_out.osm.bz2
  • proc_line_primary_out.osm.bz2
  • proc_line_railway_out.osm.bz2
  • proc_line_secondary_out.osm.bz2
  • proc_line_tertiary_out.osm.bz2
  • proc_line_trunk_out.osm.bz2
  • cities.osm.bz2
  • coastline.osm.bz2

OSMConvert

osmconvert explains that

Merging two or more Geographical Areas

Under certain conditions, OSM data files can be merged. If they hold objects (nodes, ways, relations) with the same id, they must have the same content. For example: If a way crosses the geographical border between two regional files, the way dataset must contain every node reference, even the references to nodes which do not lie within the borders of the regarding file. I.e., such a region must not have been cut out using the option --drop-brokenrefs.

Does that mean I can't do what I am trying to do? I want to keep all nodes, ways, and relations, even if the files don't have them in common.

Attempt

osmconvert polygon_aeroway_military_tourism.osm polygon_natural_landuse.osm -o=result.osm

Result

    <?xml version='1.0' encoding='UTF-8'?>
    <osm version="0.6" generator="osmconvert 0.7T">
        <node id="-10000000000001" lat="-1.0015443" lon="-80.5349586"/>
        <way id="-100000001" version="1">
            <nd ref="-10000000000001"/>
            <nd ref="-10000000000002"/>
            <nd ref="-10000000000003"/>
            <nd ref="-10000000000004"/>
            <nd ref="-10000000000005"/>
            <nd ref="-10000000000006"/>
            <nd ref="-10000000000007"/>
            <nd ref="-10000000000008"/>
            <nd ref="-10000000000009"/>
            <nd ref="-10000000000010"/>
            <nd ref="-10000000000011"/>
            <nd ref="-10000000000012"/>
            <nd ref="-10000000000013"/>
            <nd ref="-10000000000014"/>
            <nd ref="-10000000000015"/>
            <nd ref="-10000000000016"/>
            <nd ref="-10000000000017"/>
            <nd ref="-10000000000018"/>
            <nd ref="-10000000000019"/>
            <nd ref="-10000000000020"/>
            <nd ref="-10000000000021"/>
            <nd ref="-10000000000022"/>
            <nd ref="-10000000000023"/>
            <nd ref="-10000000000024"/>
            <nd ref="-10000000000025"/>
            <nd ref="-10000000000026"/>
            <nd ref="-10000000000027"/>
            <nd ref="-10000000000028"/>
            <nd ref="-10000000000029"/>
            <nd ref="-10000000000030"/>
            <nd ref="-10000000000031"/>
            <nd ref="-10000000000032"/>
            <nd ref="-10000000000033"/>
            <nd ref="-10000000000034"/>
            <nd ref="-10000000000035"/>
            <nd ref="-10000000000036"/>
            <nd ref="-10000000000037"/>
            <nd ref="-10000000000038"/>
            <nd ref="-10000000000039"/>
            <nd ref="-10000000000040"/>
            <nd ref="-10000000000041"/>
            <nd ref="-10000000000042"/>
            <nd ref="-10000000000043"/>
            <nd ref="-10000000000044"/>
            <nd ref="-10000000000045"/>
            <nd ref="-10000000000046"/>
            <nd ref="-10000000000047"/>
            <nd ref="-10000000000048"/>
            <nd ref="-10000000000049"/>
            <nd ref="-10000000000050"/>
            <nd ref="-10000000000051"/>
            <nd ref="-10000000000001"/>
        </way>
    </osm>
Michael
  • 8,362
  • 6
  • 61
  • 88
longlostbro
  • 528
  • 2
  • 7
  • 24

2 Answers2

2

It may be easier for your use case to merge the already rendered vector tiles (copy individual PBF tiles) from multiple OSM2VectorTiles extracts in MBTiles format - and merge all these into a single MBTiles file.

You may use patch utility from MBUtil package: https://github.com/mapbox/mbutil/blob/master/patch

Or the tool from https://pypi.python.org/pypi/umimbutil

These two utilities should run for a basic merging of distinct areas where individual tiles do not need to be combined together.

Alternativelly - you can write a basic script which removes from the planet MBTiles all the tiles you don't need. MBTiles is just SQLite file, so this should be doable with a basic SQL commands...

P.S. In case somebody need to merge raster (!!!) MBTiles including combining of the raster tiles in PNG or JPEG tiles together, then they need a different merging utility: https://www.maptiler.com/how-to/merge-mbtiles/ on command-line merge_mbtiles available after filling the form at http://www.maptiler.com/demo/

MapTiler
  • 1,754
  • 14
  • 22
  • Thank you, I did not know you could merge mbtiles. Now the only problem would be that it takes weeks to generate an mbtiles from an osm file when I've tried. As for removing data from the planet.mbtiles I have attempted that but with so many transactions and vacuuming such a large dataset it was estimated to take a very long time. – longlostbro Nov 30 '16 at 16:26
2

the options --complete-ways and --complex-ways are meant for usage when applying borders. If you are not clipping OSM data (options -b or -B) better do not use --complete-ways nor --complex-ways unless you are certain what they do with your data. At least I am not.

The way of choice to merge OSM data using osmconvert is this:

osmconvert input1.o5m input2.o5m input3.o5m -o=output.o5m

There is some further information about osmconvert in OSM Wiki:

https://wiki.openstreetmap.org/wiki/Osmconvert

Marqqs
  • 191
  • 4
  • Simple mistake, I can't believe I overlooked it. Thanks so much. – longlostbro Nov 30 '16 at 16:32
  • Are there any other limitations I should know about? when I combine polygon_aeroway_military_tourism.osm - 17340 nodes/ways/relations and polygon_natural_landuse.osm - 8711704 nodes/ways/relations the combined result becomes - 3 nodes/ways/relations – longlostbro Nov 30 '16 at 17:01
  • I added more info to my question. My interpretation of osmconvert is that it cannot keep all nodes and relations when merging, only those in common. – longlostbro Nov 30 '16 at 17:10
  • That sounds strange... If there are really just 3 nodes in the result file, there might be a bug in osmconvert. :-( Did you try this command? `osmconvert polygon_aeroway_military_tourism.osm polygon_natural_landuse.osm -o=result.osm` The file result.osm should then contain at least 8711704 objects. – Marqqs Nov 30 '16 at 19:51
  • yes, I've tried it in both windows, and ubuntu environments, with the exact command osmconvert polygon_aeroway_military_tourism.osm polygon_natural_landuse.osm -o=result.osm and it results in one node and one way. I added the attempt to my question, with the result. – longlostbro Nov 30 '16 at 21:07
  • Unfortunately I cannot explain this behaviour. :-( Is there a way you could send me these two files? osmconvert will show my email address if you start it with option "--help". – Marqqs Nov 30 '16 at 21:13