2

(see update at bottom)

Tilemaker is an OpenStreetMap programme to generate Mapbox vector tiles (which are themselves protocol buffers (pbf) files) from an OSM pbf data file. I have compiled it and used it to create a directory of vector tiles. I cannot parse those files in Python.

I created the vector tiles with:

tilemaker input.pbf --output=tiles/

Then I created a simple python programme, based on Google's Protocol Buffers Python Tutorial in this way:

Compiling the .proto files:

mkdir py
touch py/__init__.py
protoc --proto_path=include --python_out=./py ./include/osmformat.proto
protoc --proto_path=include --python_out=./py ./include/vector_tile.proto

This python programme pyread.py doesn't work:

import sys
import py.vector_tile_pb2

with open(sys.argv[1]) as fp:
    pbf_file_contents = fp.read()

tile = py.vector_tile_pb2.Tile()
tile.ParseFromString(pbf_file_contents)

This is the error when trying to run it:

$ python pyread.py ./tiles/13/3932/2588.pbf
Traceback (most recent call last):
  File "pyread.py", line 8, in <module>
    tile.ParseFromString(pbf_file_contents)
  File "/home/rory/.local/lib/python2.7/site-packages/google/protobuf/message.py", line 186, in ParseFromString
    self.MergeFromString(serialized)
  File "/home/rory/.local/lib/python2.7/site-packages/google/protobuf/internal/python_message.py", line 841, in MergeFromString
    if self._InternalParse(serialized, 0, length) != length:
  File "/home/rory/.local/lib/python2.7/site-packages/google/protobuf/internal/python_message.py", line 866, in InternalParse
    new_pos = local_SkipField(buffer, new_pos, end, tag_bytes)
  File "/home/rory/.local/lib/python2.7/site-packages/google/protobuf/internal/decoder.py", line 827, in SkipField
    return WIRETYPE_TO_SKIPPER[wire_type](buffer, pos, end)
  File "/home/rory/.local/lib/python2.7/site-packages/google/protobuf/internal/decoder.py", line 797, in _RaiseInvalidWireType
    raise _DecodeError('Tag had invalid wire type.')
google.protobuf.message.DecodeError: Tag had invalid wire type.

The protoc command is from the protcol buffers library. I downloaded the latest release (2.6.1) from Google's page (which links to Github) and compiled & installed it. That protoc invocation is just like what the Tilemaker Makefile does.

What's going on? How can I read this protocol buffers file in python?

UPDATE Further investigation makes me think that one of my assumptions might be wrong. Namely, that the tilemaker command has produced a valid protobuf file. I got some vector tiles from Mapzen, which should have the same format and very similar data. But this format works with the python pyread.py command, and with protoc --decode_raw and protoc --decode=vector_tile.Tile ./include/vector_tile.proto. Hence I think the problem is with the file I was looking at.

Amandasaurus
  • 58,203
  • 71
  • 188
  • 248
  • Does protobuffer from OpenStreetMap have the same version as the protoc used to generate the .py? – Ismael Infante Feb 12 '16 at 19:30
  • In this example. the Tilemaker programme's Makefile calls the `protoc` command on my machine to produce C++ files form a `.proto` file. I am using the same `protoc` binary, and the same `.proto` files to produce python files, and it doesn't work. – Amandasaurus Feb 13 '16 at 10:03
  • Are you on Windows? If so, you need to open in binary mode. `open(filename, "rb")`. (It's a good idea to do this on all systems, but it only really matters on Windows.) – Kenton Varda Feb 13 '16 at 19:09
  • @KentonVarda I'm on Ubuntu Linux, not Windows. – Amandasaurus Feb 14 '16 at 12:33

2 Answers2

1

I think the problem is that OpenStreetMap's .pbf format is not a raw protobuf. See my answer to your other question:

https://stackoverflow.com/a/35384238/2686899

Community
  • 1
  • 1
Kenton Varda
  • 41,353
  • 8
  • 121
  • 105
  • Thank you for your insight re: the OSM pbf format. That makes sense. However in this case, the result is in [MapBox Vector Tiles format](https://github.com/mapbox/vector-tile-spec), not [OSM PBF](http://wiki.openstreetmap.org/wiki/PBF_Format). And [the latest (2.1) spec](https://github.com/mapbox/vector-tile-spec/tree/master/2.1) implies that it's a pure protocol buffers file. – Amandasaurus Feb 14 '16 at 12:30
  • @Rory OK, so does `protoc --decode_raw < file` work with this file? The error you gave suggests the input is not a protobuf at all (not even with a different type). – Kenton Varda Feb 15 '16 at 06:45
  • See the update I made to the question. `protoc --decode_raw` didn't work on that file. But it (and my programme) work on another file that also should of the same format. Hence you must be right. That file isn't a valid protobuf file. – Amandasaurus Feb 15 '16 at 11:04
0

If you use tilemaker with default config, probably the tile is gzipped(compress=gzip):
https://github.com/systemed/tilemaker/blob/master/docs/CONFIGURATION.md#json-configuration

Try unzip your tile by the following command:

<tile.protoc | gunzip | protoc --decode_raw
Pham
  • 371
  • 2
  • 8