Encoding Lists in Thrift (Python)

Question

I'm wondering what's the best way to binary encode a list of structs with ThriftPy. The only way I've found is to create another wrapper struct and remove the binary prefix/suffix from the stream but this is very hacky and there should definitely be a better way.

foobar.thrift:

struct Object {
  1: i32 num1 = 0,
  2: i32 num2,
}

struct ListContainer {
    1: list<Object> objects
}

app.py

foobar = thriftpy.load('foobar.thrift', module_name="foobar_thrift")

objects = [ ... list of Objects ... ]

thrift_obj = foobar.ListContainer(objects)

trans = TMemoryBuffer()
thrift_obj.write(TBinaryProtocol(trans))

encoded_list = bytes(trans.getvalue())[3:-1]

How would you define "best way" in the context of this question? And why do you remove the header at all? IOW, **what do you want to achieve**? — JensG, Jan 13 '17 at 11:42
There's an existing API that expects an Array of Thrift objects and I want to use it. The API is implemented in Java which seems to support reading lists of objects (see [here](https://github.com/openzipkin/zipkin/blob/fe14098030ab7ee5e305376c232ac72d852890b5/zipkin/src/main/java/zipkin/internal/ThriftCodec.java#L409)) — neverlastn, Jan 13 '17 at 12:17

neverlastn · Answer 1 · 2017-01-14T19:41:47.350

I found a solution... not particularly elegant but better than the previous one.

foobar.thrift:

struct Object {
  1: i32 num1 = 0,
  2: i32 num2,
}

typedef list<Object> ObjectList

app.py:

import thriftpy
from thriftpy.transport import TMemoryBuffer
from thriftpy.protocol.binary import write_val


foobar = thriftpy.load('foobar.thrift')


def write_list(trans, val, list_type):
    ttype, spec = list_type
    write_val(trans, ttype, val, spec=spec)

val = [foobar.Object(num1=8, num2=12)]

trans = TMemoryBuffer()
write_list(trans, val, foobar.ObjectList)
encoded_list = bytes(trans.getvalue())

I tried to see what the standard thrift implementation does. It doesn't make it any easier. It doesn't generate anything for the ObjectList so in order to achieve the same you have to do thrift -gen py foobar.thrift and then:

app.py:

import sys
sys.path.append('gen-py')

from foobar.ttypes import *
from thrift.protocol import TBinaryProtocol
from thrift.transport import TTransport


def write_list(trans, val):
    protocol = TBinaryProtocol.TBinaryProtocol(trans)
    protocol.writeListBegin(TType.STRUCT, len(val))
    for values in val:
        values.write(protocol)
    protocol.writeListEnd()  # This is a nop

val = [Object(num1=8, num2=12)]

trans = TTransport.TMemoryBuffer()
write_list(trans, val)
encoded_list = trans.getvalue()

I was about to propose to just write the contents of the list (but not the list itself) one by one. But that's only a wild guess, never used Zipkin myself. Isn't there any doc or tutorial available that covers your use case? — JensG, Jan 14 '17 at 10:46
There's something little [here](https://github.com/openzipkin/pyramid_zipkin-example/blob/master/transport.py) but as you can see, it also has hackery. — neverlastn, Jan 14 '17 at 19:34
I guess the main question is, does Thrift allow encoding standalone lists of structs (not within another structs) or not? If yes, then those Python libraries should provide a way. If not (which seems likely) then Zipkin is using non-standard features. — neverlastn, Jan 14 '17 at 19:43

Encoding Lists in Thrift (Python)

1 Answers1