Apache Thrift Python 3 support

Question

I compiled my test.thrift file using:

thrift -gen py test.thrift

Then i tried to import the created files:

from test.ttypes import *

When I use Python 2.7 the import works but with Python 3.4 it raises

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/art/SerTest/addressThrift/gen-py/test/ttypes.py", line11, in <module>
from thrift.transport import TTransport
File "/usr/local/lib/python3.4/dist-
packages/thrift/transport/TTransport.py", line 20, in <module>  
from cStringIO import StringIO
ImportError: No module named 'cStringIO'

I tried to run: sudo python3 setup.py install and got many exceptions, all seems to be related to python 2 vs 3 problems. for example:

File "/usr/local/lib/python3.4/dist-
     packages/thrift/transport/TSSLSocket.py", line 99
except socket.error, e:
                   ^
SyntaxError: invalid syntax

I addition there is a warning thats seems important:

/usr/lib/python3.4/distutils/dist.py:260: UserWarning: Unknown distribution option: 'use_2to3'

Googling Thrift Python 3 support seems contradicting.
Those say that there is no support:
Does cql support python 3?
https://issues.apache.org/jira/browse/THRIFT-1857
And here I understand from the subtext that it does:
Thrift python 3.4 TypeError: string argument expected, got 'bytes'
Python 3.4 TypeError: input expected at most 1 arguments, got 3
https://issues.apache.org/jira/browse/THRIFT-2096
So does Thrift support Python 3.X? If so what did I missed?

AFAIK the support for Python 3 is about to be implemented, I remember a number of incoming patches with regard to Python3 in the last weeks. The 0.9.3 release will happen at some time in the forthcoming weeks, so you may use the current trunk as a reference for testing that out. I also suggest to ask that question on the mailing list to get a more exhaustive answer (I'm not a python expert). Also, if there are any issues, post it there or file a JIRA ticket (whatever is more appropriate). Last nolt least [we also accept pull requests](https://thrift.apache.org/docs/HowToContribute). — JensG, Sep 04 '15 at 13:32

score 7 · Answer 1 · answered Oct 27 '15 at 03:22

7

There is a better solution to this. Instead of waiting the official python 3 support, why not use our python implementation of thrift?

https://github.com/eleme/thriftpy

it fully supports python 3, and pypy, and pypy3.

answered Oct 27 '15 at 03:22

WooParadog

645
4
7

I have tried to use this library but currently (as of Dec 2016) it doesn't support Thrift type annotations. Which renders it unusable in my case. – dmitryzv Dec 19 '16 at 21:36
I see that the project/repo at that URL has been deprecated in favor of https://github.com/Thriftpy/thriftpy2. – Robert Fleming May 12 '20 at 16:40

score 4 · Answer 2 · answered Nov 27 '15 at 11:47

Install thrift with pip3 to make sure it gets converted (I think they use 2to3) and it will use the python3 syntax except ExceptionName as e as well as other python3 syntax like turning print into a function, ...

However, there are still some bugs. I found that pack() results are used as input to a StringIO as in python2 it returns a str but in python3 it was changed so that it returns a bytes. The easiest solution is to change the following files:

transport/TTransport.py:

Line 20:

from io import StringIO

to (option 1)

from io import BytesIO as StringIO

or (option 2)

from io import BytesIO

Line 55:

buff = ''

to

buff = b''

Line 142:

self.__rbuf = StringIO("")

to

self.__rbuf = StringIO(b'')

Aditional changes for option 2:

Search every

StringIO

and replace it for

BytesIO

protocol/TBinaryProtocol.py

Lines 122-123

self.writeI32(len(str))
self.trans.write(str)

to

buff = bytes(str, 'utf-8')
self.writeI32(len(buff))
self.trans.write(buff)

Line 223

str = self.trans.readAll(len)

to

str = self.trans.readAll(len).decode('utf-8')

I also made 2 functions that are equivalent to python2's pack() and unpack() for python3 that may be useful for some other problems where you can't just substitute the io for the proper one:

from io import pack, unpack

def packStr(fmt, v):
  return "".join(["\\x{:02x}".format(i) for i in list(pack(fmt, v))])

def unpackStr(fmt, v):
  return unpack(fmt, bytes([int(i, 16) for i in v.split("\\x")[1:]]))

This answer is a year old and I didn't follow developing on thrift. I actually nedt it for a backwards compatibility reason and the version that required it was supported for little time (old Apache Cassandra Driver that got swapped to CQL aware one, no longer depending on Thrift). You are free to fork, commit the changes and request a pull yourself. A link to this SO question would be apreciated. — Adirio, Nov 23 '16 at 09:29

score 3 · Answer 3 · answered May 29 '17 at 23:26

It seems that thrift 0.10.0, released in January 2017, already has built-in support for Python 2 and 3 at the same time, but beware there are some compatibility issues.

Sources: