Trying to get a Python client talk to a Java Server using thrift's TFileTransport and TFileProcessor

Question

I am trying to wrap a Java class to be called from Python using thrift's TFileTransport. I tried using two protocols TJSON and TBinary but I keep getting this exception

org.apache.thrift.transport.TTransportException: FileTransport error: bad event size
    at org.apache.thrift.transport.TFileTransport.readEvent(TFileTransport.java:327)
    at org.apache.thrift.transport.TFileTransport.read(TFileTransport.java:468)
    at org.apache.thrift.transport.TFileTransport.readAll(TFileTransport.java:439)
    at org.apache.thrift.protocol.TJSONProtocol$LookaheadReader.read(TJSONProtocol.java:263)
    at org.apache.thrift.protocol.TJSONProtocol.readJSONSyntaxChar(TJSONProtocol.java:320)
    at org.apache.thrift.protocol.TJSONProtocol.readJSONArrayStart(TJSONProtocol.java:784)
    at org.apache.thrift.protocol.TJSONProtocol.readMessageBegin(TJSONProtocol.java:795)
    at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
    at org.apache.thrift.transport.TFileProcessor.processUntil(TFileProcessor.java:69)
    at org.apache.thrift.transport.TFileProcessor.processChunk(TFileProcessor.java:102)
    at org.apache.thrift.transport.TFileProcessor.processChunk(TFileProcessor.java:111)
    at org.apache.thrift.transport.TFileProcessor.processChunk(TFileProcessor.java:118)
    at com.netflix.suro.client.SendToPyServer.startThriftServer(SendToPyServer.java:51)
    at com.netflix.suro.client.SendToPyServer.main(SendToPyServer.java:67)

This is how my Python client looks:

 def __init__(self):
        self.outFile=open("../../ThriftFile.in","a")
        self.transport = TTransport.TFileObjectTransport(self.outFile)
        self.protocol = TJSONProtocol.TJSONProtocol(self.transport)
        self.client = sendPyInterface.Client(self.protocol)
        self.transport.open()

    def send(self,routingKey, message):
        self.transport.write(pickle.dumps(self.client.send_send(routingKey, message))) 


    def configClient(self,configurationDict):       
        self.transport.write(pickle.dumps(self.client.send_ClientConfig(configurationDict)))    

if __name__ == "__main__":
    SuroClient=SuroPyClient()
    configurationDict={"ClientConfig.LB_TYPE":"static","ClientConfig.LB_SERVER":"localhost:7101"}
    SuroClient.configClient(configurationDict)
    SuroClient.send("routingKey", "testMessage")

and this is my server:

public static void startThriftServer(SendPyInterface.Processor processor) {
            try {
                File input = new File("src/main/java/com/netflix/suro/client/ThriftFile.in");
                if(!input.exists()){
                    input.createNewFile();
                }

                File output = new File("src/main/java/com/netflix/suro/client/ThriftFile.out");
                if(!output.exists()){
                    output.createNewFile();
                }

                TFileTransport inputFileTransport = new TFileTransport(input.getAbsolutePath(), true);
                TFileTransport outputFileTransport = new TFileTransport(output.getAbsolutePath(), false);
                System.out.println(input.getAbsolutePath());
                System.out.println(input.length());

                inputFileTransport.open();
                outputFileTransport.open();
                System.out.println(inputFileTransport.getBytesRemainingInBuffer());
                inputFileTransport.setTailPolicy(tailPolicy.WAIT_FOREVER);
                System.out.println("Wait ...");
                System.out.println(inputFileTransport.getBuffer());
                TFileProcessor fProcessor = 
                        new TFileProcessor(processor, new TJSONProtocol.Factory(), inputFileTransport, outputFileTransport);
                try {
                    fProcessor.processChunk();
                } catch (TTransportException e) {
                    e.printStackTrace();
                }


                System.out.println("File Thrift service started ...");
            } catch (Exception e) {
                e.printStackTrace();
            }

I have followed the example from this thread: Example on how to use TFileTransport in Thrift (Client/Server)

and from this post: http://theprogrammersguideto.com/thrift/blog/code/chapter-3-moving-bytes-with-transports/

The blog belongs to [Randy Abernethy's forthcoming book](http://www.manning.com/abernethy/), currently available as MEAP at Manning. In other words, there is more to it than just the code. — JensG, Jun 17 '14 at 23:14
When you use JSON and look into the file, does it look reasonable? — JensG, Jun 17 '14 at 23:17
Yes and yes, I know that there will be a book, but it is not there yet and there is almost zero documentation on TFileTransport and no comments in the code, but the example is reasonably easy to get. The second Yes, the JSON looks very reasonable "[1,"ClientConfig",1,0,{"1":{"map":["str","str",2,{"ClientConfig.LB_TYPE":"static","ClientConfig.LB_SERVER":"localhost:7101"}]}}]N.[1,"send",1,0,{"1":{"str":"routingKey"},"2":{"str":"testMessage"}}]N." — user2294950, Jun 17 '14 at 23:20
Again: The book can be get at Manning as MEAP, up to chapter 11 is available. Just follow the link. — JensG, Jun 17 '14 at 23:22
After taking a short look, the `esize` and `ebytes` part in `TFileTransport.readEvent()` looks strange to me. Could you verify that you can read your JSON back using python? If yes, please file a JIRA ticket. — JensG, Jun 17 '14 at 23:44
Perfect, will do that and update here too. Thanks for your help, much appreciated! — user2294950, Jun 17 '14 at 23:49
Are you sure the input/output files are empty before you start your application? Also are you sure you are using the same thrift version on both Python and Java? — Ahmed Farghal, Jun 18 '14 at 13:43
@JensG I was able to read the data using Python's TFileTransport. I will file a bug. — user2294950, Jun 18 '14 at 14:59
Yes, it looks like a bug to me, there is inconsistency between the implementations across Java and Python. — Ahmed Farghal, Jun 25 '14 at 15:29

score 6 · Accepted Answer · answered Jun 21 '14 at 21:30

6

According to the docs in the old Thrift wiki, the Java and C++ TFileTransport is actually a framed transport, which writes an additional 4-byte header preceding the data. Seems the names are a bit misleading here.

In contrast, the Python TFileObjectTransport used is just a simple wrapper. This means, that the transports are not compatible, hence the error message.

So instead of TFileTransport, the TSimpleFileTransport should be used with Java.

answered Jun 21 '14 at 21:30

JensG

13,148
4
45
55

Can not vote your answer up since I do not have enough reputation, but want to say Thank you! – user2294950 Jun 23 '14 at 19:55

Trying to get a Python client talk to a Java Server using thrift's TFileTransport and TFileProcessor

1 Answers1