0

I have a client scala code like the below

import java.net._
import java.io._
import scala.io._
import scala.pickling._        
import scala.pickling.json._  
val sk = new Socket(InetAddress.getByName("localhost"), 13373)
val output = new PrintStream(sk.getOutputStream())
val textRDD = sc.textFile("some file");
output.println( #pickle the textRDD and pass it to server)
output.flush()
sk.close()

and python server.py like below,

import SocketServer
import json
import pickle
class MyTCPServer(SocketServer.ThreadingTCPServer):
allow_reuse_address = True
class MyTCPServerHandler(SocketServer.BaseRequestHandler):
def handle(self):
    try:
        data = self.request.recv(1024)       
      #unpickle the data received here and print it.
        print data
    except Exception, e:
        print "Exception wile receiving message: ", e
server = MyTCPServer(('127.0.0.1', 13373), MyTCPServerHandler)
server.serve_forever()

How to pickle the TextRDD file in the scala client and pass it to python server to unpickle it and print the received data?

Edwin Vivek N
  • 564
  • 8
  • 28

1 Answers1

0

I don't think you will be able to do this without rewriting scala-pickling in Python first. scala-pickling is Scala-specific serialization library, it is not even intended to be able to serialize/deserialize arbitrary formats; it is designed to be used as a substitute for Java serialization - fast and precise serialization for internal purposes.

If you need to send data across different languages, you should consider using portable protocols and serialization formats, for example, Protobuf, Cap'n'Proto, Thrift or MessagePack. For most of them there are multiple libraries in different languages, including Java/Scala and Python.

Vladimir Matveev
  • 120,085
  • 34
  • 287
  • 296
  • Consider a Scala server instead of Python server. Now, no cross platforms. Scala client to Scala server, is it possible in this case? – Edwin Vivek N Jul 14 '15 at 10:21
  • Yes, if you use Scala server, then it becomes much easier - just use whatever facilities scala-pickling provides for deserialization on server side. Several examples on how to do that are available on [scala-pickling](https://github.com/scala/pickling) page. – Vladimir Matveev Jul 14 '15 at 10:24