Sending DataStream in Flink using sockets; serialization issue

Question

I want to send Stream of data from VM to host machine and I am using method writeToSocket() as shown below:

joinedStreamEventDataStream.writeToSocket("192.168.1.10", 6998) ;

Here joinedStreamEventDataStream is of type DataStream<Integer,Integer>.

Can someone please tell me how should I pass serializer to above method.

Thanks in Advance

score 3 · Answer 1 · answered Aug 22 '17 at 13:27

It depends a little bit on how you would like to read the data from the socket. If you expect it to be the String representation of the data, then you could do it via:

joinedStreamEventDataStream.map(new MapFunction<Type, String>() {
    @Override
    public String map(Type value) throws Exception {
        return value.toString();
    }
}).writeToSocket(hostname, port, new SimpleStringSchema());

If you want to keep Flink's serialization format, then you can do write:

joinedStreamEventDataStream.writeToSocket(
    hostname, 
    port, 
    new TypeInformationSerializationSchema<>(
        joinedStreamEventDataStream.getType(), 
        env.getConfig()));

If you want to output it in your own serialization format, then you have to implement your own SerializationSchema as pointed out by Alex.

score 1 · Answer 2 · answered Aug 22 '17 at 12:14

The writeToSocket() method takes 3 arguments: a socket host and port and also an implementation of SerializationSchema interface which used to serialize your data. So your implementation maybe like this:

joinedStreamEventDataStream.writeToSocket(
    "192.168.1.10",  // host name
    6998,  // port
    new SerializationSchema<Integer>() {

        @Override
        public byte[] serialize(Integer element) {
            return ByteBuffer.allocate(4).putInt(element).array();
        }
    }
);

It's true if joinedStreamEventDataStream has DataStream<Integer> type.

Sending DataStream in Flink using sockets; serialization issue

2 Answers2