Issue in deserialize protostuff object in Kafka Consumer

Question

Getting following exception while deserializing byte[] into protostuff object in Kafka Consumer

java.lang.NegativeArraySizeException
at com.dyuproject.protostuff.GraphIOUtil.mergeDelimitedFrom(GraphIOUtil.java:209)
at com.gme.protocols.protostuff.GmeTrade.readExternal(GmeTrade.java:2772)
at java.io.ObjectInputStream.readExternalData(Unknown Source)
at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.readObject(Unknown Source)

Converted protostuff object to byte[] using following code.

public static byte[] toBytes(Object o)
{
    try
    {
        ByteArrayOutputStream baos = new ByteArrayOutputStream();
        ObjectOutputStream oos = new ObjectOutputStream(baos);
        oos.writeObject(o);
        oos.close();
        byte[] b = baos.toByteArray();
        return b;
    }
    catch (IOException e)
    {
        return new byte[0];
    }
}

Sent that byte[] using Kafka producer with topic 'XX', where byte[] length is just 240. Received that record using Kafka consumer. record.value().length (byte[]) length is same 240 what I sent from producer side.

Deserialized that byte[] to object using following code.

public static Object fromBytes(byte[] bytes)
{
    try
    {
        return new ObjectInputStream(new ByteArrayInputStream(bytes)).readObject();
    }
    catch (Exception e)
    {
        e.printStackTrace();
        return null;
    }
}

Getting above mentioned exception. What I am doing wrong ? Using kafka_2.11-0.9.0.0 for your reference. Is there any other things needed?

score 0 · Accepted Answer · answered Nov 11 '16 at 13:11

I found the solution. Issue is there in toBytes and fromBytes. Need to convert it into byte[] using ProtostuffIOUtil.toByteArray method.

Serialization

public static byte[] toBytes(Foo o)
{
  LinkedBuffer BUFFER = LinkedBuffer.allocate(1024*1024);
  Schema<Foo> SCHEMA = Foo.getSchema();
  return ProtostuffIOUtil.toByteArray(o, SCHEMA, BUFFER);
}

Again need to convert byte[] to object using ProtostuffIOUtil.mergeFrom method.

Deserialization

public static Foo fromBytes(byte[] bytes)
{
  Foo tmp = Foo.getSchema().newMessage();
  ProtostuffIOUtil.mergeFrom(bytes, tmp, Foo.getSchema());
  return tmp;
}

Note : Serialization/Deserialization with ObjectOutputStream/ObjectInputStream will not work for protostuff objects.

Issue in deserialize protostuff object in Kafka Consumer

1 Answers1