Why this strange behaviour of ObjectOutputStream and ObjectInputStream throwing EOFException?

Question

I wrote a custom serializing/de-serializing logic for persisting some of the data as Java default serialization turned out to be both time and memory expensive. For this purpose I wrote readObject(ObjectInput in) and writeObject(ObjectOutput out) methods for the class(es) that needs persisting. However I noticed that if I do not use any out.writeObject(obj) in writeObject(ObjectOutput out) method then it always throws EOFException.

Consider the following example:

Data.java

public class Data implements BaseData {

private String messageUID;
private String rawData;
private String data;
private Long type;
private Boolean processed = false;
private String processedMessage;
private String processedDetaildMessage;

// getter setter

public void readObject(ObjectInput in) throws IOException, ClassNotFoundException {
    messageUID = in.readUTF();
    rawData = in.readUTF();
    data = in.readUTF();
    type = in.readLong();
    processed = in.readBoolean();
    if (processed) {
        processedMessage = in.readUTF();
        processedDetaildMessage = in.readUTF();
    }
}

public void writeObject(ObjectOutput out) throws IOException {
    out.writeUTF(messageUID);
    out.writeUTF(rawData);
    out.writeUTF(data);
    out.writeLong(type);
    out.writeBoolean(processed);
    if (processed) {
        out.writeUTF(processedMessage);
        String tempDetailsMessage[] = processedDetaildMessage.split(" more");
        out.writeUTF(tempDetailsMessage[tempDetailsMessage.length - 1]);
    }
}

However whenever I use above code the out stream is always missing some information at the end (from processedDetaildMessage field) and I get EOFException while reading it form in, stacktrace below (Data.java line 216 is processedDetaildMessage = in.readUTF());

java.io.EOFException
    at java.io.ObjectInputStream$BlockDataInputStream.readByte(ObjectInputStream.java:2766)
    at java.io.ObjectInputStream$BlockDataInputStream.readUTFChar(ObjectInputStream.java:3158)
    at java.io.ObjectInputStream$BlockDataInputStream.readUTFBody(ObjectInputStream.java:3055)
    at java.io.ObjectInputStream$BlockDataInputStream.readUTF(ObjectInputStream.java:2864)
    at java.io.ObjectInputStream.readUTF(ObjectInputStream.java:1072)
    at com.smartstream.common.Data.readObject(Data.java:216)
    at com.smartstream.common.PerformanceTest.getObjectFromBytes(PerformanceTest.java:168)
    at com.smartstream.common.PerformanceTest.access$0(PerformanceTest.java:160)
    at com.smartstream.common.PerformanceTest$1.mapRow(PerformanceTest.java:119)
    at com.smartstream.common.PerformanceTest$1.mapRow(PerformanceTest.java:1)
    at org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:92)
    at org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:60)
    at org.springframework.jdbc.core.JdbcTemplate$1.doInPreparedStatement(JdbcTemplate.java:651)
    at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:589)
    at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:639)
    at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:668)
    at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:676)
    at org.springframework.jdbc.core.JdbcTemplate.queryForObject(JdbcTemplate.java:731)
    at com.smartstream.common.PerformanceTest.readFromDb(PerformanceTest.java:109)
    at com.smartstream.common.PerformanceTest.main(PerformanceTest.java:66)

so I though I would put some extra byte/s of information at the end after writing all required fields and will not read them so that I don't reach end of file while reading. I tried all of these out.writeByte(-1), out.writeInt(-1), out.writeLong(2342343l), out.writeUTF("END_OF_STREAM") but those make no difference. finally I did this out.writeObject(new String("END_OF_STREAM")) and it works fine. Can someone please explain as to why outputstream misses some information if none of the information is written using writeObject() method. Below is how I read and write to/from streams;

private byte[] getObjectAsBytes(Data data) {
    byte[] byteArray = null;
    ByteArrayOutputStream bos = null;
    ObjectOutputStream oos = null;
    try {
        bos = new ByteArrayOutputStream();
        oos = new ObjectOutputStream(bos);
        // Use this for java default serialization
        // oos.writeObject(data);
        data.writeObject(oos);
        byteArray = bos.toByteArray();
    } catch (IOException e) {
        e.printStackTrace();
    } finally {
        if (oos != null) {
            try {
                oos.flush();
                oos.close();
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
    }
    return byteArray;
}

private Data getObjectFromBytes(byte[] byteArray) {
    Data data = new Data();
    ByteArrayInputStream bais = null;
    ObjectInputStream ois = null;
    try {
        bais = new ByteArrayInputStream(byteArray);
        ois = new ObjectInputStream(bais);
        // Use this for java default serialization
        // data = (Data) ois.readObject();
        data.readObject(ois);
    } catch (IOException e) {
        e.printStackTrace();
    } catch (ClassNotFoundException e) {
        e.printStackTrace();
    } finally {
        if (ois != null) {
            try {
                ois.close();
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
    }
    return data;
}

If anyone is interested below is what is written in the streams;

persisted data with original code (throws EOFException and missing information) (don't confuse the stacktrace with original issue this stacktrace is persisted as field processedDetailedMessage)

¬í---z-------3507319347632941385----FEEDER-----1437052314954 ---This is a random string---N---þ%J---!this is message of processed dataÛ
Caused by: java.sql.SQLException: ORA-01691: unable to extend lob segment TLM_DBO.SYS_LOB0000076335C00008$$ by 8192 in tablespace WIN_SL_TABLE
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:439)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:395)
at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:802)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:436)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:186)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:521)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:205)
at oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:1008)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1307)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3449)
at oracle.jdbc.driver.OraclePre

persisted data after writing extra string at the end using writeObject method

¬í---z-------3507319347632941385----FEEDER-----1437052314954 ---This is a random string---N---þ%J---!this is message of processed dataÛ
Caused by: java.sql.SQLException: ORA-01691: unable to extend lob segment TLM_DBO.SYS_LOB0000076335C00008$$ by 8192 in tablespace WIN_SL_TABLE
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:439)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:395)
at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:802)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:436)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:186)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:521)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:205)
at oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:1008)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1307)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3449)
at oracle.jdbc.driver.OraclePrz-----NeparedStatement.execute(OraclePreparedStatement.java:3550)
at oracle.jdbc.driver.OraclePreparedStatementWrapper.execute(OraclePreparedStatementWrapper.java:1374)
at com.ibm.ws.rsadapter.jdbc.WSJdbcPreparedStatement.pmiExecute(WSJdbcPreparedStatement.java:975)
at com.ibm.ws.rsadapter.jdbc.WSJdbcPreparedStatement.execute(WSJdbcPreparedStatement.java:642)
at com.smartstream.control.engine.config.dao.jdbc.ProcessExecutionAuditDetailDao$1.doInPreparedStatement(ProcessExecutionAuditDetailDao.java:115)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:586)
... 23t 
END_OF_STREAM

PS ---- represents unreadable bytes

This is not an answer but a recommendation to have a look at java Externalizable as it is basically a cutom serialize-interface. This might be a bit cleaner :) — red13, Jul 16 '15 at 13:47
@red13 thanks for recommendation but we can't use Externalizable because in the platform this code runs requires this class to be Serializable and it throws exception as Externalizable is not supported by that platform. So the class is Serializable but reason behind writing custom logic is memory foot print and performance. — madteapot, Jul 16 '15 at 13:49
You don't need these `readObject()` or `writeObject()` methods. They don't do anything for you that default serialization wouldn't also do. I would delete them and use `Data data = (Data)ois.readObject();` instead of `data.readObject(ois);`, and `oos.writeObject(data);` instead of `data.writeObject(oos);`. And as you have it now, those methods are public, which is a security breach. — user207421, Jul 19 '15 at 06:45
@EJP with all due respect you don't get the problem in the question. I already mentioned default serialization is memory and performance expensive. And the code snippet I've added in question is just minimal representation of the problem and not the actual code. In the actual code the class has much complex data types and java writes all of those fully qualified class names while serializing, which is completely unnecessary. In our actual code if java default serialization is used serialized class takes over 8000 bytes while using custom serializer we managed to do it under 3000 bytes. — madteapot, Jul 19 '15 at 20:40

score 3 · Accepted Answer · answered Jul 19 '15 at 05:40

Your persisted data is incomplete because you are creating your byte array before flushing the ObjectOutputStream. In getObjectAsBytes(Data) move byteArray = bos.toByteArray(); after the finally block to make it work. Alternatively, the method could be written more succinctly as follows (requires Java 7+):

private byte[] getObjectAsBytes(Data data) {
    ByteArrayOutputStream bos = new ByteArrayOutputStream();
    try (ObjectOutputStream oos = new ObjectOutputStream(bos)) {
        data.writeObject(oos);
    } catch (IOException e) {
        e.printStackTrace();
    }
    return bos.toByteArray();
}

I tested both ways in my own program and they both prevent the EOFException from being thrown.

As far as why having a writeObject was working, that's because the underlying writeObject implementation toggles block data mode at the beginning and ending of the method, and changing the block data mode performs a drain which writes all data to the underlying OutputStream, which for a ByteArrayOutputStream is effectively the same as a flush.

or just change return byteArray; to return bos.toByteArray(); — andy, Jul 20 '15 at 08:53
Thank you very much for explaining the actual reason for the issue. — madteapot, Jul 22 '15 at 16:04

score -1 · Answer 2 · answered Jan 23 '17 at 15:36

This issue is caused because of the different implementations of writeObject method and some other non-generic write* methods i.e. writeUTF. The writeObject method toggles to data block mode at the start and at the end of the method which results all the data being written to underlying OutputStream, this has same affect as calling flush on outputStream. This means that you cannot create another byteArray before flushing the remaining data to the stream. It would be best if you stick with writeObject method for now; ie

public void writeObject(ObjectOutput out) throws IOException {
    out.writeUTF(messageUID);
    out.writeUTF(rawData);
    out.writeUTF(data);
    out.writeLong(type);
    out.writeBoolean(processed);
    if (processed) {
        out.writeUTF(processedMessage);
        String tempDetailsMessage[] = processedDetaildMessage.split(" more");
        out.writeObject(tempDetailsMessage[tempDetailsMessage.length - 1]);
    }
}

Why this strange behaviour of ObjectOutputStream and ObjectInputStream throwing EOFException?

2 Answers2