3

I am trying to write my spatial data from a table to a file. But I need to know the exact size of the data on disk before writing to disk. As an example, let's say that I am writing to disk using the following code:

    FileOutputStream fos = new FileOutputStream("t.tmp",false);
    ObjectOutputStream oos = new ObjectOutputStream(fos);
    oos.writeInt(gid);
    oos.writeUTF(fullname);
    oos.writeInt(d.shape.length);
    oos.write(d.shape);

    oos.close();
    fos.close();

I was thinking that file size on disk is equal to:

size= 4B {for gid, int} + fullname.getBytes.length() {string} + 4B {d.shape.length, int} + d.shape.length

but in fact, this is very different than the real file size on disk.

I also noticed that even creating an empty file using ObjectOutputstream leads to 4B space on disk.

Any help on how to calculate the file size on disk?

(I can't write the data to disk and then read the real size. This will lower the performance. Instead, I need to calculate the size of data on disk based on data values stored in memory.)

reza
  • 1,188
  • 3
  • 17
  • 32
  • Do you really need an *Object*OutputStream? Do you use `writeObject()`? Or is your example code as sophisticated as you get? It seems that you would be better off to use a [DataOutputStream](http://docs.oracle.com/javase/6/docs/api/java/io/DataOutputStream.html). (An `ObjectOutputStream` writes stream header information and keeps track of references to objects already written so it can simply write a reference etc - all of which would hinder your efforts to pre-calculate the size.) – Greg Kopff Jun 26 '12 at 02:53
  • Also - can you expand on the performance problem that you perceive from using the file system after the bytes are written? – Greg Kopff Jun 26 '12 at 02:55
  • in terms of the performance problem. I am having 40GB of data in my table and many of them are spatial data. I am partitioning my table in a way that total size of the data stored in each file is less than a certain value (max_file_size). In first round, I calculate the size on disk of each row of my table and in the next rounds, I sum so many of them that can fit within a file with total file size less than max_file_size. Thus, writing every file and then measuring the real file size on disk is not an option. – reza Jun 26 '12 at 03:22
  • in terms of the use of ObjectOutputStream, I am not sure if that is the best way to write to disk. I am having several fields to type varchar(x) in addition to a geometry which is a byte array of variable length. I thought ObjectOutputStream would be the easiest way to read/write to file. In fact, it worked fine so far. The only problem is that I have to calculate the size on disk based on the data values in advance. Any suggestion? – reza Jun 26 '12 at 03:24
  • 1
    The usefulness of `ObjectOutputStream` is its ability to serialise a complex graph of objects automagically. This doesn't appear to be what you're doing however, so I would suggest you dispense with the `ObjectOutputStream` and just write the bytes yourself using a [`DataOutputStream`](http://docs.oracle.com/javase/6/docs/api/java/io/DataOutputStream.html). A `DataOutputStream` isn't doing anything automagically, and so you'll be able to do the size calculations that you expect. Take note of the Javadoc for `DataOutputStream.writeUTF()` as it writes 2 bytes of length data, then the String. – Greg Kopff Jun 26 '12 at 03:41
  • @reza if you have 40 GB of data, I doubt ObjectOutputStream is the best choice. It is designed to be generic, easy and flexible but is not compact or terribly efficient. If you need to know the size in advance, you are much better off using your own disk format if capacity is an issue. Why do you need to know the size anyway? – Peter Lawrey Jun 26 '12 at 06:30

2 Answers2

2

I am trying to write my spatial data from a table to a file. But I need to know the exact size of the data on disk before writing to disk.

You shouldn't use an ObjectOutputStream. An ObjectOutputStream can automatically serialise a complex graph of objects for you - but this doesn't appear to be one of your requirements. As part of this serialisation, the ObjectOutputStream writes some stream header information (this is the 4 bytes you discovered at the beginning), and also keeps track of objects written previously so that it can write special marker values rather than writing out the whole object again.

Instead, just use a DataOutputStream. It provides the same functionality you want:

A data output stream lets an application write primitive Java data types to an output stream in a portable way. An application can then use a data input stream to read the data back in.

FileOutputStream fos = new FileOutputStream("t.tmp",false);
DataOutputStream dos = new DataOutputStream(fos);
dos.writeInt(gid);                 // write 4 bytes
dos.writeUTF(fullname);            // write 2 bytes of length, then variable length string (UTF encoded)
dos.writeInt(d.shape.length);      // write 4 bytes
dos.write(d.shape);                // write a variable length byte array

dos.close();
fos.close();

There won't be any surprises here (provided you know how many bytes your UTF encoded String will end up), and you can do the arithmetic to calculate what the exact file size will be.

(If you were dealing with strings that didn't just equate to one-character-one-byte, you could render the string to a byte array first using a charset encoder).

Greg Kopff
  • 15,945
  • 12
  • 55
  • 78
  • thanks a lot. this solved the problem. Seems that I need to refresh my memory on Java streams. Do you recommend any tutorial on Java streams? – reza Jun 26 '12 at 07:15
1

Assuming you don't mind wasting some memory, you can write it all out to a ByteArrayOutputStream first, then get the size.

ByteArrayOutputStream baos = new ByteArrayOutputStream();
ObjectOutputStream oos = new ObjectOutputStream(boas);
oos.writeInt(gid);
oos.writeUTF(fullname);
oos.writeInt(d.shape.length);
oos.write(d.shape);

oos.close();
boas.close();
int size = boas.size();
wolfcastle
  • 5,850
  • 3
  • 33
  • 46
  • great. this works fine if I want to calculate the size of one row of data. I am calculating this for every row of data and store it in my table under block_size column. I then select so many of the rows that their cumulative block_size is less than a certain value. The problem is that the sum of block_size values of two rows is greater than the file size of the two rows written to disk. Any idea on how to solve this part? – reza Jun 26 '12 at 03:19
  • @reza If you need it to be accurate, you have to serialize the entire data structure in memory first (or to a temporary file) Java Serialization will remove duplicate objects and Strings to a degree so you cannot assume that double the data will take twice as much space. – Peter Lawrey Jun 26 '12 at 06:28
  • thanks, I ended up using DataOutputStream. This way, my calculated size is exactly as the file size on disk. thanks for the suggestions. – reza Jun 26 '12 at 07:25