-1

I'm modifying the source code of H2 MVStore 1.4.191 to write files by doing some thread sleep. The big change is that the file is not written in one time anymore, but by 2^16 bytes chunks. MVStore uses java nio FileChannel and ByteBuffer to write its file. The problem is that the result is different from the original version. It seems that FileChannel add space characters (0x20 in ASCII), like, more than 40 in a row. Or maybe it doesn't remove this spaces, on the contrary to the original version, I don't know.

I suppose it's due to file writing. The method file.write(buffer,position), where file is FileChannel object, and that returns the number of bytes written, sometimes returns a smaller number than the buffer size, in the original version of H2. In my version, it never happens.

Have you tips about ByteBuffer, FileChannel and my problem ?

Joss29
  • 19
  • 2
  • are you storing 8 bits character encoding or 16 bits? – Tschallacka Jan 11 '18 at 09:19
  • It writes 8 bits characters – Joss29 Jan 11 '18 at 09:37
  • Yes, you send it 8 bit characters, but if you set the encoding to a 16 bit format for the output writer, data will bet appended to fill up the remaining bits. Check your encoding settings on your output writers. – Tschallacka Jan 11 '18 at 10:05
  • I'm not sending character. Not only. H2 uses ByteBuffer to store the data before writing. I just changed the way the buffer is written (the way the write function of FileChannel is used) – Joss29 Jan 11 '18 at 10:53
  • Show your code then. so we can see how you implemented it – Tschallacka Jan 11 '18 at 11:01

2 Answers2

0

The original code call writefully function few times (it writes a header, a footer and the datas) 

 int off = 0;
            do {
                int len = file.write(src, pos + off);
                off += len;
            } while (src.remaining() > 0);

src is the ByteBuffer and file is a FileChannelImpl from sun.io. Buffer can contain more than 50MB of datas.

From this code, I developped a solution that split the ByteBuffer in 2^16-sized buffers that I write, by adding sleep function between each of them:

            int off = 0;
            byte[] buffer = src.array();
            int size = src.array().length;
            int chunkSize = 128;
            List<byte[]> splittedBuffer = new ArrayList<byte[]>();
            int i = 0;
            while (i < size) {
                int start = i;
                int end = i + chunkSize;
                if (end > size)
                {
                    //if buffer size is not a multiple of 2^16, the last 
                    //chunk will be smaller 
                    end = size;
                }
                splittedBuffer.add(Arrays.copyOfRange(src.array(), start, end));
                try {
                    Thread.sleep(5);
                } catch (InterruptedException e) {
                    // TODO Auto-generated catch block
                    e.printStackTrace();
                }
                i += chunkSize;
            }
            int offset = 0;
            for (byte[] chunk : splittedBuffer) {
                int len=file.write(ByteBuffer.wrap(chunk),pos+offset);
                offset+=len;
                try {
                    Thread.sleep(5);
                } catch (InterruptedException e) {
                    // TODO Auto-generated catch block
                    e.printStackTrace();
                }
            }

Finally, the problem is maybe not that whitespaces are added, but that a part of datas is written in a wrong place. I'm going to check it.

Joss29
  • 19
  • 2
  • I checked the writing process: In the original version, it writes 8192 B first. And it appends 53747712 B at the end . Which means 53755904 B of datas. But the file's size is finally 52367360... – Joss29 Jan 11 '18 at 13:50
0

Ok,

The problem was that I used the size of ByteBuffer to split it instead of its limit which is smaller (set by H2 during its process)

Thanks for the help

Regards

Joss29
  • 19
  • 2