0

i ask myself if there is a nice solution for receiving data via TCP and reading the data in a DataInputStream and store the data in a bytearray of "dynamic" size. I thought about some solutions like writing in buffers and finally store it in a created array that is as large as the packet i received.

Here an example: the data i receive via TCP (byte by byte) is n*13 Byte large and the end of a packet is 13 Bytes of zeros (13 Byte of zeros are unique, cant be in the data before). The next packet is m*13 Byte + 13 Byte of zeros and so on. So i want to listen to the stream and store e.g. n*13 Byte in a bytearray without zeros (i dont know the size of one dataset before).

Can you tell me how to do it a slim way?

Thanks in advance!

Chris

Chris Jung
  • 49
  • 1
  • 6

1 Answers1

0

I would read the data straight into a direct ByteBuffer. You don't need a DataInputStream as well. I would have the ByteBuffer larger than needed e.g. 1 MB or 10 MB and use the fact that unused direct buffers (like unused native memory) only uses virtual memory, not real memory. I would make sure this buffer is re-used as much as possible, ideally for the life of the application.

In a 64-bit environment you can make the buffers really large. e.g. 1 GB without much impact.

If you want to minimise memory consumption, I would process the data as you receive it. That way you avoid having to store it first. It sounds like the minimum size is 13 bytes and would make a reasonable buffer size like 512 bytes or 2 KB.

Peter Lawrey
  • 525,659
  • 79
  • 751
  • 1,130
  • thanks for your answer, but what do you mean with "process data as received"? i want to store the bytearrays in a hashmap, so there is memory usage anyway. best solution so far is an oversized buffer? – Chris Jung Mar 20 '13 at 06:21
  • i used an oversized buffer and it works great. will add code soon. – Chris Jung Mar 20 '13 at 20:03
  • Just make sure these buffers are recycled, rather than letting the GC recycle them. Letting the GC recycle the buffers can result in other pages being used which works, but isn't as efficient. – Peter Lawrey Mar 21 '13 at 23:01