Reading stream over TCP on a SocketChannel with undefined number of Bytes

Question

I am trying to read a stream on a SocketChannel without defining the number of bytes.

The alternate solution i thought about is storing different ByteBuffers of a pre-defined size into a list which will allow me afterwards to allocate a new ByteBuffer of the received size and put the result inside.

The problem is that i am on blocking-mode and cannot find a valid condition to leave the loop i made on the read method check the code:

public static final Charset charsetUTF8 = Charset.forName("UTF-8");
public static final int BUFFER_SIZE = 1024;

public static String getUnbounded(String st, SocketAddress address) throws IOException {
    SocketChannel sc = SocketChannel.open(address);
    sc.write(charsetUTF8.encode(st));
    List<ByteBuffer> listBuffers = new ArrayList<>();
    ByteBuffer buff = ByteBuffer.allocate(BUFFER_SIZE);
    while( sc.read(buff) > -1){
        if(buff.remaining() == 0){
            listBuffers.add(buff);  
            buff.clear();
        }
    }

    listBuffers.add(buff);
    ByteBuffer finalBuffer = ByteBuffer.allocate(BUFFER_SIZE * listBuffers.size());
    for(ByteBuffer tempBuff: listBuffers){
    finalBuffer.put(tempBuff);
        tempBuff.clear();
    }
    finalBuffer.flip();

    return charsetUTF8.decode(finalBuffer).toString();
}

Any idea on how to solve this?

Why use channels at all? You're doing blocking IO, and storing data into heap-allocated buffers. Do you really think you're gaining *anything* over a simple `SocketInputStream`? — kdgregory, Mar 13 '14 at 00:14
@kdgregory I need channels because i will need to use selector later — Hosni, Mar 16 '14 at 15:40

Ted Hopp · Answer 1 · 2014-03-12T23:57:41.330

2

You can't just clear() the byte buffer. You need to allocate a new one; otherwise the same buffer is being added to listBuffers repeatedly.

ByteBuffer buff = ByteBuffer.allocate(BUFFER_SIZE);
while( sc.read(buff) > -1){
    if(buff.remaining() == 0){
        listBuffers.add(buff);  
        buff = ByteBuffer.allocate(BUFFER_SIZE);
    }
}
if (buff.position() > 0) {
    listBuffers.add(buff);
}

Since the last buffer might not (probably will not) be full, you should calculate the finalBuffer size taking this into account.

edited Mar 12 '14 at 23:57

answered Mar 12 '14 at 23:52

Ted Hopp

232,168
48
399
521

clear() method clears the ByteBuffer after it's stored and then the read() method puts its new Bytes inside – Hosni Mar 12 '14 at 23:57
1

@Hosni - Yes, but you're adding the **same** `ByteBuffer` object to `listBuffers` each time. At the end, it will be full of identical copies of `buff` in whatever the state `buff` was left when the loop exits. You need a separate object for each element of `listBuffers`. – Ted Hopp Mar 12 '14 at 23:58
But as i already declared and created a ByteBuffer outside of the loop i can't just create another one with the same identifier inside it – Hosni Mar 13 '14 at 00:06
@Hosni Yes you can. It's called an 'assignment statement'. – user207421 Mar 13 '14 at 00:07
@Hosni - What behavior are you seeing? Did the behavior change when you tried my code? – Ted Hopp Mar 13 '14 at 03:02
@TedHopp i used your method to get a "clean" reception it works. thanks – Hosni Mar 13 '14 at 21:01

score 0 · Answer 2 · edited Oct 07 '21 at 06:26

0

The number of bytes in an HTTP response stream is not 'undefined'. See the RFC. It is defined by either:

EOS in the case of a connection which is closed (HTTP 1.0 or Connection: close),
The Content-Length header, or
The result of decoding the chunked-encoding format.

It is essential that it be defined in one of these ways, and maybe there are others, so that HTTP persistent connections can work, where there may be another response following this one.

I would like to know why you are implementing this at all, when the HttpURLConnection class already exists, along with various third-party HTTP clients, which already implement all this correctly, and many other things besides.

edited Oct 07 '21 at 06:26

Community

1
1

answered Mar 12 '14 at 23:53

user207421

305,947
44
307
483

Because this is an example with http stream but i cant it to work with all streams – Hosni Mar 13 '14 at 08:23
1

@Hosni I answered the question you asked. You said, before your edit, that you were working with an HTTP stream. You're now asking a completely different question. HTTP streams have different ways of defining their numbers of bytes from other streams. A *TCP* stream can just be read to EOS, as per your current code. Another application protocol might send the length ahead of each message, or use an encoding like type-length-value, XDR, XML, etc. There isn't a single universal solution. – user207421 Mar 14 '14 at 04:35

score -1 · Accepted Answer · answered Mar 13 '14 at 21:00

-1

The solution is that to get out of the loop i had to call:

sc.shutdownOutput();

Which closes the writing stream without closing the reading stream and set the sc.read(buff) to -1

answered Mar 13 '14 at 21:00

Hosni

668
8
29

That's not the solution to the question you originally asked. And you haven't clarified that you needed to call `shutdownOutput()` *at the sender.* -1 – user207421 Mar 14 '14 at 04:44
i didn't know that i would need to close writing stream, i just needed the method that would allow me to leave the loop and set read() method to -1 and i said it in the question. – Hosni Mar 16 '14 at 15:38
The point is that you shouldn't have been in the read() method at all: you should have known how many bytes were in the HTTP response. – user207421 Mar 23 '14 at 17:38
@EJP but it's not a HTTP request – Hosni Mar 23 '14 at 20:03

Reading stream over TCP on a SocketChannel with undefined number of Bytes

3 Answers3