-1

I am reading large files using the following code and sending the file content over a TCP connection buffer by buffer. At the end of each send the TCP channel adds a CRLF character. I don't want this to appear in the results unless I add it.

        final int BUFFER_SIZE = 65536;
    long bytesToSkip = 0;
    byte[] buffer = new byte[BUFFER_SIZE];

    try (RandomAccessFile rand = new RandomAccessFile(new File(requestModel.getFilePath()), "r");
    ) {

        rand.seek(bytesToSkip);
        while ((read = rand.read(buffer)) != -1) {

            MessageBuilder mb = MessageBuilder.withPayload(buffer).setHeaderIfAbsent(IpHeaders.CONNECTION_ID, connectionId);
            outMsgChannel.send(mb.build())

            buffer = new byte[BUFFER_SIZE];
        }
    }
    catch(Exceptions ..............

Sample output where new line is added. (Both the buffers are huge. I have mentioned only the lines causing problems at the end of each buffer)

Buffer One contained

A quick brown fox jumps over the lazy dog

A quick brown fox jumps over the lazy dog

A quick brown fox jumps over the

Buffer Two Contains

lazy dog

If there is no unwanted CRLF then I will not get the issue of single line splitting in to two in output.. I want to have new lines only where the file has.

learner
  • 906
  • 2
  • 10
  • 39
  • Why are you reading twice? You are *losing* data that way. And why are you ignoring the read count when building the message? And indeed apparently ignoring the *bugfffer* when sending the message? This code can't be said to work at all, let alone how you describe. – user207421 Jul 05 '17 at 08:33
  • I have edited the question. Corrected the mistakes which occured due to copy pasting the code from project to sample code. Secondly what does it mean to ignore the read bytes ? And please don't vote down the question so instantly, my real question's answer was either Spring TCP Channel has the behaviour of adding CRLF by default or the answer was that Spring provides a configuration to not to add CRLF at the end of each line. Vote down if u like to but provide the answer also. – learner Jul 05 '17 at 09:24
  • The mistake about ignoring the read count remains. – user207421 Jul 05 '17 at 09:28
  • Use it to delimit the message when you construct it.,instead of using the entire buffer. This is pretty basic. Don't ask me how to use your own API. – user207421 Jul 05 '17 at 09:35
  • Do you know the answer to my question ? I know the delimit stuff but that's a performance degrader for each read which I don't want and these unwanted bytes are ignorable at my receiver's end. – learner Jul 05 '17 at 09:37
  • I know that you're ignoring the read count, which makes your code invalid. Your statement about 'performance degradation' is complete and utter nonsense. If you don't want the right answer you can get it in zero time. The excess bytes aren't ignorable unless the receiver knows how long the message is, and at present there is no possible way to transmit that. Fix the bug. – user207421 Jul 05 '17 at 09:42
  • Then for my nonsense you are not correcting me by telling me the small piece of code which is more performance optimized. What I know is I have to split the byte array or convert it to String with desired number of read bytes which will kill the performance for 4 to 5 GB of file size reading each buffer and converting it to String. Instead of using the wrong words, you could have been really kind to guide someone if he doesn't knows the way. – learner Jul 05 '17 at 09:52
  • What you 'know' is irrelevant, as it leads to wrong answers. What you have to do, somehow, and I don't know how in your API, is not send more data than you read from the file. At present you aren't doing that, so there is no valid question here to answer. And any code you may have that relies on converting binary data to `String` has deep foundational problems that you need to address first before you get anywhere near near solving this problem. Just send the bytes. – user207421 Jul 05 '17 at 09:54
  • Can u tell me WHY there is a CRLF at the end of each TCP send using SPring Integration ? – learner Jul 05 '17 at 09:59
  • I have already done so. You are transmitting more bytes than you read. And I have also told you why that is so. Several times. – user207421 Jul 05 '17 at 10:01
  • Nope this is not the case. You just create a message as follows and still you get a CRLF at the END. I tried this before asking the question and still I was getting a newline character at the end of each received bytes at receiver end. MessageBuilder mb = MessageBuilder.withPayload("Hello World").setHeaderIfAbsent(IpHeaders.CONNECTION_ID, connectionId); outMsgChannel.send(mb.build()) – learner Jul 05 '17 at 10:04

1 Answers1

2

See the documentation.

TCP is a streaming protocol; this means that some structure has to be provided to data transported over TCP, so the receiver can demarcate the data into discrete messages. Connection factories are configured to use (de)serializers to convert between the message payload and the bits that are sent over TCP. This is accomplished by providing a deserializer and serializer for inbound and outbound messages respectively. A number of standard (de)serializers are provided.

The ByteArrayCrlfSerializer, converts a byte array to a stream of bytes followed by carriage return and linefeed characters (\r\n). This is the default (de)serializer and can be used with telnet as a client, for example.

...

You need some way to know when the message is complete - the underlying network might packetize your message so it is received in chunks.

The ByteArrayRawSerializer adds no characters to the message; it might satisfy your needs. When used on the reading side, it uses the socket EOF to indicate the message is complete.

Community
  • 1
  • 1
Gary Russell
  • 166,535
  • 14
  • 146
  • 179