serial data flow: How to ensure completion

Question

I have a device that sends serial data over a USB to COM port to my program at various speeds and lengths.

Within the data there is a chunk of several thousands bytes that starts and ends with special distinct code ('FDDD' for start, 'FEEE' for end). Due to the stream's length, occasionally not all data is received in one piece. What is the recommended way to combine all bytes into one message BEFORE parsing it? (I took care of the buffer size, but have no control over the serial line quality, and can not use hardware control with USB)

Thanks

score 0 · Accepted Answer · answered Aug 20 '14 at 07:28

One possible way to accomplish this is to have something along these lines:

# variables
#   buffer: byte buffer
#   buffer_length: maximum number of bytes in the buffer
#   new_char: char last read from the UART
#   prev_char: second last char read from the UART
#   n: index to the buffer

new_char := 0
loop forever:
    prev_char := new_char
    new_char := receive_from_uart()

    # start marker
    if prev_char = 0xfd and new_char = 0xdd
        # set the index to the beginning of the buffer
        n := 0
    # end marker
    else if prev_char = 0xfe and new_char = 0xee
        # the frame is ready, do whatever you need to do with a complete message
        # the length of the payload is n-1 bytes
        handle_complete_message(buffer, n-1)
    # otherwise
    else
       if n < buffer_length - 1
           n := n + 1
           buffer[n] := new_char

A few tips/comments:

you do not necessarily need a separate start and end markers (you can the same for both purposes)
if you want to have two-byte markers, it would be easier to have them with the same first byte
you need to make sure the marker combinations do no occur in your data stream
if you use escape codes to avoid the markers in your payload, it is convenient to take care of them in the same code
see HDLC asynchronous framing (simply to encode, simple to decode, takes care of the escaping)
handle_complete_message usually either copies the contents of buffer elsewhere or swaps another buffer instead of buffer if in hurry
if your data frames do not have integrity checking, you should check if the payload length is equal to buffer_length- 1, because then you may have an overflow

Thanks DrV. I accepted because the algorithms seems to be logical (although it is not c#). It differs significantly from what I do: While the above handles each byte, I use higher level code, reading the port as a strings, and manipulate the strings (or char arrays) as may be required. It works nice except that long messages may be divided. I am still looking for another solution that will handle the messages as char strings rather than bytes. — samtal, Aug 21 '14 at 05:00
Actually, the algorithm is not very different if you read the serial line as longer chunks. As long as the lower level does not take care of the frames, you will have to go through the message one byte at a time to find the markers. (BTW, your question does not mention any specific programming language, hence the pseudo-code.) — DrV, Aug 21 '14 at 05:13
Sorry about the c# language comment. I thought I tagged the question as c#, WinForm. In any case, the algo was clear enough to be converted to c# which I did and got it to work. The c# enables easy string handling, thus finding the beginning and ending is easy. My problem with the suggested is that I have several other messages with other starts and endings, thus it will necessitate significant modifications, but I still may use your advice. Thx — samtal, Aug 21 '14 at 06:26

samtal · Answer 2 · 2014-08-24T05:20:55.723

After several tests, I came up with the following simple solution to my own question (for c#). Shown is a minimal simplified solution. Can add length checking, etc. 'Start' and 'End' are string markers of any length.

    public void comPort_DataReceived(object sender, SerialDataReceivedEventArgs e)

    SerialPort port = (SerialPort)sender;

    inData = port.ReadExisting();
    {
    if (inData.Contains("start"))
           {
          //Loop to collect all message parts   
          while (!inData.Contains("end"))
             inData += port.ReadExisting();  
          //Complete by adding the last data chunk
           inData += port.ReadExisting();
           }                    
          //Use your collected message  
          diaplaydata(inData);

serial data flow: How to ensure completion

2 Answers2