2

I am using 2.6.32 OMAP based linux kernel. I have observed that at high speed data rate (Serial port set to 460800 baud rate) serial port HW fifo overflow happens.

The serial port is configured to generate interrupt at every 8 bytes in rx and tx both direction (i.e when the serial port HW fifo is 8 byte full serial interrupt is generated which reads the data from the serial port at once).

I am transmitting 114 bytes packet continuously (Serial driver has no clue about the packet mode, it receives data in raw mode). Based on calculations,

460800 bits/sec => 460800/10 = 46080 bytes/sec (Where 1 stop bit and 1 start bit) so in 1 second I can transmit under worst case 46080/114 => 404.21 packets without any issue.

But, I expect the serial port to handle at least 1000 packets per second as such I have configured serial driver to generate interrupt every 8 bytes.

I tried the same using windows XP and I am able to read upto 600 packets / second.

Do you think this is feasible on linux under above circumstances? or I am missing something? Let me know your comments.

could someone also, send some important configuration settings that needs to be configured in .config file. I am unable to attach .config file otherwise, I can share it.

dsolimano
  • 8,870
  • 3
  • 48
  • 63
user1867459
  • 423
  • 2
  • 8
  • 27

1 Answers1

1

There are two kind of overflows that can occur for a serial port. The first one is the one you are talking about, the driver not responding to the interrupt fast enough to empty the FIFO. They are typically around 16 bytes deep so getting a fifo overflow requires the interrupt handler to be unresponsive for 1 / (46080 / 16) = 347 microseconds. That's a really, really long time. You have to have a pretty drastically screwed up driver with a higher priority interrupt to trip that.

The second kind is the one you didn't consider and offers more hope for a logical explanation. The driver copies the bytes from the fifo into a receive buffer. Where they will sit until the user mode program calls read() to read them. Getting an overflow on that buffer will happen when don't configure any kind of handshaking with the device and the user mode program is not calling read() often enough. It looks exactly like a fifo buffer overflow, bytes just disappear. There are status bits to warn about these problems but not checking them is a frequent oversight. You didn't mention doing that either.

So start by improving the diagnostics, do check the overflow status bits to know what's going on. Then do consider enabling handshaking if you find out that it is actually a buffer overflow issue. Increasing the buffer size is possible but not a solution if this is a fire-hose problem. Getting the user mode program to call read() more often is a fix but not an easy one. Just lowering the baud rate, yeah, that always works.

Hans Passant
  • 922,412
  • 146
  • 1,693
  • 2,536
  • The serial driver gets data at 460800 baud rate and it passes data to user space. The user space code sends it to another application over network socket/Ethernet after processing it. During the data transfer only 3/4 interrupts are happening. GP_TImer, EthTx and EthRx and Serial Rx. What I observed that after sending several hundred kilo bytes the serial driver HW FIFO occurs first (printk in the serial driver code). Once, I found that I stopped sending data over the socket(reduced the overhead of the Ethernet interrupts) but just dumped to a file, this setup worked without any issues. – user1867459 Jan 11 '13 at 19:14
  • I set serial driver HW fifo receive trigger level to 8 bytes which gives me around 56 bytes time to process the next interrupt. around 1.2 ms (46080 bytes in 1 sec , time to fill 56 bytes = 56/46080 => 1.215 milliseconds). I am not able to understand whether it is Ethernet driver issue or something else? How do I make further progress? It is for sure that the serial driver HW FIFO overrun is happening since the interrupts in the system are either disabled for more than 1.2 ms or other interrupt is taking time greater than 1.2 ms. – user1867459 Jan 11 '13 at 19:17
  • 1
    I see no consideration at all for the receive buffer overflow possibility. Don't just comment out the socket send() call, replace it with sleep(1000). And verify that you still don't lose any data. I'll put a buck on that you do. – Hans Passant Jan 11 '13 at 19:59
  • Hi, Thanks for the quick reply. I don't have access to user space code right now, but once I have access, I will try inserting sleep (1000) and verify that I loose data or not. If I understand your comments correctly, according to you there is no connection between Ethernet interrupts frequency and the HW FIFO overflow? if that is the case, some where in the user space code (the one sends data to network app or the network app itself which receives data) there are some instances which triggers sleep / or makes system non-preemptive for more than 1 ms. How do I investigate this? – user1867459 Jan 12 '13 at 18:22
  • As you mentioned, HW FIFO overflow happens upon introducing delay of 1000us before send. So, the claim that Ethernet interrupts are causing this is ruled out. I also noticed that when the HW FIFO overflow happens the ISR takes 6000 to 8000 microseconds (which it self is too much time). I am unable to find relationship between **disabling the send over socket** doesn't result in HW FIFO but sending the information over socket results in HW FIFO. What causes serial driver to take so long with socket? I know ISR can't be preempted. Does it has to do with system load or scheduler? Comments – user1867459 Jan 16 '13 at 17:19
  • Sigh. You've been ignoring my advice long enough. Now that you know my answer is accurate, it is high time you start paying attention to the recommendations. Please stop asking for more help, you have what you need. – Hans Passant Jan 16 '13 at 17:51
  • I wasn't ignoring your advice at all. I followed those and found that the user space program was reading slow causing 64k buffer overflow followed by HW FIFOoverflow. Also, there were couple of issues with the sending subsystem. Thanks for all the help. Now, I am able to transfer 1000 tags/second with no issues. – user1867459 Jan 17 '13 at 21:17