3

From what I understood about buffers: a buffer is a temporarily stored data.

For example: let's assume that you wanted to implement an algorithm for determining whether something is speech or just noise. How would you do this using a constant stream flow of sound data? It would be very difficult. Therefore, by storing this into an array you can perform analysis on this data.

This array of data is called a buffer.

Now, I have a Linux command where the output is continuous:

stty -F /dev/ttyUSB0 ispeed 4800 && awk -F"," '/SUF/ {print $3,$4,$5,$6,$10,$11,substr($2,1,2),".",substr($2,3,2),".",substr($2,5,2)}' < /dev/ttyUSB0

If I were to write the output of this command to a file, I won't be able to write it, because the output is probably block buffered and only an empty text file will be generated when I terminate the output of the above command (CTRL+C).

Here is what i mean by Block Buffered.

The three types of buffering available are unbuffered, block
buffered, and line buffered. When an output stream is unbuffered, information appears on the destination file or terminal as soon as written; when it is block buffered many characters are saved up and written as a block; when it is line buffered characters are saved up until a newline is output or input is read from any stream attached to a terminal device (typically stdin). The function fflush(3) may be used to force the block out early. (See fclose(3).) Normally all files are block buffered. When the first I/O operation occurs on a file, malloc(3) is called, and a buffer
is obtained. If a stream refers to a terminal (as stdout normally does) it is line buffered. The standard error stream stderr is always unbuffered by default.

Now, executing this command,

stty -F /dev/ttyUSB0 ispeed 4800 && awk -F"," '/SUF/ {print $3,$4,$5,$6,$10,$11,substr($2,1,2),".",substr($2,3,2),".",substr($2,5,2)}' < /dev/ttyUSB0 > outputfile.txt

An empty file will be generated because the buffer block might have not been completed when I terminated the process, and since i don't know the block buffer size, there is no way to wait for the block is complete.

In order to write the output of this command to a file I have to use fflush() inside awk, which would successfully write the output into the text file, which I have already done successfully.

Here it goes:

stty -F /dev/ttyUSB0 ispeed 4800 && awk -F"," '/GGA/ {print  "Latitude:",$3,$4,"Longitude:",$5,$6,"Altitude:",$10,$11,"Time:",substr($2+50000,1,2),".",substr($2,3,2),".",substr($2,5,2); fflush(stdout) }' < /dev/ttyUSB0 | head -n 2 > GPS_data.txt

But my question is:

Is there any way to declare the buffer block size so that I would know when the buffer block in generated, so eliminating the need of using fflush()?

OR

Is there anyway to change buffer type from Block buffered to unbuffered or line buffered ?

Sufiyan Ghori
  • 18,164
  • 14
  • 82
  • 110
  • Don't quite understand your question, but you can create custom struct and function, recording information you need, like current buffer data size. – moeCake Jan 14 '14 at 11:29
  • 1
    your question explains buffer concept.. :) – Raghu Srikanth Reddy Jan 14 '14 at 11:34
  • I'm pretty sure bash calls `fclose` on `GPS_data.txt` itself which `fflush`es the buffer once the command terminates, unless it terminates with a signal, such as segmentation fault or others. That is, if you simply do `cmd > file` and the `file` does not get written (while `cmd` actually produced output), then there's something wrong with your `cmd`. – Shahbaz Jan 14 '14 at 11:40
  • the command is totally fine as far as i know...if there was any error, someone would have spotted the error by now as it has been discussed in detailed here...http://stackoverflow.com/questions/20788774/awk-command-output-to-a-text-file-the-command-is-too-complex – Sufiyan Ghori Jan 14 '14 at 11:44
  • 1
    @Shahbaz kindly look at the details added. – Sufiyan Ghori Jan 14 '14 at 14:26
  • @SufiyanGhori, wait, isn't that question exactly the same as this question? Also, you didn't mention using CTRL+C which sends the signal to kill the command, which is exactly the reason why the buffer doesn't get flushed. – Shahbaz Jan 14 '14 at 14:38
  • Hello, the two questions are different..in the previous question i had no idea how to write the output to a file and then somebody suggested to use fflush(), while in this question i wanted a way to convert from Block buffer to other buffer type to avoid using fflush(). let me add the missing details. :) – Sufiyan Ghori Jan 14 '14 at 14:40

1 Answers1

3

You can use stdbuf to run a command with a modified buffer size.

For example, stdbuf -o 100 awk ... will run awk with a 100 byte standard output buffer.

Matt Eckert
  • 1,946
  • 14
  • 16