About the unbuffered I/O in UNIX Systems

Question

I have been reading APUE recently when a basic problem turned up. My code is shown below

#include <apue.h>
#define BUFFSIZE 4096
int main()
{
    int n;
    char buf[BUFFSIZE];
    while((n = read(STDIN_FILENO, buf, BUFFSIZE)) > 0)
    {
        printf("n is %d\n", n);          //this line is added by me for testing
        if(write(STDOUT_FILENO, buf, n) != n)
            err_sys("write error");  //functions defined by the book to print error message
    }

    if(n < 0)
        err_sys("read error");
    exit(0);
}

After compiling, when I run the program as shown below

> $ ./mycat
123456[enter]
n is 7
123456
1234[enter]
n is 5
1234

It seems that it works according to the structure of my code. And I don't quite understand the function of the 'enter'. Every time I pressed 'enter', read function terminates and pass the characters including the '\n' produced by the 'enter' to the write function. So it goes inside the loop and firstly prints the number of characters read.

However, the following testing seems go against the above and against the structure of the code.

> $ ./mycat > data
123456[enter]
1234[enter]
^D
> $ cat data
123456
1234
n is 7
n is 5

It seems like that the program firstly writes all the characters to the file then prints the value of 'n', however according to my understanding, it should firstly prints as follows

n is 7
123456
n is 5
1234

I think again and again and just couldn't figure it out. Could you please help me?

alk · Answer 1 · 2019-11-06T06:23:59.907

write() is not buffered. printf()ing to stdout is buffered, but in a way depending to where the output goes.

If stdout's output goes to the console it's line buffered, if not it's fully buffered, which in your second example leads to being flushed on the program's end, whereas the outputs from the calls to write() go out immediately.

From man stdio:

[...] the standard input and output streams are fully buffered if and only if the streams do not refer to an interactive device.

Output streams that refer to terminal devices are always line buffered by default;

score 2 · Answer 2 · edited May 23 '17 at 12:05

First a solution, change read and write to fread and fwrite:

#include <apue.h>
#define BUFFSIZE 4096
int main()
{
    int n;
    char buf[BUFFSIZE];
    while((n = fread(buf, 1, BUFFSIZE, stdin)) > 0)
    {
        printf("n is %d\n", n);          //this line is added by me for testing
        if(fwrite(buf, 1, n, stdout) != n) {
            // note: if err_sys depend on errno, it may print wrong error
            err_sys("write error");
        }
    }

    if(ferror(stdin)) {
        // note: if err_sys depend on errno, it may print wrong error
        err_sys("read error");
    }
    exit(0);
}

Things to note about the code:

Here using fread is optional, because you do not read from stdio otherwise.
fread and fwrite take element size and number of elements to determine how much should be written. Partial elements will not be read, so element size 1 (not count 1) is what is usually wanted with text.
There are differences in error handling and return values, and setting of errno by stdio functions is not very well defined, see here for more.

Finally short explanation: stdio input and output is buffered. Lower level file descriptor IO (open and close, read and write etc) is not buffered and completely bypasses stdio buffering. These should not be mixed on same file, because it's easy to get mixed up on buffering details even if you try to do it so that it "should" work. Even if you get it to work on your OS, it may break when compiled for different OS and libraries. So just don't do it, instead use one or the other for the same open file.

luser droog · Accepted Answer · 2014-03-16T07:00:12.437

There are several kinds of buffering going on. The input to the program is buffered by the pseudoterminal device's Line-buffering discipline. On the output side, there is a file-system cache (a buffer in the OS for the whole file), and extra buffering in the C program when printing to a FILE * type. But read and write bypass the FILE * buffering and move data more or less directly to/from the file-system cache.

So it appears that your stdout buffer is being flushed automatically when all output is going to the terminal, but not when redirected to a file. So I'd recommend adding a call to

fflush(stdout);

after the printf call. This should explicitly flush the buffer (and enforce the ordering of the output that you want).

The important thing to be aware of is when you're using FILE *s which are a C-level structure manipulated by library functions (like fopen), and when you're using the raw file descriptor (which is just an integer, but refers to the underlying operating-system file). The FILE datatype is a wrapper around this lower level Unix implementation detail. The FILE functions implement an additional layer of buffering so the lower level can operate on larger blocks, and you can efficiently perform byte-at-a-type processing without doing lots and lots I/O handshakes.

so the input characters are line buffered and will be transmitted once 'enter' is pressed? — hercule24, Mar 16 '14 at 05:30
Yes, exactly. That happens in the pseudoterminal device and is outside of (direct) program control, so `read` and `scanf` are both buffered (or unbuffered, if you use something like `system("stty -raw")`). — luser droog, Mar 16 '14 at 06:55

About the unbuffered I/O in UNIX Systems

3 Answers3