Fastest fgets implementation in C

Question

The known fgets libc function implementation uses fgetc() inside, how can I use read() with bigger buffer or other method instead to speed up the function?

For example I read /proc/pid/maps file to search some strings. The file's format is known, currently I use the fgets implementation in the link with read(fd, &c, 1); instead of getc. I think reading single byte from file is slower than reading 200 bytes. So I want to modify the function to read N bytes from file and then finds a line break. I think replacing 1 byte read can speed up the function somehow.

What are you changing about the requirements and API of the function you want to rewrite? For example, do you know a minimum length of the returned string? That would allow to read more than one char at a time. — Yunnosch, Jul 07 '18 at 08:25
Yes, I read some known linux files with known format, that means I can give the minimum length or minimum buffer size as a parameter. — Phillip, Jul 07 '18 at 08:35
Please provide more details, show sample input. Explain the rules/syntax of the input. Show the code which you have, I understand it works and you are looking for optimisation. And please try to demonstrate that you have actually spent some effort on this, e.g. on reasearching other input functions. Explain why you don't use fscanf. Explain what probelm you want to solve by optimisation. Many people here are likely to tell you that trying to optimise something by rewriting lib functions is more indicating that your got a problem elsewhere than anything else. Show some profiling info. — Yunnosch, Jul 07 '18 at 08:48
Which documentation did you read? I recommend this one: http://pubs.opengroup.org/onlinepubs/9699919799/functions/read.html — alk, Jul 07 '18 at 08:54
I don't understand why you need to modify a standard function. If you don't like that, you may write your own function (with a similar name) and use this one. — Sir Jo Black, Jul 07 '18 at 09:02
Are the records (text lines?) you want to read all of the same length (per file)? Is this length know in advance? — alk, Jul 07 '18 at 09:04
@alk I want to write my own function indeed, the text lines length are not equal, but yes it has some limits as min/max. The data is in one line, actually I scan all the lines and count some patterns, I can't read all the file once, because of lack of big buffer. So if I can improve the function, for example it can read multiple lines in one step. This is not a problem. But I don't want it to stop in the middle of a line, I want always the data to end with a line break, because other way I can miss some patterns. — Phillip, Jul 07 '18 at 09:15
"*the text lines length are not equal [...] I don't want it to stop in the middle of a line*" Think twice, how you can (or cannot) achieves this, if not reading character by character? — alk, Jul 07 '18 at 09:31
Every line-reading function I've ever written has been slower (much slower) than `fgets`. I'm pretty sure I can't write a line-reading function that's faster than `fgets`. Nothing personal, but I doubt you can, either. — Steve Summit, Jul 07 '18 at 09:42
The code you linked to is one way to write `fgets`, but it's not the only way. My guess (I should check this sometime) is that some `fgets` implementations use a fast search in the buffer to look for the `\n` character, a search which can be much faster than using `getc` to fetch one character at a time and inspect them. I assume that's how `fgets` can be so much faster than anything I can write. But working directly with the buffer is obviously something that `fgets` can do, while you or I cannot (or at least, not at all portably). — Steve Summit, Jul 07 '18 at 09:44
@SteveSummit, this is the answer I expect, can you give an example implementation or more details/keywords? — Phillip, Jul 07 '18 at 09:47
@Phillip I took a look at a copy of glibc-2.26.tar.bz2 I had lying around. `fgets` is written in terms of an internal function called `_IO_getline`, and sure enough, right in the middle there's a call to `memchr(fp->_IO_read_ptr, delim, len)`, where `_IO_read_ptr` is a pointer into the internal read buffer. — Steve Summit, Jul 07 '18 at 10:07
@Phillip Others have asked this above, but I have to ask it again: Is `fgets` a bottleneck for you? Is your program running unacceptably slowly, and is `fgets` at fault? How do you know this? — Steve Summit, Jul 07 '18 at 10:08
@SteveSummit, I measured the fgets and strstr function execution times in my searching code. fgets is slower than strstr. I though, I can implement this function using read syscall to read more data (1024+) instead of single byte. Just I imagined that it will speed up the reading. — Phillip, Jul 07 '18 at 18:11
@Philip Using `read` to read 1 character at a time is just about guaranteed to be a loss. Using `read` to read a large number of characters at once is one of the most efficient ways of reading a file, *but* it's inflexible -- in particular, it's just about guaranteed that any newline characters will be at random positions in the buffer, meaning that it will "stop in the middle of a line", and the data won't end with a line break. — Steve Summit, Jul 07 '18 at 19:01
@Phillip Also, you said "`fgets` is slower than `strstr`", but that's not the same as saying it's a bottleneck that deserves to be sped up. `fgets` is likely to be a relatively expensive part of your program, because it's (a) doing the real work of reading characters from wherever, and (b) doing the real work of searching for line breaks, which is functionality that you want and need. But the expense of those tasks is fairly inevitable; there may not be anything you can do better. — Steve Summit, Jul 07 '18 at 19:04
@SteveSummit, This is the subject of this question here, is there any way to speed up `fgets` using bulk reading with `read()` syscall? I think I can try a state machine which reads then searches or something reads bulk data then if there is not new line char, it will read one by one until it finds new line etc... these are my ideas, what are yours? — Phillip, Jul 07 '18 at 20:05
@Phillip I know that's what you're trying to do. I don't believe there is a way, for two reasons: (1) There's no way of solving the problem of having the data not end at a line break -- in general, you're always going to have a fragmentary, leftover, partial line. (2) Once you're finished, I don't believe you'll end up with anything faster than `fgets`. — Steve Summit, Jul 08 '18 at 10:17
@Phillip If the idea is that you know the minimum line length, such that you use `read` to read that many characters all at once, then `read` one character at a time until you find the `\n`, that's easy enough to code, but I believe the overhead of calling `read` one character at a time will kill you. If you want to pursue this approach, the thing to do would be to use `fread` to read the minimum line length all at once, then call `getc` until you find the `\n`. And while that would, functionally, work, I doubt it would have any speed advantage over a well-written `fgets` implementation. — Steve Summit, Jul 08 '18 at 10:20
@SteveSummit, Thanks for your suggestions. I decided to try the partial read strategy. Read(minimal line length), if there is no `\n`, then read 1 byte until a `\n`. I hope it will be faster than `read(1)`, but I will compare with the real `fgets`. I think you are right about the performance of standart fgets. Let me try and see :) — Phillip, Jul 08 '18 at 21:40

score 6 · Answer 1 · answered Jul 07 '18 at 09:18

You're totally misunderstanding the standard I/O function. Even fgetc is buffered. Test the issuance of actual read calls with strace. On my computer, reading /proc/1/maps:

read(3, "5634f9cf6000-5634f9e44000 r-xp 0"..., 1024) = 1024
read(3, "                   /lib/x86_64-l"..., 1024) = 1024
read(3, "             /lib/x86_64-linux-g"..., 1024) = 1024
read(3, "                   /lib/x86_64-l"..., 1024) = 1024
read(3, ".0.0\n7feb2b2dc000-7feb2b4db000 -"..., 1024) = 1024
read(3, "0-7feb2b8e7000 r--p 00002000 fd:"..., 1024) = 1024
read(3, "00 rw-p 0001a000 fd:00 145004   "..., 1024) = 1024
read(3, "ux-gnu/liblzma.so.5.2.2\n7feb2c1b"..., 1024) = 1024
read(3, "6_64-linux-gnu/libgcrypt.so.20.2"..., 1024) = 1024
read(3, "000 fd:00 135558                "..., 1024) = 1024
read(3, "--p 0000e000 fd:00 136910       "..., 1024) = 1024
read(3, "001e000 fd:00 131385            "..., 1024) = 1024
read(3, "1.1.0\n7feb2da14000-7feb2da15000 "..., 1024) = 1024
read(3, "0 rw-p 00000000 00:00 0 \n7feb2de"..., 1024) = 1024
read(3, "-237.so\n7feb2e492000-7feb2e69100"..., 1024) = 1024
read(3, " \n7feb2ed15000-7feb2ed36000 rw-p"..., 1024) = 637
read(3, "", 1024)                       = 0

The read calls attempt to read 1024 bytes, not just one.

The program is

#include <stdio.h>

int main(void) {
    FILE *f = fopen("/proc/1/maps", "r");
    while (1) {
        char buf[2048];
        if (! fgets(buf, 2048, f)) {
            break;
        }
    }
}

Should 1024 bytes not be enough for you, you can change the size of the underlying buffer with setvbuf(3)!

Thank you for helping me to understand how it works. What If I use a syscall read directly, without buffering, I think it will read only one byte, thus it is not buffered? In this situation I need something like state machine, which fetches data then searches for line break, right? Actually I want to implement that logic as custom function. — Phillip, Jul 07 '18 at 09:27
@Phillip well yes if you purposefully use the low-level function (that is, [read()](https://linux.die.net/man/2/read)) then you will get exactly what you ask it, without any smart behaviors. You'll have to add that yourself, basically just re-implementing `fgetc`/`fgets`, probably slower and possibly buggy. Another option if you are willing to mess with low level stuff would be to [map the file in your address space](http://man7.org/linux/man-pages/man2/mmap.2.html). — spectras, Jul 07 '18 at 10:20
The `mmap` seems cool thing, I can search it a little bit. So that means I can make any file part of my memory and I can access anytime this file with high read speed. Cool, thanks! — Phillip, Jul 07 '18 at 20:07
@spectras **except** you cannot mmap `/proc//maps`. They've got size 0. — Antti Haapala -- Слава Україні, Jul 08 '18 at 07:44

Sir Jo Black · Answer 2 · 2018-07-07T10:11:59.327

Here an hint for you (but shall be modified to read more than one byte a time)

... And we've the function fgetsR() that uses the function read() instead of fgetc(). In the code below there're two functions:

fgets0() is a K&R fgets() function
fgetsR() is a fgets() function that uses read() instead of fgetc().

The function fgetsR is used in the main to read a file (specified into the command line) and display its contents.

#include <unistd.h>

#include <stdlib.h>
#include <time.h>
#include <malloc.h>

char * fgets0(char *dst, int max, FILE *fp);
char * fgetsR(char *dst, int max, FILE *fp);

char * fgets0(char *dst, int max, FILE *fp)
{
    int c;
    char *p;

    /* get max bytes or upto a newline */

    for (p = dst, max--; max > 0; max--) {
        if ((c = fgetc (fp)) == EOF)
            break;
        *p++ = c;
        if (c == '\n')
            break;
    }
    *p = 0;
    if (p == dst || c == EOF)
        return NULL;
    return (p);
}

char * fgetsR(char *dst, int max, FILE *fp)
{
    int c;
    char *p,x;

    /* get max bytes or upto a newline */

    for (p = dst, max--; max > 0; max--) {
        if ((c = read(fileno(fp),&x,1)) == EOF || !c)
            break;
        *p++ = x;
        if (x == '\n')
            break;
    }
    *p = 0;
    if (p == dst || c == EOF || !c)
        return NULL;
    return (p);
}


int main(int argc, char *argv[])
{
    FILE * f = NULL;
    char buffer[1024];

    if (argc<2) {
        puts("You have to specify a file to read!");
        return 1;
    }

    f=fopen(argv[1],"r");

    while(fgetsR(buffer,sizeof(buffer),f)) {
        printf("%02X %s",*buffer,buffer);
    }

    fclose(f);

    return 0;
}

I don't think that solution is the better and faster. The main() above needs some workaround to control the flow when errors occur.

Thank you for the demonstration, I'm currently using the `fgetsR` code with `open()` syscall. The problem is to modify this code to read bulk data and search random placed new line char. That's the question. — Phillip, Jul 07 '18 at 20:08
@Phillip, I don't understand your request. What do you mean saying: "to read bulk data and to search random placed new line char"? The function fgetsR stops when retrieve a new-line char! — Sir Jo Black, Jul 22 '18 at 14:12
I mean reading more than 1 byte to speed up the function when `open` syscall is used — Phillip, Jul 23 '18 at 19:04

Fastest fgets implementation in C

2 Answers2