"tail -f"-like generator

Question

I had this convenient function in Python:

def follow(path):
    with open(self.path) as lines:
        lines.seek(0, 2)  # seek to EOF

        while True:
            line = lines.readline()
            if not line:
                time.sleep(0.1)
                    continue
                yield line

It does something similar to UNIX tail -f: you get last lines of a file as they come. It's convenient because you can get the generator without blocking and pass it to another function.

Then I had to do the same thing in Go. I'm new to this language, so I'm not sure whether what I did is idiomatic/correct enough for Go.

Here is the code:

func Follow(fileName string) chan string {

    out_chan := make(chan string)

    file, err := os.Open(fileName)
    if err != nil {
        log.Fatal(err)
    }

    file.Seek(0, os.SEEK_END)
    bf := bufio.NewReader(file)

    go func() {
        for {
            line, _, _ := bf.ReadLine()

            if len(line) == 0 {
                time.Sleep(10 * time.Millisecond)
            } else {
                out_chan <- string(line)
            }
        }

        defer file.Close()
        close(out_chan)
    }()

    return out_chan
}

Is there any cleaner way to do this in Go? I have a feeling that using an asynchronous call for such a thing is an overkill, and it really bothers me.

Use `os.SEEK_END` rather than `2`. `time.Sleep(10)` sleeps for 10 nanoseconds, you probably meant `time.Sleep(10 * time.Millisecond)`. Once an `io.Reader` gives you `io.EOF` you shouldn't expect it ever give you more data; EOF == **end** of file/steam, no more data. — Dave C, Jun 29 '15 at 16:42

Charlie Tumahai · Accepted Answer · 2022-02-17T06:40:42.770

Create a wrapper around a reader that sleeps on EOF:

type tailReader struct {
    io.ReadCloser
}

func (t tailReader) Read(b []byte) (int, error) {
    for {
        n, err := t.ReadCloser.Read(b)
        if n > 0 {
            return n, nil
        } else if err != io.EOF {
            return n, err
        }
        time.Sleep(10 * time.Millisecond)
    }
}

func newTailReader(fileName string) (tailReader, error) {
    f, err := os.Open(fileName)
    if err != nil {
        return tailReader{}, err
    }

    if _, err := f.Seek(0, 2); err != nil {
        return tailReader{}, err
    }
    return tailReader{f}, nil
}

This reader can be used anywhere an io.Reader can be used. Here's how loop over lines using bufio.Scanner:

t, err := newTailReader("somefile")
if err != nil {
    log.Fatal(err)
}
defer t.Close()
scanner := bufio.NewScanner(t)
for scanner.Scan() {
    fmt.Println(scanner.Text())
}
if err := scanner.Err(); err != nil {
    fmt.Fprintln(os.Stderr, "reading:", err)
}

The reader can also be used to loop over JSON values appended to the file:

t, err := newTailReader("somefile")
if err != nil {
    log.Fatal(err)
}
defer t.Close()
dec := json.NewDecoder(t)
for {
    var v SomeType
    if err := dec.Decode(&v); err != nil {
       log.Fatal(err)
    }
    fmt.Println("the value is ", v)
}

There are a couple of advantages this approach has over the goroutine approach outlined in the question. The first is that shutdown is easy. Just close the file. There's no need to signal the goroutine that it should exit. The second advantage is that many packages work with io.Reader.

The sleep time can be adjusted up or down to meet specific needs. Decrease the time for lower latency and increase the time to reduce CPU use. A sleep of 100ms is probably fast enough for data that's displayed to humans.

There are two things that I see are an error.... [1] the time.Sleep(10*time.Millisecond) is going to cause the system to generate heat instead of sitting quietly until there is actually some data. [2] When you are at the end of the file there is no signal to the main loop so that you can decide to exit. (I have a use-case where I send some commands to a service; then monitor the logs waiting for a positive reply; Since there is no negative reply I need to exhaust the data or check the timer. Goroutines and closing the data source are not an option. — Richard, Aug 21 '17 at 20:13
@Richard (1) Adjust the sleep time up or down to balance latency vs power. (2) If you need to exhaust the data up to some deadline, then modify the code in the answer to check a deadline before sleeping. — Charlie Tumahai, Jun 03 '21 at 17:45

score 1 · Answer 2 · edited Oct 30 '19 at 08:43

1

Check out this Go package for reading from continuously updated files (tail -f): https://github.com/hpcloud/tail

t, err := tail.TailFile("filename", tail.Config{Follow: true})
for line := range t.Lines {
    fmt.Println(line.Text)
}

edited Oct 30 '19 at 08:43

Jonathan Hall

75,165
16
143
189

answered Sep 13 '19 at 05:28

Tiya Jose

1,359
14
24

"tail -f"-like generator

2 Answers2

Linked