2

I want to do this:

  1. Read a line from a text file.
  2. Process the line.
  3. Delete the line.

My first thought was to read the entire file into memory with ioutil.Readfile(),
but I'm not sure how to update the text file after the line has been processed,
and what happens if extra lines is added to the text file after it has been read into memory?

I normally write shell scripts and would do something like this:

while read -r line; do
    echo "${line}"
    sed -i 1d "${myList}"
done < "${myList}"

What is the best way to do this in Golang?

John Smith
  • 381
  • 1
  • 5
  • 12
  • 4
    You can't delete a line from the front of a file without re-writing the whole file. Your sed example has the same problem with losing a line added while it's doing it's thing. The race is smaller in sed given that you only lose the line if it was written while you were processing the previous last line – David Budworth Jul 31 '16 at 15:05
  • 1
    Is the purpose of deleting the line to protect from failure, so you can pick up where you left off? In that case, you might be better off writing a state file with the most recent line processed, checking for it on startup, then begin processing from that location if it exists. When you hit EOF, you know you can safely delete the file. – jxstanford Jul 31 '16 at 16:07
  • You could loop through each line, concat the "processed" line to a string variable, then (once loop is done) rewrite the file use contents of string variable. – openwonk Aug 01 '16 at 02:06
  • 1
    @DavidBudworth I haver never thought about sed in a shell script having the same problem, it's only now that I'm learning Go and having to deal with this directly that I learn these things. – John Smith Aug 03 '16 at 01:12

2 Answers2

11

Use the bufio package.

Here's the basic syntax for opening a text file and looping through each line.

package main

import (
    "bufio"
    "fmt"
    "os"
)

func main() {
    // Open the file.
    f, _ := os.Open("C:\\programs\\file.txt")
    // Create a new Scanner for the file.
    scanner := bufio.NewScanner(f)
    // Loop over all lines in the file and print them.
    for scanner.Scan() {
      line := scanner.Text()
      fmt.Println(line)
    }
}
openwonk
  • 14,023
  • 7
  • 43
  • 39
  • Perfect, this is what I need. I will change the requirement of deleting the line, because this seems not to be common, like `Wget` will not delete any lines if getting URL's from a file. – John Smith Aug 03 '16 at 01:07
  • Sounds good. Figured once you have a skeleton for looping through the file, you could do as you want... Please upvote if you like. – openwonk Aug 03 '16 at 14:51
1

you have some options:
1- read file, process it, then write it back (you need to lock that file).
2- use binary file and invent (make use of) special data structure (like linked list) to optimize text processing (with line locking).
3- use ready made databases.
4- use Virtual filesystem inside your file, and treat each line like one file, see: https://github.com/lotrfan/vfs and https://github.com/blang/vfs

using file manager (like database server) solves the file locking dilemma.

and if the purpose of using file is one way communication which sender program just adds new line and receiver program just removes it, it is better to use os pipes (named pipe (FIFO)) or other interop methods.

see for Linux: Unix FIFO in go?
for Windows: https://github.com/natefinch/npipe

sample file writer: 

package main

import (
    "bufio"
    "fmt"
    "os"
    "time"
)

func main() {
    f, err := os.OpenFile("/tmp/file.txt", os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0666)
    if err != nil {
        panic(err)
    }
    defer f.Close()
    for i := 0; ; i++ {
        w := bufio.NewWriter(f)
        _, err := fmt.Fprintln(w, i)
        if err != nil {
            panic(err)
        }
        w.Flush() // Flush writes any buffered data to the underlying io.Writer.
        f.Sync()  // commit the current contents of the file to stable storage.
        fmt.Println("write", i)
        time.Sleep(500 * time.Millisecond)
    }
}

sample file reader: 

package main

import (
    "fmt"
    "os"
    "time"
)

func main() {
    f, err := os.OpenFile("/tmp/file.txt", os.O_RDWR, 0666)
    if err != nil {
        panic(err)
    }
    defer f.Close()

    i := 0
    for {
        n, err := fmt.Fscanln(f, &i)
        if n == 1 {
            fmt.Println(i)
        }
        if err != nil {
            fmt.Println(err)
            return
        }
        time.Sleep(500 * time.Millisecond)
    }
}
Community
  • 1
  • 1