Synchronization in a Go-lang webapplication

Question

I'm writing a golang web application. The web application accesses the file system (reading and writing) and an sqlite3 database file.

Question 1: How can I synchronize file system access in Go?

type DataObject struct {
  data []byte
}

func (*d DataObject) Write() {
   //
   // Synchronization ?
   //
   ioutil.WriteFile("file.name", d.data, 0644)
   //
   // Stop synchronization ?
   //
}

Question 2: Do I need to synchronize sqlite3 database file access?

type SqlObject struct {
  sqldata string
}

func (*s SqlObject) Store() error {
  //
  // Open the file, do I need some sort of synchronization?
  //
  con, err := sqlite.Open("database/datafile.db")
  if err != nil {
    return err
  }
  defer con.Close()

  err = con.Exec("INSERT INTO data(sqldata) values(?)", s.sqldata)
  if err != nil {
    return err
  }
  return nil
}

I'm using the gosqlite3 driver (http://code.google.com/p/gosqlite/).

score 7 · Accepted Answer · answered Feb 08 '13 at 20:12

For files it depends on your application. If you only have one goroutine writing to the file, you shouldn't need to. If more than one then it depends:

If you are coordinating between different processes (programs) you can use flock (and it probably won't be fun).

If you are coordinating multiple goroutines in your program you can use mutexes or you can see if you can re-organize the program so just one routine writes to the file and the others send updates via channels.

For SQLite I believe the easiest way would be to just keep one sqlite connection open and use that from the various goroutines; though it does support multiple processes having it open at once and if your system does many concurrent reads that might be faster (it uses a global lock for writing).

Thank you. I assume a web application in Go is concurrent, and every request is handled in a new go routine, thus I should synchronize? — Kiril, Feb 09 '13 at 12:50
Yes, each http request runs in its own goroutine so you have to synchronize between them or use a separate "helper" goroutine (with channels) to handle things that needs to be serialized. — Ask Bjørn Hansen, Feb 21 '13 at 18:04

score 4 · Answer 2 · answered Feb 08 '13 at 20:08

Question 1

The "sync" package provides Lock and RWLock for synchronizing access to resources in the conventional way. While there's nothing wrong with that, I like playing around with idioms so I might do something like this:

package main

import "fmt"

func SyncFile(path string) chan<- func(string) {
    ch := make(chan func(string))
    go func() {
        for process := range ch {
            process(path)
        }
    }()
    return ch
}

func main() {
    data := SyncFile("data.name")
    data <- func(path string) {
        fmt.Println(path)
    }
}

Then if you only access the file through this channel of funcs, access will be synchronized. This is reliable, but you may be able to find something more efficient.

Question 2

The gosqlite3 driver is just a Go binding to libsqlite3, so the SQLite FAQ applies (and it looks like you're in the clear on this). If you want both of these to be synchronized with each other, then just wrap the SQLite usage inside the file access synchronization code.

Your answer to Question 2 is wrong. If you open the database connection once, you can concurrently re-use it. If you cannot, then your "database/sql" driver implementation has a bug: http://golang.org/src/pkg/database/sql/doc.txt?s=1209:1611#L31 — voidlogic, Feb 08 '13 at 22:11
Thank you for your answer. @voidlogic So, once I have a connection I can use it the entire time, and have many go-routines which are using it? — Kiril, Feb 09 '13 at 12:49
@Kiril: Yes, if your app creates a single instance of *sql.DB, you can then use that from many goroutines without any extra synchronization in your code. If you run into a problem doing this make sure to report the issue to the SQL driver implementer because this means their implementation is not compliment/has a bug. — voidlogic, Feb 10 '13 at 16:45

score 3 · Answer 3 · answered Feb 09 '13 at 07:05

1) You should use a read/write mutex (in the go std library). Code would look something like:

import "sync" // http://golang.org/pkg/sync/
const (
    filename = "file.name"
)
var globalFileLock sync.RWMutex

type DataObject struct {
  data []byte
}

func (*d DataObject) Write() {
   globalFileLock.Lock()
   defer globalFileLock.Unlock()
   ioutil.WriteFile(filename, d.data, 0644)
}

func (*d DataObject) Read() {
   globalFileLock.RLock()
   defer globalFileLock.RUnlock()
   d.data = ioutil.ReadFile(filename)
}

2) As you haven't posted the 'import' section from your program, I don't know which sqlite driver you're using.

If you are using database/sql to open the database connection, the driver provides concurrency control between goroutines.

Hi, thank you. I'm using this driver: http://godoc.org/code.google.com/p/gosqlite/sqlite. It is imported by: import "code.google.com/p/gosqlite/sqlite" — Kiril, Feb 09 '13 at 12:47

score -1 · Answer 4 · answered May 26 '14 at 19:26

I know its an old question but since I had the same "db lock" problem, and a transaction resolved it for me, I thought I'd mention it here:

db, err := sql.Open("sqlite3", db_path)
if err != nil {
    log.Printf("Cannot connect to database: %s\n", err ) 
}
defer db.Close()
tx, _ := db.Begin() 
var value string
err = tx.QueryRow("SELECT X FROM Y").Scan(&value)
tx.Rollback() // or: tx.Commit()

Synchronization in a Go-lang webapplication

4 Answers4