1

I'm using Go Gob to transfer large files (~ 1 GB) or many small files (~ 30 MB). Server is running in a loop, and will receive files when clients send it.

My code is working if I send one large file, or few small files, but when sending a large file for the second time, it returns a 'fatal error: runtime: out of memory'. If I send a large file, stops the program, then starts again and send another large file, it works.

It looks after receiving file via Gob and writing into a file, it is not releasing memory.

Server code

type FileGob struct {
  FileName    string
  FileLen     int
  FileContent []byte
}

func handleConnection(conn net.Conn) {
  transf := &FileGob{}
  dec := gob.NewDecoder(conn)
  err := dec.Decode(transf)       // file from conn to var transf
  if err != nil {
    fmt.Println("error to decode into buffer:", err)
  }

  conn.Close()
  file, err := os.Create("sent_" + transf.FileName)
  if err != nil {
    fmt.Println("error to create file:", err)
  }
  file.Write(transf.FileContent)     // writes into file

  fileStat, err := file.Stat()
  if err != nil {
    fmt.Println("error to get File Stat:", err)
  }

  file.Close()
  fmt.Printf("File %v was transferred\n", transf.FileName)
  fmt.Printf("Transferred: %d, Expected: %d\n", fileStat.Size(), transf.FileLen)
}
Jonathan Hall
  • 75,165
  • 16
  • 143
  • 189
grbueno
  • 11
  • 1
  • 1
    Show a full [MCVE]. Probably, some reference to your gigantic file data remains somewhere. Consider also *explicitly* clearing it and firing the GC. – Basile Starynkevitch Oct 17 '18 at 18:51
  • 2
    This seems like a really poor use case for Gob, because it requires you to decode the entire file contents into memory before you can do anything with it. You'd be better served using Gob for the metadata then just streaming the contents directly to file to conserve memory. – Adrian Oct 17 '18 at 18:52
  • Would this be a job for a loop using https://godoc.org/io#LimitedReader ? – RayfenWindspear Oct 17 '18 at 19:19
  • You shouldn't accept filenames from clients unchecked. With the code you've shown, clients can overwrite arbitrary files the server has access to, and create arbitrary files in directories that the server has write access to. Even if you trust the client right now, those clients change over time, new clients pop up eventually, etc. Trust, but verify. Typical validation includes checking for directory separators and `/../` segments. – Peter Oct 18 '18 at 06:29
  • hi Adrian, indeed it is a poor use of Gob. It takes ~15 sec to read from conn to FileContent, and other 10 - 15 sec to read from FileContent to file. I'll write the bytes from conn directly to file, and will use the struct just for metadata. – grbueno Oct 19 '18 at 14:25

0 Answers0