Too Many Open File Error in Golang

Question

I am simply reading the /proc/diskstats file. My code is:

func ReadFromFile(filepath string)(string){
    defer func() {
        if err1 := recover(); err1 != nil {
            fmt.Println("!!!!!!!!!!!!!!!!Panic Occured and Recovered in readFromFile(), Error Info: ", err1)
        }
     }()

    strData := ""

    data, err := ioutil.ReadFile(filepath)
    if err != nil{
        fmt.Println("File read error: ", err)
        return ""
    }

    strData = string(data)
    return strData
}

The error I am getting is:

File read error: open /proc/diskstats: too many open files

Not only for this file, I am also getting the same error for some other files.

I have also run this command:

root@golang:~# lsof|wc -l

785

Please guide me.

i restarted my service then i didn't get any error,but output of lsof|wc -l output is keep on incresing. — GKV, Jun 14 '16 at 07:28
`io.ReadFile` does not provide a close file function directly. It's included into the `ReadFile` function itself. So it's not needed to close the file explicitly. It will close automatically. — Endre Simo, Jun 14 '16 at 07:37
ioutil.ReadFile closes after the read. And dump whole content. — khrm, Jun 14 '16 at 07:38
How are you calling the function? Leakage seems to be from somewhere else. — khrm, Jun 14 '16 at 07:39
`lsof|wc -l` counts open files system wide. You have to run it with the `-p` option for your process. It also helps to actually look at the output of `lsof` to see what the open fds are to to help identify the problem. — JimB, Jun 14 '16 at 13:13

Michael Dorner · Answer 1 · 2022-06-18T07:15:47.917

16

I ran into the same problem (maybe different circumstances or setup) and fixed it differently:

func some_func(file_name []string) {
    for _, file_name := range file_names {
        f, _ := os.Create(file_name)
        // defer f.Close() // bad idea
        _, _ = f.Write([]byte("some text"))
        f.Close() // better idea
    }
}

The problem is that defer will be executed, when the function will return, which could take a while - depending on the loop size (bad idea). So just do it explicit (better idea).

edited Jun 18 '22 at 07:15

answered Aug 04 '17 at 07:47

Michael Dorner

17,587
13
87
117

f.Write requires a []byte as an input, i.e. `[]byte("some text")` – 030 Feb 18 '21 at 20:38

score 7 · Answer 2 · answered Jun 14 '16 at 08:08

7

Basically in UNIX platforms, the OS places a limit to the number of open file descriptors that a process may have at any given time.
The error too many open files is raised since you have reached the limit of file (and or pipe or socket)currently opened and you are trying to open a new file (and or pipe or socket).
To avoid this problem you must close the file when you have finished to use the open file using the Close() function

answered Jun 14 '16 at 08:08

Tinwor

7,765
6
35
56

1

To be honest, I would've expected ioutil.ReadFile to close the file (although it may well be that there's other file descriptors being opened elsewhere causing the problem). – Vatine Jun 14 '16 at 08:45
If we look on ReadFile we can see that the Close() function is only called when the open file fails (https://golang.org/src/io/ioutil/ioutil.go?s=1464:1510#L39). There is another option to call automatically the Close() function and is using the return statement (as void so you must return nothing) – Tinwor Jun 14 '16 at 08:51
3

No, there's a "defer f.Close()" on line 54, causing the file to be closed during the return from the function. There's an early return if the Open call fails (so as to not call the Close() method on an undefined value). – Vatine Jun 14 '16 at 09:50
You are right @Vatine I didn't see the earlier return, my fault – Tinwor Jun 14 '16 at 10:05

score 7 · Answer 3 · answered Nov 15 '19 at 23:00

The OP does not provide a Minimal, Reproducible Example. The error in question is caused by code that was not posted. An easy way to demonstrate this is to simply run the provided code in a minimal example (with no other activity), and see that it does not fail.

The function ioutil.ReadFile, of course, closes the file. It has been implicated, in this case, simply because it is attempting to open a new file when the resource limit has already been reached.

One common gotcha in Go is failure to close streams that were opened implicitly. A specific case of this is that a stream is opened when using the client functions of the http library.

Examples: (1), (2)

The client must close the response body when finished with it:

resp, err := http.Get("http://example.com/")
if err != nil {
    // handle error
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)

Such requests should always include a call to Close, using a form as shown above.

There may be other, similar, cases of implicit streams being opened...

This is a particularly tricky issue because, for trivial programs, you'll never know the difference. You won't know there is a problem until you run this through hundreds, or thousands, of iterations. And, then, the error may often be presented as a failure of some unrelated function call -- just as demonstrated by the OP.

// Body is the request's body. // // For client requests, a nil body means the request has no // body, such as a GET request. The HTTP Client's Transport // is responsible for calling the Close method. // // For server requests, the Request Body is always non-nil // but will return EOF immediately when no body is present. // The Server will close the request body. The ServeHTTP // Handler does not need to. Body io.ReadCloser — Brent Bradburn, Feb 19 '20 at 20:08
If `defer` is used in the context of a larger function, it may be a good idea to wrap that portion in a separate function (maybe an [IIFE](https://stackoverflow.com/q/40081996/86967)) -- since the execution is deferred ["until the surrounding function returns"](https://tour.golang.org/flowcontrol/12). — Brent Bradburn, Apr 11 '20 at 03:26

Too Many Open File Error in Golang

3 Answers3