2

I found strange occupy heap when convert byte[] to string with below code

package main

import (
  "bytes"
  "fmt"
  "net/http"
  _ "net/http/pprof"
  "strings"
  "time"
)

var (
  c = make(chan int, 500000)
)

func main() {
  go func() {
    http.ListenAndServe(":8080", nil)
  }()
  f := func(ss []string) {
    fmt.Println(ss)
    time.Sleep(time.Millisecond)
    <-c
  }
  for {
    c <- 1
    bs := bytes.NewBufferString("A,B,C,D,E,F,G,H,I,J,K,L,M,N,O,P,Q,R,S,T,U,V,W,X,Y,Z").Bytes()
    fmt.Println(bs) // will raise memory leak after marked as comment???
    s := string(bs)
    ss := strings.Split(s, ",")
    go f(ss)
  }
}
  1. without fmt.Println(bs) will gradually exhausting memory.

  2. with fmt.Println(bs) work fine. i can't understand what happened? i'm worked with version go1.9.2 darwin/amd64

Nelson
  • 183
  • 3
  • 9

1 Answers1

5

No, there is no memory leak:
You are using 500000 concurrent goroutines, you just need to limit (reduce) the number of concurrent goroutines, e.g.:

c := make(chan int, runtime.NumCPU())

Try this ( and see the end of this edit):

package main

import (
    "bytes"
    "fmt"
    "runtime"
    "strings"
    "time"
)

func main() {
    c := make(chan int, runtime.NumCPU())
    for {
        c <- 1
        bs := bytes.NewBufferString("A,B,C,D,E,F,G,H,I,J,K,L,M,N,O,P,Q,R,S,T,U,V,W,X,Y,Z").Bytes()
        s := string(bs)
        ss := strings.Split(s, ",")
        go func(ss []string) {
            fmt.Println(ss)
            time.Sleep(time.Millisecond)
            <-c
        }(ss)
    }
}

Your Code:

package main

import (
    "bytes"
    "fmt"
    "net/http"
    _ "net/http/pprof"
    "strings"
    "time"
)

var (
    c = make(chan int, 500000)
)

func main() {
    go func() {
        http.ListenAndServe(":8080", nil)
    }()
    f := func(ss []string) {
        fmt.Println(ss)
        time.Sleep(time.Millisecond)
        <-c
    }
    for {
        c <- 1
        bs := bytes.NewBufferString("A,B,C,D,E,F,G,H,I,J,K,L,M,N,O,P,Q,R,S,T,U,V,W,X,Y,Z").Bytes()
        // fmt.Println(bs) // will raise memory leak after marked as comment???
        s := string(bs)
        ss := strings.Split(s, ",")
        go f(ss)
    }
}

It reaches steady state after a while and even reduced Memory usage:

// Mem          CPU time:
// 5,464,208K 0:1:20
// 5,468,208K 0:2:20
// 5,469,608K 0:3:20
// 5,469,844K 0:4:20
// 5,469,844K 0:5:20
// 5,469,848K 0:6:20
// 5,469,848K 0:7:20 fixed
// 5,469,848K 0:8:20 fixed
// 5,469,616K 0:9:20 reduced
wasmup
  • 14,541
  • 6
  • 42
  • 58
  • have you try add fmt.Println(bs)? that will reduce memory occupy – Nelson Dec 03 '17 at 09:34
  • Yes, Sure. Using `fmt.Println(bs)` makes GC(garbage collector) more time to collect Memory and the memory usage is `25520KBytes` for me with `fmt.Println(bs)` . – wasmup Dec 03 '17 at 09:53
  • Yes, fmt.Println(bs) makes GC more times. But no print should not occupy more and more memory until system limit. – Nelson Dec 03 '17 at 10:10
  • In my testing, no print will occupy more than 3GB memory – Nelson Dec 03 '17 at 10:13
  • I still can’t understand why no print will exhausting system memory and with print work fine. No print should trigger GC periodly too? – Nelson Dec 03 '17 at 10:22
  • see [Go scheduler](http://morsmachine.dk/go-scheduler) and [this](http://www.sarathlakshman.com/2016/06/15/pitfall-of-golang-scheduler) – wasmup Dec 03 '17 at 10:51