4

Basically I want to find if my program is leaking goroutines over time. So I want to see how many goroutines are running over time. Is there any way to do this through pprof?

I've done go tool pprof http://localhost:8888/debug/pprof/block.

Which gives me how long is being spent blocked but not how many routines are running.

tommy_p1ckles
  • 595
  • 2
  • 6
  • 17
  • 5
    Are you aware of [`runtime.NumGoroutine()`](https://golang.org/pkg/runtime/#NumGoroutine) which returns the number of goroutines that currently exist? – icza Jul 15 '16 at 17:43

3 Answers3

19

Open http://localhost:8888/debug/pprof/ in your browser. You'll see two relevant links: "goroutine" (http://localhost:8888/debug/pprof/goroutine?debug=1) and "full goroutine stack dump" (http://localhost:8888/debug/pprof/goroutine?debug=2).

The first one will show all goroutines that share the same code as one entry, with the number of such goroutines before their name. For example:

1 @ 0x42f223 0x42f2e4 0x40542f 0x404f4b 0x4a0586 0x4600a1
#   0x4a0586    gitlab.com/gitlab-org/gitlab-ci-multi-runner/commands.(*RunCommand).startWorkers+0x56   /home/me/go/src/gitlab.com/gitlab-org/gitlab-ci-multi-runner/commands/multi.go:164

1 @ 0x42f223 0x43dfd7 0x43d532 0x4a04ed 0x4600a1
#   0x4a04ed    gitlab.com/gitlab-org/gitlab-ci-multi-runner/commands.(*RunCommand).processRunners+0x45d    /home/me/go/src/gitlab.com/gitlab-org/gitlab-ci-multi-runner/commands/multi.go:147

There's one of both of these goroutines, that's what the 1 before the @ means.

The full dump is extremely useful for finding leaks, it'll show you every goroutine separately, as well as its stack trace and what it's doing (e.g. how long it has been waiting to receive from a channel):

goroutine 49 [chan receive, 2 minutes]:
gitlab.com/gitlab-org/gitlab-ci-multi-runner/commands.(*RunCommand).startWorkers(0xc820103ee0, 0xc820274000, 0xc820274060, 0xc8201d65a0)
    /home/me/go/src/gitlab.com/gitlab-org/gitlab-ci-multi-runner/commands/multi.go:164 +0x56
created by gitlab.com/gitlab-org/gitlab-ci-multi-runner/commands.(*RunCommand).Run
    /home/me/go/src/gitlab.com/gitlab-org/gitlab-ci-multi-runner/commands/multi.go:294 +0x41b

goroutine 50 [select]:
gitlab.com/gitlab-org/gitlab-ci-multi-runner/commands.(*RunCommand).processRunners(0xc820103ee0, 0x0, 0xc820274060, 0xc8201d65a0)
    /home/me/go/src/gitlab.com/gitlab-org/gitlab-ci-multi-runner/commands/multi.go:147 +0x45d
created by gitlab.com/gitlab-org/gitlab-ci-multi-runner/commands.(*RunCommand).startWorkers
    /home/me/go/src/gitlab.com/gitlab-org/gitlab-ci-multi-runner/commands/multi.go:165 +0x96
user1431317
  • 2,674
  • 1
  • 21
  • 18
  • 4
    When debug=1, what does addresses after @ signify? - ("0x42f223 0x42f2e4 0x40542f 0x404f4b 0x4a0586 0x4600a1") – lafolle Feb 03 '18 at 19:25
1

As icza pointed out the answer is to use runtime.NumGoroutine().

tommy_p1ckles
  • 595
  • 2
  • 6
  • 17
0

  • As stated above we can definitely use runtime.NumGoroutine() to get the current number of goroutines which may either be in running or in waiting state.

  • We can use Lookup function to fetch a number of profiles. They can be either pre-defined profiles like goroutine, heap, allocs, mutex, etc. or custom profiles also. More here

  • We can chain WriteTo method defined on Profile to write to either stdout or to a file. Example pprof.Lookup("goroutine").WriteTo(writer, 2)
  • This will give you significant details about each go-routine like how much time it spent waiting, what was it waiting on, was it sleeping, etc.
  • You can use that info to point out where the real issue is.
Abhishek Srivastava
  • 503
  • 7
  • 8