0

I am writing streamlink-alike CLI tool, but capable of downloading multiple twitch.tv live streams simultaneously using goroutines.

After a few hours memory usage is 37% (2 GB RAM) and unix file descriptors hits limit.

  1. Increasing 'ulimit -n' is not an option.
  2. Yes, I use resp.Body.Close() and 'Connection': 'close' tag and defer too.

I give up and need help, read articles, googled it, tried all I know. Please, have a look at code and logs.

  1. Logs: https://i.stack.imgur.com/GApmO.jpg
  2. Code: https://pastebin.com/87HW0prZ

P.S: Is there any CLI tool to detect memory leaks?

wmw
  • 1
  • Those commented out `defer` statements... are we to assume that in the actual application they aren't commented out? Did you comment them out just for the sake of example, why? Why not leave them uncommented, then you wouldn't have to use defensive claims like *"Yes, I use resp.Body.Close()"*... – mkopriva Aug 26 '21 at 06:24
  • `defer` never triggers inside loop. I use VS Code, it says using defer in never ending loops is useless, so I use "resp.Body.Close()" a few lines down instead of "defer resp.Body.Close()". Of course I tried both way. It is commented out because it does not help. I made checks. – wmw Aug 26 '21 at 06:29
  • 1
    Note that generally it is preferred that you put the code directly into the question. If you believe it to be too big to share directly you should try to create a [mcve] and share that instead. Sharing *only* links to code on some other site is not considered ok. – mkopriva Aug 26 '21 at 06:29
  • So when `resp.StatusCode != 200` is `true` and you invoke `continue` where do you call `resp.Body.Close()` then? Or do you expect a non-200 reponse to not have a body to close? – mkopriva Aug 26 '21 at 06:32
  • Note that if you want to use `defer` within a loop you can wrap the loop's body into a immediately invoked closure. https://play.golang.org/p/TkGxu_xfIaM – mkopriva Aug 26 '21 at 06:34
  • Thanks for pointing out, but its not it. If I open 6.2MB log and search for "Received HTTP" I get 0 matches. Means, I never get non 200 status from Twitch servers. Yes, I should handle it, but it does not cause leaks. Am I right? – wmw Aug 26 '21 at 06:37
  • Thanks for https://play.golang.org/p/TkGxu_xfIaM example. I thought about using anonymous func, if func returns it means GC gonna clean it later. We basically just leaving all the dirty job for GC to cleanup after us. Since, I came from Rust, I really want to know where it leaks excatly. – wmw Aug 26 '21 at 06:41
  • https://golang.org/doc/diagnostics or google "go memory profiling" and you'll get lots of other blog posts on that topic. – mkopriva Aug 26 '21 at 06:43
  • HAHA! I tried profiling and using `lsof -p 103072 | wc -l`. Every cycle it creates 5 file descriptors. Not 100% sure, but turns out my code is OK and our "black hole" is `pl, err := twitchpl.Get(u.User)` from external lib `import "github.com/wmw9/twitchpl"`. – wmw Aug 26 '21 at 07:27
  • https://github.com/wmw9/twitchpl/blob/main/twitchpl.go#L201 – Martin Gallagher Aug 26 '21 at 07:55
  • Thanks, Martin! I consider my problem being fully solved! – wmw Aug 26 '21 at 09:40

0 Answers0