Creating slice byte goroutine hangs

Question

I have an upload program that I am working on and I am running into an issue. I have n go routines that handle uploading the parts to a big file. Essentially it will split the file into 100MB chunks and upload them concurrently depending on the amount of concurrent processes you specify in the config.

The issue I'm having is when I create a buffer to read the file and upload the make([]byte, 100000000) hangs... but only if it's in a go routine. (I'm using 100000000 to simplify the upload calculations)

Here is an example.

This works: https://play.golang.org/p/tkn8JVir9S

package main

import (
    "fmt"
)

func main() {
    buffer := make([]byte,  100000000)
    fmt.Println(len(buffer))
}

This doesn't: https://play.golang.org/p/H8626OLpqQ

package

 main

import (
    "fmt"
)

func main() {
    go createBuffer()
    for {
    }

}

func createBuffer() {
    buffer := make([]byte, 100000000)
    fmt.Println(len(buffer))
}

It just hangs... I'm not sure if there is a memory constraint for a go routine? I tried to research and see what I could find but nothing. Any thoughts would be appreciated.

EDIT: Thanks everyone for the feedback. I will say I didn't explain the real issue very well and will try to provide more of a holistic view next time. I ended up using a channel to block to keep my goroutines ready for new files to process. This is for a DR backup uploading to a 3rd party all that requires large files to be split into 100mb chunks. I guess I should have been more clear as to the nature of my program.

You have a busy loop in your program. There is never a reason to use an empty loop. It's just consuming 100% of a cpu for no reason, and will eventually block the runtime. — JimB, Nov 22 '17 at 14:59
Try to use `select{}` instead of `for{}`. Is not the best approach (you should use channels for this to check when the go routines finish), but for testing purposes is ok. — ajnavarro, Nov 22 '17 at 15:27
Allocating a byte slice in a goroutine is fine as you can see here https://play.golang.org/p/8QawOUTFyb, the problem in your example, as already pointed out by JimB, is the for loop. — mkopriva, Nov 22 '17 at 15:28
Well I saw the for {} as an example for demonizing goroutines. I essentially have goroutines that are processing files and I needed a way to demonize them. — MattA, Nov 23 '17 at 15:50

score 1 · Answer 1 · answered Nov 22 '17 at 15:31

1

This program hangs because there is an infinite loop in your code. Try running the code just like this to prove it to yourself. The goroutine is not what is causing the hanging.

func main() {
    for {
    }
}

If you just want to see fmt.Println(..) print, then I'd recommend having a time.Sleep call or similar.

If you would like to wait for a bunch of goroutines to complete, then I'd recommend this excellent answer to that exact question.

answered Nov 22 '17 at 15:31

Frank Bryce

8,076
4
38
56

Well I have goroutines that are processing files and need to remain open and continue to remain open. The issue that I'm having is running the app as a daemon and that is what spurred this issue. – MattA Nov 23 '17 at 16:11
Looks like [this post](https://stackoverflow.com/questions/23736046/how-to-create-a-daemon-process-in-golang) is a good place to start if you would like to make a daemon. The main problem is you have to not have a busy-wait loop so you don't waste CPU cycles. – Frank Bryce Nov 25 '17 at 00:57

peterSO · Answer 2 · 2017-11-23T00:30:34.380

0

Package runtime
import "runtime"
func Gosched
func Gosched()
Gosched yields the processor, allowing other goroutines to run. It does not suspend the current goroutine, so execution resumes automatically.

When you do something strange (for {} and 100MB), you get strange results. Do something reasonable. For example,

package main

import (
    "fmt"
    "runtime"
)

func main() {
    go createBuffer()
    for {
        runtime.Gosched()
    }
}

func createBuffer() {
    buffer := make([]byte, 100000000)
    fmt.Println(len(buffer))
}

Output:

100000000
^Csignal: interrupt

edited Nov 23 '17 at 00:30

answered Nov 22 '17 at 22:07

peterSO

158,998
31
281
276

Well I have several go routines running that are processing files, several of which use a config to determine concurrent processes. The issue is that I need to slice a file into 100MB chucks to send via an http post, so the 100MB isn't strange. As far as the for loop this issue disappeared, and now I know why, when I added my channel to return the finished files. – MattA Nov 23 '17 at 16:04
@MattA: Your example is very contrived. Your real-world case is different. Also, using 100MB of memory per chunk to send 100MB chunks concurrently is inefficient. How much memory will you need to support 10,000 users? You need a better strategy. Google uses Go internally for downloading: [dl.google.com now served by Go](https://groups.google.com/forum/#!topic/golang-nuts/BNUNbKSypE0) and [dl.google.com: Powered by Go](https://talks.golang.org/2013/oscon-dl.slide#1). – peterSO Nov 23 '17 at 16:33
I agree, I should have been a little more explicit with the exact issue and why I was doing it. I was more curious as to the reason behind it and now that I see the responses it makes sense. I actually need to have a channel that blocks and let's the goroutines continue until an error. As far as the inefficiency goes, this is an upload utility to a DR backup provider so there are no users. You can define max concurrent processes in the config file and that is the limit for the memory consumption. The DR api requires large files to be split into chunks and sent in the body of a POST. – MattA Nov 23 '17 at 18:01

Creating slice byte goroutine hangs

2 Answers2