0

I was Using PProf to analyze tracing for a data pipeline with producer and consumer. I'm not sure how I should read this, but from the scheduler latency profile, chanrecv and chansend accounts for a decent portion of my total run time.

I find it hard to believe, so does this suggest go channel send/receive taking that long, or this only referring t producer lagging behind that leads to chanrecv blocking. And potentially how I can make this more efficient?

Sample code:

func consume(in chan User) {
    out := make(chan UserDetail)
    go func() {
        for user := range in {
            if userDetail, err := getUserDetails(user); err == nil {
                out <- userDetail
            }
        }
        close(out)
    }()
    return out
}

func produce() {
    out := make(chan User)
    go func() {
        // could be issuing a paginated api call to get users list
        for user := range users {
            out <- user
        }
        close(out)
    }()
    return out
}


func main() {
    userChan := produce() // get all users
    userDetailChannels := []chan User

    // fanout to 8 workers
    for workerIndex := 0; workerIndex < 8; workerIndex++ {
        userDetailChannels = append(userDetailChannels, consume(userChan))
    }
}

enter image description here

jub0bs
  • 60,866
  • 25
  • 183
  • 186
smilence
  • 349
  • 4
  • 9
  • 3
    The easiest first step is setting the channel buffer size to some number > 0. – Hymns For Disco Jun 21 '21 at 23:31
  • 6
    As the code you _showed_ doesn't do anything else than send and receive this is expected. If you need help you must provide a reproducable example. – Volker Jun 22 '21 at 05:23
  • Voting to close as we can't reproduce the problem from the details shown – Vorsprung Jun 22 '21 at 18:47
  • @HymnsForDisco thank you! it does cut the latency in half but still curious if the `chanrecv` is referring to consumer waiting for producer or it's just the channel receive would take that long in go – smilence Jun 23 '21 at 21:09

0 Answers0