0

I am writing a de- and encoder for a custom video format (QTC). The decoding process consists of multiple stages, where the output of each stage is passed to the next stage:

  1. deserialize the input stream
  2. generate a sequence of symbols with a range coder
  3. generate a stream of images from the symbol stream
  4. serialize the image stream into an output format

Steps three and four take up almost all of the processing time, step three takes roughly 35% and step four takes about 60%, the first and the last step are rather trivial.

What is the recommend and ideomatic way to runs the four steps in parallel? I am mostly interested in how to handle the communication between the parts. I plan to use one Goroutine for step two and one for step three, the routines are connected with a buffered channel. Is this the right way?

fuz
  • 88,405
  • 25
  • 200
  • 352
  • I would say your plan is good! It also means that you can speed things up by having multiple independent workers in steps 3 and 4 (if the algorithm allows it) by making more go routines. All the parallel workers can read from the same channel and write to the same channel. – Nick Craig-Wood Apr 08 '13 at 18:31
  • @Nick The algorithm encodes video data. This especially means that the encoding of each frame depends on the previous frame, therefore one has to wait for one frame to finish encoding in order to process the next frame. – fuz Apr 08 '13 at 19:09

2 Answers2

3

For some tasks having a "shared" data structure "protected" by careful use of mutexes is easier, but a buffered channel would be the "standard" way to do this in Go and for your task it sounds like the right solution, too.

Are you running into any problems with it since you are asking here?

Ask Bjørn Hansen
  • 6,784
  • 2
  • 26
  • 40
  • I prefer thinking about my problems before writing code. It seemed that the preffered method to share streams of data is to use the io.Reader and io.Write sets of interfaces. But on the other hand there are channels that provide virtually the same functionality. Using a shared data structure never sprung into my mind as the Go credo says: *Don't communicate by sharing memory; share memory by communicating.* – fuz Apr 06 '13 at 20:54
  • Using channels will be nicer to work with than using io.Reader/io.Writer handles (though I hadn't considered that as an option...). If nothing else then channels save you unpacking/packing the structs/data all the time. – Ask Bjørn Hansen Apr 06 '13 at 23:12
0

Do not over think. Just write some code. Go code is easy to change - compiler holds your hand. Write some tests / benchmarks to keep yourself honest.

Alex

alex
  • 2,178
  • 16
  • 14
  • Sorry. I prefer having a rough idea of how to tackle a problem before writing up a solution. – fuz Sep 14 '14 at 14:33