C++ - Producer / Consumer only allow consumption in defined chunks

Question

There's two threads A (Producer) and B (Consumer).

The data A produces is only meant to be read in chunks, hence B shall only be allowed to read once A has produced a whole chunk. A single piece of data is a simple struct and the chunk length is variable. For example once it could be that B is allowed to read after 50 pieces of data are produced, another time it might be 200.

I've found this implementation of a producer/consumer queue that I'd like to use: https://github.com/cameron314/readerwriterqueue

My current idea is that A writes its data into a std::vector and then pass the std::vector into the queue. But I doubt this works since the queue does not know how much memory an std::vector will take and it wants to allocate the memory beforehand.

I hope anybody knows an easier solution for this.

Have you tried your solution, and if so, what went wrong with storing a `std::vector` in the queue? — Yakk - Adam Nevraumont, Jan 28 '16 at 14:53
Your chunk is a message of variable size. If the queue your've chosen doesn't allow for variable-size messages, you need another queue. — SergeyA, Jan 28 '16 at 14:54
' A writes its data into a std::vector and then pass the std::vector into the queue' - try it and see:) — Martin James, Jan 28 '16 at 15:03
Okay, thanks guys, I just thought this might not work - as the queue that I've linked allocated memory based on the size of its items. — keyboard, Jan 28 '16 at 15:17

score 2 · Accepted Answer · answered Jan 28 '16 at 15:05

2

Regardless of what you produce or consume, you will need a concurrent queue for communication between the producer(s) and the consumer(s). If we are doing it C++ style, you'll end up with something like:

template<typename T, typename Alloc>
class concurrent_queue;

(note that some libraries already give you such containers, Intel TBB for example).

The template parameter T is what you exchange between producers and consumers. As you asked to consume chunks, here T = your_chunk_type. Let's say your chunks are variable size: Chunk = std::vector<something>. With the lib you've linked on github, you could just use ReaderWriterQueue<Chunk> as the queue sharing the work.

answered Jan 28 '16 at 15:05

Synxis

9,236
2
42
64

So you say I could well use std::vector as the template parameter of the ReaderWriterQueue that I linked? – keyboard Jan 28 '16 at 15:16
Yes you can. `std::vector` has a well-defined size that is known at compile time. The queue will allocate memory for its `std::vector`s, and each `std::vector` will allocate additional memory for its items – Synxis Jan 28 '16 at 15:17
Ah okay, I think I start to get it. It doesn't matter how many elements are in the vector, right? Since the memory of the elements in the vector is somewhere else. – keyboard Jan 28 '16 at 15:20
Yes, exactly. It is just a layer that handle a memory buffer for you, in a type-safe manner. – Synxis Jan 28 '16 at 15:23

score 0 · Answer 2 · answered Jan 28 '16 at 15:05

0

You can stick with a standard sized data chunk and have one or more framing chunks that either indicate the start and end of a message, or perhaps just a start chunk that also has the length of the message stored in it.

answered Jan 28 '16 at 15:05

Paul Evans

27,315
3
37
54

C++ - Producer / Consumer only allow consumption in defined chunks

2 Answers2