Sliding window with 50% overlap with producer/consumer threads

Question

I have the following scenario in Java:

1 producer thread stores event objects into a queue. Blocking it is not an option. It should always just store each element at the end of the queue and exit (so no bounded queues).
1 consumer thread waits for the queue to have WINDOW_SIZE number of events in it. It should then retrieve all WINDOW_SIZE events from the queue for processing, but only remove half of them (i.e. WINDOW_SIZE/2), for a 50% overlap.

My question is, which (concurrent) collection would you use to implement this efficiently? The events come in at 100Hz on a resource-limited device (a mobile phone running Android). I thought of using the following, none of which seem to be a proper fit:

A ConcurrentLinkedQueue, checking for queue size each time it is modified, and using peek()/poll() in the consumer when WINDOW_SIZE events are available. This seems a bit cumbersome.
An ArrayBlockingQueue, again checking for queue size, and using drainTo(). However, that method has the following documentation: "[...] Further, the behavior of this operation is undefined if the specified collection is modified while the operation is in progress. [...]". This seems a bit odd for a concurrent collection.

Here's some example code:

import java.util.Queue;

import com.google.common.collect.Queues;

public class AccelerometerProcessor implements Runnable {

    private static final int WINDOW_SIZE = 128;

    private final Queue<AccelerometerEvent> eventQueue = Queues.newConcurrentLinkedQueue();

    @Override
    public void run() {
        while (!Thread.interrupted()) {
            try {
                synchronized (eventQueue) {
                    while (eventQueue.size() < WINDOW_SIZE) {
                        eventQueue.wait();
                    }

                    // We have WINDOW_SIZE eventQueue, start processing
                }
            } catch (InterruptedException e) {
                // Do nothing
            }
        }
    }

    public void addAccelerometerEvent(AccelerometerEvent accelerometerEvent) {
        synchronized (eventQueue) {
            eventQueue.add(accelerometerEvent);
            eventQueue.notifyAll();
        }
    }
}

I'm using Google Guava also, by the way, so if there's a nice collection in there I haven't heard about, please refer me.

So: Any good ideas how to solve this efficiently and cleanly?

score 1 · Accepted Answer · answered Jan 16 '13 at 10:39

1

If you're always going to consume WINDOW_SIZE/2 events en bloc, why doesn't the producer thread (you said there's only one) fill an array of size WINDOW_SIZE/2 and pass it to the queue once it's full?

answered Jan 16 '13 at 10:39

class stacker

5,357
2
32
65

For a 50% overlap, the consumer needs to process WINDOW_SIZE events, but only consume/remove half of them. But I guess your answer is still valid. I'll consider it, thanks! – Markus Jan 16 '13 at 10:45
@MarkusWüstenberg You're right, sorry. My solution wasn't fully described. The producer still has to provide two of those arrays. But the consumer can then call `data = poll(); peek();` which I wouldn't consider totally unclear. The cleanest approach would be a counting semaphore I guess, but you didn't sound as if you wanted to implement something manually. – class stacker Jan 16 '13 at 11:01
No, I'd rather not go into semaphores. But I'll try your approach (actually passing a List of WINDOW_SIZE samples to a queue of lists). Thanks again! – Markus Jan 16 '13 at 11:16
@MarkusWüstenberg Wouldn't that give you a 100% overlap every other time? Can you live with that? Didn't expect that. – class stacker Jan 16 '13 at 11:29
Well, half of the content of the list passed is removed from the event queue, the other half is copied. :) This should be 50% overlap. – Markus Jan 16 '13 at 11:34
@MarkusWüstenberg Maybe I'm not getting what you have on your mind, but the producer has to provide `List`s with `WINDOW_SIZE` events in it before it provides them via the queue. And your consumer can tell only by looking at the queue whether the producer generated any data at all, right? Then, the consumer would only be allowed to process the second half of the first `List` when the second `List` is provided. That implies that the producer has provided `WINDOW_SIZE` events instead of `WINDOW_SIZE/2` events, right? – class stacker Jan 16 '13 at 11:46
Maybe it's easier to show in code: `producerEventQueue.add(accelerometerEvent); if (producerEventQueue.size() < WINDOW_SIZE) { return; } List eventList = Lists.newArrayList(); // Remove the first half producerEventQueue.drainTo(eventList, WINDOW_SIZE / 2); // Peek the second half, for 50% overlap for (AccelerometerEvent accelerometerEvent2 : producerEventQueue) { eventList.add(accelerometerEvent2); } consumerEventListQueue.add(eventList);` – Markus Jan 16 '13 at 11:53
@MarkusWüstenberg And the consumer is working on `consumerEventListQueue`? – class stacker Jan 16 '13 at 12:11
Exactly. :) Gets a list of size WINDOW_SIZE from the queue each time. Works great! (Except the garbage collector can't collect the used objects, still figuring that one out...) Thanks for your help! – Markus Jan 16 '13 at 13:56
@Markus Wüstenberg: Concerning the GC, you can use a (non-concurrent) queue instead of the proposed arrays. – maaartinus Jan 16 '13 at 16:16
@maaartinus Right... or set the processed array entries to `null`. The initial post was asking for high efficiency. However, it appears that Markus now passes an ArrayList from producer to consumer. – class stacker Jan 16 '13 at 16:26

Sliding window with 50% overlap with producer/consumer threads

1 Answers1