-2

I am searching for a data structure that works similar to a Apache Kafka topic with one partition:

  • A sender will always append to the end of the journal
  • Each appended message gets a strictly increasing increasing offset
  • The messages are persisted for at least a given retention time
  • Consumers can start consuming at any offset or the oldest or the newest offset
  • Consuming messages will not delete them

If there is not suitable existing structure I will implement it myself but I would prefer to use something existing.

Christian Schneider
  • 19,420
  • 2
  • 39
  • 64

2 Answers2

1

I think there is no such existing data structure,you have to write custom logic for removing element after certain time,as per my understanding you can use any list family of collection with little customization.

vivekdubey
  • 484
  • 2
  • 7
0

I was able to find a suitable structure based on ConcurrentNavigableMap:

import java.util.Iterator;
import java.util.Map.Entry;
import java.util.concurrent.ConcurrentNavigableMap;
import java.util.concurrent.ConcurrentSkipListMap;
import java.util.concurrent.atomic.AtomicLong;

public class Journal<T> {
    private static final long INITIAL_OFFSET = 0;
    private AtomicLong nextOffset = new AtomicLong();
    private ConcurrentNavigableMap<Long, T> messages = new ConcurrentSkipListMap<>();

    public long append(T message) {
        Long offset = nextOffset.getAndIncrement();
        messages.put(offset, message);
        return offset;
    }


    public long getFirstOffset() {
        Iterator<Long> it = messages.keySet().iterator();
        return it.hasNext() ? it.next() : INITIAL_OFFSET;
    }

    public Entry<Long, T> getNext(long offset) {
        return this.messages.higherEntry(offset);
    }

}
Christian Schneider
  • 19,420
  • 2
  • 39
  • 64
  • I omitted the retention for now but it can be easily added. My main issue was finding a suitable data structure for the main duties of a journal. Btw. You can find my full code at https://github.com/apache/aries-journaled-events/blob/master/org.apache.aries.events.memory/src/main/java/org/apache/aries/events/memory/Journal.java – Christian Schneider Jan 02 '19 at 16:49