1

I'm looking for a way to efficiently maintain a set of values from a 1 minute sliding window from a given datastream (~100k values/sec).

I'm looking for solution with at most logarithmic insertion time (since basic time-ordered vector of values has o(n))

discobot
  • 31
  • 4
  • How will you access elements inside the window? A deque allows amortized constant-time additions and deletions from the front and back (respectively) of the set, but only O(n) access to random items in the middle. – chepner May 13 '13 at 18:43

1 Answers1

1

Unless I misunderstand your question, this can be done in constant time ( constant per insertion ), with a circular buffer. Allocate a buffer with is a power of 2 length larger than the maximum number of elements in your window. E.g. max 100k values per second 6 million values per minute, so allocate a buffer that is 8388608 bytes long. Keep an absolute index into this buffer, but insert into it, and remove elements from it by masking the index with 0x7FFFFF.

Rafael Baptista
  • 11,181
  • 5
  • 39
  • 59
  • If it's a *sliding* window, I believe you only need a buffer of size equal to the window size, though that will change things a bit (requiring an offset rather than mask). – Bernhard Barker May 13 '13 at 17:38
  • No one will delete items after one minute, so this data structure should track values age by itself. – discobot May 13 '13 at 19:34