4

Context

Consider having a stream of tuples (string, timestamp), with the goal of having a bin containing a Map of the minimal timestamp received for each string. Another constraint is that the update for this Map will be atomic.

For this input example:

("a", 1)
("b", 2)
("a", 3)

the expected output: Map("a" -> 1, "b" -> 2) without the maximal timestamp 3.

The current implementation I chose is using CDT of list to hold the timestamps, so my result is Map("a" -> ArrayList(1), "b" -> ArrayList(2)) as follows:

private val boundedAndOrderedListPolicy = new ListPolicy(ListOrder.ORDERED, ListWriteFlags.INSERT_BOUNDED)

private def bundleContext(str: String) =
      CTX.mapKeyCreate(Value.get(str), MapOrder.UNORDERED)

private def buildInitTimestampOp(tuple: (String, Long)): List[Operation] = {
      // having these two ops together assure that the size of the list is always 1
      val str = tuple._1
      val timestamp = tuple._2
      val bundleCtx: CTX = bundleContext(str)
      List(
        ListOperation.append(boundedAndOrderedListPolicy, initBin, Value.get(timestamp), bundleCtx),
        ListOperation.removeByRankRange(initBin, 1, ListReturnType.NONE, bundleCtx), // keep the first element of the order list - earliest time
      )
    }

This works as expected. However, if you have a better way to achieve this without the list and in an atomic manner - I would love to hear it.

My question

What does not work for me is retaining the max timestamp received for each input str. For the example above, the desired result should be Map("a" -> ArrayList(3), "b" -> ArrayList(2)). My implementation attempt is:

private def buildLastSeenTimestampOp(tuple: (String, Long)): List[Operation] = {
      // having these two ops together assure that the size of the list is always 1
      val str = tuple._1
      val timestamp = tuple._2
      val bundleCtx: CTX = bundleContext(str)
      List(
        ListOperation.append(boundedAndOrderedListPolicy, lastSeenBin, Value.get(timestamp), bundleCtx),
        ListOperation.removeByRankRange(lastSeenBin, 1, ListReturnType.NONE | ListReturnType.REVERSE_RANK, bundleCtx), // keep the last element of the ordered list - last time
      )
    }

Any idea why doesn't it work?

guniata
  • 101
  • 5

1 Answers1

6

So, i've solved it:

private def buildLastSeenTimestampOp(tuple: (String, Long)): List[Operation] = {
      // having these two ops together assure that the size of the list is always 1
      val str = tuple._1
      val timestamp = tuple._2
      val bundleCtx: CTX = bundleContext(str)
      List(
        ListOperation.append(boundedAndOrderedListPolicy, lastSeenBin, Value.get(command.timestamp.getMillis), bundleCtx),
        ListOperation.removeByRankRange(lastSeenBin, -1, ListReturnType.NONE | ListReturnType.INVERTED, bundleCtx), // keep the last element of the ordered list - last time
      )
    }

When dealing with Rank/Index (removeByIndexRange for indexes) in ascending order -1 represent the max Rank/Index.

Using the ListReturnType.INVERTED is retaining the range (or element in this case) that is selected by initial rank/index until count - by deleting all elements of the list that aren't in the selected range.

guniata
  • 101
  • 5