2

In my application I have up to N consumers working in parallel and a producer. Consumers grab resources from the producer, do their work, append results to an updateQueue and ask for more resources. Producer has some resources available initially and can generate more by applying updates from the updateQueue. It is important to apply all available updates before a new resource is emitted to a consumer. I've tried using a following generator, requesting updates "in bulk" whenever a consumer makes a request and setting aside new resources (which are not needed by the consumer but may be later requested by other consumers) in a ticketQueue:

      def updatesOrFresh: Process[Task, Seq[OptimizerResult] \/ Unit] =
        Process.await(updateQueue.size.continuous.take(1).runLast) {
          case Some(size) =>
            println(s"size: $size")
            if (size == 0)
              wye(updateQueue.dequeueAvailable, ticketQueue.dequeue)(wye.either)
            else
              updateQueue.dequeueAvailable.map(_.left[Unit])
        }.take(1) ++ Process.suspend(updatesOrFresh)

It doesn't work - initially available resource are emitted from the ticketQueue.dequeue and then it appears to block on the wye, logging:

size: 0
<<got ticket>>
size: 0
<<got ticket>>
size: 0   // it appears the updateQueue did not receive the consumer output yet, but I can live with that, it should grab an update from the wye anyway
<<blocks>>

when there were two resources available initially on the ticketQueue. However, if I change it to just

 val updatesOrFresh = wye(updateQueue.dequeueAvailable, ticketQueue.dequeue)(wye.either)

It works as expected (although without the "apply updates before emitting new resource" guarantee). How can I make it work ensuring the updates are applied at the right time?


Edit: I've solved it using the following code:

  val updatesOrFresh: Process[Task, Seq[OptimizerResult] \/ Unit] =
    Process.repeatEval {
      for {
        sizeOpt <- updateQueue.size.continuous.take(1).runLast
        nextOpt <-
          if (sizeOpt.getOrElse(???) == 0)
            wye(updateQueue.dequeueAvailable, ticketQueue.dequeue)(wye.either).take(1).runLast
          else
            updateQueue.dequeueAvailable.map(_.left[Unit]).take(1).runLast
      } yield nextOpt.getOrElse(???)
    }

However the question why the original def didn't work remains...

Pyetras
  • 1,492
  • 16
  • 21

0 Answers0