Monix write to single output stream from parallel Tasks

Question

how can we use Monix Tasks to write a single output stream from a sequence of tasks (can run in parallel): for example, I have N tasks that can be run in parallel and get some response from HTTP call/server, and I'm writing the response back to a file stream (as is there constraint Limited memory) but I get the stream closed error, was wondering if there is a way to achieve this using Monix tasks or any other API

java.io.IOException: Stream closed

Sample Code snippet:

 import java.io.{FileOutputStream, OutputStream, OutputStreamWriter}

    import monix.eval.Task
    import monix.execution.ExecutionModel.AlwaysAsyncExecution
    import monix.execution.Scheduler

    import scala.util.Random

    object TaskTest extends App {

      implicit val scheduler = Scheduler(
        AlwaysAsyncExecution
      )
      val list = List(1 to 100)

      val filePath = System.currentTimeMillis() + "_test.txt"
      val outputStreamWriter = new OutputStreamWriter(new FileOutputStream(filePath))
      val futureTask = autoClose(outputStreamWriter) {
        outputStream =>
          val totalCount = SeqOfTasks(outputStream)
          totalCount.map(_.sum)
      }

      futureTask.runToFuture.onComplete {
        case scala.util.Success(value) =>
          println(s"fetched a total of $value")

        case scala.util.Failure(exception) =>

          println("there is some error occurred ")
          println(exception)
      }

//close this stream when all tasks completed.
      def autoClose[A <: AutoCloseable, B](resource: A)(code: A ⇒ B): B = {
        try code(resource)
        finally resource.close()
      }


      def SeqOfTasks(outPutStream: OutputStreamWriter): Task[Seq[Int]] = {

        val list = List(0 to 10: _*)
        Task.wanderUnordered(list)(l => makeHttpCall(outPutStream, l))
      }

      // each taks will fetch some data and write to single out put stream that is shared by other tasks.
      def makeHttpCall(outputStream: OutputStreamWriter, value: Int): Task[Int] = {

        Task {
          val result = getDummyValue(value)
          val bytes = result.mkString("\t") + "\n"
          outputStream.write(bytes)
          //return some value
          result.size
        }
      }

      //return a dummy response for http
      def getDummyValue(n: Int): Seq[Int] = {
        println(s"dummy value : $n")
        Seq.fill(n)(Random.nextInt)
      }

Technical question aside, are you aware that this will not work? Having multiple asynchronous processes write into a single `OutputStream` will result in garbled data, because it happens character by character. What you want to do is write each response in a different file. — Markus Appel, Jun 21 '19 at 12:39
Hi Markus, The output stream is sync and thread-safe, so it will not result in garbled data and output is being to streamed to cloud storage and requires data written to a single file. — vkt, Jun 24 '19 at 15:19

score 0 · Accepted Answer · answered Jun 24 '19 at 15:25

The issue is due to Task is being Lazy and async, but autoClose is neither.

The solution is to make autoClose Task-aware:

def autoClose[A <: AutoCloseable, B](resource: A)(code: A ⇒ Task[B]): Task[B] = {
    code(resource).guarantee(Task(resource.close()))
  }

Note: This is answered by Oleg Pyzhcov on Monix Gitter chat

Monix write to single output stream from parallel Tasks

1 Answers1