Iteratee for stream data upload

Question

Following this post to implement stream based file upload using custom parser/Iteratee. Of course code in that post is old and doesn't compile anymore. Following is what I am trying with.

case class UploadIteratee(state: Symbol = 'Cont, input: Input[Array[Byte]] = Empty, received: Int = 0) extends Iteratee[Array[Byte], Either[Result, Int]] {
def fold[B](folder: Step[Array[Byte], Either[Result, Int]] => Future[B])(implicit ec: ExecutionContext): Future[B] = {
  folder(
    Step.Cont(in => in match {
      case in: El[Array[Byte]] => copy(input = in, received = received + in.e.length)
      case Empty => copy(input = in)
      case EOF => copy(state = 'Done, input = in)
      case _ => copy(state = 'Error, input = in)
    }))
}
}

def send = Action(BodyParser(rh => new UploadIteratee).map(Right(_))) { request =>
Ok("Done")

}

Does that look right and sufficient to accept stream from file upload? I must be doing something silly to get following compile error.

type mismatch; found : controllers.MyController.UploadIteratee required: 
 play.api.libs.iteratee.Iteratee[Array[Byte],Either[play.api.mvc.SimpleResult,?]]

EDIT I am on Play 2.2.2. My bad that I was looking at play 2.3 source code. now following compiles

case class UploadIteratee(state: Symbol = 'Cont, input: Input[Array[Byte]] = Empty, received: Int = 0) extends Iteratee[Array[Byte], Either[SimpleResult, Int]] {
def fold[B](folder: Step[Array[Byte], Either[SimpleResult, Int]] => Future[B])(implicit ec: ExecutionContext): Future[B] = {
  folder(
    Step.Cont(in => in match {
      case in: El[Array[Byte]] => copy(input = in, received = received + in.e.length)
      case Empty => copy(input = in)
      case EOF => copy(state = 'Done, input = in)
      case _ => copy(state = 'Error, input = in)
    }))
}
}
def send = Action(BodyParser(rh => new UploadIteratee).map(Right(_))) { request =>
 Ok("Done")
}

wingedsubmariner : I am trying to write a body parser which takes in stream of data for file upload functionality. Subsequently I'll have to populate another data holding case class with each record received in the stream. My understanding about Iteratee is still raw so any pointers will be appreciated.

Usually you don't need to create your own `Iteratee` subclass and can instead create an `Iteratee` using `Iteratee.foreach` or `Iteratee.fold`. What does this code need to do? — wingedsubmariner, Jul 02 '14 at 03:59
In addition to what wingedsubmariner said, the compile error is that your iteratee produces an Either[Result, whatever] but play expects it to produce a Either[SimpleResult, whatever]. so Result vs SimpleResult — johanandren, Jul 02 '14 at 08:48
Please also post what version of Play you're using. In the newest version (2.3), SimpleResult is deprecated and one should use Result instead. — Carsten, Jul 02 '14 at 10:39
@wingedsubmariner : updated question with more details. Would like to hear your thoughts. Thanks — user2066049, Jul 02 '14 at 12:16
It doesn't look to me like you should be writing your own body parser. What are you trying to parse? — wingedsubmariner, Jul 02 '14 at 15:20
@wingedsubmariner : I am trying to implement file upload which is totally stream based (doesn't want to store incoming data into a file) but put incoming records (from excel upload) to the db directly. Am I on a wrong path? — user2066049, Jul 02 '14 at 15:29
@wingedsubmariner : I went into that direction(of writing custom body parser) after reading http://www.playframework.com/documentation/2.2.1/ScalaFileUpload where in "Writing your own body parser" section it mentions about writing own parser if one didn't want to store into a temp file etc. — user2066049, Jul 02 '14 at 15:45

wingedsubmariner · Answer 1 · 2014-07-02T16:22:05.093

-1

There is no need to write a custom Iteratee implementation for most tasks. Instead, use the factory methods in the companion object of Iteratee. In this case, we can use Iteratee.foreach to create a body parser that does something with each block of data as it is received:

val bodyParser = BodyParser { requestHeader =>
  Iteratee.foreach[Array[Byte]] { block =>
    // Do something with each block as it is received
  }.map(Right(_))
}

A more complex implementation might use an Enumeratee to break the incoming stream of bytes into distinct records, which could then be consumed by an Iteratee.

edited Jul 02 '14 at 16:22

answered Jul 02 '14 at 15:51

wingedsubmariner

13,350
1
27
52

But in order to use a parser in conjunction with Action like `def send = Action(BodyParser(rh =>bodyParser).map(Right(_))) { request =>` `bodyParser` must return `Either[SimpleResult,?]` , no? `type mismatch; found : play.api.libs.iteratee.Iteratee[Array[Byte],Unit] required: play.api.libs.iteratee.Iteratee[Array[Byte],Either[play.api.mvc.SimpleResult,?]]` – user2066049 Jul 02 '14 at 15:57
Sorry, I had forgotten the extra wrapping to make it a body parser. Edited. – wingedsubmariner Jul 02 '14 at 16:13
@wingedsbmariner :- Isn't that(`foreach`) side effecting and only returns `Unit`? `.map(Right)` gives `type mismatch; found : util.Right.type required: Unit => ?` – user2066049 Jul 02 '14 at 16:17
:- Thanks. Do you still think that having more flexible body parser isn't required here? meaning I am looking to create a reusable parser function which can be used to plug into any action which wants to handle file upload in a streaming/iteratee way (with no temp file storage). And each action can choose to decide how/what they want to do with incoming `Array[Byte]`. – user2066049 Jul 02 '14 at 18:09
Simplest use case is to concatenate all incoming (`Array[Byte]`) chunks, like a buffer, from file upload. Isn't `Iteratee.fold` required here? – user2066049 Jul 02 '14 at 18:16
If you do that, you aren't processing the file as a stream, you're just buffering it in memory instead of using a temporary file. Be careful of running out of memory. – wingedsubmariner Jul 02 '14 at 21:22
@wingedsbmariner :messed up with my example in the last comment. What I really need is out of `Array[Byte]` need to create a db record, so need to populate a `case class` for each record which are nothing but data holder class for `Slick`. – user2066049 Jul 03 '14 at 00:33

Iteratee for stream data upload

1 Answers1