connect Akka HTTP to Akka stream

Question

I would like to build a REST service with Akka HTTP that connect to an existing Sink (with Kafka reactive stream) but I cannot figure out how to chain an HTTP flow to an Akka stream sink...

Should I go for the low level Akka HTTP API that use Flows ?

My requirement is to have:

backpressure on the complete flow
200 response code when all events are acknowledged by kafka sink
500 when backpressure is too high ? Is it possible ?

Here is my code current code

// flow to split group of lines into lines
  val splitLines = Flow[String].mapConcat(_.split("\n").toList)

// sink to produce kafka records in kafka
val kafkaSink = Flow[String]
    .map(new ProducerRecord[Array[Byte], String](topic, _))
    .toMat(Producer.plainSink(ProducerSettings(system,new ByteArraySerializer, new StringSerializer)))(Keep.right)

val routes = {
    path("ingest") {
      post {
        logger.info("starting ingestion")
        entity(as[GenericEvent]) { eventIngest =>
          ????      
        }~
        entity(as[GenericEventList]) { eventIngestList =>
          ????
        }
      }
    }
  }

Http(actorSystem).bindAndHandle(routes, config.httpInterface, config.httpPort)

Stefano Bonetti · Accepted Answer · 2017-02-27T15:32:46.290

There are several ways of going about this. One suggestion could be to stream the data straight from your request entity into your kafka sink. The extractDataBytes directive helps you do exactly that (more info here).

Try something along the lines of the example below. I added a ??? flow to allow for your case-specific transformation to correctly split/transform your request entity bytes. You can use the likes of Framing.delimiter to split the entity byte stream (more info here).

  (extractDataBytes & extractMaterializer) { (byteSrc, mat) =>
    val f = byteSrc.via(???).runWith(kafkaSink)(mat)
    onComplete(f){
      case Success(value) => complete(s"OK")
      case Failure(ex)    => complete((StatusCodes.InternalServerError, s"An error occurred: ${ex.getMessage}"))
    }
  }

Alternatively, if you want to unmarshal your entity to some domain object, you can do something like

  (entity(as[Event]) & extractMaterializer) { (event, mat) =>
    val f = Source.single(event).via(???).runWith(kafkaSink)(mat)
    onComplete(f){
      case Success(value) => complete(s"OK")
      case Failure(ex)    => complete((StatusCodes.InternalServerError, s"An error occurred: ${ex.getMessage}"))
    }
  }

To come to your last question, should Kafka backpressure, your stream will never complete. You should expect the server to give you back a 500 after the configured request-timeout (citing the docs below):

A default request timeout is applied globally to all routes and can be configured using the akka.http.server.request-timeout setting (which defaults to 20 seconds).

is there any way to unmarshall the stream with entity(as[event]) ? — vgkowski, Feb 27 '17 at 15:11
yes, you can always run your unmarshalled event against the kafka sink. I have put another example in the answer — Stefano Bonetti, Feb 27 '17 at 15:33
this is a new flow in the Akka HTTP flow, doesn't t introduce some useless overhead ? — vgkowski, Feb 27 '17 at 15:44
if you want to keep using the high-level routing DSL it's the only possible way at the moment. See a related question here http://stackoverflow.com/questions/36294428/akka-http-complete-request-with-flow. If you are that concerned about the materialization overhead, you should consider moving to the low-level DSL (http://doc.akka.io/docs/akka-http/10.0.0/scala/http/introduction.html#low-level-http-server-apis). — Stefano Bonetti, Feb 27 '17 at 15:58

connect Akka HTTP to Akka stream

1 Answers1

Linked