How to integrate akka streams kafka (reactive-kafka) into akka http application?

Question

I have a basic scala akka http CRUD application. See below for the relevant classes.

I'd simply like to write an entity id and some data (as json) to a Kafka topic whenever, for example, an entity is created/updated.

I'm looking at http://doc.akka.io/docs/akka-stream-kafka/current/producer.html, but am new to scala and akka, and unsure of how to integrate it into my application?

For example, from the docs above, this is the example of a producer writing to kafka, so I think I need to something similar, but whereabouts in my application should this go? Can I just add another map call in the create method in my service after I have created the user?

Many thanks!

val done = Source(1 to 100)
  .map(_.toString)
  .map { elem =>
    new ProducerRecord[Array[Byte], String]("topic1", elem)
  }
  .runWith(Producer.plainSink(producerSettings))

Or do I need to do something like the example here https://github.com/hseeberger/accessus in the bindAndHandle() method in my Server.scala?

WebServer.scala

object System {

  implicit val system = ActorSystem()
  implicit val dispatcher = system.dispatcher
  implicit val actorMaterializer = ActorMaterializer()

}

object WebServer extends App {

  import System._

  val config = new ApplicationConfig() with ConfigLoader
  ConfigurationFactory.setConfigurationFactory(new LoggingConfFileConfigurationFactory(config.loggingConfig))
  val injector = Guice.createInjector(new MyAppModule(config))
  val routes = injector.getInstance(classOf[Router]).routes
  Http().bindAndHandle(routes, config.httpConfig.interface, config.httpConfig.port)

}

Router.scala

def routes(): Route = {
    post {
      entity(as[User]) { user =>
        val createUser = userService.create(user)
        onSuccess(createUser) {
          case Invalid(y: NonEmptyList[Err]) =>  {
            throw new ValidationException(y)
          }
          case Valid(u: User) => {
              complete(ToResponseMarshallable((StatusCodes.Created, u)))
          }
        }
      }
    } ~
    // More routes here, left out for example  
}

Service.scala

def create(user: User): Future[MaybeValid[User]] = {
    for {
      validating <- userValidation.validateCreate(user)
      result <- validating match {
        case Valid(x: User) =>
          userRepo.create(x)
            .map(dbUser => Valid(UserConverters.fromUserRow(x)))
        case y: DefInvalid =>
          Future{y}
      }
    } yield result
  }

Repo.scala

def create(user: User): Future[User] = {
    mutateDbProvider.db.run(
      userTable returning userTable.map(_.userId)
        into ((user, id) => user.copy(userId = id)) +=
        user.copy(createdDate = Some(Timestamp.valueOf(LocalDateTime.now())))
    )
  }

Just to be clear on what you want: you are looking to send the `user.userId` to kafka everytime `create` returns valid user? — Ramón J Romero y Vigil, Sep 07 '17 at 15:12

score 2 · Accepted Answer · answered Sep 07 '17 at 15:32

2

Since you have written your Route to unmarshall just 1 User from the Entity I don't think you need Producer.plainSink. Rather, I think Producer.send will work just as well. Also, as a side note, throwing exceptions is not "idiomatic" scala. So I changed the logic for invalid user:

val producer : KafkaProducer = new KafkaProducer(producerSettings)

val routes : Route = 
  post {
    entity(as[User]) { user =>
      val createUser = userService.create(user)
      onSuccess(createUser) {
        case Invalid(y: NonEmptyList[Err]) =>  
          complete(BadRequest -> "invalid user")
        case Valid(u: User) => { 
          val producerRecord = 
            new ProducerRecord[Array[Byte], String]("topic1",s"""{"userId" : ${u.userId}, "entity" : "User"}""")

          onComplete(producer send producerRecord) { _ =>
            complete(ToResponseMarshallable((StatusCodes.Created, u)))
          }
        }
      }
    }
  }

answered Sep 07 '17 at 15:32

Ramón J Romero y Vigil

17,373
7
77
125

@Rory You are welcome. I am not familiar with any testkit similar to the example you provided. – Ramón J Romero y Vigil Sep 08 '17 at 10:22
Do you see any problems with *not* using onComplete to call 'producer send producerRecord' (so we're not waiting for the send to kafka to complete before sending the HTTP API response to the user)? Basically just doing this: producer send producerRecord; complete(ToResponseMarshallable((StatusCodes.Created, u))) – Rory Oct 03 '17 at 09:38
@Rory By not using `onComplete` it becomes possible that the client receives a `Created` response code but the message doesn't send. If this bifurcated state is ok then there are no problems, but the usual pattern is to return a valid response only after the record has been submitted. It's a design choice: do you want fast responses where the given code may be incorrect, or do you want to wait for a validated code? – Ramón J Romero y Vigil Oct 03 '17 at 11:11
Thanks Ramon, for my use case, I always return a Created response to users even if the send to kafka fails (I'm handling these errors in a different way), so I think I'm ok to remove the onComplete. Thanks again. – Rory Oct 03 '17 at 12:06

How to integrate akka streams kafka (reactive-kafka) into akka http application?

1 Answers1

Linked