2

I'm learning Akka Streams and as an exercise I would like to insert logs into Cassandra. The issue is that I could not manage to make the stream insert logs into the database.

I naively tried the following :

object Application extends AkkaApp with LogApacheDao {

  // The log file is read line by line
  val source: Source[String, Unit] = Source.fromIterator(() => scala.io.Source.fromFile(filename).getLines())

  // Each line is converted to an ApacheLog object
  val flow: Flow[String, ApacheLog, Unit] = Flow[String]
    .map(rawLine => {
      rawLine.split(",") // implicit conversion Array[String] -> ApacheLog
    })

  // Log objects are inserted to Cassandra
  val sink: Sink[ApacheLog, Future[Unit]] = Sink.foreach[ApacheLog] { log => saveLog(log) }

  source.via(flow).to(sink).run()

}

saveLog() is defined in LogApacheDao like this (I omitted the column values for a clearer code):

val session = cluster.connect()

session.execute(s"CREATE KEYSPACE IF NOT EXISTS $keyspace WITH replication = {'class':'SimpleStrategy', 'replication_factor':1};")

session.execute(s"DROP TABLE IF EXISTS $keyspace.$table;")

session.execute(s"CREATE TABLE $keyspace.$table (...)")

val preparedStatement = session.prepare(s"INSERT INTO $keyspace.$table (...) VALUES (...);")

def saveLog(logEntry: ApacheLog) = {
    val stmt = preparedStatement.bind(...)

    session.executeAsync(stmt)
  }

The conversion from Array[String] to ApacheLog when entering in the sink happens without issue (verified with println). Also, the keyspace and table are both created, but when the execution comes to saveLog, it seems that something is blocking and no insertion is made.

I do not get any errors but Cassandra driver core (3.0.0) keeps giving me :

Connection[/172.17.0.2:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
Connection[/172.17.0.2:9042-2, inFlight=0, closed=false] heartbeat query succeeded

I should add that I use a dockerized Cassandra.

blakelead
  • 1,780
  • 2
  • 17
  • 28
  • 1
    I only read about akka streams so it's just a guess, are you sure that `saveLog` is not throwing an exception that is being swallowed? I would try/catch it with a print on the exception to be sure. Those cassandra logs show that the connection is inactive for 30 seconds and after those a heartbeat is sent to keep it open. – Ende Neu May 11 '16 at 21:17
  • Probably this is related http://stackoverflow.com/questions/35631754/why-is-akka-streams-swallowing-my-exceptions – Ende Neu May 11 '16 at 21:20
  • I will try to catch eventual Exceptions from saveLog, thanks for the advice – blakelead May 11 '16 at 21:27
  • Even when handling Exceptions (with onComplete(...)), I can't get any error message. To me it seems that execution cannot perform the `bind()` method. – blakelead May 11 '16 at 21:51

1 Answers1

0

Try using the Cassandra Connector in alpakka.

Ramón J Romero y Vigil
  • 17,373
  • 7
  • 77
  • 125