Questions tagged [alpakka]

Alpakka is the collective name for various Akka Streams connectors, integration patterns, and data transformations.

Alpakka is a community-driven initiative that provides connectors, integration patterns, and data transformations that are built with Akka Streams. This toolkit is meant to be a "modern alternative to Apache Camel" (hence its name, which is a homophone of "alpaca," a relative of the camel, and was first used as a code name for an old akka-camel module).

From an introductory blog post:

Akka Streams already has a lot that are useful for integrations. Defining processing pipelines is what the Akka Streams DSL is all about and that is exactly what you need for operating on streaming data that cannot fit in memory as a whole. It handles backpressure in an efficient non-blocking way that prevents out-of-memory errors, which is a typical problem when using unbounded buffering with producers that are faster than consumers.

Connectors:

  • AMQP
  • Apache Geode
  • AWS DynamoDB
  • AWS Kinesis
  • AWS Lambda
  • AWS S3
  • AWS SNS
  • AWS SQS
  • Azure Storage Queue
  • Cassandra
  • Elasticsearch
  • File
  • FTP
  • Google Cloud Pub/Sub
  • HBase
  • IronMq
  • JMS
  • MongoDB
  • MQTT
  • Server-sent Events (SSE)
  • Slick (JDBC)
  • Spring Web

Integration Patterns:

  • Splitter

Data Transformations:

  • Parsing Lines
  • JSON
  • Compressing/decompressing
  • Comma-Separated Values (CSV)
  • RecordIO Framing
  • Extensible Markup Language (XML)

Additional Links:

217 questions
0
votes
2 answers

How do you transform a `FixedSqlAction` into a `StreamingDBIO` in Slick?

I'm creating an akka-stream using Alpakka and the Slick module but I'm stuck in a type mismatch problem. One branch is about getting the total number of invoices in their table: def getTotal(implicit session: SlickSession) = { import…
tonicebrian
  • 4,715
  • 5
  • 41
  • 65
0
votes
0 answers

How to to stream a directory of parquet files through akka/alpakka streams

I trying to stream a directory of parquet files through akka/alpakka streams, the problem is although I am setting the directory as the source, it streams only one file. Any advice will be greatly appreciated This is my last experiment: val reader:…
igx
  • 4,101
  • 11
  • 43
  • 88
0
votes
2 answers

akka streaming file lines to actor router and writing with single actor. how to handle the backpressure

I want to stream a file from s3 to actor to be parsed and enriched and to write the output to other file. The number of parserActors should be limited e.g application.conf akka{ actor{ deployment { HereClient/router1 { …
igx
  • 4,101
  • 11
  • 43
  • 88
0
votes
0 answers

akka streams kafka consumer stops in few seconds

I am reading from kafka consumer, then persist message to db and send to producer on different topic. My akka streams app stops running in few seconds after launch. Here is how my stream looks like. Consumer.committableSource(consumerSettings,…
priya
  • 109
  • 4
0
votes
1 answer

how to initialize a continous running stream using alpakka, spring boot & Akka-stream?

All, I am developing an application, which use alpakka spring boot integration to read data from kafka. I have most of the code ready, the only place i am stuck is how to initialize a continuous running stream, as this is going to be a backend…
chetank
  • 392
  • 3
  • 17
0
votes
1 answer

Kafka consumers (same group-id) stucked in reading always from same partition

I have 2 consumers running with the same group-id and reading from topic having 3 partititons and parsing messages with KafkaAvroDeserializer. The consumer has these settings: def avroConsumerSettings[T <: SpecificRecordBase](schemaRegistry:…
Truman
  • 72
  • 8
0
votes
1 answer

How to read parquet file from S3 using akka streams or alpakka

Im trying to read parque file from S3 using akka streams following the official doc but I am getting this error java.io.IOException: No FileSystem for scheme: s3a this is the code that triggered that exception. I will highly appreciate any…
igx
  • 4,101
  • 11
  • 43
  • 88
0
votes
1 answer

Not able to read data from elasticsearch using alpakka

I was trying to read document(json data stored in ES) from elasticsearch using alpakka. I got this alpakka-Elasticsearch. Here it says that you can stream messages from or to Elasticsearch using the ElasticsearchSource, ElasticsearchFlow or the…
md samual
  • 305
  • 3
  • 15
0
votes
1 answer

No implicits found for parameter sprayJsonReader: JsonReader[T]

I have created an sbt project to learn how to use elastic search with akka. I came across alpakka which provides this feature (to connect with elasticsearch). According to docs, to search from ES we have following code: val connectionSettings =…
md samual
  • 305
  • 3
  • 15
0
votes
0 answers

Slick Query with calling a procedure does not terminate the program

I have a Main program that queries data from DB using slick. It constructs JSON from the record it gets from DB and then publishes to some kafka topics. Almost at the end of the program, I m trying to call a stored procedure that updates the…
user9920500
  • 606
  • 7
  • 21
0
votes
1 answer

Alpakka and Akka Version Compatibility Issue

When I try to sbt build my project, the project build fails with an "extracting product structure failed" error. I am suspecting something related to the versions I am using for Alpakka and Akka. Here is my build.sbt file: lazy val akkaHttpVersion =…
0
votes
1 answer

Adjusting parallism based on number of partitions assigned in Consumer.committablePartitionedSource

I am trying to use Consumer.committablePartitionedSource() and creating stream per partition as shown below public void setup() { control = Consumer.committablePartitionedSource(consumerSettings, …
Pokuri
  • 3,072
  • 8
  • 31
  • 55
0
votes
1 answer

MongoDB Reactive Streams run-time dependency error with Alpakka Mongo Connector ClassNotFoundException

I'm trying to integrate the Alpakka Mongo Connector into an application that heavily relies on the Akka libraries for stream processing. The application utilizes Akka HTTP as well. I am encountering a dependency issue at run-time. In particular, I'm…
0
votes
1 answer

How to download from Google Cloud Storage by Alpakka-gcs without providing secret-key?

I'm using Alpakka-gcs connecting to GCS from google-compute-engine perfectly if I provide gcs-secret-key on application.conf like the below. alpakka.google.cloud.storage { project-id = "project_id" client-email = "client_email" private-key =…
0
votes
1 answer

Why does my Alpakka SFTP connector never connect?

I'm new to Alpakka/Akka Streams and I'm trying to set up a stream where I stream data between two SFTP servers with my system in the middle, here's the code. import akka.actor.ActorSystem import akka.stream.ActorMaterializer import…
conorgriffin
  • 4,282
  • 7
  • 51
  • 88